Partial fix Error doing JSON conversion xf_report.content_info

[dethfire]

Affected version

2.1

Had a few hundred of these show up after upgrade from 1.5.22 to 2.1RC2

ErrorException: Error doing JSON conversion xf_report.content_info (report_id=24745): Malformed UTF-8 characters, possibly incorrectly encoded src/XF/Error.php:75

Stack trace

#0 src/XF.php(187): XF\Error->logError('Error doing JSO...', true)
#1 src/XF/Install/InstallHelperTrait.php(173): XF::logError('Error doing JSO...', true)
#2 src/XF/Install/InstallHelperTrait.php(117): XF\Install\Upgrade\AbstractUpgrade->updateResultsToJson(Array, 'xf_report', Array, Array)
#3 src/XF/Install/InstallHelperTrait.php(42): XF\Install\Upgrade\AbstractUpgrade->tableColumnsToJson('xf_report', Array, Array, 23008, Array, false)
#4 src/XF/Install/Upgrade/2010010-210a.php(464): XF\Install\Upgrade\AbstractUpgrade->entityColumnsToJson('XF:Report', Array, 23008, Array)
#5 src/XF/Cli/Command/Upgrade.php(134): XF\Install\Upgrade\Version2010010->step21(23008, Array, Object(XF\Install\Controller\Upgrade))
#6 src/vendor/symfony/console/Command/Command.php(255): XF\Cli\Command\Upgrade->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#7 src/vendor/symfony/console/Application.php(953): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#8 src/vendor/symfony/console/Application.php(248): Symfony\Component\Console\Application->doRunCommand(Object(XF\Cli\Command\Upgrade), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#9 src/vendor/symfony/console/Application.php(148): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#10 src/XF/Cli/Runner.php(63): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 cmd.php(15): XF\Cli\Runner->run()
#12 {main}

Request state

array(1) {
["cli"] => string(18) "cmd.php xf:upgrade"
}

 

[Mike]

If you look in the xf_report table for report_id 24745, can you see the content type this entry relates to?

It's possible that this could be from a custom content type that was logging some binary data here (or potentially a DB change that triggered this). We could consider "forcing" these conversions, though we didn't want to take that approach through the beta process as it would make debugging harder. This is the first report we've had of it though.

 

[dethfire]

If you look in the xf_report table for report_id 24745, can you see the content type this entry relates to?

alterego, so it's an addon on problem, thanks!

 

[Mike]

This error log entry is mostly as designed, though I have just made some subsequent changes. Previously, for rows that triggered this, we would simply not convert the row's content at all (it would generally remain serialized).

For the next release, we will now do a partial conversion on error (JSON_PARTIAL_OUTPUT_ON_ERROR). This will convert as much data as possible to JSON for the specified column. However, this will potentially remove some data that may be important, depending on the add-on (though note that if the data is significant, the add-on likely wouldn't work in 2.1 without changes anyway). In order to combat this data loss, we will now log the original data into a row in the newly created xf_json_convert_error table (which logs table, column, primary key value and the original data). The developer can potentially use this table to access the full original data in an add-on upgrade step.

If the data that triggered this error is legacy or not significant, then this change should avoid issues that may have occurred from the conversion not proceeding.

This error message will still be logged to bring the data conversion issues to your attention.

Note that the partial conversion changes will only apply when upgrading to 2.1. (If you've already upgraded, this won't apply; it would require upgrading from 2.0 again.)

@Xon -- so you're aware, as I talked a bit about this in the alter ego thread.

 

[Xon]

Thanks, the data gets trimmed anyway as most of it isn't required any more in the XF2 version of the add-on. I've pushed an update so any users on XF2 have their alterego report records updated to properly convert to XF2.1 format.