Fixed Serialised content array not converted to json

[Naz]

Affected version

2.1

I had an active report before I updated and it looks like the serialised array in content_info was not converted to json leaving me with the following:

ErrorException: [E_NOTICE] Undefined index: node_id in src/XF/Report/Post.php at line 12
XF::handlePhpError() in src/XF/Report/Post.php at line 12
XF\Report\Post->canViewContent() in src/XF/Report/AbstractHandler.php at line 19
XF\Report\AbstractHandler->canView() in src/XF/Repository/Report.php at line 110
XF\Repository\Report->XF\Repository\{closure}()
array_filter() in src/XF/Mvc/Entity/AbstractCollection.php at line 181
XF\Mvc\Entity\AbstractCollection->filter() in src/XF/Repository/Report.php at line 111
XF\Repository\Report->filterViewableReports() in src/XF/Pub/Controller/Report.php at line 35
XF\Pub\Controller\Report->actionIndex() in src/XF/Mvc/Dispatcher.php at line 321
XF\Mvc\Dispatcher->dispatchClass() in src/XF/Mvc/Dispatcher.php at line 248
XF\Mvc\Dispatcher->dispatchFromMatch() in src/XF/Mvc/Dispatcher.php at line 100
XF\Mvc\Dispatcher->dispatchLoop() in src/XF/Mvc/Dispatcher.php at line 50
XF\Mvc\Dispatcher->run() in src/XF/App.php at line 2155
XF\App->run() in src/XF.php at line 389
XF::runApp() in index.php at line 20

Weirdly, running the step manually converts the field in to json.

 

[Russ]

Was just about to post this , getting the same error.

 

[Mike]

Just to confirm, the data is still serialized in the column? Is that for all rows?

We didn't have any problem with conversion here. The only thing that comes to mind could involve non-UTF-8 data here which might cause the conversion process to fail, and given that we don't know what data add-ons have inserted in the past, we might have to handle that a bit differently.

 

[Mike]

Actually, something else to check -- are your tables InnoDB or MyISAM?

(I believe there may be a bug with this conversion due to the query relying on InnoDB's data ordering.)

 

[Naz]

Yup, all rows were still serialised and there was nothing particularly special about the content of the reports.

My tables are InnoDB.

 

[Mike]

I would certainly expect some things to be converted to JSON as presumably it should manage 1 "set" before any potential failure.

Right now, the main thing I have to go on is the missing order clause. Can you confirm your MySQL version? (The only reason I could see nothing being converted is if MySQL is failing the regex, though I would have expected that to happen more generally.)

If possible, would you be able to test another upgrade to 2.1 with this code change to src/XF/Install/InstallHelperTrait.php?

Find:

AND (" . implode(' OR ', $whereSql) . ")

And after it add:

ORDER BY {$primaryKeys[0]}

 

[Russ]

@Mike I just tried your edit and ran another update, I can access the forum now, however, the error only shows on the reports area now (before I couldn't even load the forum at all).

I can provide access to my 2 (2.0.10 and 2.1) test forums if you'd like.

 

[Chris D]

Thanks Russ

If you can submit a ticket with the login details (Admin CP and PhpMyAdmin) that would be ideal, thank you.

 

[Naz]

I'm using MariaDB 10.3.10.

I just tried that change but same thing happened. Serialised data still remains and none of the rows in the table are converted.

 

[Chris D]

@NixFifty would you be able to do the same as Russ, please? I guess if not, then an SQL dump of the databases would be ok, but then that depends whether it's a version/server specific thing.

Only significant thing that springs to mind is whether there's something especially different in MariaDB 10.3, being that it's fairly bleeding edge and may explain why this isn't affecting everyone.

@Russ which version of MySQL/MariaDB are you running BTW?

 

[Naz]

This was in a local environment so I sent you a SQL dump instead.

 

[Russ]

Submitted a ticket with the info.

 

[Wildcat Media]

Watching. Same problem here. Since it is our testing forum, can I simply delete or modify something in the database to get this running?

FWIW, we have InnoDB tables, and an older MySQL version (5.7.18). There were no errors during the upgrade, and the admin control panel works perfectly well.

 

[Mike]

We have a working theory that we're investigating. It looks like it may not be related to this specific code.

For those of you who upgraded, did you do a web or CLI upgrade?

 

[Russ]

Web

 

[Wildcat Media]

Web upgrade.

Also, I just truncated xf_report and xf_report_comment, and the site is working again. (We only had four test reports and seven comments.) However, I did copy the two tables to a backup in case they're needed for anything.

 

[Mike]

There are very likely many other tables that didn't do the JSON conversion as expected. We would likely recommend a re-upgrade after this issue is resolved (hence why we don't recommend upgrading production sites to betas ).

 

[Wildcat Media]

If someone doesn't see this bug report and hasn't had an issue, will the next beta version have a means to do the JSON conversion if it was missed during Beta 1? (I'm only thinking that if others don't know there is a problem with Beta 1, they would not be aware they would need to restore from a backup before upgrading to Beta 2.)

 

[Mike]

We've now identified the cause and it was a slightly more general web-based upgrader issue that incorrectly maintained the "position" value across steps. This effectively means that earlier data in the tables became more likely to be skipped. This will be fixed in Beta 2, which we will very likely be releasing tomorrow (Wednesday).

The CLI upgrader was not affected by the issue.

Because of the way this issue has manifested, it will not be resolved automatically if you already upgraded to 2.1.0 Beta 1. At this point, if you did a test upgrade using the web upgrader, regardless of whether you noticed issues, we would recommend restoring from a pre-upgrade backup and redoing the upgrade once beta 2 is out (or redoing the upgrading using the CLI upgrader). If this is absolutely not possible, then there are some alternatives, but the restore and re-upgrade approach is what we'd recommend.

 

[akhan]

We've now identified the cause and it was a slightly more general web-based upgrader issue that incorrectly maintained the "position" value across steps. This effectively means that earlier data in the tables became more likely to be skipped. This will be fixed in Beta 2, which we will very likely be releasing tomorrow (Wednesday).

The CLI upgrader was not affected by the issue.

Because of the way this issue has manifested, it will not be resolved automatically if you already upgraded to 2.1.0 Beta 1. At this point, if you did a test upgrade using the web upgrader, regardless of whether you noticed issues, we would recommend restoring from a pre-upgrade backup and redoing the upgrade once beta 2 is out (or redoing the upgrading using the CLI upgrader). If this is absolutely not possible, then there are some alternatives, but the restore and re-upgrade approach is what we'd recommend.

After upgrading to Beta 1, I didn't restore it from backup. I did a Beta 2 upgrade on Beta 1.What may have been affected and incomplete?