Missing records in Elastic Index with Enhanced Search

[sajal]

We are using XF Enhanced Search and I'm trying to index it. We've around 5M records on the hand, and after indexing it seems that it indexes around 4M records. So, it almost skips around 1M records during indexing.

Does anyone have faced similar issues of missing records in Elastic index?

Does this plugin index all the posts/pages etc.. or it skips records that are delete/moderated etc.

Thanks in advance.

 

[Jeremy P]

Does this plugin index all the posts/pages etc.. or it skips records that are delete/moderated etc.

It should index the core content types regardless of their visibility. I believe the only time content is not indexed is if it is in an orphaned/inconsistent state, like a post with no corresponding thread or forum, or a profile post with no corresponding profile user.

Where are you pulling the source number (5M) from?

 

[sajal]

@Jeremy P, It's the count of xf_posts table only. Also, we're using CloudFlare, so during indexing, it breaks in-between, but then I click on "run deferred" link and that resumes the process, so I'm not sure if that could also cause issues with these missing records.

 

[Jeremy P]

Oh is this on XF1? I haven't worked with XF1 in a very long time, but at a cursory glance it appears only visible content is indexed there.

 

[sajal]

@Jeremy P, it's XF 1.5.21.

 

[Jeremy P]

Ah okay. Yeah if I'm not mistaken only content that is visible is indexed, so any moderated or soft-deleted content would be excluded. On XF2+, content is indexed regardless of visibility.

 

[Chris D]

Yes, that's correct.



XF 1.5 rarely (if ever) indexed non-visible content. This is different starting with XF 2.0 where we simply store whether an indexed record is hidden, or not, and then decide during a search whether to show those records or not.