XF 1.5 my xf_search table is massive all of a sudden, rebuild doesn't help

[sross]

Hi, not sure what is going on here. I noticed a backup taking too long to find my xf_search table had blown out to 85 GB. My database backups are usually 1.1 GB. Forum has 2.4mil posts. I am running elastisearch. I emptied the table and re-ran the indexing. It's been going for hours and table is now 25 GB and rising. Not sure what to do. Thanks for any help.

 

[sross]

in case this helps anyone with similar issue. I might be in the clear.. I updated my robots.txt and google claims to follow this. I then went to search console at google and re-submitted my updated robots.txt which included the disallow for /find-new/

I then had google test the url and the bot claimed it was blocked. Will see if this sorts it out in the next day or so



*update

so far no change, so although google claims their bot will honour the robots.txt doesn't seem to unless it takes a while to kick in. I'll see if I can find any support at google regarding this

*update

you can see where this started all of a sudden



*one more update

I might be in the clear now. I discovered several url variations in my google search console and those did not contain the updated robots.txt blocking /find-new/ block. I updated that and the pounding seems to have stopped. My server load has dropped from 15.0 to 0.70 still monitoring, also xf_search is not growing and lurking around 15mb