XF 1.5 my xf_search table is massive all of a sudden, rebuild doesn't help

sross

Active member
Hi, not sure what is going on here. I noticed a backup taking too long to find my xf_search table had blown out to 85 GB. My database backups are usually 1.1 GB. Forum has 2.4mil posts. I am running elastisearch. I emptied the table and re-ran the indexing. It's been going for hours and table is now 25 GB and rising. Not sure what to do. Thanks for any help.
 

sross

Active member
ok it was enabled and elastisearch installed a while back. I tried re-installing. I have this on my config page:

Screen Shot 2022-01-12 at 9.35.31 PM.png

Am I missing something? Database is still blowing up at 28gb and rising. How do I stop it using xf_search without disabling all search functions?

Thanks @AndyB
 

sross

Active member
throwing an error that it can't find it hmm, guess i have to create the folder somewhere? why are there no instructions for setting this up? I have looked everywhere and can find no info
 

Deebs

Well-known member
throwing an error that it can't find it hmm, guess i have to create the folder somewhere? why are there no instructions for setting this up? I have looked everywhere and can find no info
You can leave it blank (didn't see the bit where it uses your database name as the index name). Even if you specified one it should instruct ES to create the necessary index. You do not need to mess with the filesystem.

What version of ES are you running?
 

sross

Active member
maybe es is not even running on my server at the moment, i tried starting/stopping it the way I used to and got no response. is there an easy way to tell if it's running? whm server with ssh access
 

sross

Active member
Ok after some help from Andy, got my elastisearch index built again and in settings elastisearch is enabled. I emptied my xf_search table after this which grew to 35 GB. I was hoping after enhanced search rebuilt that the xf_search would remain empty. Or does it not remain empty? It's already grown to 1 GB and seems to still be growing. With elastisearch running and selected, why is xf_search still growing? Thanks
 

Brogan

XenForo moderator
Staff member
To clarify, the xf_search_index table contains the indexed content when using MySQL search. It is not used when XFES is installed.

The xf_search table contains records of searches performed by visitors and is used for both MySQL search and XFES.
 

sross

Active member
To clarify, the xf_search_index table contains the indexed content when using MySQL search. It is not used when XFES is installed.

The xf_search table contains records of searches performed by visitors and is used for both MySQL search and XFES.
Thanks Brogan,

My xf_search_index is only 5kb but my xf_search is growing without stopping, it gained 25 GB overnight while I slept and is still growing. It is flooded with xengallery media inserts. My gallery is not THAT busy, so maybe a ddos or some glitch in the gallery hitting endless searches? If anyone has any ideas how I can track it down let me know. I'll check web logs and see if I can find a rogue ip hitting the gallery. @AndyB as well

Here is a screenshot of most recent inserts

Screen Shot 2022-01-13 at 9.26.22 AM.png
 

sross

Active member
ok that was useless, the only thing that stopped it was disabling access to the gallery for unregistered, but the bot is still hammering /forum/find-new which is recent posts and new posts, so that is filling up my xf_search as well and almost acting like a DDoS, ugh
 

sross

Active member
I made a typo so set it to:

Disallow: /forum/find-new/

No change. I tried disabling search to unregistered users but unregistered can still use any /find-new searches

I guess my only option is to find a way to block unregistered users from using /find-new ? If that is doable

from google:

Google usually checks your robots.txt file every 24-36 hours at the most. Google obeys robots directives. If it looks like Google is accessing your site despite robots.txt disallow you may want to use reverse DNS to verify and ensure it is not a bad actor pretending to be Googlebot.

i'll give it some time to see if it will recognise /find-new/
 
Last edited:
Top