XF 1.5 my xf_search table is massive all of a sudden, rebuild doesn't help

[sross]

Hi, not sure what is going on here. I noticed a backup taking too long to find my xf_search table had blown out to 85 GB. My database backups are usually 1.1 GB. Forum has 2.4mil posts. I am running elastisearch. I emptied the table and re-ran the indexing. It's been going for hours and table is now 25 GB and rising. Not sure what to do. Thanks for any help.

 

[AndyB]

Sounds like you don’t have Enhanced search enabled and are rebuilding the standard search.

 

[sross]

Sounds like you don’t have Enhanced search enabled and are rebuilding the standard search.

right, I thought I had it enabled. Investigating now. Thanks! Been so long since I've dealt with it.

 

[sross]

ok it was enabled and elastisearch installed a while back. I tried re-installing. I have this on my config page:



Am I missing something? Database is still blowing up at 28gb and rising. How do I stop it using xf_search without disabling all search functions?

Thanks @AndyB

 

[sross]

I also see no option for "enhanced search" on my options page, eventhough it's installed. Maybe there is a direct link I can use?

 

[sross]

ok i created an index name and in tools->elastisearch am re-building it. once this is done I just empty the xf_search table? cheers!

 

[sross]

throwing an error that it can't find it hmm, guess i have to create the folder somewhere? why are there no instructions for setting this up? I have looked everywhere and can find no info

 

[Deebs]

throwing an error that it can't find it hmm, guess i have to create the folder somewhere? why are there no instructions for setting this up? I have looked everywhere and can find no info

You can leave it blank (didn't see the bit where it uses your database name as the index name). Even if you specified one it should instruct ES to create the necessary index. You do not need to mess with the filesystem.

What version of ES are you running?

 

[sross]

You can leave it blank (didn't see the bit where it uses your database name as the index name). Even if you specified one it should instruct ES to create the necessary index. You do not need to mess with the filesystem.

What version of ES are you running?

 

[sross]

maybe es is not even running on my server at the moment, i tried starting/stopping it the way I used to and got no response. is there an easy way to tell if it's running? whm server with ssh access

 

[AndyB]

What is the server error log you are showing?

 

[AndyB]

is there an easy way to tell if it's running? whm server with ssh access

At the terminal enter:

service elasticsearch status

 

[AndyB]

To start elasticsearch enter:

service elasticsearch start

 

[sross]

Ok after some help from Andy, got my elastisearch index built again and in settings elastisearch is enabled. I emptied my xf_search table after this which grew to 35 GB. I was hoping after enhanced search rebuilt that the xf_search would remain empty. Or does it not remain empty? It's already grown to 1 GB and seems to still be growing. With elastisearch running and selected, why is xf_search still growing? Thanks

 

[Paul B]

To clarify, the xf_search_index table contains the indexed content when using MySQL search. It is not used when XFES is installed.

The xf_search table contains records of searches performed by visitors and is used for both MySQL search and XFES.

 

[sross]

To clarify, the xf_search_index table contains the indexed content when using MySQL search. It is not used when XFES is installed.

The xf_search table contains records of searches performed by visitors and is used for both MySQL search and XFES.

Thanks Brogan,

My xf_search_index is only 5kb but my xf_search is growing without stopping, it gained 25 GB overnight while I slept and is still growing. It is flooded with xengallery media inserts. My gallery is not THAT busy, so maybe a ddos or some glitch in the gallery hitting endless searches? If anyone has any ideas how I can track it down let me know. I'll check web logs and see if I can find a rogue ip hitting the gallery. @AndyB as well

Here is a screenshot of most recent inserts

 

[sross]

ok, i have found the offender, it is a google bot smashing my gallery. Now just need to figure out how to stop it.

 

[sross]

I'm trying this in robots.txt

Disallow: /forum/find-new

that is what the bot is hitting 100,000s of times

 

[sross]

ok that was useless, the only thing that stopped it was disabling access to the gallery for unregistered, but the bot is still hammering /forum/find-new which is recent posts and new posts, so that is filling up my xf_search as well and almost acting like a DDoS, ugh

 

[sross]

I made a typo so set it to:

Disallow: /forum/find-new/

No change. I tried disabling search to unregistered users but unregistered can still use any /find-new searches

I guess my only option is to find a way to block unregistered users from using /find-new ? If that is doable

from google:

Google usually checks your robots.txt file every 24-36 hours at the most. Google obeys robots directives. If it looks like Google is accessing your site despite robots.txt disallow you may want to use reverse DNS to verify and ensure it is not a bad actor pretending to be Googlebot.

i'll give it some time to see if it will recognise /find-new/