XF 2.2 Crawler add-on. How?

[Anatoliy]

Greeting,

I'd like to build and add-on that will visit say 500 threads from a list (database table?) every 15 mnts using cronjob.
The goal is to keep them cashed for guests (as my forum is not heavily visited).

What would be the direction to move?
Thanks in advance.

 

[VBX Co]

What are you hoping this will achieve? Faster loading for visitors?

What caching do you have setup on your server?

 

[Anatoliy]

What are you hoping this will achieve? Faster loading for visitors?

yep, for guest visitors. to keep those 500 pages always in status "cashed for guests".

What caching do you have setup on your server?

Redis + Xon's Redis add-on. changed default 600 secs to 900 in the config.

 

[Anatoliy]

I created an array with several urls in the actionIndex in the controller, and it looks like all that I need is to send requests with

foreach ($array AS $key => $value)
{
    send request
}

So what would be the code for "send request"?
file_get_contents("$value"); ?

 

[VBX Co]

Have you tested the loading speed of your pages? A non-cached thread vs the same thread once it has been cached? Is it considerably faster?

This just seems like a waste of server resources, insomuch as you've already said the site isn't heavily visited, so who is going to benefit from these 500 threads being churned into cache all the time?

It might be more useful to look at ways of improving overall load time for the site as a whole. What are your server specs and how many threads, messages and members do you have? Have you done any testing?

 

[Anatoliy]

Have you tested the loading speed of your pages? A non-cached thread vs the same thread once it has been cached? Is it considerably faster?

pagespeed insights gives 90 on a first scan and then 95-96 for 15 mnts.

so who is going to benefit from these 500 threads being churned into cache all the time?

google bot of course )

What are your server specs and how many threads, messages and members do you have?

• 8 GB RAM,
• 4 CPU
• 90 GB SSD
• 100 Mbit port

17k threads. 230k posts.

P.S. Dude, you're killing a spirit of a nub add-on developer. Of course first projects will be as silly as 'hello world". )

 

[VBX Co]

P.S. Dude, you're killing a spirit of a nub add-on developer. Of course first projects will be as silly as 'hello world". )

Sorry, it just seemed like a bit of a fruitless exercise, but we didn't realise it was a first add-on project; we'll leave you to it. Others should be able to offer guidance on the coding, it's not something we've ever got into. Good luck.

 

[Anatoliy]

Good luck.

thanks)