XF 2.2 Crawler add-on. How?

Anatoliy

Well-known member
Greeting,

I'd like to build and add-on that will visit say 500 threads from a list (database table?) every 15 mnts using cronjob.
The goal is to keep them cashed for guests (as my forum is not heavily visited).

What would be the direction to move?
Thanks in advance.
 

VBX Co

Active member
What are you hoping this will achieve? Faster loading for visitors?

What caching do you have setup on your server?
 

Anatoliy

Well-known member
I created an array with several urls in the actionIndex in the controller, and it looks like all that I need is to send requests with
PHP:
foreach ($array AS $key => $value)
{
    send request
}

So what would be the code for "send request"?
file_get_contents("$value"); ?
 

VBX Co

Active member
Have you tested the loading speed of your pages? A non-cached thread vs the same thread once it has been cached? Is it considerably faster?

This just seems like a waste of server resources, insomuch as you've already said the site isn't heavily visited, so who is going to benefit from these 500 threads being churned into cache all the time?

It might be more useful to look at ways of improving overall load time for the site as a whole. What are your server specs and how many threads, messages and members do you have? Have you done any testing?
 

Anatoliy

Well-known member
Have you tested the loading speed of your pages? A non-cached thread vs the same thread once it has been cached? Is it considerably faster?
pagespeed insights gives 90 on a first scan and then 95-96 for 15 mnts.

psi.jpg



so who is going to benefit from these 500 threads being churned into cache all the time?
google bot of course )
What are your server specs and how many threads, messages and members do you have?

  • 8 GB RAM,
  • 4 CPU
  • 90 GB SSD
  • 100 Mbit port
17k threads. 230k posts.

P.S. Dude, you're killing a spirit of a nub add-on developer. Of course first projects will be as silly as 'hello world". )
 

VBX Co

Active member
P.S. Dude, you're killing a spirit of a nub add-on developer. Of course first projects will be as silly as 'hello world". )

Sorry, it just seemed like a bit of a fruitless exercise, but we didn't realise it was a first add-on project; we'll leave you to it. Others should be able to offer guidance on the coding, it's not something we've ever got into. Good luck. :)
 
Top