XF 2.1 Where are the Spiders

[Ben44]

I have two forums, one using Vbulletin 5 and more a list of recipes than an actual forum and the other
is a proper forum using Xenforo.

My Vbulletin forum is being crawled but my Xenforo isn't, any ideas why ?

 

[djbaxter]

Are you sure it isn't? Have you checked the origins and nature of Guests?

https://{your forum.com}/online/

Should see 4 tabs:



But not all bots are identified as robots unless you use an addon. Some of those listed under Guests are bots.

 

[Ben44]

Xenforo


VB


That's how it stands at this moment. Sometimes there are in excess of 20 crawlers online.

 

[Ben44]

Well it would seem that Vbulletin is a better coded forum than Xenforo.
If spiders are finding Vbulletins forums and not finding Xenforo, that says to me that VB is better written and the SEO is
of a higher quality, so it looks like I will have to ditch this one and move my forum to the better quality Vbulletin.

Unless someone can come on and reassure me that this is not the case ?

 

[iCafe]

bot not stay but come and go.

 

[sbj]

Right now for me:



I am assuming your site is new as you have no robots. It is normal that in the early stage that you get no bots at all. As bots only come if they know the site. They can only know your site, if someone visits your site. And it looks like you have no activity, nobody visiting your site, so no bots online.

 

[Ozzy47]

This could be a issue as well.

 

[HotRodCarts]

It can't be helping the issue.

 

[Ben44]

The URL is

Westcountry Link

westcountry link

westcountrylink.com

as in my sig

 

[briansol]

as above, your ssl cert is broken. google won't crawl you when you have errors like this.

 

[Ozzy47]

Also the content is very thin, 99% of the site is RSS feeds. Nothing for search engines to see and index.

 

[djbaxter]

as above, your ssl cert is broken. google won't crawl you when you have errors like this.

Yes your certificate expired like 9 months ago.

Warning: Potential Security Risk Ahead

Firefox detected a potential security threat and did not continue to thewestcountrylink.com. If you visit this site, attackers could try to steal information like your passwords, emails, or credit card details.

Learn more…

Websites prove their identity via certificates, which are valid for a set time period. The certificate for thewestcountrylink.com expired on 8/6/2019.

Error code: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT

 

[Brad Padgett]

Do you have a robots.txt file? That is what usually will point spiders to your site to index it if they aren't showing up already.

Search it in Google to see how to make one. It basically tells the robots to come to your site.

 

[Max Taxable]

Do you have a robots.txt file? That is what usually will point spiders to your site to index it if they aren't showing up already.

Search it in Google to see how to make one. It basically tells the robots to come to your site.

I mean... Actually it nicely asks them to do this/please don't do that, once they arrive. It doesn't increase bot traffic. It's not a siren call.

 

[Brad Padgett]

I mean... Actually it nicely asks them to do this/please don't do that, once they arrive. It doesn't increase bot traffic. It's not a siren call.

I realize that. It still helps them show up.

 

[Max Taxable]

It still helps them show up.

How? It doesn't call out to them. It doesn't do anything until the bot visits. And even then, it's just a "please follow my directions" kind of thing.

You're not getting crawled until the search engines are aware of your site, and the SSL is correct, and there's content.

 

[Brad Padgett]

How? It doesn't call out to them. It doesn't do anything until the bot visits. And even then, it's just a "please follow my directions" kind of thing.

Actually if they're not showing up already then it does help them come to your site. I'm not here to argue. Was trying trying to give a helpful suggestion.

 

[Max Taxable]

Actually if they're not showing up already then it does help them come to your site

How? I'm not arguing, I am asking. How does it help spiders find and come to your site?

 

[Brad Padgett]

How? I'm not arguing, I am asking. How does it help spiders find and come to your site?

You may be a bit right but once they show up if you have it set to allow then I can guarantee it has an effect on how long they stay. That much I'm sure of.

 

[Max Taxable]

but once they show up if you have it set to allow then I can guarantee it has an effect on how long they stay

For the "noble" spiders that even bother to use it and obey it, probably.

If the OP wants crawlers he's got to get them. Needs to register his site with google to start with and verify it, and fix his SSL issues otherwise any robots.txt file he has is irrelevant.