Sitemap

[Spartan]

As I have been saying since the 90s, all SEO for forums starts with on page SEO — and it's not initially about user generated content but about things like

• site title
• site description
• titles/names of categories
• description of each category
• titles/names of nodes
• description of each node
• make sure that the descriptions are always visible, not tooltips or popups, and that they are visible to non-logged in guests (searchbots)
• do the same for subforums/subnodes
• to an extent, the order of your nodes: move the chatter stuff to the bottom, not at the top of the categories
• check the box for specific nodes to NOT index the chatter stuff — index the meat of the site, not the silly stuff that doesn't even represent what your forums are about
• always use seo-friendly URLs

These are all opportunities to get a variety of defining and relevant search terms in front of search bots. In today's world of search term generated search results, this is critical.

Set up a Google Search Console account for your XF site and manually submit a sitemap to kickstart crawling.

Create a robots.txt file like this to filter out duplicates:

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /members/
Disallow: /admin.php
Disallow: /tags/
Disallow: /editprefix
Allow: /

Sitemap: https://{your domain}.com/sitemap.xml

Then review your GSC results on a semi-regular basis looking for where you need to improve titles and descriptions.

If I want my prefixes indexed, should I eliminate "Disallow: /editprefix"?

'Also, I'm a bit confused. Should the sitemap.xml file be readable? Mine looks odd:

https://the-boneyard.com/sitemap.xml

 

[JoyFreak]

If I want my prefixes indexed, should I eliminate "Disallow: /editprefix"?

'Also, I'm a bit confused. Should the sitemap.xml file be readable? Mine looks odd:

https://the-boneyard.com/sitemap.xml

Yes, remove it if you want it indexed.

That looks correct. When a site is as big as yours, it breaks the sitemap down into separate files.

 

[MySiteGuy]

That's incorrect advice. Allowing the edit prefix page to be crawled won't allow crawling prefixes. Guest can't view edit prefix pages. Search engines will crawl an error page. If you want to see what search engines see, log out and then view the page(s) you want to check.

 

[JoyFreak]

That's incorrect advice. Allowing the edit prefix page to be crawled won't allow crawling prefixes. Guest can't view edit prefix pages. Search engines will crawl an error page. If you want to see what search engines see, log out and then view the page(s) you want to check.

It’s not incorrect. If he removes anything from that disallow list, it means it will index. Of course, if guests can’t access it, then robots can’t access it so it’s pointless on there either way.

 

[MySiteGuy]

He asked "If I want my prefixes indexed"
Allowing that page will not get prefixes indexed. It will get an error page indexed.

Your advice was incorrect and doubling down to explain your way out of it ain't helping.

 

[JoyFreak]

He asked "If I want my prefixes indexed"
Allowing that page will not get prefixes indexed. It will get an error page indexed.

Your advice was incorrect and doubling down to explain your way out of it ain't helping.

I’m not looking to score points about who is correct or not. I just hope the OP got his answer either way.

 

[MySiteGuy]

Joyfreak, I'm not interested in whatever spurred your doubling and tripling down.

@Spartan,
Be very cautious about advice for SEO. There's as much, if not more, poor advice out there as there is good advice. If it were me, I would not concern myself with Google indexing a list of thread prefixes. By themselves they aren't content. Things like that, Google is pretty good at ferreting out as thin content, "soft 404", or keyword stuffed pages. Use them as part of overall title and H tag structure to help classify site content with your focus on being helpful to the user. If you make it easier for people to find what they are looking for on your site, you're making it easier on search engines too (though there are exceptions).

 

[Spartan]

That's incorrect advice. Allowing the edit prefix page to be crawled won't allow crawling prefixes. Guest can't view edit prefix pages. Search engines will crawl an error page. If you want to see what search engines see, log out and then view the page(s) you want to check.

Im not sure what an edit prefix page is. I'm referring to thread prefixes. These are the threads I most want indexed.

 

[MySiteGuy]

I was referring to the editprefix URL you asked about.

Don't worry about prefixes and Google indexing. It will crawl all threads regardless of prefix. Google isn't a "your wish is our command" search engine. Giving them an alternate crawling route to those prefixed threads will not change how they rank them. When it crawls both regular threads and those with prefixes they decide which to index in search, and how well, based on ranking factors of each page individually and the site as a whole.