Google indexing problem

[Ekonomist]

i just puplished a forum about 1 month ago. first two week google indexed everything perfectly. When i and my members started to post a lot threads google stoped to indexing. Discovered - currently not indexed pages increasing fastly.
And now, i recognized that my old index wich was indexed first, now not on google. Google started to delete my index on search results.

im sure theres no problem with robots.txt or any other noindex tag about my page. i see all kind of google bots are all time on my website.

what do you think? whats the wrong about my site? and what can i do to solve this problem?

thank you.

 

[djbaxter]

Are you talking about "indexed" as in the Google Search Console? or in actual search results for your forum, e.g., site:yourforum.com?

 

[Ekonomist]

@djbaxter i mean was when i search as site:myforum.com same of indexed before are not seeing now.

 

[djbaxter]

You're probably looking at a variant of Google's "new site" effect. Google doesn't initially know how to rank or apportion a googlebot crawling budget to a brand new site so it may initially give it the benefit of the doubt. Once the algorithms have had time to properly evaluate the site, it will drop back to normal processing.

Make sure you set up a Google Search Console account for the forum and submit an initial link to your sitemap., i.e., https://yourforum.com/sitemap.php. This will help to boost crawling your site.

Also create a robots.txt file in the root folder of your domain, something like this:

User-agent: *
Disallow: /forum/whats-new/
Disallow: /forum/account/
Disallow: /forum/attachments/
Disallow: /forum/goto/
Disallow: /forum/posts/
Disallow: /forum/login/
Disallow: /forum/admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.php

 

[Ekonomist]

Thank you. @djbaxter I already done all of theme and more. I think i need to wait. So i will check after a few months again.

 

[Joe Kuhn]

...Make sure you set up a Google Search Console account for the forum and submit an initial link to your sitemap., i.e., https://yourforum.com/sitemap.php. This will help to boost crawling your site.

Also create a robots.txt file in the root folder of your domain, something like this:

User-agent: *
Disallow: /forum/whats-new/
Disallow: /forum/account/
Disallow: /forum/attachments/
Disallow: /forum/goto/
Disallow: /forum/posts/
Disallow: /forum/login/
Disallow: /forum/admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.php

This new guy appreciates your note:

1. Setup a Google Search Console account - done
2. Submit an initial link to my sitemap - done
3. Create a robots.txt file in the root folder, something like this... - questions:

I'm sorry, but I'm not seeing any of those Disallow directories off of public_html in CPanel. Is your 'forum' directory something I should replace with an actual path in my installation? I did find attachments at public_html/data/attachments and another path, but I'm not sure that's what you're referring to. Include them all? Perhaps 'forum' is a place holder that robots understand to mean - ignore all 'attachments' directories in this forum.

[edit: Some of your paths don't exist at all. Others yield search results with hundreds of files with that part in their names: login.]

Thanks!

 

[djbaxter]

Apologies. I copied that from another forum where the actual forum is in a subfolder called forum.

If your forum is in the root, then use this:

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.php

 

[Joe Kuhn]

Apologies. I copied that from another forum where the actual forum is in a subfolder called forum.

If your forum is in the root, then use this:

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.php

Hm, still don't see those directories. My XF version is 2.1.7.

 

[djbaxter]

They're not directories. They are XF "functions".

 

[Joe Kuhn]

3. Create a robots.txt file in the root folder, something like this. - Done

Thanks!!!

 

[Joe Kuhn]

Found another discussion that said:

/account/ = https://your-site.com/account/

So I checked them all for my site. Some I got the "requested page not found" message, so I took them out of my robots.txt file:

Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/

 

[Chromaniac]

sorry to barge in this thread. i had a related query. google has been indexing post permalink urls on my board. which it is not really a big deal. was just wondering how would one disallow these urls:

https://xenforo.com/community/threads/google-indexing-problem.177402/post-1408771

these links have nofollow on them of course.

 

[djbaxter]

In robots.txt:

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /members/
Disallow: /admin.php
Allow: /

 

[Chromaniac]

Disallow:/threads/*/post-* worked!