XF 2.2 Problems from Vbulletin to Xenforo (with data)

[qwer81]

Hi guys, a few days ago I opened a topic on the decline in page views (but also users) after the transition from Vbulletin 4 to Xenforo (from the end of August).

Today, checking Search console, I noticed that the pages excluded from indexing are 888,000 (but the number was high even with Vbulletin) while those valid, to date, are 243.

523,000 scanned but not indexed

361,000 with redirect (always not indexed)

243 indexed (but not submitted via sitemap) (???)

In September the valid pages also reached 1600

I didn't have a Robots file until yesterday. I forgot to set it. The one inserted yesterday I report below.

I specify that all the url have been redirected from Vbulletin 4 + Dbseo to Xenforo:

Old Url: https://www.milanworld.net/milan-dalot-pole-aurier-e-odriozola-le-alternative-vt105324.html#post2378219

New url: https://www.milanworld.net/threads/...-odriozola-le-alternative.105324/#post2378219

What happens?!

Robots
User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /goto/
Disallow: /login/
Disallow: /members/
Disallow: /admin.php
Disallow: /tags/
Allow: /

User-agent: Mediapartners-Google*
Disallow:

Sitemap: https://www.milanworld.net/sitemap.php

 

[MySiteGuy]

The number seems to show Google can't get to the pages, and the issue we talked about with FB open graph is the same: guests do not have view permissions on many sections of the forum.

If it were me, I'd set guest permissions to:
Allow guests to view all forum pages except (a) private sections like moderator/admin sections, and (b) any premium (paid) membership pages.

 

[javakhir]

delete robots.txt & be happy

or leave one line only:

User-agent: *