XF 2.2 Enabling crawler access for Google AdSense

[36degrees]

Hi,

There are a few threads on setting up crawler access for Google AdSense already (like here, here and here) although none of them are conclusive. As far as I can tell from reading the existing threads, the expectation is that you can configure the AdSense crawler to POST to login/login with a login and password and then the crawler would be able to 'log in' and access pages that are otherwise not accessible by guests.

I've tried setting this up and according to Google the status is 'Crawling' with a nice green dot, however we're still not seeing ads in threads not visible to guests, and the user that we created for the crawler has had no activity and has no IP addresses associated with it other than my own from when I set it up.

As far as I can tell from testing the issue is that without a valid CSRF token in the POST data, it's always going to fail CSRF validation? Unless I'm missing something that's meant to work around this?

Does anyone think they've actually managed to get this working?

Ollie

 

[djbaxter]

No, that's not how it works.

AdSense needs to evaluate the content of pages on your site in order to

1. determine what ads are appropriate or best for the content of that page and the viewers of that page; and
2. confirm that the content of that page confirms to AdSense ToS.

The AdSense bot cannot register and login to your forum. Therefore, it will always conform to the Guest usergroup.

If Guests can't see something on your forum, neither can bots including the AdSense bot.

Id AdSense can't see it, you won't get any AdSense ads on that page.

 

[Black Tiger]

The AdSense bot cannot register and login to your forum.

Just out of curiosity.... if that is the case, why can you provide login credentials for the adsense bot?

Display ads on login-protected pages - Google AdSense Help

After your account has been activated, you can display Google ads on pages behind a login by creating a crawler login. This will give the AdSense ads crawler access to visit your pages and deli

support.google.com

 

[djbaxter]

Let me rephrase what I said above:

Under normal circumstances, the AdSense bot cannot register and login to your forum. Therefore, it will always conform to the Guest usergroup.

If you specifically provide login details to AdSense, that of course changes. But I don't advise anyone do that. What you are doing by that action is letting the AdSense not see EVERYTHING on your site that a registered logged in member can see. That is giving a lot of power to Google and arguably they already have too much of that.

Apart from that, consider the value to your forum of having AdSense ads displayed on a page that most potential members will only see once and never again at a time when all they are trying to do is register, not click on ads to take them off the registration page. How many click do you think you are going to get? How many additional impressions are you going to get? (Are you getting hundreds or thousands of new valid non-spam registrations a day? If not, you're not going to see any real difference in earnings.)

 

[Black Tiger]

I wouldn't give a crawler user access anyway, but I just was wondering about your statement.
Since you rephrased that, I understand how you mean it and fully agree with you!

 

[36degrees]

We have some forums that are closed to guests just to encourage registration. Other than that, these pages are not sensitive in any way and there is no reason not to allow the AdSense crawler to be able to see them.