Not a bug Google following url's in code tag?

[dutchbb]

Affected version

2.0

Not sure if this also happens on xf1, but had this crawl error in webmaster tools. Both give a 405 code and were placed in xenforo [code ] [/ code] on this forum. Wouldn't have noticed if they were not add-reply and add-thread url's

Placed xenforo url's in it so can be checked. Same format as how i placed it, was from a server error.

array(3) {
  ["url"] => string(77) "https://xenforo.com/community/threads/google-following-urls-in-code-tag.153950/add-reply"
  ["_GET"] => array(0) {
  }

array(3) {
  ["url"] => string(77) "https://xenforo.com/community/forum/bugs/add-thread"
  ["_GET"] => array(0) {
  }

 

[Chris D]

How sure are you that this actually happens through a non-linked URL vs them just discovering the link in some other way, such as a link a user posted somewhere?

I guess it is possible that Google is following URLs that are not linked, but there’s not exactly anything we can do about that.

 

[dutchbb]

100% sure. With webmaster tools you can check where the links are coming from (most of the time). Both of them from 2 seperate threads on this forum with those exact links.

It is also not normal that google would follow those and indeed not much to do about it, but interesting to know. Maybe a bug or test from their side. Will check if there is more to find about this.

 

[Chris D]

I found something that suggested that theoretically they do it as part of understanding brand awareness of certain brands.

For example it’s potentially useful for them to pick out things like domain names such as google.com which wouldn’t ordinarily be links, but perhaps they’ve taken it one further and they seliberaly scrape anything that looks like a URL.

Whether or not they treat it in the same way in terms of actually following it with the intention of indexing the content or whatever, I don’t know. Certainly seems odd that they’d log it and report it in webmaster tools.

 

[dutchbb]

Google is indeed following those (recently), got an error today from an url in code tag on this forum. I'm thinking this is because they are between ".."

Yep, pretty sure that is it. Found some 404's also from url's like /deferred.php and add-on url's that are not visible for guests. All are between ".." posted on xf as error logs between code tags.