understanding the behavior of google bots

[Diana]

I don't know much about crawlers or SEO, and I don't understand how google bots operate.

Watching the behavior of google bots on my forum it appears that they're stuck in some kind of netherland. All it shows is that they're "viewing unknown page." Once in awhile they view the "events calendar." I've never seen them view a thread, and there are quite a lot of them now.

Can someone help me understand this process better, and is there anything I can do to optimize how google crawls my forum?

btw, I do have a robots file.

User-agent: *
Disallow: /forum/find-new/
Disallow: /forum/forums/-/
Disallow: /forum/account/
Disallow: /forum/attachments/
Disallow: /forum/goto/
Disallow: /forum/posts/
Disallow: /forum/login/
Disallow: /forum/admin.php
Allow: /

 

[Jake Bunce]

That "unknown" page might be an addon that has failed to define its location. To discover the unknown location you need to look in the xf_session_activity table in the database. What is the controller name and action in the associated record? Are there any params? That will tell us what page they are viewing.

But you can't control or direct the bots. They just follow links.

 

[Dean]

Keep tweaking the robots.txt file, then checking the webmaster tools to see how your site is showing up. It takes a while to get things optimized.

For instance "Disallow: /forum/calendar/ might make a nice addition to your robots.txt file.

Googlebots remind me of these things, they go everywhere they can from what I can tell..:

 

[Diana]

That "unknown" page might be an addon that has failed to define its location. To discover the unknown location you need to look in the xf_session_activity table in the database. What is the controller name and action in the associated record? Are there any params? That will tell us what page they are viewing.

But you can't control or direct the bots. They just follow links.

Hi Jake, thanks for your reply.

Doing a search in the tables for xf_session_activity, I get this: The mcrypt extension is missing. Please check your PHP configuration

Checking the database I could only find this for xf_session_activity.

xf_session_activity (is unchecked)

xenforo.xf_session_activity
note : The storage engine for the table doesn't support check

These were also unchecked:

xenforo.xf_thread_view
note : The storage engine for the table doesn't support check

xenforo.xf_attachment_view
note : The storage engine for the table doesn't support check

It's all greek to me.

 

[Diana]

Keep tweaking the robots.txt file, then checking the webmaster tools to see how your site is showing up. It takes a while to get things optimized.

For instance "Disallow: /forum/calendar/ might make a nice addition to your robots.txt file.

Googlebots remind me of these things, they go everywhere they can from what I can tell..:

Which webmaster tools would you recommend?

 

[Jake Bunce]

Checking the database I could only find this for xf_session_activity.

xf_session_activity (is unchecked)

xenforo.xf_session_activity
note : The storage engine for the table doesn't support check

These were also unchecked:

xenforo.xf_thread_view
note : The storage engine for the table doesn't support check

xenforo.xf_attachment_view
note : The storage engine for the table doesn't support check

It's all greek to me.

You only need to do this if you want to discover the exact location of a user that is currently online.

You need to browse the records in xf_session_activity to see the information.

 

[Diana]

Jake, I'm sorry, you're talking to a complete idiot where databases are concerned. Nothing I see makes sense to me. I'm just stumbling around in the database, not finding anything you mentioned above.

This is what I see:

 

[Diana]

Does this help?

 

[Jake Bunce]

Jake, I'm sorry, you're talking to a complete idiot where databases are concerned. Nothing I see makes sense to me. I'm just stumbling around in the database, not finding anything you mentioned above.

This is what I see:

That's the one.

What is the controller name and action in the associated record? Are there any params? That will tell us what page they are viewing.

 

[Diana]

hehe, like I said, it's all greek to me. This is what shows up when I hover over the URL google was visiting:

/forum/index.php?misc/style&redirect=%2Fforum%2Findex.php%3Fmisc%2Fstyle%26redirect%3D%252Fforum%252Findex.php%253Fmisc%252Fstyle%2526redirect%253D%25252Fforum%25252Findex.php%25253Fmisc%25252Fstyle%252526redirect%25253D%2525252Fforum%2525252Findex.php%2525253Fmisc%2525252Fstyle%25252526redirect%2525253D%252525252Fforum%252525252Findex.php%252525253Fmisc%252525252Fstyle%2525252526redirect%252525253D%25252525252Fforum%25252525252Findex.php%25252525253Fmisc%25252525252Fstyle%252525252526redirect%25252525253D%2525252525252Fforum%2525252525252Findex.php%2525252525253Fmisc%2525252525252Fstyle%25252525252526redirect%2525252525253D%252525252525252Fforum%252525252525252Findex.php%252525252525253Fmisc%252525252525252Fstyle%2525252525252526redirect%252525252525253D%25252525252525252Fforum%25252525252525252Findex.php%25252525252525253Fmisc%25252525252525252Fstyle%252525252525252526redirect%25252525252525253D%2525252525252525252Fforum%2525252525252525252Findex.php%2525252525252525253Fmisc%2525252525252525252Fstyle%25252525252525252526redirect%2525252525252525253D%252525252525252525252Fforum%252525252525252525252Findex.php%252525252525252525253Fmisc%252525252525252525252Fstyle%2525252525252525252526redirect%252525252525252525253D%25252525252525252525252Fforum%25252525252525252525252Findex.php%25252525252525252525253Fevents%25252525252525252525252Fweekly%25252525252525252525252F06.2015%25252525252525252525252F


Guess that means it was looking at the style chooser:

As far as what you asked here:

What is the controller name and action in the associated record? Are there any params?

I can't see any way of finding that information.

 

[Jake Bunce]

Yep. It looks like a style chooser URL.

Is that one URL? It's all messed up.

 

[dieketzer]

Which webmaster tools would you recommend?

hes talking about googles webmaster tools. it will allow you to inspect/alter various aspects of your sites interaction with google.
http://www.google.com/webmasters/tools

 

[Dean]

hes talking about googles webmaster tools. it will allow you to inspect/alter various aspects of your sites interaction with google.
http://www.google.com/webmasters/tools

There you go, that is indeed what I was referring to. Thanks!

 

[Diana]

Yep. It looks like a style chooser URL.

Is that one URL? It's all messed up.

It's messed up the way it accesses it. It's normally a popup.

 

[Diana]

hes talking about googles webmaster tools. it will allow you to inspect/alter various aspects of your sites interaction with google.
http://www.google.com/webmasters/tools

Thank you!

 

[Diana]

There you go, that is indeed what I was referring to. Thanks!

And thank you too

 

[Diana]

Yep. It looks like a style chooser URL.

Is that one URL? It's all messed up.

As I check each visit from google it appears that's the only URL it's accessing. No wonder it says "unknown page."

Why is it doing that?

 

[Diana]

I'm also seeing some strange queries.

What is this?

* jQuery Tools 1.2.5 - The missing UI library for the Web
*
* [tabs, tooltip, tooltip.slide, scrollable, scrollable.navigator, overlay, overlay.apple, dateinput, rangeinput, toolbox.history, toolbox.expose, toolbox.mousewheel]
*
* NO COPYRIGHTS OR LICENSES. DO WHAT YOU LIKE.
*
* http://flowplayer.org/tools/
*
* jquery.event.wheel.js - rev 1
* Copyright (c) 2008, Three Dub Media (http://threedubmedia.com)
* Liscensed under the MIT License (MIT-LICENSE.txt)
* http://www.opensource.org/licenses/mit-license.php
* Created: 2008-07-01 | Updated: 2008-07-14
*

 

[Diana]

Why are they looking at these files?

/forum/js/xenforo/xenforo.js?_v=21d5099e

 

[Jake Bunce]

As I check each visit from google it appears that's the only URL it's accessing. No wonder it says "unknown page."

Why is it doing that?

Add it to your robots file.