• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

understanding the behavior of google bots

Diana

Active member
#1
I don't know much about crawlers or SEO, and I don't understand how google bots operate.

Watching the behavior of google bots on my forum it appears that they're stuck in some kind of netherland. All it shows is that they're "viewing unknown page." Once in awhile they view the "events calendar." I've never seen them view a thread, and there are quite a lot of them now.

Can someone help me understand this process better, and is there anything I can do to optimize how google crawls my forum?

btw, I do have a robots file.

User-agent: *
Disallow: /forum/find-new/
Disallow: /forum/forums/-/
Disallow: /forum/account/
Disallow: /forum/attachments/
Disallow: /forum/goto/
Disallow: /forum/posts/
Disallow: /forum/login/
Disallow: /forum/admin.php
Allow: /
 

Jake Bunce

XenForo moderator
Staff member
#2
That "unknown" page might be an addon that has failed to define its location. To discover the unknown location you need to look in the xf_session_activity table in the database. What is the controller name and action in the associated record? Are there any params? That will tell us what page they are viewing.

But you can't control or direct the bots. They just follow links.
 

Dean

Well-known member
#3
Keep tweaking the robots.txt file, then checking the webmaster tools to see how your site is showing up. It takes a while to get things optimized.

For instance "Disallow: /forum/calendar/ might make a nice addition to your robots.txt file.

Googlebots remind me of these things, they go everywhere they can from what I can tell..:

 

Diana

Active member
#4
That "unknown" page might be an addon that has failed to define its location. To discover the unknown location you need to look in the xf_session_activity table in the database. What is the controller name and action in the associated record? Are there any params? That will tell us what page they are viewing.

But you can't control or direct the bots. They just follow links.
Hi Jake, thanks for your reply.

Doing a search in the tables for xf_session_activity, I get this: The mcrypt extension is missing. Please check your PHP configuration

Checking the database I could only find this for xf_session_activity.

xf_session_activity (is unchecked)

xenforo.xf_session_activity
note : The storage engine for the table doesn't support check

These were also unchecked:

xenforo.xf_thread_view
note : The storage engine for the table doesn't support check

xenforo.xf_attachment_view
note : The storage engine for the table doesn't support check

It's all greek to me.
 

Diana

Active member
#5
Keep tweaking the robots.txt file, then checking the webmaster tools to see how your site is showing up. It takes a while to get things optimized.

For instance "Disallow: /forum/calendar/ might make a nice addition to your robots.txt file.

Googlebots remind me of these things, they go everywhere they can from what I can tell..:


Which webmaster tools would you recommend?
 

Jake Bunce

XenForo moderator
Staff member
#6
Checking the database I could only find this for xf_session_activity.

xf_session_activity (is unchecked)

xenforo.xf_session_activity
note : The storage engine for the table doesn't support check

These were also unchecked:

xenforo.xf_thread_view
note : The storage engine for the table doesn't support check

xenforo.xf_attachment_view
note : The storage engine for the table doesn't support check

It's all greek to me.
You only need to do this if you want to discover the exact location of a user that is currently online.

You need to browse the records in xf_session_activity to see the information.
 

Diana

Active member
#7
Jake, I'm sorry, you're talking to a complete idiot where databases are concerned. Nothing I see makes sense to me. I'm just stumbling around in the database, not finding anything you mentioned above.

This is what I see:
 

Attachments

Jake Bunce

XenForo moderator
Staff member
#9
Jake, I'm sorry, you're talking to a complete idiot where databases are concerned. Nothing I see makes sense to me. I'm just stumbling around in the database, not finding anything you mentioned above.

This is what I see:
That's the one.

What is the controller name and action in the associated record? Are there any params? That will tell us what page they are viewing.
 

Diana

Active member
#10
hehe, like I said, it's all greek to me. This is what shows up when I hover over the URL google was visiting:

/forum/index.php?misc/style&redirect=%2Fforum%2Findex.php%3Fmisc%2Fstyle%26redirect%3D%252Fforum%252Findex.php%253Fmisc%252Fstyle%2526redirect%253D%25252Fforum%25252Findex.php%25253Fmisc%25252Fstyle%252526redirect%25253D%2525252Fforum%2525252Findex.php%2525253Fmisc%2525252Fstyle%25252526redirect%2525253D%252525252Fforum%252525252Findex.php%252525253Fmisc%252525252Fstyle%2525252526redirect%252525253D%25252525252Fforum%25252525252Findex.php%25252525253Fmisc%25252525252Fstyle%252525252526redirect%25252525253D%2525252525252Fforum%2525252525252Findex.php%2525252525253Fmisc%2525252525252Fstyle%25252525252526redirect%2525252525253D%252525252525252Fforum%252525252525252Findex.php%252525252525253Fmisc%252525252525252Fstyle%2525252525252526redirect%252525252525253D%25252525252525252Fforum%25252525252525252Findex.php%25252525252525253Fmisc%25252525252525252Fstyle%252525252525252526redirect%25252525252525253D%2525252525252525252Fforum%2525252525252525252Findex.php%2525252525252525253Fmisc%2525252525252525252Fstyle%25252525252525252526redirect%2525252525252525253D%252525252525252525252Fforum%252525252525252525252Findex.php%252525252525252525253Fmisc%252525252525252525252Fstyle%2525252525252525252526redirect%252525252525252525253D%25252525252525252525252Fforum%25252525252525252525252Findex.php%25252525252525252525253Fevents%25252525252525252525252Fweekly%25252525252525252525252F06.2015%25252525252525252525252F


Guess that means it was looking at the style chooser:

As far as what you asked here:

What is the controller name and action in the associated record? Are there any params?
I can't see any way of finding that information.
 

Diana

Active member
#18
I'm also seeing some strange queries.

What is this?

* jQuery Tools 1.2.5 - The missing UI library for the Web
*
* [tabs, tooltip, tooltip.slide, scrollable, scrollable.navigator, overlay, overlay.apple, dateinput, rangeinput, toolbox.history, toolbox.expose, toolbox.mousewheel]
*
* NO COPYRIGHTS OR LICENSES. DO WHAT YOU LIKE.
*
* http://flowplayer.org/tools/
*
* jquery.event.wheel.js - rev 1
* Copyright (c) 2008, Three Dub Media (http://threedubmedia.com)
* Liscensed under the MIT License (MIT-LICENSE.txt)
* http://www.opensource.org/licenses/mit-license.php
* Created: 2008-07-01 | Updated: 2008-07-14
*