1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

understanding the behavior of google bots

Discussion in 'XenForo Questions and Support' started by Diana, Nov 7, 2011.

  1. Diana

    Diana Active Member

    I don't know much about crawlers or SEO, and I don't understand how google bots operate.

    Watching the behavior of google bots on my forum it appears that they're stuck in some kind of netherland. All it shows is that they're "viewing unknown page." Once in awhile they view the "events calendar." I've never seen them view a thread, and there are quite a lot of them now.

    Can someone help me understand this process better, and is there anything I can do to optimize how google crawls my forum?

    btw, I do have a robots file.

    User-agent: *
    Disallow: /forum/find-new/
    Disallow: /forum/forums/-/
    Disallow: /forum/account/
    Disallow: /forum/attachments/
    Disallow: /forum/goto/
    Disallow: /forum/posts/
    Disallow: /forum/login/
    Disallow: /forum/admin.php
    Allow: /
     
  2. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    That "unknown" page might be an addon that has failed to define its location. To discover the unknown location you need to look in the xf_session_activity table in the database. What is the controller name and action in the associated record? Are there any params? That will tell us what page they are viewing.

    But you can't control or direct the bots. They just follow links.
     
  3. Dean

    Dean Well-Known Member

    Keep tweaking the robots.txt file, then checking the webmaster tools to see how your site is showing up. It takes a while to get things optimized.

    For instance "Disallow: /forum/calendar/ might make a nice addition to your robots.txt file.

    Googlebots remind me of these things, they go everywhere they can from what I can tell..:

    [​IMG]
     
    Degrinch likes this.
  4. Diana

    Diana Active Member

    Hi Jake, thanks for your reply.

    Doing a search in the tables for xf_session_activity, I get this: The mcrypt extension is missing. Please check your PHP configuration

    Checking the database I could only find this for xf_session_activity.

    xf_session_activity (is unchecked)

    xenforo.xf_session_activity
    note : The storage engine for the table doesn't support check

    These were also unchecked:

    xenforo.xf_thread_view
    note : The storage engine for the table doesn't support check

    xenforo.xf_attachment_view
    note : The storage engine for the table doesn't support check

    It's all greek to me.
     
  5. Diana

    Diana Active Member

    Which webmaster tools would you recommend?
     
  6. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    You only need to do this if you want to discover the exact location of a user that is currently online.

    You need to browse the records in xf_session_activity to see the information.
     
  7. Diana

    Diana Active Member

    Jake, I'm sorry, you're talking to a complete idiot where databases are concerned. Nothing I see makes sense to me. I'm just stumbling around in the database, not finding anything you mentioned above.

    This is what I see:
     

    Attached Files:

  8. Diana

    Diana Active Member

    Does this help?

    googlebot.png
     
  9. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    That's the one.

    What is the controller name and action in the associated record? Are there any params? That will tell us what page they are viewing.
     
  10. Diana

    Diana Active Member

    hehe, like I said, it's all greek to me. This is what shows up when I hover over the URL google was visiting:

    /forum/index.php?misc/style&redirect=%2Fforum%2Findex.php%3Fmisc%2Fstyle%26redirect%3D%252Fforum%252Findex.php%253Fmisc%252Fstyle%2526redirect%253D%25252Fforum%25252Findex.php%25253Fmisc%25252Fstyle%252526redirect%25253D%2525252Fforum%2525252Findex.php%2525253Fmisc%2525252Fstyle%25252526redirect%2525253D%252525252Fforum%252525252Findex.php%252525253Fmisc%252525252Fstyle%2525252526redirect%252525253D%25252525252Fforum%25252525252Findex.php%25252525253Fmisc%25252525252Fstyle%252525252526redirect%25252525253D%2525252525252Fforum%2525252525252Findex.php%2525252525253Fmisc%2525252525252Fstyle%25252525252526redirect%2525252525253D%252525252525252Fforum%252525252525252Findex.php%252525252525253Fmisc%252525252525252Fstyle%2525252525252526redirect%252525252525253D%25252525252525252Fforum%25252525252525252Findex.php%25252525252525253Fmisc%25252525252525252Fstyle%252525252525252526redirect%25252525252525253D%2525252525252525252Fforum%2525252525252525252Findex.php%2525252525252525253Fmisc%2525252525252525252Fstyle%25252525252525252526redirect%2525252525252525253D%252525252525252525252Fforum%252525252525252525252Findex.php%252525252525252525253Fmisc%252525252525252525252Fstyle%2525252525252525252526redirect%252525252525252525253D%25252525252525252525252Fforum%25252525252525252525252Findex.php%25252525252525252525253Fevents%25252525252525252525252Fweekly%25252525252525252525252F06.2015%25252525252525252525252F


    Guess that means it was looking at the style chooser:

    As far as what you asked here:

    I can't see any way of finding that information.
     
  11. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    Yep. It looks like a style chooser URL.

    Is that one URL? It's all messed up.
     
  12. dieketzer

    dieketzer Well-Known Member

    hes talking about googles webmaster tools. it will allow you to inspect/alter various aspects of your sites interaction with google.
    http://www.google.com/webmasters/tools
     
  13. Dean

    Dean Well-Known Member

    There you go, that is indeed what I was referring to. Thanks!
     
  14. Diana

    Diana Active Member

    It's messed up the way it accesses it. It's normally a popup.
     
  15. Diana

    Diana Active Member

    Thank you!
     
  16. Diana

    Diana Active Member

    And thank you too ;)
     
    Dean likes this.
  17. Diana

    Diana Active Member

    As I check each visit from google it appears that's the only URL it's accessing. No wonder it says "unknown page."

    Why is it doing that?
     
  18. Diana

    Diana Active Member

    I'm also seeing some strange queries.

    What is this?

    * jQuery Tools 1.2.5 - The missing UI library for the Web
    *
    * [tabs, tooltip, tooltip.slide, scrollable, scrollable.navigator, overlay, overlay.apple, dateinput, rangeinput, toolbox.history, toolbox.expose, toolbox.mousewheel]
    *
    * NO COPYRIGHTS OR LICENSES. DO WHAT YOU LIKE.
    *
    * http://flowplayer.org/tools/
    *
    * jquery.event.wheel.js - rev 1
    * Copyright (c) 2008, Three Dub Media (http://threedubmedia.com)
    * Liscensed under the MIT License (MIT-LICENSE.txt)
    * http://www.opensource.org/licenses/mit-license.php
    * Created: 2008-07-01 | Updated: 2008-07-14
    *
     
  19. Diana

    Diana Active Member

    Why are they looking at these files?

    /forum/js/xenforo/xenforo.js?_v=21d5099e
     
  20. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    Add it to your robots file.
     

Share This Page