1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.4 Forum Chewing Up Too Many Shared Server Resources. How Do I Diagnose The Issue?

Discussion in 'Troubleshooting and Problems' started by Rambro, Sep 9, 2014.

  1. Rambro

    Rambro Well-Known Member

    Thought this might be the better place for this rather than Server Config. If you guys want to move it feel free.

    Over the past 2 days I've been getting intermittent 403 errors and random slowdowns. I thought it might be a hosting issue, so I opened a support ticket and asked them what was going on. Apparently, I'm using up too many of my system resources. Prior to this, I'd never had a problem, so I'm not exactly sure what the hell is going on. This was their response email:

    This is my resource usage for the past 30 days:

    View attachment 83247

    Seems that I'm mostly chewing up the CPU. My problem is that I have no idea how to go about finding what's causing the (seemingly) excess CPU usage. I'm 100% positive its not from forum growth, because we have less than 100 total users and only about 10-20 active at any time. There are a lot of images though. Any thoughts on how to attempt to diagnose and fix this?
     

    Attached Files:

  2. Rambro

    Rambro Well-Known Member

    My host got back to me with something interesting:

    On checking further I see that the forum folder is consuming more CPU. Please see the details below.

    ============
    root@chi10 [~]# /usr/local/cpanel/bin/dcpumonview | grep dressedw
    <tr bgcolor=#FFFFFF><td>dressedw</td><td>dressedwell.net</td><td>9.90</td><td>0.01</td><td>0.1</td></tr>
    <tr><td>Top Process</td><td>%CPU 44.0</td><td colspan=3>lsphp:/home/dressedw/public_html/forum/index.php</td></tr>
    <tr><td>Top Process</td><td>%CPU 41.0</td><td colspan=3>lsphp:/home/dressedw/public_html/forum/index.php</td></tr>
    ============
     
  3. Tracy Perry

    Tracy Perry Well-Known Member

    Have you disabled all add-ons and then started re-enabling them one at a time to see if it is specific add-on that is causing it? Have you recently added any new add-ons?
     
  4. Rambro

    Rambro Well-Known Member

    No, not yet. A few new addons over the past month. Nothing horribly resource intensive like a portal though.
     
  5. Rambro

    Rambro Well-Known Member

    Apparently, I'm under attack by robots. From the host support:

    From below details you can see that bots are trying to access forum which can cause the issue. Please add Captcha code in your forum which will prevent any bots to access it automatically. It is better to contact your developer for further help.

    ===========
    37.58.100.235 - - [09/Sep/2014:00:25:03 -0500] "GET /forum/goto/post?id=41248 HTTP/1.1" 301 20 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"
    199.21.99.207 - - [09/Sep/2014:00:25:11 -0500] "GET /forum/sportsbook/nfl-week-5-thursday-night-game.59/ HTTP/1.1" 200 8144 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
    199.21.99.207 - - [09/Sep/2014:00:25:14 -0500] "GET /forum/attachments/img_0743-jpg.384/ HTTP/1.1" 404 6731 "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
    83.163.204.141 - - [09/Sep/2014:00:25:15 -0500] "POST /forum/posts/59464/like HTTP/1.1" 200 487 "http://www.dressedwell.net/forum/threads/menswear-experts-do-as-i-say-not-as-i-do.982/page-2" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
    83.163.204.141 - - [09/Sep/2014:00:25:24 -0500] "POST /forum/posts/59467/like HTTP/1.1" 200 475 "http://www.dressedwell.net/forum/threads/menswear-experts-do-as-i-say-not-as-i-do.982/page-2" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
    199.21.99.207 - - [09/Sep/2014:00:25:28 -0500] "GET /forum/attachments/image-jpg.1814/ HTTP/1.1" 200 77124 "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
    83.163.204.141 - - [09/Sep/2014:00:25:27 -0500] "GET /forum/threads/menswear-experts-do-as-i-say-not-as-i-do.982/page-3 HTTP/1.1" 200 17560 "http://www.dressedwell.net/forum/threads/menswear-experts-do-as-i-say-not-as-i-do.982/page-2" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
    83.163.204.141 - - [09/Sep/2014:00:25:31 -0500] "GET /forum/css.php?css=GoToTop_purecss,Gritter,GritterEXTRA,attachment_editor,bb_code,bbm_buttons,bbm_editor_redactor_autocss,conversationbutton_postbit_avatar,conversationbutton_postbit_user_info,editor_ui,hide_signatures,message,message_user_info,moderator_bar,nat_public_css,quick_reply,thread_view&style=38&dir=LTR&d=1410161250 HTTP/1.1" 200 12409 "http://www.dressedwell.net/forum/threads/menswear-experts-do-as-i-say-not-as-i-do.982/page-3" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
    83.163.204.141 - - [09/Sep/2014:00:25:34 -0500] "POST /forum/deferred.php HTTP/1.1" 200 21 "http://www.dressedwell.net/forum/threads/menswear-experts-do-as-i-say-not-as-i-do.982/page-3" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
    83.163.204.141 - - [09/Sep/2014:00:25:35 -0500] "POST /forum/deferred.php HTTP/1.1" 200 22 "http://www.dressedwell.net/forum/threads/menswear-experts-do-as-i-say-not-as-i-do.982/page-3" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
    199.21.99.207 - - [09/Sep/2014:00:25:40 -0500] "GET /forum/attachments/image-jpg.1680/ HTTP/1.1" 200 197077 "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
    37.58.100.81 - - [09/Sep/2014:00:25:44 -0500] "GET /forum/find-new/328583/posts HTTP/1.1" 303 20 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"
    199.21.99.207 - - [09/Sep/2014:00:25:53 -0500] "GET /forum/data/avatars/m/0/105.jpg?1368740065 HTTP/1.1" 404 1148 "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
    37.58.100.93 - - [09/Sep/2014:00:26:08 -0500] "GET /forum/misc/quick-navigation-menu?selected=node-18&fullSite=1 HTTP/1.1" 200 7088 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"
    37.58.100.86 - - [09/Sep/2014:00:26:19 -0500] "GET /forum/find-new/327017/posts HTTP/1.1" 303 20 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"
    ===========
     
  6. CyclingTribe

    CyclingTribe Well-Known Member

    AhrefsBot - YandexBot - there will likely be others in your web server access log file too - are crawlers/robots. The come along and index your site and if enough of them do it all at once it can impact on your site's performance.

    You don't appear to have a robots.txt file in place (goes in the root of your site file structure) - consider adding one to tell selected crawlers and spiders NOT to index / scrape content from your site. (This should lessen the load).

    You can use Google to research robots.txt and bots/crawlers, but to give you an example this is my robots.txt file for CycleChat.net:

    Code:
    User-agent: AhrefsBot
    Disallow: /
    
    User-agent: Baidu
    Disallow: /
    
    User-agent: Baiduspider
    Disallow: /
    
    User-agent: Baiduspider-video
    Disallow: /
    
    User-agent: Baiduspider-image
    Disallow: /
    
    User-agent: Cliqzbot
    Disallow: /
    
    User-agent: Diffbot
    Disallow: /
    
    User-agent: DotBot
    Disallow: /
    
    User-agent: EasouSpider
    Disallow: /
    
    User-agent: Exabot
    Disallow: /
    
    User-agent: linkdexbot
    Disallow: /
    
    User-agent: linkdexbot-mobile
    Disallow: /
    
    User-agent: magpie-crawler
    Disallow: /
    
    User-agent: meanpathbot
    Disallow: /
    
    User-agent: MJ12bot
    Disallow: /
    
    User-agent: NaverBot
    Disallow: /
    
    User-agent: omgilibot
    Disallow: /
    
    User-agent: proximic
    Disallow: /
    
    User-agent: Rogerbot
    Disallow: /
    
    User-agent: SiteBot
    Disallow: /
    
    User-agent: sogou
    Disallow: /
    
    User-agent: sogou spider
    Disallow: /
    
    User-agent: Sogou web spider
    Disallow: /
    
    User-agent: spbot
    Disallow: /
    
    User-agent: trendictionbot
    Disallow: /
    
    User-agent: Twiceler
    Disallow: /
    
    User-agent: URLAppendBot
    Disallow: /
    
    User-agent: Yandex
    Disallow: /
    
    User-agent: YoudaoBot
    Disallow: /
    
    User-agent: Yeti
    Disallow: /
    
    User-agent: Mediapartners-Google
    Disallow:
    
    User-agent: *
    Disallow: /account/
    Disallow: /admin.php
    Disallow: /attachments/
    Disallow: /conversations/
    Disallow: /find-new/
    Disallow: /goto/
    Disallow: /login/
    Disallow: /logos/
    Disallow: /posts/
    
    Hope this helps.

    Cheers,
    Shaun :D
     
    Last edited: Sep 9, 2014
  7. Rambro

    Rambro Well-Known Member

    I was thinking about this last night. How would I test this? Would I have to leave the addons off for specific periods of time to see if one was causing the CPU to spike repeatedly?
     
  8. Mike

    Mike XenForo Developer Staff Member

    Since the load and issues are sporadic, unfortunately yes.
     
  9. Rambro

    Rambro Well-Known Member

    Son of a. That's what I figured.
     

Share This Page