XF 1.4 Forum Chewing Up Too Many Shared Server Resources. How Do I Diagnose The Issue?

The Grand Potentate

The Grand Potentate

Well-known member
Thought this might be the better place for this rather than Server Config. If you guys want to move it feel free.

Over the past 2 days I've been getting intermittent 403 errors and random slowdowns. I thought it might be a hosting issue, so I opened a support ticket and asked them what was going on. Apparently, I'm using up too many of my system resources. Prior to this, I'd never had a problem, so I'm not exactly sure what the hell is going on. This was their response email:

I have checked your concern in detail and see that your account has exceeded its resource limit for several times. You can analyze it from cPanel --> Resource Usage.

Basically resources usage depends upon the content hosted on the server under your domain hosting account. As we are using CloudLinux in our servers, if any account's resource usage went beyond the limit, the site and cPanel will become slow. Once an account reaches the limit of set resources, its sites will begin to slow down.

The account consuming too many resources will temporarily stop working until their resource usage returns to normal. Meanwhile, the other tenants on the server will continue to run normally.

So I would suggest you to contact your developer and optimize your site contents by following the methods given below:

====
1.Try deactivating all plugins and check to see if this resolves the problem. If it does, activate the plugins one by one until you find the problematic plugin(s).
2.Try using "WP Super Cache" plug-in and don't forget to delete unnecessary plug-ins.
3.Try changing themes or using the default one and see if there is a speed increase.
4.If you are loading pretty large images, you can try optimizing those.
5. Optimise your database through phpmyadmin.
6. In case you have developed your own scripts, try reducing the number of MySQL queries for optimizing your scripts so that they need less processing time
Click to expand...

This is my resource usage for the past 30 days:

View attachment 83247

Seems that I'm mostly chewing up the CPU. My problem is that I have no idea how to go about finding what's causing the (seemingly) excess CPU usage. I'm 100% positive its not from forum growth, because we have less than 100 total users and only about 10-20 active at any time. There are a lot of images though. Any thoughts on how to attempt to diagnose and fix this?
 

Attachments

  • lvechart.cgi.webp
    lvechart.cgi.webp
    60.4 KB · Views: 19
Sort by date Sort by votes
My host got back to me with something interesting:

On checking further I see that the forum folder is consuming more CPU. Please see the details below.

============
root@chi10 [~]# /usr/local/cpanel/bin/dcpumonview | grep dressedw
<tr bgcolor=#FFFFFF><td>dressedw</td><td>dressedwell.net</td><td>9.90</td><td>0.01</td><td>0.1</td></tr>
<tr><td>Top Process</td><td>%CPU 44.0</td><td colspan=3>lsphp:/home/dressedw/public_html/forum/index.php</td></tr>
<tr><td>Top Process</td><td>%CPU 41.0</td><td colspan=3>lsphp:/home/dressedw/public_html/forum/index.php</td></tr>
============
 
Upvote 0 Downvote
Have you disabled all add-ons and then started re-enabling them one at a time to see if it is specific add-on that is causing it? Have you recently added any new add-ons?
 
Upvote 0 Downvote
Apparently, I'm under attack by robots. From the host support:

From below details you can see that bots are trying to access forum which can cause the issue. Please add Captcha code in your forum which will prevent any bots to access it automatically. It is better to contact your developer for further help.

===========
37.58.100.235 - - [09/Sep/2014:00:25:03 -0500] "GET /forum/goto/post?id=41248 HTTP/1.1" 301 20 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"
199.21.99.207 - - [09/Sep/2014:00:25:11 -0500] "GET /forum/sportsbook/nfl-week-5-thursday-night-game.59/ HTTP/1.1" 200 8144 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
199.21.99.207 - - [09/Sep/2014:00:25:14 -0500] "GET /forum/attachments/img_0743-jpg.384/ HTTP/1.1" 404 6731 "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
83.163.204.141 - - [09/Sep/2014:00:25:15 -0500] "POST /forum/posts/59464/like HTTP/1.1" 200 487 "http://www.dressedwell.net/forum/threads/menswear-experts-do-as-i-say-not-as-i-do.982/page-2" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
83.163.204.141 - - [09/Sep/2014:00:25:24 -0500] "POST /forum/posts/59467/like HTTP/1.1" 200 475 "http://www.dressedwell.net/forum/threads/menswear-experts-do-as-i-say-not-as-i-do.982/page-2" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
199.21.99.207 - - [09/Sep/2014:00:25:28 -0500] "GET /forum/attachments/image-jpg.1814/ HTTP/1.1" 200 77124 "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
83.163.204.141 - - [09/Sep/2014:00:25:27 -0500] "GET /forum/threads/menswear-experts-do-as-i-say-not-as-i-do.982/page-3 HTTP/1.1" 200 17560 "http://www.dressedwell.net/forum/threads/menswear-experts-do-as-i-say-not-as-i-do.982/page-2" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
83.163.204.141 - - [09/Sep/2014:00:25:31 -0500] "GET /forum/css.php?css=GoToTop_purecss,Gritter,GritterEXTRA,attachment_editor,bb_code,bbm_buttons,bbm_editor_redactor_autocss,conversationbutton_postbit_avatar,conversationbutton_postbit_user_info,editor_ui,hide_signatures,message,message_user_info,moderator_bar,nat_public_css,quick_reply,thread_view&style=38&dir=LTR&d=1410161250 HTTP/1.1" 200 12409 "http://www.dressedwell.net/forum/threads/menswear-experts-do-as-i-say-not-as-i-do.982/page-3" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
83.163.204.141 - - [09/Sep/2014:00:25:34 -0500] "POST /forum/deferred.php HTTP/1.1" 200 21 "http://www.dressedwell.net/forum/threads/menswear-experts-do-as-i-say-not-as-i-do.982/page-3" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
83.163.204.141 - - [09/Sep/2014:00:25:35 -0500] "POST /forum/deferred.php HTTP/1.1" 200 22 "http://www.dressedwell.net/forum/threads/menswear-experts-do-as-i-say-not-as-i-do.982/page-3" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
199.21.99.207 - - [09/Sep/2014:00:25:40 -0500] "GET /forum/attachments/image-jpg.1680/ HTTP/1.1" 200 197077 "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
37.58.100.81 - - [09/Sep/2014:00:25:44 -0500] "GET /forum/find-new/328583/posts HTTP/1.1" 303 20 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"
199.21.99.207 - - [09/Sep/2014:00:25:53 -0500] "GET /forum/data/avatars/m/0/105.jpg?1368740065 HTTP/1.1" 404 1148 "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
37.58.100.93 - - [09/Sep/2014:00:26:08 -0500] "GET /forum/misc/quick-navigation-menu?selected=node-18&fullSite=1 HTTP/1.1" 200 7088 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"
37.58.100.86 - - [09/Sep/2014:00:26:19 -0500] "GET /forum/find-new/327017/posts HTTP/1.1" 303 20 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"
===========
 
Upvote 0 Downvote
AhrefsBot - YandexBot - there will likely be others in your web server access log file too - are crawlers/robots. The come along and index your site and if enough of them do it all at once it can impact on your site's performance.

You don't appear to have a robots.txt file in place (goes in the root of your site file structure) - consider adding one to tell selected crawlers and spiders NOT to index / scrape content from your site. (This should lessen the load).

You can use Google to research robots.txt and bots/crawlers, but to give you an example this is my robots.txt file for CycleChat.net:

Code: 
User-agent: AhrefsBot
Disallow: /

User-agent: Baidu
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Baiduspider-video
Disallow: /

User-agent: Baiduspider-image
Disallow: /

User-agent: Cliqzbot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: EasouSpider
Disallow: /

User-agent: Exabot
Disallow: /

User-agent: linkdexbot
Disallow: /

User-agent: linkdexbot-mobile
Disallow: /

User-agent: magpie-crawler
Disallow: /

User-agent: meanpathbot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: NaverBot
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: proximic
Disallow: /

User-agent: Rogerbot
Disallow: /

User-agent: SiteBot
Disallow: /

User-agent: sogou
Disallow: /

User-agent: sogou spider
Disallow: /

User-agent: Sogou web spider
Disallow: /

User-agent: spbot
Disallow: /

User-agent: trendictionbot
Disallow: /

User-agent: Twiceler
Disallow: /

User-agent: URLAppendBot
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: YoudaoBot
Disallow: /

User-agent: Yeti
Disallow: /

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /account/
Disallow: /admin.php
Disallow: /attachments/
Disallow: /conversations/
Disallow: /find-new/
Disallow: /goto/
Disallow: /login/
Disallow: /logos/
Disallow: /posts/

Hope this helps.

Cheers,
Shaun :D
 
Last edited:
Upvote 0 Downvote
Tracy Perry said:
Have you disabled all add-ons and then started re-enabling them one at a time to see if it is specific add-on that is causing it? Have you recently added any new add-ons?
Click to expand...
I was thinking about this last night. How would I test this? Would I have to leave the addons off for specific periods of time to see if one was causing the CPU to spike repeatedly?
 
Upvote 0 Downvote
You must log in or register to reply here.

Similar threads

iaresee
  • Question Question
XF 1.5 User complaining about losing likes, how do I diagnose?
Replies
4
Views
876
iaresee
iaresee
Back
Top Bottom