Php-fpm configuration: encountered and error

[Floren]

@Tracy Perry, you will get a bunch of 503's, because I block those requests with Nginx. For a normal user it will be transparent. Try to refresh quickly one page on AXIVO site, you will get first images not loading and eventually a 503 error page.

It was self explanatory on the Siege test... you did not see the huge list of red 503's displayed?
If I disable the Nginx DDoS protection, you will see results like these.

 

[Floren]

Just ran a test on @eva2000's server, no more issues with the file descriptors:

# siege -u https://community.centminmod.com/ -c 2000 -d 30 -t 1M
Lifting the server siege...      done.

Transactions:               4937 hits
Availability:             100.00 %
Elapsed time:              59.79 secs
Data transferred:          71.08 MB
Response time:               0.52 secs
Transaction rate:          82.57 trans/sec
Throughput:               1.19 MB/sec
Concurrency:              42.56
Successful transactions:        4937
Failed transactions:              0
Longest transaction:           5.78
Shortest transaction:           0.00

The numbers are not very good, obviously because the server hardware. It will take a bit until George will have 2,000 online users. Actually not to long... from his good work. Time to tweak those server settings! Sysctl is your friend.

 

[TPerry]

If I disable the Nginx DDoS protection, you will see results like these.

Need to check... I honestly don't remember if I did any DDOS related in my vhost... but for the low use of the site, it works fine right now. If it starts getting busier I'll tweak it some more.

 

[Floren]

@Tracy Perry, set verbose to true in siegerc, is set by default to false in Debian.
Then, you will see a lot useful info....

 

[TPerry]

@Tracy Perry, set verbose to true in siegerc, is set by default to false in Debian.
Then, you will see a lot useful info....

Will try that in a few... right now working on setting up the fastCGI cache... had never got around to it. Think I've got it working so now time to figure out what not to cache.

 

[eva2000]

Just ran a test on @eva2000's server, no more issues with the file descriptors:

# siege -u https://community.centminmod.com/ -c 2000 -d 30 -t 1M
Lifting the server siege...      done.

Transactions:               4937 hits
Availability:             100.00 %
Elapsed time:              59.79 secs
Data transferred:          71.08 MB
Response time:               0.52 secs
Transaction rate:          82.57 trans/sec
Throughput:               1.19 MB/sec
Concurrency:              42.56
Successful transactions:        4937
Failed transactions:              0
Longest transaction:           5.78
Shortest transaction:           0.00

The numbers are not very good, obviously because the server hardware. It will take a bit until George will have 2,000 online users. Actually not to long... from his good work. Time to tweak those server settings! Sysctl is your friend.

Still pretty sure the file descriptor problems were on your system (system that siege runs from) see Matts example of testing against Axivo https://xenforo.com/community/threa...ncountered-and-error.79759/page-4#post-804130 he had socket/file descriptor errors and that's specific to his system running Siege not Axivo.

root@debian:~# siege -u https://www.axivo.com/forums/ -c 5000 -d 30 -t 1M
siege: invalid option -- 'u'
siege: invalid option -- 'u'
** SIEGE 2.70
** Preparing 5000 concurrent users for battle.
The server is now under siege..      done.
siege aborted due to excessive socket failure; you
can change the failure threshold in $HOME/.siegerc
Transactions:                    210 hits
Availability:                   3.45 %
Elapsed time:                  36.14 secs
Data transferred:               4.97 MB
Response time:                 15.45 secs
Transaction rate:               5.81 trans/sec
Throughput:                     0.14 MB/sec
Concurrency:                   89.77
Successful transactions:         210
Failed transactions:            5876
Longest transaction:            8.36
Shortest transaction:           0.34
FILE: /var/log/siege.log
You can disable this annoying message by editing
the .siegerc file in your home directory; change
the directive 'show-logfile' to false.
root@debian:~#

But yeah my forums are on a 4 cpu threads and 4GB XEN VPS isn't going be comparable to the full Intel Ivy Bridge or Hashwell based 4 core + 4 thread = 8 cpu thread based servers you folks use. Number of Nginx worker processes used can make a difference

Also Siege I use as well and it is cpu intensive in itself, so you don't want to be running it from same system you're testing but remotely.. so Siege results can vary depending on where you launch Siege from and can be dependent on the systems server hardware and configuration setup i.e. try launching Siege test from a single cpu thread, 256MB memory based VPS and see

 

[Floren]

Still pretty sure the file descriptor problems were on your system

Ya, as the settings you have were astronomical.
No idea why it flopped on my side, I have them set to a very generous value on my machine... Did not changed anything and your test ran fine with same test settings... Just patched AXIVO siege rpm with few minor file location corrections in manuals. If you have it installed do a reinstall.

 

[Floren]

Think I've got it working so now time to figure out what not to cache.

To see if it works, add $upstream_cache_status to your log_format. Example:

# tail -n 3 /var/log/nginx/localhost.access.log
70.51.12.14 - Floren [05/Aug/2014:00:05:29 -0400] "HIT" GET /css.php?css=article_update,article_view,article_view_header,article_view_tabs,bb_code,facebook,google,login_bar,message_simple,rating,sidebar_share_page,twitter&style=1&dir=LTR&d=1407201484 HTTP/1.1 "200" 6592 "https://www.axivo.com/articles/centos-7-released.21/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.58 Safari/537.36" "-" "3.85"
70.51.12.14 - Floren [05/Aug/2014:00:05:31 -0400] "HIT" GET / HTTP/1.1 "304" 0 "https://www.axivo.com/articles/centos-7-released.21/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.58 Safari/537.36" "-" "-"
70.51.12.14 - Floren [05/Aug/2014:00:05:31 -0400] "-" POST /deferred.php HTTP/1.1 "200" 42 "https://www.axivo.com/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.58 Safari/537.36" "-" "0.52"

Note how POST is bypassed automatically, 3rd log entry hidden by quote expansion.

 

[eva2000]

Ya, as the settings you have were astronomical.
No idea why it flopped on my side, I have them set to a very generous value on my machine... Did not changed anything and your test ran fine with same test settings... Just patched AXIVO siege rpm with few minor file location corrections in manuals. If you have it installed do a reinstall.

Centmin Mod auto installs a custom source compiled Siege 3.0.6 version not RPM hehe

 

[Floren]

@eva2000, oh my... Are you going to install the entire Open Source with it?
Where is your siege.log located?

 

[eva2000]

@eva2000, oh my... Are you going to install the entire Open Source with it?
Where is your siege.log located?

source compiled kernels next... j/k

cat /etc/redhat-release
CentOS release 6.5 (Final)

cat /etc/centminmod-release
1.2.3-eva2000.07

locate siege.log              
/usr/local/var/siege.log

siege -v
SIEGE 3.0.6

 

[eva2000]

Just ran a test on @eva2000's server, no more issues with the file descriptors:

# siege -u https://community.centminmod.com/ -c 2000 -d 30 -t 1M
Lifting the server siege...      done.

Transactions:               4937 hits
Availability:             100.00 %
Elapsed time:              59.79 secs
Data transferred:          71.08 MB
Response time:               0.52 secs
Transaction rate:          82.57 trans/sec
Throughput:               1.19 MB/sec
Concurrency:              42.56
Successful transactions:        4937
Failed transactions:              0
Longest transaction:           5.78
Shortest transaction:           0.00

The numbers are not very good, obviously because the server hardware. It will take a bit until George will have 2,000 online users. Actually not to long... from his good work. Time to tweak those server settings! Sysctl is your friend.

You had me curious so did a quick test myself launching Siege from my remote server in Las Vegas (Xeon E3-1240v3 32GB, 250GB SSD) to test my forums in Freemont, California (Linode 4 cpu thread, 4GB Xen VPS) - geographical distance factors into it for RTT so slightly better results

siege -c2000 -d30 -t1M https://community.centminmod.com
Lifting the server siege...      done.

Transactions:                   5345 hits
Availability:                 100.00 %
Elapsed time:                  59.05 secs
Data transferred:              76.38 MB
Response time:                  0.37 secs
Transaction rate:              90.52 trans/sec
Throughput:                     1.29 MB/sec
Concurrency:                   33.22
Successful transactions:        5345
Failed transactions:               0
Longest transaction:            2.32
Shortest transaction:           0.10

edit: FYI, there's also overhead involved in Centmin Mod when using ngx_pagespeed http://centminmod.com/nginx_ngx_pagespeed.html basically the faster page load times come at a cost of pure throughput - maybe ~10-40% cost. Also ~20% overhead cost for virtualised platforms like OpenVZ, KVM and Xen compared to non-virtualised servers i.e. full dedicated.

 

[Floren]

@eva2000, those are the results I get on my tiny server, when I disable the fastcgi_cache:

# siege -u https://www.axivo.com/forums/ -c 2000 -d 30 -t 1M
Lifting the server siege... done.

Transactions: 7126 hits
Availability: 100.00 %
Elapsed time: 59.16 secs
Data transferred: 61.61 MB
Response time: 0.32 secs
Transaction rate: 120.45 trans/sec
Throughput: 1.04 MB/sec
Concurrency: 38.46
Successful transactions: 7126
Failed transactions: 0
Longest transaction: 1.57
Shortest transaction: 0.02

With fastcgi_cache enabled:

# siege -u https://www.axivo.com/forums/ -c 2000 -d 30 -t 1M
Lifting the server siege... done.

Transactions: 18042 hits
Availability: 100.00 %
Elapsed time: 59.57 secs
Data transferred: 151.95 MB
Response time: 0.05 secs
Transaction rate: 302.87 trans/sec
Throughput: 2.55 MB/sec
Concurrency: 15.06
Successful transactions: 18042
Failed transactions: 0
Longest transaction: 0.38
Shortest transaction: 0.00

Why don't you move to a dedicated OVH server and get rid of ngx_pagespeed insanity?
A full dedicated with 16GB of ram is 42$/month... I don't even have those specs BTW, I run on a i5-3570S with 8GB of ram...

 

[eva2000]

how many addons you installed on your forum again ? mine = 110+ addons too

OVH too far latency wise for predominant Asian visitor audiences and well I like ngx_pagespeed and it serves as a test bed for me via my forums to tune and tweak to improve Centmin Mod Nginx + ngx_pagespeed integration

Also testing Linode VPSes to make sure I am up to speed with their offerings i.e. I also test Amazon EC2, GoGrid, Rackspace, PheonixNAP SecureCloud, IWStack, DigitalOcean, Linode, Wable, Vultr etc

 

[Floren]

@eva2000, what is the point to like a product that actually hurts your server performance for the "benefit" of doing something you can do yourself directly into XenForo? You are not running a hosting service with thousands of shared accounts. Friendly speaking, I find the product inappropriate for your needs as well your user base.

And there are other hosting providers beside OVH, if you worry about latency... Personally I find their prices excellent. If Asia wants your products, then they can wait few seconds, the same way you wait when you visit an US based server from Australia.

 

[eva2000]

Pageload speed vs throughput.. ngx_pagespeed has faster page load speeds from my stats it's visual rendering difference of ~0.5 seconds. Siege won't show you that benefit at all in a visitors page load experience.

And well it's all about learning and expanding one's knowledge of ngx_pagespeed. I have along way to go before I need that max throughput versus what has immediate benefit right now for visitors = page load speed. When I do eventually need high throughput, I'll upgrade my server setup as needed Right now it's all about learning and hands on experience

edit: as to do it yourself optimisations, I wouldn't even know where to begin to manually be able to serve on Xenforo pages only the css elements needed to render the browser's viewport above the fold for each and every device out there. So on mobile and tablet devices, would only be served css elements needed to render the above fold view of the page and all images on the site would have optimal sizes specific for mobile and tablet devices. While desktop users would be their optimal image sizes via auto webp conversion and render only the css elements needed to render above fold in their desktop browsers.

 

[Floren]

@eva2000, my point is: everything you do with ngx_pagespeed can be done yourself directly, without the added server overhead. I commend your desire to learn, for sure... it is great. But realistically and based on my personal experience, I do not find the product appropriate for your user niche. IMO, most Centminmod users lack the Linux knowledge necessary to manage a server and perform appropriate tune-ups, which explains why they choose your installer. Basically, they rely directly on your script to install everything for them. "I'll just run the script installer, George will do everything else for me." Luckily, you provide them an optimized setup. However, most don't understand 70% of what you did inside Centminmod, so technically you "tell them" what to do.

I would make sure your users know the negative impact ngx_pagespeed brings in general, for the benefit of gaining 0.5 seconds. I'm curious to see a breakdown of all numbers, I'm sure you are right and have detailed stats. But I doubt those numbers remain accurate on the long run, I explained to you previously how ngx_pagespeed drops the code modification after a certain number of milliseconds. This probably happens often on a shaky and non-optimized network. IMO, ngx_pagespped was designed primarily to allow host providers save on network resources at the detriment of server resources, which are way easier to address.

An even better question: why none of the large single domain sites use ngx_pagespeed? Could be that is still a beta product or because it performs very bad on a large scale setup?
I did some basic research and found this thread, looks like target and wallgreens are using it:
http://bigqueri.es/t/how-many-sites-are-using-mod-pagespeed/86

 

[eva2000]

well point is to learn .. without that drive to learn where would we all be

mod_pagespeed and ngx_pagespeed have different headers and some sites opt to hide them so you won't have an accurate number anyway

definitely there's more sites out there using ngx_pagespeed, just myself I have like 40 sites over 20 servers with Centmin Mod Nginx + ngx_pagespeed in play

FYI, Centmin Mod Nginx can also disable ngx_pagespeed too and it's disabled by default on installs. And you can further remove it from Nginx compiles with a setting change http://centminmod.com/nginx_ngx_pagespeed.html#howtodisable.

Also not every site I run has ngx_pagespeed enabled - it depends on the site

 

[eva2000]

Oh and example of some stuff you just can't manually optimise for - auto user uploaded image attachment optimisation and conversion to webp format via ngx_pagespeed

auto optimised PNG conversion where browser doesn't support WebP

Shrinking image `*attachmentlink*/2014/05/198_upload_*.png' (24863 bytes) to `*attachmentlink*/2014/05/x198_upload_*.png.pagespeed.ic.Qrx6HYAfML.png' (11796 bytes)
Shrinking image `*attachmentlink*/2014/05/199_upload_*.png' (42496 bytes) to `*attachmentlink*/2014/05/x199_upload_*.png.pagespeed.ic.gWB4NG1biA.png' (20769 bytes)
Shrinking image `*attachmentlink*/2014/05/200_upload_*.png' (53259 bytes) to `*attachmentlink*/2014/05/x200_upload_*.png.pagespeed.ic.Uh4t8-wfgy.png' (24965 bytes)

auto optimised conversion of user attachments from PNG to WebP format where browser supports WebP

hment-files/2014/05/198_upload_*.png' (24863 bytes) to `*attachmentlink*/2014/05/x198_upload_*.png.pagespeed.ic.3Exc-iTnt7.webp' (6268 bytes)
*threadlink*/:1241: Shrinking image `*attachmentlink*/2014/05/199_upload_2014-*.png' (42496 bytes) to `*attachmentlink*/2014/05/x199_upload_2014-*.png.pagespeed.ic.MtXR6WHd7d.webp' (12718 bytes)
*threadlink*/:1248: Shrinking image `*attachmentlink*/2014/05/200_upload_2014-*.png' (53259 bytes) to `*attachmentlink*/2014/05/x200_upload_2014-*.png.pagespeed.ic.QRsR_7kaYd.webp' (14484 bytes)

excerpt from pagespeed stats ~52% reduction in image sizes served to visitors

image_rewrite_total_bytes_saved:                              243473
image_rewrite_total_original_bytes:                           468526

image_file_count_reduction:                                        0
image_rewrites:                                                   14
image_resized_using_rendered_dimensions:                           0
image_norewrites_high_resolution:                                  0
image_rewrites_dropped_intentionally:                              7
image_rewrites_dropped_decode_failure:                             0
image_rewrites_dropped_mime_type_unknown:                          0
image_rewrites_dropped_server_write_fail:                          0
image_rewrites_dropped_nosaving_resize:                            0
image_rewrites_dropped_nosaving_noresize:                          7
image_rewrites_dropped_due_to_load:                                0
image_rewrites_squashing_for_mobile_screen:                        0
image_rewrite_total_bytes_saved:                              243473
image_rewrite_total_original_bytes:                           468526
image_rewrite_uses:                                             1958
image_inline:                                                   1142
image_webp_rewrites:                                               7
image_rewrite_latency_total_ms:                                 6619
image_ongoing_rewrites:                                            0
image_webp_conversion_gif_timeouts:                                0
image_webp_conversion_png_timeouts:                                0
image_webp_conversion_jpeg_timeouts:                               0
image_webp_alpha_timeouts:                                         0
image_webp_opaque_timeouts:                                        0
in_place_oversized_opt_stream:                                     0
in_place_uncacheable_rewrites:                                     0

love to see you manually optimise each and every user uploaded image attachment + do webp conversion only for browsers which support webp

 

[Floren]

love to see you manually optimise each and every user uploaded image attachment + do webp conversion only for browsers which support webp

That is quite easy, there is an option in GD to do that automatically. All you have to do is use proxy.php to determine if the browser supports webp or not. Then serve the appropriate file to client.

Regardless, I find this of advantage for sites that have large image galleries. Perfect example is OKCupid, not sure if they use ngx_pagespeed but they do offer webp and jpeg to users based on used browser. Other than that, the default XenForo setup is more than sufficient to serve static files through Nginx at blazing speeds.