Php-fpm configuration: encountered and error

[eva2000]

That is quite easy, there is an option in GD to do that automatically. All you have to do is use proxy.php to determine if the browser supports webp or not. Then serve the appropriate file to client.

Interesting... proxy.php ? you mean to externally posted images ? what about local attachments ? I think short of you writing up a guide on how to do that, not even the majority of veteran Xenforo owners/users would know how to set that up let alone me who still isn't that familiar with Xenforo.

No doubt Nginx has fast static file performance, but ngx_pagespeed isn't about raw throughput shown in Siege, it's about serving optimally only what needs to be served in terms of css, js and images (particularly above fold on mobile and desktop devices) to all visitors and not just guests (nginx guest caching). You can still have a high static file request rate (Siege) but still have a slow user page loading experience.

 

[Floren]

@eva2000, few interesting threads:
http://xenforo.com/community/thread...images-with-webp-extension.68092/#post-717079
http://xenforo.com/community/threads/google-webp-support.76626/

I believe what you do with pagespeed can be done programatically.
I think @digitalpoint released an add-on for that already...

 

[rdn]

I think @digitalpoint released an add-on for that already...

No, I don't think soo.

 

[eva2000]

@eva2000, few interesting threads:
http://xenforo.com/community/thread...images-with-webp-extension.68092/#post-717079
http://xenforo.com/community/threads/google-webp-support.76626/

I believe what you do with pagespeed can be done programatically.
I think @digitalpoint released an add-on for that already...

you might be thinking of https://marketplace.digitalpoint.com/image-proxy.2306/item i use that on my forums as Xenforo default one uses proxy.php file which will bypass ngx_pagespeed. Shawn's addon allows proxied external files to use /proxy/ so ngx_pagespeed to auto optimise proxied external images

But nothing to do with auto webp conversion. But yes if you have the programming chops, you could do auto webp to be served via supported browsers. Ilya Grigorik touched on this with WebP at https://www.igvita.com/2013/03/07/faster-smaller-and-more-beautiful-web-with-webp/ and is part of why ngx_pagespeed and mod_pagespeed exists.

In fact, in the time-honored tradition of dogfooding own products, there is already a large and growing list of Google properties (Gmail, Drive, Picasa, Instant Previews, Play Magazines, Image Search, YouTube, ...) with WebP support. Most recently, Chrome Web Store switched to WebP, saw ~30% byte reduction on average, and is now saving several terabytes of bandwidth per day!

In parallel, there are now over 300,000 sites using the open-source PageSpeed libraries, which enable transparent WebP transcoding on Apache (mod_pagespeed) and Nginx (ngx_pagespeed), and there is a growing list of commercial products (Torbit, EdgeCast) which can do similar optimizations - the bandwidth savings are hard to argue against!

article is 17+ months old

https://www.igvita.com/ uses ngx_pagespeed itself too

 

[Floren]

Ilya Grigorik touched on this with WebP at https://www.igvita.com/2013/03/07/faster-smaller-and-more-beautiful-web-with-webp/ and is part of why ngx_pagespeed and mod_pagespeed exists.

His Nginx config is pretty heavy, there are a lot easier ways to achieve the same results programatically and serve them through Nginx. All we need is an option in XenForo to export the attached .data files as either .jpg or .webp and serve them based on client. Or it can be done with cwebp also. An example of a much simpler Nginx configuration:

http {
    ...
    map $http_accept $webp {
        default    '';
        ~*webp    '.webp';
    }
    ...
    server {
        ...
        location ~* \.(?:ico|css|gif|jpe?g|js|png|swf)(\?.+)?$ {
            location ~* \.(?:jpe?g|png)(\?.+)?$ {
                add_header    Vary Accept;
                try_files     $uri$webp =404;
            }
            access_log        off;
            log_not_found     off;
            expires           1y;
        }
        ...
    }
    ...
}

 

[eva2000]

nice maybe try it on your forums ? i am using http://xenforo.com/community/resources/bd-attachment-store.1779/ to push attachments to files with image extensions which ngx_pagespeed then chews and spits out the optimised version

 

[Floren]

maybe try it on your forums ?

Nah, I'll let this to @Mike and @Kier... I rather focus on OpenSSl 1.0.1i, I will have it released today (way too many patches to redo, so I could not do it yesterday).

 

[rdn]

@RoldanLT, try these settings and siege your server. Post the results.
pm.max_children = 16
pm.start_servers = 10
pm.min_spare_servers = 4
pm.max_spare_servers = 16

I'm still hitting "server reached max_children"
3 to 5 times a day.
I just changed my config to

pm.max_children = 32
pm.start_servers = 10
pm.min_spare_servers = 4
pm.max_spare_servers = 16

 

[Floren]

@RoldanLT, children's are very high, you allocate 16GB of RAM wow. Your site does not justifies those numbers, you should also lower the servers. What's the number of requests, 100? Lower it to 50.

Something is not right on your setup, how many requests/sec you do in average?

 

[rdn]

I just changed my config back to:

pm = dynamic
pm.max_children = 16
pm.start_servers = 6
pm.min_spare_servers = 4
pm.max_spare_servers = 10
pm.max_requests = 100

Last time I'm using this:

m.max_children = 16
pm.start_servers = 10
pm.min_spare_servers = 4
pm.max_spare_servers = 16
pm.max_requests = 1000

 

[rdn]

how many requests/sec you do in average?

How can I get this value?
Right now my forum is not that busy (8AM here).
My forum spikes start from 12 noon until 12 evening.

 

[Floren]

How can I get this value?

Use Siege with a reasonable number of users, it will tell you the number of requests/sec while you tailf your logs to see if you get the error. But is mostly related to the heuristic settings you have, try this:

# grep ^pm /etc/php-fpm.d/www.conf
pm = dynamic
pm.max_children = 8
pm.start_servers = 6
pm.min_spare_servers = 4
pm.max_spare_servers = 8
pm.max_requests = 100

# tail -n 5 /etc/php-fpm.d/www.conf
php_admin_value[error_log] = /var/log/php-fpm/$pool.log
php_admin_value[date.timezone] = America/Montreal
php_admin_value[memory_limit] = 256M
php_admin_value[post_max_size] = 1M
php_admin_value[upload_max_filesize] = 1M

Should be MORE than enough for your site. I ran on those settings a Siege bombardment (yes it has a script called like that ) with 2,000 users, no issues with fastcgi_cache disabled.

If you still run into issues, you should look into your overall server setup, something is definitely wrong. @eva2000 should able to help you on his forums.

 

[rdn]

If you still run into issues, you should into your overall server setup, something is definitely wrong.

Yeah
If time permits, I will start building a server from scratch using all Axivo repos

 

[rdn]

Siege bombardment (yes it has a script called like that )

Maybe this: http://manpages.ubuntu.com/manpages/hardy/man1/bombardment.1.html ?

 

[rdn]

Try this:

# grep ^pm /etc/php-fpm.d/www.conf
pm = dynamic
pm.max_children = 8
pm.start_servers = 6
pm.min_spare_servers = 4
pm.max_spare_servers = 8
pm.max_requests = 100

Should be MORE than enough for your site.

Just tried now, with off peak (low visitors) still got this warning:

WARNING: [pool www] server reached pm.max_children setting (8), consider raising it

 

[Floren]

@RoldanLT, sorry I updated the previous post.
I have this bad habit to add new things while I'm still thinking... Wait like 10min before replying to be sure I'm done, I don't like to create multiple posts.

I will start building a server from scratch

Check with @MattW, I think he already moved to it.

 

[rdn]

php_admin_value[post_max_size] = 1M php_admin_value[upload_max_filesize] = 1M

Seems to be really low? Why?
Default value of PHP I think is 15MB.

 

[eva2000]

i think it's low because Floren disabled attachments on his forum ? default is 15MB for Centmin Mod setups only

As to siege it has it's limitations as those 2000 concurrent users aren't doing much to hog php processes unlike real world users would i.e. concurrent users uploading large image and file attachments which RoldanLT has for his user activity patterns.

edit: to elaborated posted a simulated siege test with php files where one has a sleep 10s delay added https://community.centminmod.com/th...setting-50-consider-raising-it.791/#post-4330

excerpt

small light test siege -c50 -d30 -t1M

with simulated 10s sleep delay during siege

pool:                 www
process manager:      dynamic
start time:           08/Aug/2014:20:54:44 -0700
start since:          78
accepted conn:        69
listen queue:         19
max listen queue:     37
listen queue len:     65535
idle processes:       0
active processes:     8
total processes:      8
max active processes: 8
max children reached: 1
slow requests:        0

with simulated 10s sleep delay after siege

pool:                 www
process manager:      dynamic
start time:           08/Aug/2014:20:54:44 -0700
start since:          121
accepted conn:        88
listen queue:         0
max listen queue:     37
listen queue len:     65535
idle processes:       7
active processes:     1
total processes:      8
max active processes: 8
max children reached: 1
slow requests:        0

Transactions:                     39 hits
Availability:                  68.42 %
Elapsed time:                  59.32 secs
Data transferred:               0.00 MB
Response time:                 17.95 secs
Transaction rate:               0.66 trans/sec
Throughput:                     0.00 MB/sec
Concurrency:                   11.80
Successful transactions:          39
Failed transactions:              18
Longest transaction:           30.00
Shortest transaction:           0.00

Pay attention to the above max children reached, listen queue and max listen queue values during siege with simulated 10s sleep.

listen queue - the current number of connections that have been initiated, but not yet accepted. If this value is non-zero it typically means that all the available server processes are currently busy, and there are no processes available to serve the next request. Raising pm.max_children (provided the server can handle it) should help keep this number low. This property follows from the fact that PHP-FPM listens via a socket (TCP or file based), and thus inherits some of the characteristics of sockets.

max listen queue - the maximum value the listen queue has reached since the server was started.

max children reached - the number of times that pm.max_children has been reached since the php-fpm server started (only applicable if the process manager is ondemand or dynamic)

Siege has it's place in load and stress testing but it isn't 100% reflective of real world for PHP performance especially for forum type PHP loads which are non-cached PHP based and are not straight forward (php based attachment uploads).

 

[Floren]

Seems to be really low? Why?
Default value of PHP I think is 15MB.

IMO 15MB is HUGE. Max allowed attachments are 1MB on my site, I think XenForo scales them down anyways when you set the image dimensions (max 1024px for AXIVO), something to 200KB max. I doubt many sites allow users uploading large images... well not me. Therefore, I limit the POST and file size to 1MB, never had an issue so far. Client size is also 1MB on Nginx.

 

[MattW]

IMO 15MB is HUGE. Max allowed attachments are 1MB on my site, I think XenForo scales them down anyways when you set the image dimensions (max 1024px for AXIVO), something to 200KB max. I doubt many sites allow users uploading large images... well not me. Therefore, I limit the POST and file size to 1MB, never had an issue so far. Client size is also 1MB on Nginx.

I do, hence why I have over 22GB of attachments