XF 1.3 How sensitive is the image proxy to slow origin servers?

[DeltaHF]

I've been using the image proxy for a few weeks now on my large forum (9.7 million posts) and for the most part I've been happy with it.

However, I've noticed that the forum now can hang up a bit at times. While investigating with NewRelic, I've found these hiccups almost always correspond to slow external servers (taking more than 15 seconds to respond) which are being accessed from XenForo's proxy.php. See these graphs from earlier today, for example, which show how a 100-second response from a slow website apparently had a significant impact on all my visitors (I personally received a 504 Nginx timeout error during this incident):




It bothers me that a slow origin server can impact the performance of my entire site, though I'm still a bit surprised that it's happening. How is the image proxy designed to deal with these situations, and is there anything I can do to mitigate it? I'm running Nginx/PHP-FPM on CentOS via @eva2000's Centminmod.

 

[Brogan]

I believe there is a 10 second timeout in the PHP for any proxied image URLs.

 

[eva2000]

probably is PHP related somewhat but depends on the external site and size of the externally referenced images ?

For my forums I use Shawn's addon https://xenforo.com/community/resources/digital-point-image-proxy.2747/ as the proxy url is /proxy and not /proxy.php so I can utilise ngx_pagespeed to auto optimise all static images that are proxied

 

[DeltaHF]

This just happened again, and NewRelic appears to tell the same story, though it was a bit more ugly.

I found the images referenced from gt5.zqz.me (which doesn't appear to be loading at all), changed the file request parameters (so the requests wouldn't be cached), and tried posting them myself. Sure enough, the entire site grinds to a halt. I tried referencing the images here in the XenForo test forum and the same thing happened - I was unable to load any pages, though it did only last for about 10 seconds.

@Brogan Is the 10-second timeout a 1.4 thing? I'm on 1.3, and at this point, it seems to be a trivial way to knock any XenForo installation offline for a few minutes if they're using the image proxy.

@eva2000 These images don't appear to be unusual, but the server that's hosting them is completely offline. It just seems odd to me that one blocking call in proxy.php would completely wipe out PHP-FPM. Maybe adjusting the timeout for proxy.php in the Nginx config would resolve this?

 

[Mike]

If one blocking call to proxy.php is taking out your server, that sounds like an issue with your configuration (the number of FPM slots you have available?). PHP will be active while waiting for the service to come back so it will hold a slot.

If you reproduced it here, what you were likely seeing was your browser preventing any additional connections to our server (see: http://stackoverflow.com/questions/...-of-data-transfered-waiting-for-available-soc). This would be a limit specific to your browser; it wouldn't apply to anyone else. If you only load a couple images as well, you can reload another tab without issue.

 

[rdn]

Ow and this one is related to my problem: https://xenforo.com/community/threads/php-fpm-configuration-encountered-and-error.79759/

 

[eva2000]

If one blocking call to proxy.php is taking out your server, that sounds like an issue with your configuration (the number of FPM slots you have available?). PHP will be active while waiting for the service to come back so it will hold a slot.

If you reproduced it here, what you were likely seeing was your browser preventing any additional connections to our server (see: http://stackoverflow.com/questions/...-of-data-transfered-waiting-for-available-soc). This would be a limit specific to your browser; it wouldn't apply to anyone else. If you only load a couple images as well, you can reload another tab without issue.

Yeah that's probably what is happening when using default XF proxy.php.

@RoldanLT if it's what you're experiencing then it's basically the same as what I replicated with siege testing and a 10 second delay PHP file at https://community.centminmod.com/th...setting-50-consider-raising-it.791/#post-4601 and you have tested this on my forums with Shawn's image proxy addon that uses /proxy instead and no problems for my forum and no errors in error log https://community.centminmod.com/threads/testing-your-php-fpm.1224/. But I am using multiple PHP-FPM pools so would of alleviated any such PHP process blocking https://community.centminmod.com/pages/mstats/

 

[eva2000]

can confirm if using Xenforo 1.3.5 default proxy images for broken images can add some time to page load completion somewhat my PHP-FPM config has multi PHP-FPM pools so probably impact isn't that much




didn't happen with Digitalpoint proxy image add on but it seems that addon isn't working with PHP 5.6.0 i updated last night Needed Digitalpoint proxy image addon so that proxied images are served via non-PHP extension url /proxy/image instead of Xenforo default proxy.php

checking PHP-FPM pool full status output shows proxy.php requests to such broken image urls taking 4555+ seconds or 4.55 billion microseconds

pid:                  11842
state:                Finishing
start time:           29/Aug/2014:11:33:08 +0000
start since:          5846
requests:             14
request duration:     4555840125
request method:       GET
request URI:          /proxy.php?image=http%3A%2F%2Fgt5.zqz.me%2Fcd2%2Fimg.php%3Fs%3D1384830052000%26f%3DDigitalSans%26fc%3D%2523000000%26bc%3D%25234169e1%26fs%3D2asdf4%262&hash=d69d7b116cb55bedc4d1af464036447f
content length:       0
user:                 -
script:               /proxy.php
last request cpu:     0.00
last request memory:  0

 

[eva2000]

Suggestion posted XF https://xenforo.com/community/threads/removing-php-extension-from-proxy-php-images.81988/