1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.3 How sensitive is the image proxy to slow origin servers?

Discussion in 'XenForo Questions and Support' started by DeltaHF, Aug 26, 2014.

  1. DeltaHF

    DeltaHF Well-Known Member

    I've been using the image proxy for a few weeks now on my large forum (9.7 million posts) and for the most part I've been happy with it.

    However, I've noticed that the forum now can hang up a bit at times. While investigating with NewRelic, I've found these hiccups almost always correspond to slow external servers (taking more than 15 seconds to respond) which are being accessed from XenForo's proxy.php. See these graphs from earlier today, for example, which show how a 100-second response from a slow website apparently had a significant impact on all my visitors (I personally received a 504 Nginx timeout error during this incident):

    Screen Shot 2014-08-26 at 6.23.03 PM.png
    Screen Shot 2014-08-26 at 6.24.30 PM.png

    It bothers me that a slow origin server can impact the performance of my entire site, though I'm still a bit surprised that it's happening. How is the image proxy designed to deal with these situations, and is there anything I can do to mitigate it? I'm running Nginx/PHP-FPM on CentOS via @eva2000's Centminmod.
     
  2. Brogan

    Brogan XenForo Moderator Staff Member

    I believe there is a 10 second timeout in the PHP for any proxied image URLs.
     
  3. eva2000

    eva2000 Well-Known Member

  4. DeltaHF

    DeltaHF Well-Known Member

    This just happened again, and NewRelic appears to tell the same story, though it was a bit more ugly.

    I found the images referenced from gt5.zqz.me (which doesn't appear to be loading at all), changed the file request parameters (so the requests wouldn't be cached), and tried posting them myself. Sure enough, the entire site grinds to a halt. I tried referencing the images here in the XenForo test forum and the same thing happened - I was unable to load any pages, though it did only last for about 10 seconds.

    @Brogan Is the 10-second timeout a 1.4 thing? I'm on 1.3, and at this point, it seems to be a trivial way to knock any XenForo installation offline for a few minutes if they're using the image proxy. :confused:

    @eva2000 These images don't appear to be unusual, but the server that's hosting them is completely offline. It just seems odd to me that one blocking call in proxy.php would completely wipe out PHP-FPM. Maybe adjusting the timeout for proxy.php in the Nginx config would resolve this?

    Screen Shot 2014-08-28 at 5.25.24 PM.png
    Screen Shot 2014-08-28 at 5.25.10 PM.png
     
  5. Mike

    Mike XenForo Developer Staff Member

    If one blocking call to proxy.php is taking out your server, that sounds like an issue with your configuration (the number of FPM slots you have available?). PHP will be active while waiting for the service to come back so it will hold a slot.

    If you reproduced it here, what you were likely seeing was your browser preventing any additional connections to our server (see: http://stackoverflow.com/questions/...-of-data-transfered-waiting-for-available-soc). This would be a limit specific to your browser; it wouldn't apply to anyone else. If you only load a couple images as well, you can reload another tab without issue.
     
    Abracadaniel likes this.
  6. RoldanLT

    RoldanLT Well-Known Member

  7. eva2000

    eva2000 Well-Known Member

    Yeah that's probably what is happening when using default XF proxy.php.

    @RoldanLT if it's what you're experiencing then it's basically the same as what I replicated with siege testing and a 10 second delay PHP file at https://community.centminmod.com/th...setting-50-consider-raising-it.791/#post-4601 and you have tested this on my forums with Shawn's image proxy addon that uses /proxy instead and no problems for my forum and no errors in error log https://community.centminmod.com/threads/testing-your-php-fpm.1224/. But I am using multiple PHP-FPM pools so would of alleviated any such PHP process blocking https://community.centminmod.com/pages/mstats/
     
    RoldanLT likes this.
  8. eva2000

    eva2000 Well-Known Member

    can confirm if using Xenforo 1.3.5 default proxy images for broken images can add some time to page load completion somewhat my PHP-FPM config has multi PHP-FPM pools so probably impact isn't that much

    xenforo_proxy_image_hog_process_newrelic_01.png
    xenforo_proxy_image_hog_process_newrelic_02.png

    didn't happen with Digitalpoint proxy image add on but it seems that addon isn't working with PHP 5.6.0 i updated last night :( Needed Digitalpoint proxy image addon so that proxied images are served via non-PHP extension url /proxy/image instead of Xenforo default proxy.php

    checking PHP-FPM pool full status output shows proxy.php requests to such broken image urls taking 4555+ seconds or 4.55 billion microseconds

    Code:
    pid:                  11842
    state:                Finishing
    start time:           29/Aug/2014:11:33:08 +0000
    start since:          5846
    requests:             14
    request duration:     4555840125
    request method:       GET
    request URI:          /proxy.php?image=http%3A%2F%2Fgt5.zqz.me%2Fcd2%2Fimg.php%3Fs%3D1384830052000%26f%3DDigitalSans%26fc%3D%2523000000%26bc%3D%25234169e1%26fs%3D2asdf4%262&hash=d69d7b116cb55bedc4d1af464036447f
    content length:       0
    user:                 -
    script:               /proxy.php
    last request cpu:     0.00
    last request memory:  0
     
    Last edited: Aug 29, 2014
  9. eva2000

    eva2000 Well-Known Member

Share This Page