XF 2.0 PayPal Instant Payment Notification Warning

[PumpinIron]

I’m running the latest stable version of XenForo, but as of today I just received the following email from PayPal:

Please check your server that handles PayPal Instant Payment Notifications (IPN). IPNs sent to the following URL(s) are failing:

https://wranglertjforum.com/payment_callback.php?_xfProvider=paypal

If you do not recognize this URL, you may be using a service provider that is using IPN on your behalf. Please contact your service provider with the above information. If this problem continues, IPNs may be disabled for your account.

Thank you for your prompt attention to this issue.

Thanks,

PayPal



I checked the address and this isn’t a phishing scam, it’s a legit email from PayPal. What I can’t figure out is why I am getting this email. I have used upgraded enabled and the @AddonFlare “Paid Registrations” add-on which uses PayPal as well.

I’ve never once had an issue with a payment going through though, and I don’t see anything in the logs about it, so I’m really confused as to what’s going on.

 

[Mike]

Just to confirm, that is the correct URL, right?

For any request from PayPal that hits that script, there should be a log entry in the payment provider log. If you're not seeing error entries in the payment log (or server errors that could be related), then it doesn't sound like XF is directly the culprit. We did recently run into a server that was blocking PayPal's requests -- seemingly to give them a CAPTCHA -- so the request never actually reached PHP/XF.

You should be able to look at the IPN history within PayPal. I believe it's accessible via a link on the IPN configuration page. That should show you whether requests are failing (based on a non-200 response code). What does that show?

 

[PumpinIron]

Just to confirm, that is the correct URL, right?

For any request from PayPal that hits that script, there should be a log entry in the payment provider log. If you're not seeing error entries in the payment log (or server errors that could be related), then it doesn't sound like XF is directly the culprit. We did recently run into a server that was blocking PayPal's requests -- seemingly to give them a CAPTCHA -- so the request never actually reached PHP/XF.

You should be able to look at the IPN history within PayPal. I believe it's accessible via a link on the IPN configuration page. That should show you whether requests are failing (based on a non-200 response code). What does that show?

That should be the correct URL, yes. But my question is where do I specify that URL? Is there somewhere within PayPal or XenForo where I am supposed to specify the correct URL?

I will say that that URL looks correct, as that is the path to the payment_callback.php file on my server.

So I logged in to PayPal to check the IPN history (which is conveniently hidden in the PayPal account), and this is what I see in the IPN log:



In addition, I saw that the Instant Payment (IPN) was turned off, so I enabled it like so:



That of course required me entering the URL and turning it on.

But then this leads me to wonder... was it turned off the entire time? And if it was turned off, how was it that I've been able to receive payments this entire time without issue until recently?

 

[PumpinIron]

Also, on the payments that failed, the response code it's giving is a 503.

Here's an example of one of the failed transactions that it continually keeps retrying:

 

[Mike]

So 503 isn't actually a response code that our code returns from this script. It's also the same response code we found in the other case where this came up. If possible, have a look at your web server error log -- not the one in XenForo, but the one that you may find in your hosting control panel. I suspect you may find something along the lines of this:

rdbm_handle_postscan_block: hostname=domain.com, client_ip='111.111.111.111', would captcha uri: '/payment_callback.php'

(Some elements edited as they may vary.)

In your web server access log, you may also find a similar reference with a "503" response code.

The error message appears to indicate that it is serving a CAPTCHA to the client (or just blocking it). Unfortunately, I don't know what is triggering this and in the case with the other customer, the host wasn't able to provide any useful information. If you can identify a line like that, you could try with your host to see if maybe someone there can at least point to what that's coming from. (Google isn't helpful.)

 

[PumpinIron]

So 503 isn't actually a response code that our code returns from this script. It's also the same response code we found in the other case where this came up. If possible, have a look at your web server error log -- not the one in XenForo, but the one that you may find in your hosting control panel. I suspect you may find something along the lines of this:

rdbm_handle_postscan_block: hostname=domain.com, client_ip='111.111.111.111', would captcha uri: '/payment_callback.php'

(Some elements edited as they may vary.)

In your web server access log, you may also find a similar reference with a "503" response code.

The error message appears to indicate that it is serving a CAPTCHA to the client (or just blocking it). Unfortunately, I don't know what is triggering this and in the case with the other customer, the host wasn't able to provide any useful information. If you can identify a line like that, you could try with your host to see if maybe someone there can at least point to what that's coming from. (Google isn't helpful.)

I’ll get in touch with my web host and see what (if anything) they find and report back.

Are you thinking this is something that is related to the web host and not XF itself then? Or could this be something on PayPal’s end?

 

[PumpinIron]

Another IPN fail just happened, basically they are all failing now. Not sure what's changed, but up until the past few days, this wasn't an issue whatsoever.

Waiting to hear back from my web host (Liquid Web) as we speak:

 

[Mike]

Are you thinking this is something that is related to the web host and not XF itself then? Or could this be something on PayPal’s end?

If it's what I think, it's entirely the web host. They appear to be blocking (or requiring a CAPTCHA of) the PayPal callback "bot", and thus the request is never even reaching XF.

 

[Snog]

I don't know if this IS the problem, but I did find reference to the exact same error with WordPress. It turned out to be a server Spam security setting that needed to be turned off.

I can't read German, but this is what had to be turned off. Perhaps it's something similar...

Changes in WordPress backend result in downloads of PHP files (some POST requests are blocked)

When I try to create a user in WordPress, the download of the file "user-new.php" starts. But no user is created. The same problem if I want to change the settings. Then the download of "options.php"

stackoverflow.com

 

[PumpinIron]

If it's what I think, it's entirely the web host. They appear to be blocking (or requiring a CAPTCHA of) the PayPal callback "bot", and thus the request is never even reaching XF.

My web host got back to me and he said he searched the logs for any errors similar to what you posted above, but he did say this:


We did try to grep for some of those things or key words from that line that the developer stated we should see and there was nothing in the error logs We did see multiple 200 ok entries like the following though.

10.20.4.104 - - [03/Jan/2019:00:26:19 -0500] "GET /favicon.ico HTTP/2.0" 200 391 "https://wranglertjforum.com/payment_callback.php?_xfProvider=paypal" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 Safari/537.36"

So these requests are not throwing an error.

However after looking further down I did find the following

173.0.81.1 - - [02/Jan/2019:11:26:56 -0500] "POST /payment_callback.php?_xfProvider=paypal HTTP/1.1" 503 - "-" "PayPal IPN ( https://www.paypal.com/ipn )"

However its not really providing us with anything viable. I mean its on a Post from a customer and it is a 503 however it does not give us a reason why, there is no mod security blocking it. 173.0.81.1 the IP is not being blocked at all. So let me do some additional looking at this to see what else we can find out.


Does any of that mean anything?

 

[PumpinIron]

I don't know if this IS the problem, but I did find reference to the exact same error with WordPress. It turned out to be a server Spam security setting that needed to be turned off.

I can't read German, but this is what had to be turned off. Perhaps it's something similar...

Changes in WordPress backend result in downloads of PHP files (some POST requests are blocked)

When I try to create a user in WordPress, the download of the file "user-new.php" starts. But no user is created. The same problem if I want to change the settings. Then the download of "options.php"

stackoverflow.com

View attachment 191864

It could have something to do with some sort of spam setting, I agree. I'm just not sure what. I'm pretty familiar with cPanel / WHM, but I'm no system admin by any chance, so I'm hoping the people at Liquid Web can find out what the culprit might be.

Odd that this never happened until a few days ago.

 

[PumpinIron]

@Mike,

Any idea how I can "test" this?

My web host got back to me and said this:

"Unfortunately without a way of testing this is may be quite difficult to get to the bottom of the issue. Can you possibly provide me with the IP address Paypal is using to send these requests that are returning a 503?"

I'm not entirely sure what IP address PayPal would be using, nor am I sure how to test this without actually running a live transaction through PayPal.

 

[Mike]

The request that shows the IP and a 503 result appears to be 173.0.81.1.

If I search the code for "503", there are very few places where this can occur. The most significant is when your forum is closed or being upgraded, but that isn't something that is checked with these payment callbacks anyway. If that were the case, then you visiting the URL in question should trigger a 503 error. (It triggers a 403 if you view it directly, because PayPal requires validating the callback against their servers, which clearly can't happen when you view it; this is a different and unrelated element.)

Unfortunately, I don't really have much more I can recommend. The result PayPal appears to be getting isn't something XenForo would ever send in a situation like this, so I don't see how XF can be the one causing this. (We'd normally also output a reason -- the access log entry for the 503 shows no content length output.)

 

[PumpinIron]

The request that shows the IP and a 503 result appears to be 173.0.81.1.

If I search the code for "503", there are very few places where this can occur. The most significant is when your forum is closed or being upgraded, but that isn't something that is checked with these payment callbacks anyway. If that were the case, then you visiting the URL in question should trigger a 503 error. (It triggers a 403 if you view it directly, because PayPal requires validating the callback against their servers, which clearly can't happen when you view it; this is a different and unrelated element.)

Unfortunately, I don't really have much more I can recommend. The result PayPal appears to be getting isn't something XenForo would ever send in a situation like this, so I don't see how XF can be the one causing this. (We'd normally also output a reason -- the access log entry for the 503 shows no content length output.)

Okay, thanks for the response.

My web host isn't finding anything, and after speaking with PayPal themselves (their tech support), they aren't finding anything on their end. They are saying it's got to be web host related (which is what you're saying as well).

Yet somehow my web host seems unable to find anything. Sheesh!

What blows my mind is why this didn't start happening until the past few days.

 

[sbj]

Any idea how I can "test" this?

I'm not in a position to give support or know how these things work, but for "testing", how about you set up a transaction for the lowest amount possible (0.01$ maybe?) and try to see if you can trigger errors by doing transactions? Just an idea.

 

[PumpinIron]

I'm not in a position to give support or know how these things work, but for "testing", how about you set up a transaction for the lowest amount possible (0.01$ maybe?) and try to see if you can trigger errors by doing transactions? Just an idea.

Yes, I can do that (and probably will). My web host seems perplexed, even though I'm literally providing them with exact times and dates of the failed transactions. Seems like it would be easy enough to search the logs for the exact time and date and look for anything off.

 

[sbj]

Yes, I can do that (and probably will). My web host seems perplexed, even though I'm literally providing them with exact times and dates of the failed transactions. Seems like it would be easy enough to search the logs for the exact time and date and look for anything off.

Try different scenarios like enabling/disabling Paypal related Addons and see if anything is off.
Or I am not sure if it is possible, but put your stuff on localhost and do a transaction there to see if it is server related or not? It would have no access I guess... stupid idea

 

[PumpinIron]

Even more interesting is that in the past 30 days, I've had 302 transactions, and only 4 of them failed. The others were all sent successfully without the IPN failing.

The mystery deepens. Why are some failing and others aren't?

 

[Mike]

I suppose it could be something specific in the request from PayPal, as they include a fair amount of POST data that wouldn't appear in the access log. At the least, I would hope that the anti-spam/mod_security-style system could be adjusted to give more verbose logging to make it clearer when a rule is being triggered.

 

[PumpinIron]

I suppose it could be something specific in the request from PayPal, as they include a fair amount of POST data that wouldn't appear in the access log. At the least, I would hope that the anti-spam/mod_security-style system could be adjusted to give more verbose logging to make it clearer when a rule is being triggered.

It must be. I just had another transaction come through (user upgrade) an hour ago, and the IPN went through without any issue at all.

So some of them fail, while some of them go through. Problem is, when I look at the transactions, it doesn't give much info other than "503 HTTP response code".