Fixed Sporadic Registration rejection due to startTime check in RegisterForm

[keithg]

Affected version

v2.0.7

We are getting a large number of registration failures (upto 40% failure rate) being reported on a new system where a small set of users have been invited to join.

One of the symptoms is the user completes the registration form correctly and then gets presented with a blank registration form. This is sporadic and can not be reproduced at will. The typical user device is a low powered Android smartphone.

We have tracked the blank form issue to the startTime check in \Services\User\RegisterForm in isValidRegistrationAttempt:
if (!$this->startTime || ($this->startTime + $options->registrationTimer) > time())

In the failed cases the $this->startTime has the same value as time() call returns. I logged $this->startTime, $options->registrationTimer, and time() and was given 1532520108, 15, 1532520108.

I personally managed to recreate the error just once. I was using an Android mobile phone, filled out the registration form correctly and then waited some time for the phone screen saver to kick in. I then unlocked the phone and hit the register button. The above code rejected my registration. I could only create this failure once. I was able to successfully register at least 10 other times by following the exact same procedure. Sometimes I waited as long as 5 minutes to submit. I mention this since maybe the time delay is involved in some way?

Additional background info:
The user registration process is slow as there are a large number of data fields that get collected as the system builds "areas of interest" user profile information. There are also 6 mandatory fields and a question captcha required. For new users the large amount of fields most likely means that it will take several minutes to fill in the registration form.

On the failed registration attempts all of the form data was filled in correctly (we watched them do it).
None of the failed registration attempts have come from automated bots (so they had to wait 15 seconds for the Registration button to become active at the bottom of the screen).

So the basic question is: Why does the RegisterForm object sometimes have a current timestamp? i.e. $this->startTime == time()

 

[Chris D]

This actually may be related to another bug, here:
https://xenforo.com/community/threads/register-fails-if-ip-changes.148759/

The registration start time and other state are actually stored in the user's session. If their IP address changes or their session expires while they are registering then this can happen.

That said, the session expiring naturally while they're filling in the registration form would be highly unusual - our default session expiry time is 4 hours. So it could point to, as I suggested, something changing their IP address during registration, a much lower session expiry time (the default is 4 hours but it can be changed with some config/code changes) or a misconfigured caching layer where session caching is enabled.

A common cause for IP address changes on mobile devices (apart from a flaky connection) would be data saving or proxy features built into some browsers.

Another cause could be something like CloudFlare or some other proxy, especially if you notice that the IP addresses for the users are actually coming from the proxy server rather than the user's real IP address.

 

[keithg]

Thanks for the quick feedback.

I have no idea if the problem in the field is "ip change" related - my single failure could be ip related as the phone was on wifi in my office (it is a test phone with no sim card) and I have dual active internet connections via a load balancer.

Since the registration fails i have no idea what actual network the real users are using - they are all based in Kenya, I am in Canada.

Regardless, we need a fix as that IS the network situation that the real users are using.

 

[keithg]

Some more information - as noted before this is happening to a large number of registration attempts. We added some logging and have detected 2 different symptoms:
- multiple retries on registration as some users do not fill in the required fields or the captcha. Not sure if this is contributing to the problem.

We are now also seeing a combo of "incorrect unique key" as well as the "incorrect starttime" noted above. Both symptoms of the session getting wiped.

My investigation reveals that these pieces of information are extracted from the session. Where is the session retained? Cookies? Is it possible that these failures are self inflicted by users that refuse to allow cookies on their smart phones?

 

[Chris D]

The session storage depends very much how your server is configured.

By default we will store it in the xf_session table.

However it’s also possible to use a caching layer such as Redis or Memcached and use that for session storage. If this is misconfigured then it’s possible it could be causing this.

You generally can’t start/proceed with registration if cookies are blocked so that seems unlikely.

 

[keithg]

We have a bare bones simple Apache MySQL system with no cache layer.

I did a quick test with cookies blocked and I got popups in my browser telling me about my bad cookie setting. I assume these are java script? What happens when java script is also disabled? No popups?

We are getting such a high field failure rate in Kenya that is not reproducible here in Canada that I suspect the root cause is something environmental. Either user setting or browser version. Sigh.

I will dig some more.

 

[Chris D]

Without JavaScript enabled, you’d get the same error, but the user would see it on a new page rather than in an overlay.

 

[keithg]

Thanks. No Silver bullet then. I will continue to investigate. From our logging it appears that the user session is being reset / not correctly retained when the user does multiple registration submits over a short period of time. We will turn off the iP matching to see if this magically makes everything work.

The multiple registration attempts are generally due to the user failing to fill in required custom user fields or a simple captcha question. The unique key mismatch generally happens after a few attempts.

 

[keithg]

And some feedback - after disabling the IP check we have 1 full day of no errors with about 30 new registrations.

 

[Chris D]

So the question is, what is making those users IP change so often?

Worth noting that XF has actually been working in the same way since day one, over 8 years ago. It generally isn't/shouldn't be a problem, and it really doesn't present itself as a problem apart from, coincidentally, two different users reporting the same thing recently...

If these users are on the same IP or using VPN/proxy software or unreliable internet connections or similar then that could explain the IP address changing frequently, but for the most part, this really shouldn't be a frequent problem.

 

[keithg]

So the question is, what is making those users IP change so often?

Worth noting that XF has actually been working in the same way since day one, over 8 years ago. It generally isn't/shouldn't be a problem, and it really doesn't present itself as a problem apart from, coincidentally, two different users reporting the same thing recently...

If these users are on the same IP or using VPN/proxy software or unreliable internet connections or similar then that could explain the IP address changing frequently, but for the most part, this really shouldn't be a frequent problem.

Agreed. This surprised me as well. It was extremely difficult to narrow this down as the IP behaviour makes no sense to me. But the users are in Kenya and maybe their networking world is different?

 

[keithg]

Agreed. This surprised me as well. It was extremely difficult to narrow this down as the IP behaviour makes no sense to me. But the users are in Kenya and maybe their networking world is different?

P.S. I claim our large set of registration questions means that some peeps fail to answer all questions and so require a couple of attempts. Normally you would expect only 1. Likely the need for a second or a third attempt reveals the IP hopping behaviour.

 

[Chris D]

I think you mentioned in the other thread that you noticed some of the IPs were Google proxies. This suggests they might be using the Google Data Saver feature in their browser.

That might be the root cause here, but there is code for us to replace the Google IP with the user's real IP and therefore it's possible there are still issues for us to fix here. We will look into it.

 

[keithg]

I think you mentioned in the other thread that you noticed some of the IPs were Google proxies. This suggests they might be using the Google Data Saver feature in their browser.

That might be the root cause here, but there is code for us to replace the Google IP with the user's real IP and therefore it's possible there are still issues for us to fix here. We will look into it.

Thanks. Since I disabled the IP address checks registrations have been going well. Should we close this thread and move all future info to the other one?

In the worst nasty cases of "way too many IPs by a single user" I see a mix of several google proxy IP addresses along with several local telco IP addresses - often mulitple addresses with a few minutes of each other. This is every early days for the site (we just turned it on less than a week ago) so we do not yet have a large user base or a long history for enough users.

Cheers...

 

[keithg]

Some real metrics from the field: the # of IP addresses for the top users based upon # of messages posted is:
63, 42, 7, 23, 32, 3, 6, 5, 2, 8, 2, 3...

The first are international users so I expect higher counts. The others are locals that do not travel.

 

[Chris D]

@giffenk although I cannot reproduce any issues with how our Google Data Saver stuff works, and it seems to be returning the correct "real" IP where appropriate, we are actually removing the IP address session invalidation stuff for guests.

Ultimately, that solves the issue you reported. That fix will be in XF 2.0.10. Thanks for your help.