In my test it is just did not got deleted. I don't know whether this is caused by a cache.
But after the user changes one of the other user fields after the deletion of the one field, the previously deleted field is also (correctly) deleted from $user['customFields'].
This is expected an mentioned in the comment Chris quoted. You can rebuild user info if you really want to get rid of it, but there isn't an expectation that a field would be accessed that doesn't exist (and we never loop through the custom field cache for display).