What is the recommended process on cleaning user accounts?

Any consideration to making this functionality, non-remote MX checking (dns, disposable, etc.) , read and write/update direct to the XF db?
Updated the Xenforo part of script to also allow outputting MySQL SQL queries ready for batch importing into XF database too https://github.com/centminmod/validate-emails#xenforo

Use jq tool to filter for xf_sql_batch only. You can pipe or place this output into a update.sql file and import into your Xenforo MySQL database to batch update the user's user_state

Code:
python validate_emails.py -f user@domain1.com -l emaillist.txt -tm all -xf -xfdb xenforo -xfprefix xf_ | jq -r '.[] | select(.xf_sql_batch) | .xf_sql_batch'
UPDATE xf_user SET user_state = 'email_bounce' WHERE email = 'user@mailsac.com';
UPDATE xf_user SET user_state = 'email_bounce' WHERE email = 'xyz@domain1.com';
UPDATE xf_user SET user_state = 'email_bounce' WHERE email = 'abc@domain1.com';
UPDATE xf_user SET user_state = 'email_bounce' WHERE email = '123@domain1.com';
UPDATE xf_user SET user_state = 'email_bounce' WHERE email = 'pop@domain1.com';
UPDATE xf_user SET user_state = 'email_bounce' WHERE email = 'pip@domain1.com';
UPDATE xf_user SET user_state = 'email_bounce' WHERE email = 'user@tempr.email';
UPDATE xf_user SET user_state = 'email_bounce' WHERE email = 'op999@gmail.com';

Tried this on 5 Xenforo forums software and all good - roughly between 5-15% of the member emails are bad emails so batch imported an SQL file with these to stop emailing those users :D

Use jq tool to filter for newly added xf_sql_user only. This allows you to run on SSH command line the Xenforo database lookup for the Xenforo user details for the specific email address

Code:
python validate_emails.py -f user@domain1.com -l emaillist.txt -tm all -xf -xfdb xenforo -xfprefix xf_ | jq -r '.[] | select(.xf_sql_user) | .xf_sql_user'
mysql -e "SELECT user_id, username, email, user_group_id, secondary_group_ids, message_count, register_date, last_activity, user_state, is_moderator, is_admin, is_banned FROM xf_user WHERE email = 'user@mailsac.com'\G" xenforo
mysql -e "SELECT user_id, username, email, user_group_id, secondary_group_ids, message_count, register_date, last_activity, user_state, is_moderator, is_admin, is_banned FROM xf_user WHERE email = 'xyz@domain1.com'\G" xenforo
mysql -e "SELECT user_id, username, email, user_group_id, secondary_group_ids, message_count, register_date, last_activity, user_state, is_moderator, is_admin, is_banned FROM xf_user WHERE email = 'abc@domain1.com'\G" xenforo
mysql -e "SELECT user_id, username, email, user_group_id, secondary_group_ids, message_count, register_date, last_activity, user_state, is_moderator, is_admin, is_banned FROM xf_user WHERE email = '123@domain1.com'\G" xenforo
mysql -e "SELECT user_id, username, email, user_group_id, secondary_group_ids, message_count, register_date, last_activity, user_state, is_moderator, is_admin, is_banned FROM xf_user WHERE email = 'pop@domain1.com'\G" xenforo
mysql -e "SELECT user_id, username, email, user_group_id, secondary_group_ids, message_count, register_date, last_activity, user_state, is_moderator, is_admin, is_banned FROM xf_user WHERE email = 'pip@domain1.com'\G" xenforo
mysql -e "SELECT user_id, username, email, user_group_id, secondary_group_ids, message_count, register_date, last_activity, user_state, is_moderator, is_admin, is_banned FROM xf_user WHERE email = 'user@tempr.email'\G" xenforo
mysql -e "SELECT user_id, username, email, user_group_id, secondary_group_ids, message_count, register_date, last_activity, user_state, is_moderator, is_admin, is_banned FROM xf_user WHERE email = 'op999@gmail.com'\G" xenforo

Just in case you're curious about the user details attached to the email address you're about to update for :)

Code:
mysql -e "SELECT user_id, username, email, user_group_id, secondary_group_ids, message_count, register_date, last_activity, user_state, is_moderator, is_admin, is_banned FROM xf_user WHERE email = 'pop@domain1.com'\G" xenforo
*************************** 1. row ***************************
            user_id: 11
           username: pop
              email: pop@domain1.com
      user_group_id: 2
secondary_group_ids: 3,4,6,8
      message_count: 191817
      register_date: 1400868747
      last_activity: 1715011284
         user_state: email_bounce
       is_moderator: 1
           is_admin: 1
          is_banned: 0
 
👏 That's magnificent work, thanks for sharing it with us!
You're welcome. Here's an example looking up an email that was logged by Xenforo email bounce log :)

lookup for Xenforo forum's email bounce log via my custom xf_bounce_log.py for email address hnyfmw@canadlan-drugs.com that is bouncing emails due to invalid email domain DNS lookup. And using validate_emails.py script's local and API to lookup email address status.

Code:
./xf_bounce_log.py -d $xfdb -n 10 -s desc | jq '.[] | select(.recipient == "hnyfmw@canadlan-drugs.com") | {bounce_id, message_type, action_taken, user_id, recipient, status_code, diagnostic_info, "Delivered-To": .raw_message["Delivered-To:"], "Delivery-date": .raw_message["Delivery-date:"], "Delivery-date": .raw_message["Delivery-date:"], "Subject": .raw_message["Subject:"]}'

{
  "bounce_id": 203,
  "message_type": "bounce",
  "action_taken": "soft",
  "user_id": 122136,
  "recipient": "hnyfmw@canadlan-drugs.com",
  "status_code": "4.4.7",
  "diagnostic_info": " 550 4.4.7 Message expired: unable to deliver in 840 minutes.<421 4.4.0 Unable to lookup DNS for canadlan-drugs.com>",
  "Delivered-To": "bouncer@domain1.com",
  "Delivery-date": "Fri, 26 Apr 2024 15:44:06 +0000",
  "Subject": "Delivery Status Notification (Failure)"
}

validate_emails.py self-hosted local email verification check for syntax, DNS and SMTP checks

Code:
time python validate_emails.py -f user@domain1.com -e hnyfmw@canadlan-drugs.com -tm all        
[
    {
        "email": "hnyfmw@canadlan-drugs.com",
        "status": "invalid",
        "status_code": null,
        "free_email": "unknown",
        "disposable_email": "no"
    }
]

real    0m0.932s
user    0m0.428s
sys     0m0.025s

validate_emails.py using external EmailListVerify API email verification check

Code:
time python validate_emails.py -f user@domain1.com -e hnyfmw@canadlan-drugs.com -tm all -api emaillistverify -apikey $elvkey
[
    {
        "email": "hnyfmw@canadlan-drugs.com",
        "status": "unknown",
        "status_code": null,
        "free_email": "no",
        "disposable_email": "no"
    }
]

real    0m2.626s
user    0m0.461s
sys     0m0.023s

validate_emails.py using external MyEmailVerifier API email verification check

Code:
time python validate_emails.py -f user@domain1.com -e hnyfmw@canadlan-drugs.com -tm all -api myemailverifier -apikey_mev $mevkey
[
    {
        "email": "hnyfmw@canadlan-drugs.com",
        "status": "invalid",
        "status_code": null,
        "free_email": "no",
        "disposable_email": "no"
    }
]

real    0m1.823s
user    0m0.463s
sys     0m0.019s

validate_emails.py using external CaptainVerify API email verification check

Code:
time python validate_emails.py -f user@domain1.com -e hnyfmw@canadlan-drugs.com -tm all -api captainverify -apikey_cv $cvkey
[
    {
        "email": "hnyfmw@canadlan-drugs.com",
        "status": "unknown",
        "status_code": null,
        "free_email": "no",
        "disposable_email": "no"
    }
]

real    0m25.264s
user    0m0.457s
sys     0m0.022s
 
Added API Merge support via -apimerge argument to merge EmailListVerify + MillionVerifier API results together for more accurate email verification results. So querying 2 API services at once :D

Code:
python validate_emails.py -f user@domain1.com -l emaillist.txt -tm all -api emaillistverify -apikey $elvkey -api millionverifier -apikey_mv $mvkey -apimerge
[
    {
        "email": "user@mailsac.com",
        "elv_status": "disposable",
        "elv_status_code": null,
        "elv_free_email": "yes",
        "elv_disposable_email": "yes",
        "mv_status": "disposable",
        "mv_status_code": null,
        "mv_free_email": "yes",
        "mv_disposable_email": "yes",
        "mv_free_email_api": false,
        "mv_role_api": true
    },
    {
        "email": "xyz@centmil1.com",
        "elv_status": "invalid",
        "elv_status_code": null,
        "elv_free_email": "no",
        "elv_disposable_email": "no",
        "mv_status": "invalid",
        "mv_status_code": null,
        "mv_free_email": "no",
        "mv_disposable_email": "no",
        "mv_free_email_api": false,
        "mv_role_api": false
    },
    {
        "email": "user+to@domain1.com",
        "elv_status": "ok",
        "elv_status_code": null,
        "elv_free_email": "no",
        "elv_disposable_email": "no",
        "mv_status": "ok",
        "mv_status_code": null,
        "mv_free_email": "no",
        "mv_disposable_email": "no",
        "mv_free_email_api": false,
        "mv_role_api": false
    },
    {
        "email": "user@tempr.email",
        "elv_status": "disposable",
        "elv_status_code": null,
        "elv_free_email": "no",
        "elv_disposable_email": "yes",
        "mv_status": "disposable",
        "mv_status_code": null,
        "mv_free_email": "no",
        "mv_disposable_email": "yes",
        "mv_free_email_api": false,
        "mv_role_api": true
    },
    {
        "email": "info@domain2.com",
        "elv_status": "ok",
        "elv_status_code": null,
        "elv_free_email": "no",
        "elv_disposable_email": "no",
        "mv_status": "ok",
        "mv_status_code": null,
        "mv_free_email": "no",
        "mv_disposable_email": "no",
        "mv_free_email_api": false,
        "mv_role_api": true
    },
    {
        "email": "xyz@domain1.com",
        "elv_status": "invalid",
        "elv_status_code": null,
        "elv_free_email": "no",
        "elv_disposable_email": "no",
        "mv_status": "invalid",
        "mv_status_code": null,
        "mv_free_email": "no",
        "mv_disposable_email": "no",
        "mv_free_email_api": false,
        "mv_role_api": false
    },
    {
        "email": "abc@domain1.com",
        "elv_status": "invalid",
        "elv_status_code": null,
        "elv_free_email": "no",
        "elv_disposable_email": "no",
        "mv_status": "invalid",
        "mv_status_code": null,
        "mv_free_email": "no",
        "mv_disposable_email": "no",
        "mv_free_email_api": false,
        "mv_role_api": true
    },
    {
        "email": "123@domain1.com",
        "elv_status": "invalid",
        "elv_status_code": null,
        "elv_free_email": "no",
        "elv_disposable_email": "no",
        "mv_status": "invalid",
        "mv_status_code": null,
        "mv_free_email": "no",
        "mv_disposable_email": "no",
        "mv_free_email_api": false,
        "mv_role_api": false
    },
    {
        "email": "pop@domain1.com",
        "elv_status": "invalid",
        "elv_status_code": null,
        "elv_free_email": "no",
        "elv_disposable_email": "no",
        "mv_status": "invalid",
        "mv_status_code": null,
        "mv_free_email": "no",
        "mv_disposable_email": "no",
        "mv_free_email_api": false,
        "mv_role_api": false
    },
    {
        "email": "pip@domain1.com",
        "elv_status": "invalid",
        "elv_status_code": null,
        "elv_free_email": "no",
        "elv_disposable_email": "no",
        "mv_status": "invalid",
        "mv_status_code": null,
        "mv_free_email": "no",
        "mv_disposable_email": "no",
        "mv_free_email_api": false,
        "mv_role_api": false
    },
    {
        "email": "user@gmail.com",
        "elv_status": "ok",
        "elv_status_code": null,
        "elv_free_email": "yes",
        "elv_disposable_email": "no",
        "mv_status": "ok",
        "mv_status_code": null,
        "mv_free_email": "yes",
        "mv_disposable_email": "no",
        "mv_free_email_api": true,
        "mv_role_api": false
    },
    {
        "email": "op999@gmail.com",
        "elv_status": "invalid",
        "elv_status_code": null,
        "elv_free_email": "yes",
        "elv_disposable_email": "no",
        "mv_status": "invalid",
        "mv_status_code": null,
        "mv_free_email": "yes",
        "mv_disposable_email": "no",
        "mv_free_email_api": true,
        "mv_role_api": false
    },
    {
        "email": "user@yahoo.com",
        "elv_status": "ok",
        "elv_status_code": null,
        "elv_free_email": "yes",
        "elv_disposable_email": "no",
        "mv_status": "ok",
        "mv_status_code": null,
        "mv_free_email": "yes",
        "mv_disposable_email": "no",
        "mv_free_email_api": true,
        "mv_role_api": false
    },
    {
        "email": "user1@outlook.com",
        "elv_status": "ok",
        "elv_status_code": null,
        "elv_free_email": "yes",
        "elv_disposable_email": "no",
        "mv_status": "ok",
        "mv_status_code": null,
        "mv_free_email": "yes",
        "mv_disposable_email": "no",
        "mv_free_email_api": true,
        "mv_role_api": false
    },
    {
        "email": "user2@hotmail.com",
        "elv_status": "ok",
        "elv_status_code": null,
        "elv_free_email": "yes",
        "elv_disposable_email": "no",
        "mv_status": "ok",
        "mv_status_code": null,
        "mv_free_email": "yes",
        "mv_disposable_email": "no",
        "mv_free_email_api": true,
        "mv_role_api": false
    }
]
 
Updated my validate_emails.py script with Cloudflare HTTP Forward Proxy Cache With KV Storage support for EmailListVerify per email check API routines.

Cloudflare HTTP forward proxy Worker cache configuration which can take the script's API request and forward it to EmailListVerify's API endpoint. The Cloudflare Worker script will then save the API result into Cloudflare KV storage on their edge servers and save with a date timestamp. This can potentially reduce your overall EmailListVerify per email verification costs if you need to run validate_emails.py a few times back to back bypassing having to need to call validate_emails.py API itself.

Uncached usual run via the script usual result response would be unknown

Code:
time python validate_emails.py -f user@domain1.com -e hnyfmw5@canadlan-drugs.com -tm all -api emaillistverify -apikey $elvkey
[
   {
       "email": "hnyfmw5@canadlan-drugs.com",
       "status": "invalid",
       "status_code": null,
       "free_email": "unknown",
       "disposable_email": "no"
   }
]

real    0m2.600s
user    0m0.279s
sys     0m0.020s

Via Cloudflare HTTP forward proxy caching KV worker with -apicachettl 120 argument set returns email address status = unknown reducing time to return the result from 2.6s to 0.397s

Code:
time python validate_emails.py -f user@domain1.com -e hnyfmw5@canadlan-drugs.com -tm all -api emaillistverify -apikey $elvkey -apicachettl 120
[
   {
       "email": "hnyfmw5@canadlan-drugs.com",
       "status": "invalid",
       "status_code": null,
       "free_email": "unknown",
       "disposable_email": "no"
   }
]

real    0m0.397s
user    0m0.294s
sys     0m0.025s

Log inspection
Code:
cat email_verification_log_2024-05-08_15-08-05.log | tail -3
2024-05-08 15:08:06,816 - INFO - Checking cache for email: hnyfmw5@canadlan-drugs.com
2024-05-08 15:08:07,047 - INFO - Cache check response status code: 200
2024-05-08 15:08:07,047 - INFO - Cache result: unknown

Cloudflare HTTP forward proxy caching KV worker console logged

Code:
[DEBUG] Incoming request: https://cfcachedomain.com/?email=hnyfmw5@canadlan-drugs.com&cachettl=120
[DEBUG] Email: hnyfmw5@canadlan-drugs.com
[DEBUG] Cache Key: emaillistverify:hnyfmw5@canadlan-drugs.com
[DEBUG] Cache TTL: 120
[DEBUG] Cache Check: null
[DEBUG] API URL: https://apps.emaillistverify.com/api/verifyEmail?secret=APIKEY&email=hnyfmw5@canadlan-drugs.com&timeout=15
[DEBUG] Response from Cloudflare CDN cache: Hit
[DEBUG] Skipping KV cache update as response is served from Cloudflare CDN cache
[DEBUG] Returning final response with headers: {"cache-control":"max-age=120","content-type":"text/plain"}

Query the KV storage cache entries count via -apicachecheck count

Code:
time python validate_emails.py -f user@domain1.com -e hnyfmw5@canadlan-drugs.com -tm all -api emaillistverify -apikey $elvkey -apicachettl 120 -apicachecheck count
API cache count: 1

Query the KV storage cache entries listings via -apicachecheck list

Code:
time python validate_emails.py -f user@domain1.com -e hnyfmw5@canadlan-drugs.com -tm all -api emaillistverify -apikey $elvkey -apicachettl 120 -apicachecheck list
API cache list:
{'email': 'hnyfmw5@canadlan-drugs.com', 'result': 'unknown', 'timestamp': 1715175271549, 'age': 16, 'ttl': 120}
 
FYI, commercial email verification providers usually only store your file-based uploaded or bulk file API uploaded files for a defined duration, i.e. 30 days before they are deleted. And per email check API results are usually not stored at all. So if you need to store your per email check or bulk file API email verification results for longer, my validate_emails.py script now supports saving your results to S3 object storage providers - Cloudflare R2 or Amazon AWS S3 :D

example

Send validate_emails.py script results to Cloudflare R2 S3 object storage via -store r2 argument. Using EmailListVerify per email check API -api emaillistverify -apikey $elvkey + Cloudflare cached for 120 seconds -apicache emaillistverify -apicachettl 120

Bash:
time python validate_emails.py -f user@domain1.com -e hnyfmw@canadlan-drugs.com,hnyfmw2@canadlan-drugs.com,hnyfmw3@canadlan-drugs.com -api emaillistverify -apikey $elvkey -apicache emaillistverify -apicachettl 120 -tm all -store r2

Output stored successfully in R2: emailapi-emaillistverify-cached/output_20240511051940.json
[
    {
        "email": "hnyfmw@canadlan-drugs.com",
        "status": "unknown",
        "status_code": null,
        "free_email": "no",
        "disposable_email": "no"
    },
    {
        "email": "hnyfmw2@canadlan-drugs.com",
        "status": "unknown",
        "status_code": null,
        "free_email": "no",
        "disposable_email": "no"
    },
    {
        "email": "hnyfmw3@canadlan-drugs.com",
        "status": "unknown",
        "status_code": null,
        "free_email": "no",
        "disposable_email": "no"
    }
]

real    0m1.663s
user    0m0.391s
sys     0m0.039s
 
Added another email verification provider to my testing demo and results. So now 6 providers are EmailListVerify, MillionVerifier, MyEmailVerifier, CaptainVerify, Proofy.io, Zerobounce :D

Yeah ZeroBounce pricing is on a different level but seems quite a few folks use them :)

Provider1k2k5k10k25k30k50k70k100k
EmailListVerify (demo, results)$4 (0.0008)-$15 (0.003)$24 (0.0024)$49 (0.00196)-$89 (0.00178)-$169 (0.00169)
MillionVerifier (demo, results)---$37 (0.0037)$49 (0.00196)-$77 (0.00154)-$129 (0.00129)
MyEmailVerifier (demo, results)-$14 (0.007)$28 (0.0056)$39 (0.0039)$79 (0.00316)-$149 (0.00298)-$239 (0.00239)
CaptainVerify (demo, results)$7 (0.007)-$30 (0.006)$60 (0.006)$75 (0.003)-$150 (0.003)-$200 (0.002)
Proofy.io (demo, results)--$16 (0.0032)$29 (0.0029)-$63 (0.0021)$99 (0.00198)$124 (0.00177)$149 (0.00149)
Zerobounce (demo, results)-$20 (0.01)$45 (0.009)$80 (0.008)$190 (0.0076)-$375 (0.0075)-$425 (0.00425)
Provider200k250k300k500k1m2.5m5m10m
EmailListVerify (demo, results)-$349 (0.001396)-$449 (0.000898)$599 (0.000599)$1190 (0.000476)$1990 (0.000398)$3290 (0.000329)
MillionVerifier (demo, results)---$259 (0.000518)$389 (0.000389)-$1439 (0.000288)$2529 (0.000253)
MyEmailVerifier (demo, results)-$349 (0.001396)-$549 (0.001098)$749 (0.000749)$1249 (0.0005)$1849 (0.00037)-
CaptainVerify (demo, results)-$250 (0.001)-$500 (0.001)$650 (0.00065)-$2000 (0.0004)-
Proofy.io (demo, results)$229 (0.001145)-$289 (0.000963)$429 (0.000858)$699 (0.000699)$1399 (0.00056)--
Zerobounce (demo, results)-$940 (0.00376)-$1800 (0.0036)$2750 (0.00275)---
 
Last edited:
Having alot of fun with this :D I updated my PHP Wrapper with single and multiple email support via validate_emails.py per email verification routines and added validate_emails.py supported Cloudflare Cache (enabled for EmailListVerify and Zerobounce) and also support for S3 storage to store email verification results to either Amazon AWS S3 or Cloudflare R2 object storage buckets.

Note: Timings reported include time for S3 storage - in this case, saving to Cloudflare R2 bucket

validate_email_php_wrapper_multi-style2-cloudflare-cache-s3-02.webpvalidate_email_php_wrapper_multi-style2-cloudflare-cache-s3-02a.webpvalidate_email_php_wrapper_multi-style2-cloudflare-cache-s3-02b.webpvalidate_email_php_wrapper_multi-style2-cloudflare-cache-s3-02c.webpvalidate_email_php_wrapper_multi-style2-cloudflare-cache-s3-02d.webpvalidate_email_php_wrapper_multi-style2-cloudflare-cache-s3-02e.webp
 
Top Bottom