Duplicate sitemap xmls visible when generation disabled

[Marcus]

Affected version

2.3 beta 6

I disabled to generate sitemaps in admin.php?options/groups/sitemap/ and today saw a spike in cloudflares useage telling me it transferred more than 30GB.

Yesterday my server transferred 3GB of sitemaps all in the format /sitemap-235.xml, /sitemap-256.xml, /sitemap-888.xml ...

# awk -F'\t' '$3 ~ /03\/May\/2024/ {split($4, a, " "); if (a[2] ~ /\/sitemap/) sum += $6} END {print sum / 1073741824 " GB"}' log.log
2.82386 GB

My issue is also not that it is possible to see the sitemap (I disabled the generation, not the view) but that there are hundreds of -xxx sitemaps available where my community is smaller than XenForos.

65.21.136.254   -       [03/May/2024:00:34:10 +0000]    "GET /sitemap-5331.xml HTTP/2.0"        200     188285  "-"     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"        "65.21.136.254" "www.redacted.com" rt=0.000        ua="unix:/run/php/php-fpm.sock" us="200"        ut="0.000"      ul="188341"     cs=-   cfip="65.21.136.254"     cfcountry="FI"   cfssl="{\x22scheme\x22:\x22https\x22}"  cfproto="https" cfipcity="Helsinki"     cfipcountry="FI"        cfipcontinent="EU"      cfiplongitude="24.93470"        cfiplatitude="60.17190" cfregion="Uusimaa"       cfregioncode="18"       cfmetrocode="-" cfpostalcode="00100"    cftimezone="Europe/Helsinki"
207.180.242.82  -       [03/May/2024:00:34:10 +0000]    "GET /sitemap-409.xml HTTP/2.0" 200     186898  "-"     "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"      "207.180.242.82" "www.redacted.com" rt=0.000        ua="unix:/run/php/php-fpm.sock" us="200"        ut="0.000"      ul="186958"     cs=-    cfip="207.180.242.82"   cfcountry="DE" cfssl="{\x22scheme\x22:\x22https\x22}"    cfproto="https" cfipcity="Nuremberg"    cfipcountry="DE"        cfipcontinent="EU"      cfiplongitude="11.16170"        cfiplatitude="49.40500" cfregion="Bavaria"     cfregioncode="BY"        cfmetrocode="-"  cfpostalcode="90475"    cftimezone="Europe/Berlin"
54.36.232.187   -       [03/May/2024:00:34:10 +0000]    "GET /sitemap-2308.xml HTTP/2.0"        200     185387  "-"     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 OPR/109.0.0.0"  "54.36.232.187" "www.redacted.com" rt=0.000        ua="unix:/run/php/php-fpm.sock" us="200"        ut="0.000"      ul="185445"    cs=-     cfip="54.36.232.187"    cfcountry="FR"   cfssl="{\x22scheme\x22:\x22https\x22}"  cfproto="https" cfipcity="-"    cfipcountry="FR"        cfipcontinent="EU"      cfiplongitude="2.33870" cfiplatitude="48.85820" cfregion="-"    cfregioncode="-" cfmetrocode="-" cfpostalcode="-"        cftimezone="Europe/Paris"
116.202.224.86  -       [03/May/2024:00:34:10 +0000]    "GET /sitemap-3084.xml HTTP/2.0"        200     184578  "-"     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"        "116.202.224.86"        "www.redacted.com" rt=0.000        ua="unix:/run/php/php-fpm.sock" us="200"        ut="0.000"      ul="184637"    cs=-     cfip="116.202.224.86"   cfcountry="DE"   cfssl="{\x22scheme\x22:\x22https\x22}"  cfproto="https" cfipcity="Willanzheim"  cfipcountry="DE"        cfipcontinent="EU"      cfiplongitude="10.22910"       cfiplatitude="49.68080"  cfregion="Bavaria"       cfregioncode="BY"       cfmetrocode="-" cfpostalcode="97348"    cftimezone="Europe/Berlin"
135.181.214.38  -       [03/May/2024:00:34:10 +0000]    "GET /sitemap-920.xml HTTP/2.0" 200     185358  "-"     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"        "135.181.214.38"        "www.redacted.com" rt=0.000        ua="unix:/run/php/php-fpm.sock" us="200"        ut="0.000"      ul="185414"     cs=-   cfip="135.181.214.38"    cfcountry="FI"   cfssl="{\x22scheme\x22:\x22https\x22}"  cfproto="https" cfipcity="Helsinki"     cfipcountry="FI"        cfipcontinent="EU"      cfiplongitude="24.93470"        cfiplatitude="60.17190" cfregion="Uusimaa"       cfregioncode="18"       cfmetrocode="-" cfpostalcode="00100"    cftimezone="Europe/Helsinki"
54.36.232.187   -       [03/May/2024:00:34:10 +0000]    "GET /sitemap-3083.xml HTTP/2.0"        200     183559  "-"     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 OPR/109.0.0.0"  "54.36.232.187" "www.redacted.com" rt=0.000        ua="unix:/run/php/php-fpm.sock" us="200"        ut="0.000"      ul="183613"    cs=-     cfip="54.36.232.187"    cfcountry="FR"   cfssl="{\x22scheme\x22:\x22https\x22}"  cfproto="https" cfipcity="-"    cfipcountry="FR"        cfipcontinent="EU"      cfiplongitude="2.33870" cfiplatitude="48.85820" cfregion="-"    cfregioncode="-" cfmetrocode="-" cfpostalcode="-"        cftimezone="Europe/Paris"
116.202.224.86  -       [03/May/2024:00:34:10 +0000]    "GET /sitemap-6352.xml HTTP/2.0"        200     187210  "-"     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"        "116.202.224.86"        "www.redacted.com" rt=0.000        ua="unix:/run/php/php-fpm.sock" us="200"        ut="0.000"      ul="187269"    cs=-     cfip="116.202.224.86"   cfcountry="DE"   cfssl="{\x22scheme\x22:\x22https\x22}"  cfproto="https" cfipcity="Willanzheim"  cfipcountry="DE"        cfipcontinent="EU"      cfiplongitude="10.22910"       cfiplatitude="49.68080"  cfregion="Bavaria"       cfregioncode="BY"       cfmetrocode="-" cfpostalcode="97348"    cftimezone="Europe/Berlin"

 

[Chromaniac]

Ok. I am sort of confused by the question. But it's likely that you were affected by the Beta 5 bug that got fixed in Beta 6?

Fixed Thread 'Infinite Sitemap Generation in 2.3B5'

Apr 22, 2024

So, I just noticed this accidentally. The sitemap generator was going on since Saturday morning.



I had to restart php to stop it. I tried creating it again and it again went crazy. This was definitely not happening on Beta 4.

I tried on my test install which barely has one thread. And the same behavior is happening there as well. I have disabled the sitemap cron for now.

• Chromaniac
• sitemap
• Replies: 5
• Forum: Resolved bug reports

• 

 

[Marcus]

wooooooow thats one busy Sitemap generator there! Thank you for pointing me to this bug report.

I do run beta 6 now but on 22nd April my generator according to admin.php?logs/sitemap/ pushed out 387.211.179 URLs in 7.745 files just on that day alone. That was the last entry in the statistics.

I guess the sitemaps were served and public thanks to Cloudflare. I do wonder about yesterdays 3 GB of transfer from my server alone for these sitemap-123.xml transfers.

 

[Chromaniac]

i assume xenforo would not delete the extra files it created on beta 5 by itself. i suppose just cleaning up the sitemap folder and running a fresh sitemap generation on beta 6 should bring the sitemap files updated and to accurate number.

 

[Marcus]

for any beta testers who read this thread, you can delete the falsly generated sitemaps in /internal_data/sitemaps