I host my sitemaps on s3.
I recently had a bug with an addon that got my sitemap stuck in an infinite loop (my fault).
However, i noticed that due to the extra $30 in s3 costs for the day! on this mistake, it writes to the bucket seemingly on each record.
My egress fees hit 827,000+ transactions.
Suggest to write the records to memory/a buffer of some sort -- even local temp/ disk and then do a bulk one-write per xml file.
a few moments later:
and....
you get the idea. It shouldn't write this temp file to S3 for every record append.
Write once. we have ram, use it. we have local disk if needed, use it.
I recently had a bug with an addon that got my sitemap stuck in an infinite loop (my fault).
However, i noticed that due to the extra $30 in s3 costs for the day! on this mistake, it writes to the bucket seemingly on each record.
My egress fees hit 827,000+ transactions.
Suggest to write the records to memory/a buffer of some sort -- even local temp/ disk and then do a bulk one-write per xml file.
a few moments later:
and....
you get the idea. It shouldn't write this temp file to S3 for every record append.
Write once. we have ram, use it. we have local disk if needed, use it.
Upvote
1