Behind the scenes: How we host Ars Technica

sub_ubi

Well-known member
Pretty fascinating look at how ArsTechina is hosted.

The relevant Xenforo section,

arx-prod-service-r.png


At Ars, we have one master task called arx-production-web-apps (or just “Arx,” in Ars dev lingo) that contains everything we need to run the site. The Arx task runs four containers:

arx_nginx
arx_acta_php
arx_civis_php
arx_taberna_php

The first container, arx_nginx, runs an Nginx reverse proxy. Incoming HTTP traffic hits Nginx first, and Nginx then routes the traffic appropriately to one of the three PHP containers. The three PHP containers run php-fpm and actually do the dynamic processing and page rendering for Ars and its components.

Those PHP containers are arx_acta, which runs the whole WordPress PHP application; arx_civis, which runs the Xenforo-powered OpenForum; and arx_taberna, which runs the Ars web store (where you can purchase fine merchandise like a Moonshark t-shirt or an Ars mug!)

Depending on the time of day, more tasks will be invoked to handle load as it ramps up. I’m writing this at about 7:00 am US Eastern Time early in the morning of July 12, and right now, the AWS console says we’re running 24 instances of the “Arx” task. That usually scales up to 30-ish instances by 9:00 am Eastern as more people turn on their computers and hit up the Ars homepage for news. (As more tasks are added, the Application Load Balancer seamlessly distributes traffic between them to prevent any one task’s processes from becoming saturated.)
 
Last edited:
  • Like
Reactions: fly
It's crazy to me that large sites use AWS instances. I mean it's definitely convenient, but it can get so expensive. Last time I priced it (granted, it was about a decade ago, so maybe things have changed), it would have cost me somewhere north of $50,000/month to host what I was doing. At the time I was paying about $1,500/month to colocate my own equipment (that cost has since gone down to $815 because I moved from an area that had close to the most expensive electricity in the US to one of the cheapest). Getting charged per HTTP request and per database query priced them out of what I was even remotely willing to pay.
 
The price for them is likely much lower since they're owned by Condé Nast

AWS was always the preferred option, yeah—cost was a factor, and like i said in the piece, we get to cheat a bit by taking advantage of conde's enterprise negotiated rates. But a bigger factor was the huge availability of AWS docs—especially and including reference designs—that we could consult and crib from when coming up with our hosting strategy.

 
The price for them is likely much lower since they're owned by Condé Nast



AWS is still prohibitively expensive. Unless you need their Geo and redundancy features and an outage would cost you dearly... they aren't worth it. We used to colocate our own hardware like @digitalpoint above and it was way cheaper. We bought used servers and equipment and did all the setup and maintenance ourselves. Definitely more work... but also 1/10th the price :(

Now Linode and Digital Ocean are relatively cheap and the hardware headache is gone so that's what we use.
 
AWS is still prohibitively expensive. Unless you need their Geo and redundancy features and an outage would cost you dearly... they aren't worth it. We used to colocate our own hardware like @digitalpoint above and it was way cheaper. We bought used servers and equipment and did all the setup and maintenance ourselves. Definitely more work... but also 1/10th the price :(

Now Linode and Digital Ocean are relatively cheap and the hardware headache is gone so that's what we use.

Ironically, AWS does not have a good track record with outages.
 
  • Wow
Reactions: FTL
Why am I not surprised that they use AWS? Use the best.

When I used it for my own small forum project, it was pretty slick and reliable. XF Cloud runs on it and that should be an endorsement as the devs here know what they're doing so just wouldn't choose something second rate to run their business on.

Gonna look at that article.

AWS is still prohibitively expensive. Unless you need their Geo and redundancy features and an outage would cost you dearly... they aren't worth it. We used to colocate our own hardware like @digitalpoint above and it was way cheaper. We bought used servers and equipment and did all the setup and maintenance ourselves. Definitely more work... but also 1/10th the price :(
Yeah, they're a bit pricey alright; I had to be very careful which resources I used to keep costs down. It boiled down to one small EC2 server and an RDS instance along with an automated backup regime and a few other bits and pieces.

I'm not surprised you saved a ton by doing it all yourself with your own equipment. After all, one is actually renting resources on AWS, so that's why users are called tenants and renting is expensive in whatever form it takes.

Ironically, AWS does not have a good track record with outages.
I hope they've improved. They were completely reliable when I used them.
 
It's crazy to me that large sites use AWS instances. I mean it's definitely convenient, but it can get so expensive. Last time I priced it (granted, it was about a decade ago, so maybe things have changed), it would have cost me somewhere north of $50,000/month to host what I was doing. At the time I was paying about $1,500/month to colocate my own equipment (that cost has since gone down to $815 because I moved from an area that had close to the most expensive electricity in the US to one of the cheapest). Getting charged per HTTP request and per database query priced them out of what I was even remotely willing to pay.
+1 totally agree. As server hardware and cpus get more powerful, over time that trickles down to lower server hardware too. I'd imagine your next dedicated/colo server hardware will last you even longer - AMD EPYC Genoa-X has just been released https://www.phoronix.com/review/amd-epyc-9684x-benchmarks. Imagine this hardware in 2-3 yrs time in the cheaper price ranges :)
 
Last edited:
+1 totally agree. As server hardware and cpus get more powerful, over time that trickles down to lower server hardware too. I'd imagine your next dedicated/colo server hardware will last you even lower - AMD EPYC Genoa-X has just been released https://www.phoronix.com/review/amd-epyc-9684x-benchmarks. Imagine this hardware in 2-3 yrs time in the cheaper price ranges :)

The long term reliability of the hardware has also improved, so that's a big bonus. Except for rust drives, consumer grade rust reliability has gone down in recent years. WD 10K Enterprise rust, however, have been extremely robust in my experience.
 
XF Cloud runs on it and that should be an endorsement as the devs here know what they're doing so just wouldn't choose something second rate to run their business on.

Where do you get this idea? The site in your signature that you moved to XF Cloud has an IP that belongs to vultr 🤔
 
Where do you get this idea? The site in your signature that you moved to XF Cloud has an IP that belongs to vultr 🤔
I asked one of the devs at the time I took it out and they said AWS. Just checked it and it is indeed showing as vultr, whom I've never heard of. That's disappointing. :confused:

Now, there's one of three possiblities here:
1 I was lied to
2 It's been quietly changed since
2 Other

I'm gonna go with the second option as the more likely one as I can't see a reason for them to lie.

At the time, I just took them at their word and I didn't make a note of the IP address, which would change with different hosts and the data takes time to transfer over and set up, but I've not seen any little outages, either; it's been rock solid.

It wouldn't surprise me if it's been changed, since when taking out the hosted service, they don't specify where it's hosted. I checked just now and still don't see anything.
 
Last edited:
I'm gonna go with the second option as the more likely one as I can't see a reason for them to lie.

Looking through your ticket of moving to Cloud, the only mentions of AWS are when we logged into your account to perform the migration. I think you must have mis-remembered as we have never used AWS.
 
Looking through your ticket of moving to Cloud, the only mentions of AWS are when we logged into your account to perform the migration. I think you must have mis-remembered as we have never used AWS.
Maybe I misremembered, I'm not sure. I just remember AWS and I've never heard of vultr. A name like that I'd remember. I'll see if I can find it. As I said to Brogan though, please don't hold your breath.
 
@Brogan @Slavik You know what, perhaps it's IPS that's hosted on AWS and over time I confused it with XF? That sounds like a definite possibility.
 
Last edited:
Back
Top Bottom