> Just create separate AWS accounts for separate services
My understanding is that different AWS accounts have different mappings of availability zones, so it's very easy to suddenly find yourself with an unexpected bandwidth bill due to all the cross-az traffic.
I've been irritated at AWS (and the other large cloud providers) that they charge $0.01/GB for cross-az traffic. That's $3.24/Mbps -- about the same I was paying for internet transit (as in: from London to anywhere in the world) 20 years ago, and this is just between two datacenters in the same city controlled by the same organisation, markup must be 10,000x or more considering these places are cross-connected with massive bundles of fiber!
> My understanding is that different AWS accounts have different mappings of availability zones, so it's very easy to suddenly find yourself with an unexpected bandwidth bill due to all the cross-az traffic.
As far as I remember, accounts within the same organization will have the same mapping. You also can use stable zone names these days, instead of the regular mappings.
And yeah, egress traffic pricing is freaking insane at this point. It's the biggest reason to NOT use AWS.
Insanely high S3 storage charges too. $23/TB/month? Even with the insane HDD pricing that we see today, that's paying off a drive in 1 month (at retail) that will last for 50-100 months. Sure, there's probably some encoding overhead, but it's still mad.
S3 is pretty competitive if you want similarly-performing storage with consistent millisecond-level latency, high scalability, and at least 3x redundancy. Try looking at how much it's going to cost you in enterprise SSDs :)
Is it 3x redundancy forever? I always just kinda assumed it was RS encoded after a while, so only 30-50% larger than a single copy. Plus, almost all object storage is written to / read from hard disks, not to SSDs. Unless they're in a caching layer that is.
I know Azure has done a bunch of work around Pyramid Codes (essentially a locally repairable EC/RS variant), and Google obviously have the Colossus infrastructure that allows variable encodings, I'd be surprised if AWS is still triple-replicated everywhere.
Yes, and S3 is multiply redundant and is designed to survive a total AZ failure. So your data is physically replicated into at least 2 different AZs and might be multiply-redundant within them. They also provide a crazy SLA for data integrity, meaning that data must never be lost.
S3 also has a reduced redundancy tier and infrequent access tiers that are quite a bit cheaper.
It _is_ expensive, but once you crunch all the numbers, it's actually not unreasonable. I'd argue that using the real S3 is overkill for most scenarios that don't need infinite scalability.
GCS / Azure can survive a total AZ failure too, think of 3 x Replicas as RAID-1, whereas Erasure Coding is more like RAID-6. Only it's actually more resilient:
Let's say you have 10 data blocks, and you have 4 parity blocks. You can now lose 4 servers containing a block and still be able to repair the data, whereas in 3 x Replica you can only lose 2, and have to store everything 300% of size, instead of only 140%.
And yes, it is unreasonable how much they charge for both storage and inter-az bandwidth.
The problem with erasure coding is that if a disk fails, you suddenly need to read 3-5x more data to reconstruct the missing data from parity blocks. This is especially problematic if your replicas are split across zones. The inter-zonal bandwidth is large, but not infinite.
So I'm pretty sure that you need to have at least 2 full copies in different AZs, and then likely at least some additional redundancy within a single AZ (in the form of erasure codes or a full mirror).
So that's at least 3-4x the amount of data. 1Tb of NVMe SSD capacity is around $200 and with 3x redundancy that's $600, or about 2 years of AWS S3 storage. As I said, it's expensive but not unreasonable.
That's what Locally Recoverable Codes (including Pyramid Codes) are designed to address: you can repair a missing block using only a subset of the blocks, which you can make sure are placed in just one zone, eliminating the cross-az bandwidth requirements. Sure, if you lose multiple blocks, you are going to end up needing that extra bandwidth, but the chances of having two blocks offline at once is very low -- if it's just for a failure rather than extended maintenance, then you have probably already repaired the first block by the time the second fails.
In fact, the RS configuration is often in the >50 data blocks and >10 parity blocks range (albeit with an LRC/nested RS config) for object stores because it's more important to have that recoverability than repair efficiency. While one large provider I worked at did have a system whereby they did effectively have two copies of the RS-encoded data (so that 130% turned into 260%) across two AZs, they were actively in the process of swapping to the blocks being evenly distributed across the AZs, near-halving the total required disk space.
As I said before, most object storage is not on SSD, it's on hard disk: it's 20% of the price per TB, and most objects are read very infrequently. I can promise you that they're not paying $200 for 1TB of SSD either... I realise prices are higher than sensible in the last 6 months, but it was fairly easy to pick up SSDs for under $50/TB at retail pricing (and hard disks for under $10/TB) only a year ago.
Someone came up with a good theory a while ago that I'm inclined to believe: The social media companies (esp. Meta as I understand it) were looking at huge fines for showing adult content to under-18s, so they lobbied hard to ensure that the burden of proof for age verification was on anyone else but themselves, hence why the OS vendors are being targeted now.
Ultimately, they seem to have realised that they can't stop adult content from being shared, so the easiest way to get there was to mark anything even vaguely possible of being adult, and require age verification -- which comes with a lot of political cover vs. just deleting it.
Of course, if you stoke up the right people, you end up with lots of support from the puritanical brigades, and label all naysayers as putting children in harm's way.
They could stop adult content from being shown to minors; it would just take effort on their part to do so, so why not shift the effort on to everyone else?
>> They could stop adult content from being shown to minors; it would just take effort on their part to do so
If you voluntarily sensor content, you might be in danger of being held responsible for various things since you control what people see. Phone companies in the US are "common carriers" which means they just connect people, but are not responsible for what people do over the phone (plotting crime or whatever). Social Media is still trying to have it both ways - censor some stuff but not be responsible for anything. IMHO that will eventually fail.
Section 230 allows for as much censoring as you want, you are not liable for user generated content as an interactive computer service provider if you censor or don't.
Showing adult content to minors is also probably not an insignificant part of their business (certaintly a major part if the classification of social media as adult becomes more widespread), and having age be an os-user property might give children more opportunity to subvert the verification. And if enough applications end up behind the maturity wall, they can count on children to badger their parents into setting their account to adult, and the industry will absolve itself of all responsibility once more.
I'm really quite confident I don't want these companies collecting face and ID scans to prove age, so no I think this being an OS problem is actually a very reasonable solution.
That's the point, though. An SSH key gives authentication, not authorization. Generally a certificate is a key signed by some other mutually trusted authority, which SSH explicitly tried to avoid.
SSH does support certificate based auth, and it’s a great upgrade to grant yourself if you are responsible for a multi human single user system. It grants revocation, short lifetime, and identity metadata for auditing, all with vanilla tooling that doesn’t impose things on the target system.
They are remarkably common in long lived enterprise Linux servers. Think eg database servers or web servers where they are of the (much longer lived) pet era not cattle era.
Not sure why you need to belittle one example just to add another
But what I found, empirically, is that a substantial number of observable SSH public keys are (re)used in way that allows a likely-unintended and unwanted determination of the owner's identities.
This consequence was likely not foreseen when SSH pubkey authentication was first developed 20-30 years ago. Certainly, the use and observability of a massive number of SSH keys on just a single servers (ssh [email protected]) wasn't foreseen.
I've never quite understood why there couldn't be a standardised "reverse" HTTP connection, from server to load balancer, over which connections are balanced. Standardised so that some kind of health signalling could be present for easy/safe draining of connections.
The idea is attractive (especially for draining), but once you try to map arbitrary inbound client connections onto backend-initiated "reverse" pipes, you end up needing standardized semantics for multiplexing, backpressure, failure recovery, identity propagation, and streaming! So, you're no longer just standardizing "reverse HTTP", you’re standardizing a full proxy transport + control plane. In practice, the ecosystem standardized draining/health via readiness + LB control-plane APIs and (for HTTP/2/3) graceful shutdown signals, which solves the draining problem without flipping the fundamental accept/connect roles.
Whether the load balancer connects to the server or reverse, nothing changes. A modern H2 connection is pretty much just that: one persistent connection between the load balancer and server, who initiates it doesn't change much.
The connection being active doesn't tell you that the server is healthy (it could hang, for instance, and you wouldn't know until the connection times out or a health check fails). Either way, you still have to send health checks, and either way you can't know between health checks that the server hasn't failed. Ultimately this has to work for every failure mode where the server can't respond to requests, and in any given state, you don't know what capabilities the server has.
They're 47 inches long. Amazon (UK) has 48 inch long zip ties for $14.45 (pack of 12), 60 inch long for $18. Not quite as thick or wide, sure... But that's not what was in the headline :P
Also utility plenums. Or other things. Literally available retail at Home Depot up to 36" for a dollar a tie. We use them to hold temporary fencing together.
> An extra 131 GB of bandwidth per download would have cost Steam several million dollars over the last two years
Nah, not even close. Let's guess and say there were about 15 million copies sold. 15M * 131GB is about 2M TB (2000 PB / 2 EB). At 30% mean utilisation, a 100Gb/s port will do 10 PB in a month, and at most IXPs that costs $2000-$3000/month. That makes it about $400k in bandwidth charges (I imagine 90%+ is peered or hosted inside ISPs, not via transit), and you could quite easily build a server that would push 100Gb/s of static objects for under $10k a pop.
It would surprise me if the total additional costs were over $1M, considering they already have their own CDN setup. One of the big cloud vendors would charge $100M just for the bandwidth, let alone the infrastructure to serve it, based on some quick calculation I've done (probably incorrectly) -- though interestingly, HN's fave non-cloud vendor Hetzner would only charge $2M :P
Isn't it a little reductive to look at basic infrastructure costs? I used Hetzner as a surrogate for the raw cost of bandwidth, plus overheads. If you need to serve data outside Europe, the budget tier of BunnyCDN is four times more expensive than Hetzner.
But you might be right - in a market where the price of the same good varies by two orders of magnitude, I could believe that even the nice vendors are charging a 400% markup.
Yea, I always laugh when folks talk about how expensive they claim bandwidth is for companies. Large “internet” companies are just paying a small monthly cost for transit at an IX. They arent paying $xx/gig ($1/gig) like the average consumer is. If you buy a 100gig port for $2k, it costs the same if you’re using 5 GB a day or 8 PB per day.
> I imagine 90%+ is peered or hosted inside ISPs, not via transit
How hosting inside ISPs function? Does ISP have to MITM? I heard similar claims for Netflix and other streaming media, like ISPs host/cache the data themselves. Do they have to have some agreement with Steam/Netflix?
Yea netflix will ship a server to an ISP (Cox, comcast, starlink, rogers, telus etc) so the customers of that ISP can access that server directly. It improves performance for those users and reduces the load on the ISP’s backbone/transit. Im guessing other large companies will do this as well.
A lot of people are using large distributed DNS servers like 8.8.8.8 or 1.1.1.1 and these cansometimes direct users to incorrect CDN servers, so EDNS was created to help with it. I always use 9.9.9.11 instead of 9.9.9.9 to hopefully help improve performance.
The CDN/content provider ships servers to the ISP which puts them into their network. The provider is just providing connectivity and not involved on a content-level, so no MITM etc needed.
Honestly, I'd give FUSE a second chance, you'd be surprised at how useful it can be -- after all, it's literally running in userland so you don't need to do anything funky with privileges. However, if I starting afresh on a similar project I'd probably be looking at using 9p2000.L instead.
I may be confusing two systems but I believe that AFS system was also encompassed the first iteration of “AWS Glacier” I encountered in the wild. A big storage that required queuing a job to a tape array or pinging an undergrad to manually load something for retrieval.
I know a lot of people who use it, in fact I'm one of them.
I have an @gmail.com account with about 20 years of stuff associated with it, from purchases to YouTube subscriptions, from calendars to GCP accounts.
However, I use a vanity email ([email protected]) that everyone I know uses to get hold of me. Until about 10 years ago I could just forward emails but that slowly became unworkable as more and more stuff just broke due to SPF etc. So, I've been using POP pickup (and accepting the 5-30 minute delay) ever since.
As I understand it, I can't move all my gmail.com data into a GWork profile easily, and POP has worked for years. This is very frustrating.
My understanding is that different AWS accounts have different mappings of availability zones, so it's very easy to suddenly find yourself with an unexpected bandwidth bill due to all the cross-az traffic.
I've been irritated at AWS (and the other large cloud providers) that they charge $0.01/GB for cross-az traffic. That's $3.24/Mbps -- about the same I was paying for internet transit (as in: from London to anywhere in the world) 20 years ago, and this is just between two datacenters in the same city controlled by the same organisation, markup must be 10,000x or more considering these places are cross-connected with massive bundles of fiber!
reply