Hacker Newsnew | past | comments | ask | show | jobs | submit | more BiteCode_dev's commentslogin

Yes, perfs are better on bare metal, and it's cheaper too. In fact, even a cheap VPS will perform better than the cloud for much less money, and scale vertically to incredible highs.

You do have to do a lot of things manually, and even doing that, you won't get the high availability and elasticity you have the potential to get from a cloud offer. Potential being the key word though.

But honestly, most projects don't need it.

Crashing used to terrify me. One day I started to work for a client that had his service down once a month.

Guess what ? Nothing happened. No customers ever complained. No money was lost, in fact, the cash flow kept growing.

Most services are not so important they can't get switched off once in a while.

Not to mention, monoliths are more robust they are given credit for.


> you won't get the high availability

Debatable.

I've seen a fair amount of outages caused by the extra complexity brought on by making a system distributed for the purposes of high availability.

Hardware is actually quite reliable nowadays, and I'll trust a hardware single-point of failure running a monolithic application more than a distributed microservice-based system with lots of moving parts.

Sure, in theory, the distributed system should win, but in practice, it fails more often (due to operator error or unforeseen bugs) than hardware.


> Sure, in theory, the distributed system should win, but in practice, it fails more often (due to operator error or unforeseen bugs) than hardware.

Isn't this because of the rampant introduction of accidental complexity whenever you attempt to make a system horizontally scalable - e.g. for whatever reason the developers or the people in charge suddenly try to cram in as many technological solutions as possible because apparently that's what the large companies are doing?

There's no reason why you couldn't think about which data can or cannot be shared, and develop your system as one that's scalable, possibly modular, but with the codebase still being largely monolithic in nature. I'd argue that there's a large gray area between the the opposite ends of that spectrum - monoliths vs microservices.

I actually wrote down some thoughts in a blog post of mine, called "Moduliths: because we need to scale, but we also cannot afford microservices": https://blog.kronis.dev/articles/modulith-because-we-need-to...


Not to say that some systems don't introduce accidental complexity, however, getting distributed systems right/correct is necessarily complicated. Design for failure and you will get a good picture of what tools you need to employ.


I think that you can keep systems simple and HA if you forget about multi-master components and design active/passive solutions instead, whose behaviour in failure scenarios is much easier to understand and much less likely to go wrong, especially if you also forgo attempting to autorecover failed components after a failover event.

If no HA is fine 90% of the time, simple HA is fine 99% of the time.


I think another issue is that there are plenty of robust, battle-tested building blocks such as databases for monolithic, vertically-scalable applications, but much fewer for horizontally-scalable ones, meaning you'll need to roll your own (at the application layer) and most likely screw it up.


I do agree with you in that regard, however, that's also a dangerous line of thinking.

There are attempts to provide horizontal scalability for RDBMSes in a transparent way, like TiDB https://pingcap.com/ (which is compatible with the MySQL 5.7 drivers), however, the list of functionality that's sacrificed to achieve easily extensible clusters is a long one: https://docs.pingcap.com/tidb/stable/mysql-compatibility

There are other technologies, like MongoDB, which sometimes are more successful at a clustered configuration, however most of the traditional RDBMSes work best in a leader-follower type of replication scenario, because even those aforementioned systems oftentimes have data consistency issues that may eventually pop up.

Essentially, my argument is that the lack of good horizontally scalable databases or other data storage solutions is easily explainable by the fact that the problem itself isn't solvable in any easy way, apart from adopting eventual consistency, which is probably going to create more problems than it will solve in case of any pre-existing code that makes assumptions about what ways it'll be able to access data and operate on it: https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...

To that end, i'd perhaps like to suggest an alternative: use a single vertically scalable RDBMS instance when possible, with a hot standby if you have the resources for that. Let the architecture around it be horizontally scalable instead, and let it deal with the complexities of balancing the load and dealing with backpressure - introduce a message queue if you must, maybe even an in-memory one for simplicity's sake, or consider an event based architecture where "what needs to be done" is encapsulated within a data structure that can be passed around and applied whenever possible. In my eyes, such solutions can in many cases be better than losing the many benefits of having a single source of truth.

Alternatively, when you actually hit issues with the above approach (and only then), consider sharding as a possibility, or, alternatively, do some domain driven design, figure out where to draw some boundaries and split your service into multiple ones that cover the domain with which you need to work with. Then you have one DB for sales, one for account management, one for reports and so on, all with services that are separated by something as simple as REST interfaces and with rate limits or any of the other mechanisms.

If, however, neither of those two groups of approaches don't seem to be suitable for the loads that you're dealing with, then you probably have a team of very smart people and a large amount of resources to figure out what will work best.

To sum up, if there are no good solutions in the space, perhaps that's because the problems themselves haven't been solved yet. Thus, sooner or later, they'll need to be sidestepped and their impact mitigated in whatever capacity is possible. Not all components can or should scale horizontally.


Have you ever seen CockroachDB, or Yugabyte? Both are not eventually consistent, but are horizontally scalable. It looks like to me it is a solved problem.


The biggest benefit of HA architectures is IMO not resilience from crashing or overloaded systems, but more that it is often a prerequisite for doing zero-downtime updates and partial rollout of updates.

No more emergency revert to last version because the new version didn't start up, only to realize there was a schema update so you also must restore your data from a snapshot, which nobody knows how to do. All this under stress from the service being completely down.


Yeah, I edited my answer because that's fair. Even AS 430 main frames with crazy uptimes are not unheard of after all.


> ...and it's cheaper too...

> You do have to do a lot of things manually...

Doing those manual things has a cost, though.

In most cases, it's only cheaper if you value engineering time at $0. Fine for a personal project, but the math changes when you're paying a fully-loaded cost of $100-200/hr for engineers and you have a long backlog of more valuable things for them to work on.

That's the real reason companies use cloud services: Not because it's cheaper or faster, but because it lets the engineers focus their efforts on the things that differentiate the company.

From another perspective: I can change the oil in my car cheaper and faster than driving to a shop and having someone else do it. However, if my time is better spent doing an hour of freelancing work then I'll gladly pay the shop to change my oil while I work.


> In most cases, it's only cheaper if you value engineering time at $0.

Clearly not at $0 since the cloud is much more expensive and you need more of it because it's slower. If you could have someone do setup and maintenance for free, obviously you'd be way ahead.

So the question is really how much does it cost and whether it's cheaper than the cloud tax.

At a previous startup our AWS bill would've been enough to hire about 3 to 4 full time sysadmins at silicon valley rates. Our workload wasn't huge. I estimated at the time we could've taken one month of AWS cost to buy more than enough equipment to run everything in-house with redundancy, then hire two people to run it all and bank the rest.

At current startup, we're still very small and have no real customer traffic, but the AWS bill is getting close, not there yet, to pay for one full time person. For a workload that could be hosted on a few ancient laptops over a home DSL line if we wanted.

Yes, there's quite the convenience in being able to just click a new instance into existence, but the true cost is also very high. Sure it's just VC money we're all lighting on fire, but I often wonder how much more could be accomplished if we didn't collectively just hand over most of the funding to AWS.


Genuine question, why can't "sys-admin" or are you overloaded with work?


Eh, everything can be automated just as well on bare metal. "The cloud" tends to add complexity not remove it, or at best replace one kind of complexity with another. Bare metal tooling is less familiar and could use some advancement, but basically anything that a cloud compute provider can do can be done on bare metal.

A lot of orgs just never bothered to learn how to automate bare metal and ended up doing a lot of manual work.


> Eh, everything can be automated just as well on bare metal.

Right, but that's not free.

Anything can be automated to match the level of commercial cloud providers given sufficient time and money, but it's far from free.

It's easy to underestimate engineering costs when they're hidden behind a fixed annual salary, but once you're looking at the cost/benefit analysis and actually accounting for all the time spent managing bare metal, it's not a good tradeoff for small or even medium size companies.


People universally build big complicated systems around cloud providers which are equally not free.

Unless you are doing something extremely basic, this is what gets done, with a good deal of experience it seems the complexity of what people do to manage and automate cloud providers is more or less equal to what they do to manage bare metal hardware well.


Well, not really. If you are using a cloud solution, you usually needs an engineer that knows this particular solution. Outside of the HN bubble, it's a rare breed and it cost a lot more than your traditional linux admin, that you probably already have anyway.

Then you need to design, maintain and debug the distributed cloud system, which is more complex.

So you'll have a dedicated person or team for that in both cases.

On the other end, setuping a linux box for the common tasks (web server, cache, db, etc) never takes me more than a day.


> Outside of the HN bubble, it's a rare breed and it cost a lot more than your traditional linux admin, that you probably already have anyway.

Strongly disagree. It's much easier to find people with experience deploying to particular cloud services than it is to find someone who knows how to setup, managed, and maintain a bare metal Linux system from scratch. It's also much easier to teach a cloud solution than to teach someone everything to know about bootstrapping and maintaining Linux servers in a safe and secure manner.

> On the other end, setuping a linux box for the common tasks (web server, cache, db, etc) never takes me more than a day.

That's ignoring all of the future overhead of maintaining it and keeping up with updates. Not to mention the required documentation and knowledge sharing to help others understand your custom system so the bus factor isn't 1.

It's always easy to get started with one-off systems, but that's missing the bulk of the work.


Kinda playing the devil’s advocate, but cloud solution aren’t inherently more complex than administrating a traditional linux.

Thinking of securing the OS, configuring a deploy mechanism, managing updates, dealing with lib compatibilities once you have more than 1 app running, etc. That’s a pretty vast and deep set of skills needed.

You assume there is already one of such admin in the company, but going cloud first reduce all of the above to Docker/container knowledge, and understanding cloud configurations, which can go from a heroku setup, a GCE set of instances up to a small GKE cluster. You can introduce as much complexity as you are comfortable with.


>You can introduce as much complexity as you are comfortable with.

Reading HN, people still have complexity issues due various reasons.


> On the other end, setuping a linux box for the common tasks (web server, cache, db, etc) never takes me more than a day.

You can get a lightsail instance setup in less than an hour.


Oh like that one system I saw once with an uptime of 10 years that was happily chugging away data (not web facing tho).

Bare metal servers with a propper fallback and a working monitoring/notification system can be incredibly reliable and for most purposes definitely enough.


>"You do have to do a lot of things manually, and even doing that, you won't get the high availability and elasticity you get from a cloud offer."

I run my things on bare metal. In case of hardware failure it takes less than 1 hour to restore servers from backup and there was exactly zero hardware failures for all its lifetime anyways. I also have configurations with standby servers but I am questioning it now as again there was not a single time (bar testing) when standby was needed.

As for performance - I use native C++ backends and modern multicore CPU with lots of RAM. Those babies can process thousands of requests per second sustainably without ever breaking a sweat. It is more than enough for any reasonable business. All while costing fractions of a cloudy stuff.


With ZFS or even RAID, there should ideally never be a need to "restore from backup" because of a conventional hardware failure; storage drive malfunctions nowadays can and (IMO) should be resolved online.

This is of course not a reason to avoid backups, but nowadays "restoring from backups" should be because of operator error or large-scale disaster (fire, etc), not because of storage drive failure.

Nowadays I'd be more worried about compute hardware failure - think CPU, RAM or the system board. Storage redundancy is IMO a long-solved problem provided you don't cheap out.


Can you educate me on the typical setup you would use to achieve that goal?


Either ZFS or some form of RAID with redundancy cranked up to the max, so that it can tolerate a high number of drives failing and still continue operating. ZFS configured in RAIDZ3 mode for example can tolerate 3 drives failing and still be able to operate (and rebuild itself once the failed drives are replaced).

You are very unlikely to have 3 drives fail at the same time, so you're pretty much immune to storage hardware failures. Again, this is not an argument against backups (they're useful for other reasons), but with this amount of redundancy on the storage level, I'd expect something else in the system to die before the storage layer fails catastrophically.


> You are very unlikely to have 3 drives fail at the same time, so you're pretty much immune to storage hardware failures.

You have to be careful. If you take three storage devices off the production line and use them as a mirrored array and one fails, there's a good chance they all fail, because you're not just dealing with storage hardware, but storage software too.

HP had two batches of SSDs that failed when the power on time hit some rollover point. And I don't think they're the only one. Fabrication issues are also likely to turn up simultaneously if given equal write loads.

If failures were independent events, you're spot on, but they may not be.


Yes that is correct; I assumed it was somewhat common knowledge if you're going for (and have the budget for) such levels of redundancy.


I would add to this that certain combinations of modern SSDs (like nvme) can put disk performance equal to or in excess of RAM performance, too, so there are other considerations besides resilience, too.

I've seen people do this for DB-intenstive things like large search server deployments and map rendering. Doesn't take many nvme mirrors to get past DDR4 speed.


Would you happen to be freelancing a bit?

I'd like to hire somebody for a day to teach me those things. Remote is ok.


Feel free to send me an email (address in my profile) and we can discuss. I don't think I'm the right person to teach system administration, but we can always have an informal conversation and I can hopefully point you to better resources.


Take a look at iocage and the FreeNAS project.

They probably have the best implementation of all of this wrapped up into a cloud-like web interface.


I have a service at work that only needs to be up during a couple of 4 hour intervals every week. It's one of the least stressful elements of my job.

There's a cost to always-on systems, and I don't think we're accounting those costs properly (externalities). It's likely that in many cases the benefits do not outweigh the costs.

I think it comes down to your ability to route around a problem. If the budgeting or ordering system is down, there are any number of other tasks most of your people can be doing. They can make a note, and stay productive for hours or days.

If you put all of your software on one server or one provider, and that goes down, then your people are pretty much locked out of everything. Partial availability needs to be prioritized over total availability, because of the Precautionary Principle. The cost of everything being down is orders of magnitude higher than the cost of having some things down.


One thing I don't get is why not have an on prem solution with cloud fallback? It's hard but not super hard. You would just need a cloud data store. And depending on your app, you can have on prem data store that periodically backs up (and no egress under normal operation) if you can design with an eventual consistency models.


It's not a dead end with Signal. But it requires field work, as they used to do 50 years ago.

Now, cops and politicians want to solve all the problems from their desk.

No, sorry, my freedom is not to be sold for their convenience.

You want to catch a bad guy, you get a trained investigation team that follows people, that wires their house, that interrogates neighborhood, etc.

Is it more work ? Yes. Is it more dangerous ? Hell, yes.

But don't say you can't catch criminals because of Signal. What you can't do, is click on a button to spy on people. It's a good thing.

This mantra is just an excuse to chew off chunks of democracy.


> What you can't do, is click on a button to spy on people

There's a subpoena in this process that you're glossing over. You can argue that's too easy or too secretive or something, and that's more than fair, but it's not just 'clicking a button'.


I'm not arguing about the subpoena, I'm arguing against the idea that encrypted solutions are bad.

If you have a subpoena to open a safe, and you realize that you have no tools that are strong enough to open that safe, you don't suddenly blame safes. You don't tell banks they should stop using safes. You don't ask them to create weaker safes robbers can break into.

You try another route.

A subpoena is fair. Asking signal to preemptively not encrypt the data in case we need it later is not.


azinman2 didn't say that there should not be encryption, just that there's a cost, and I think that's a fair statement. Sometimes, 'other methods' are not viable and you're not going to be able to stop the bad guys.


Sub poena is basically a rubber stamp after filling out a form. Often done in secret with the barest of oversight. A warrant requires a bit more justification at least.


> Sub poena is basically a rubber stamp after filling out a form.

Do you have a source for that?


Oldie but a goodie, and I doubt the numbers have improved - but essentially they are never denied for FISA at least. Which is basically the definition of rubber stamping no? [https://www.google.com/amp/s/www.zdnet.com/google-amp/articl...]


The obvious counter-argument, unfortunately, is that the government has a clear idea of which requests the FISA court would reject, and therefore never bothers making such requests. That would mean the court is correctly preventing executive overreach and there's nothing to worry about.


Even agents randomly screwing up would produce higher reject rates - 99%+ approval rates practically means no one is even looking no?


Have you heard about parallel construction [https://en.m.wikipedia.org/wiki/Parallel_construction] and AT&T’s room 641a? [https://en.m.wikipedia.org/wiki/Room_641A]

Those are the tip of the iceberg.

If there is an automated way to get this data, the agencies will do it, and then retroactively generate the paperwork to make it look ok in court if they feel it’s worth their time.

If structurally there is no privacy except some friction with generating paperwork, then we already have solid evidence here in the US in modern times that you defacto have no privacy.


This is only true if the companies you're asking for data refuse to provide it _without_ a subpoena. Many companies (let's us AT&T as an example) will provide law enforcement whatever data they ask for without requiring a subpoena.


Bad example: regulate telecommunications companies are required by statute to provide certain information on request. A better example is Amazon with their law enforcement partnerships through Ring.


I assume parent was probably referring more to the subpoena- / warrant- less "creative" solutions that have been discovered, than the typical exhaustion process.


50 years ago police would obtain a suspect's phone records. This has been part of criminal investigation for as long as phones have existed.

If you have someone being blackmailed or defrauded online, what "field work" are you suggesting here?

> "cops [...] want to solve all the problems from their desk."

This literally couldn't be more wrong.


So you want cost the taxpayer significantly with potentially months of unneeded work and expose cops to potentially more danger to ultimately arrive at the same result? How exactly is this better?


> to ultimately arrive at the same result

[citation needed]

Further:

Wiretapping is illegal without a warrant. I believe the spirit of the law there implied that wiretapping of [previous, historical conversations] was _always_ illegal, since a wiretap could only be tracking future conversations by its very nature.

The nature of communication has changed, such that all conversations theoretically have a permanent, historical record, despite the intention of those conversations to not have that historical record. It's called "instant messaging", after all, not "perpetual letter writing". It's meant to be an analogue to talking directly with one another.

The path we've gone down where everyone uses a third party to communicate with each other, and that that third party could theoretically record and retain all communications back and forth in perpetuity does not change the _intent_ of the laws as they were written.

The laws were to protect everyone from unreasonable review of their historical actions.

Perhaps you remember that story - I've completely forgotten the source and am having trouble finding it - about the person taken in the night and thrown in front of a judge. He asked what his crimes were, and the judge said "that's what we're here to find out", as they were going to go through everything he'd ever done to find something to charge him with.

edit: another instance would be Lavrentiy Beria, a police chief under Stalin (https://www.oxfordeagle.com/2018/05/09/show-me-the-man-and-i...)

"Show me the man, I'll show you the crime."


>"Show me the man, I'll show you the crime."

That's nothing new, either.

"If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him." -- Attributed (possibly apocryphal) to Cardinal Richelieu (1585-1642).


Ironically, under totalitarian regimes, citizens find themselves faced with the choice between either following the unwritten rules of the regime (which require them to break the laws as written) or following the written laws (which are an impediment to the regime, and thus cause one to be guilty of more nebulous crimes like "sedition").

I don't disagree that overly broad laws are a problem in non-totalitarian countries, but I think that regimes like Stalin's are a special case.


>So you want cost the taxpayer significantly with potentially months of unneeded work and expose cops to potentially more danger to ultimately arrive at the same result? How exactly is this better?

Because my privacy and that of most others who are decent, law-abiding citizens is more important than not making police do their jobs.

How do you think police caught people before apps like Signal? With real police work. Perhaps if they had to spend more time doing that, they wouldn't have time to beat and kill as many unarmed civilians.


If the alternative is a mass surveillance state (which we are sliding to) and the end of democracy, yes. Yes it's way better.


Let go? Sued.


- Your privacy is being invaded. Look at snowden, PRISM, facebook!

- Don't have anything to hide.

- At least use Firefox.

- Nah.

Sorry but no, it's definitly apathy.


bandwith


> "bandwith"

Leave it to HackerNews to downvote the only sensible answer in this thread, rooted in a simple technical understanding of how distributed caching works.


It's probably not downvoted because it's wrong, but it's because it doesn't help anyone who don't have "simple technical understanding of how distributed caching works."


Down With the Bandness


Rate limiting for all would be fair. You can torrent the whole db anyhow.


Can you give us more details about this ? Do you have a particular story in mind ? It seems a lot of fresh blood is comming to the python project recently, so I'm interested in what you have to say.

But why are you using a throwaway account ? And that you that have been answering the other comments with also newly created accounts ?


I can share the sentiment that PSF and the core team has become something like “old boys club”, and no, new core developers didn’t change this ambience you get when you read their forums/mailing lists and twitters (better avoid twitters though).

I don’t have anything about a community being an old boys’ club (would be nice to be in one some day, I reckon it’s what older people’s equivalent of a “safe space” is), it’s that they try hard to pretend they are something else that irks me.

But for the life of me I couldn’t find “python” and “minibosses” mentioned in the same document on the internet, except for an integration testing package and a computer game. So I think the story alluded to is bogus.

There’s a minuscule likelihood of it all being a twitter drama with key characters then nuking their tweets or deleting their accounts, but if it was of any significance some blog would have echoed it.


The only reference I can find to "minibosses" is this ancient python-dev thread here (https://mail.python.org/pipermail/python-dev/2001-July/01571...)

> I'd prefer if you didn't feel the need to play miniboss on the Python project

Disclosure: I am a current core dev


I find it harsh to suggest that the names in that statement didn't do real work. They've been at it for decades. And the newcommer report is outstanding in quality.

I'm rather interested in why you feel that way however. Do you think your work, or the work of some specific project/person hasn't received the recognition it deserves ? Or that the attention of the community diverted from important work toward subjects that are more eye catching ?


Yaml has so many problems. Python 3.10 raised a new one to my attention when the core devs realized their arrays of versions contained twice 3.1 and no 3.10. Indeed, if write unquotted ascii, yaml gives you strings. Except if it can cast it to a number that is.

TOML is better, but it still has more gotchas that necessary. So much I find it easier to just edit a python file

I'm thinking of giving a try to cue. Any feedback ?


Yes, they also probably have also an AI collecting what makes your activity a unique profile even if you are not logged in. Thanks to google search, map, analytics, android, dns, amp, google fonts and the like, you almost always load something from a google server if you browse the web.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: