Do you use any external dedicated host?

Hetzner.de

It's a 1 person part-time job. Most of the work goes into coordinating package updates with Ansible. I can include $1000 in the calculation if that makes you feel better ;)

Everything is deployed with Ansible, monitored with Monit. The servers have redundant PSUs and SATA hot-swappable HDDs, so you can fix minor hardware issues without having to reboot. As the result, we have more than a year of uptime on a typical server.

On the other side, we've been hit with several S3 outages. My feeling is that the cloud needs fully automatic fail-over because it is failing much more often than all our bare metal server combined.

If I could have engineers for 1000$ who are able to build and operate infra better than AWS then Jeff Bezos fortune would look pale in comparison.

An iron rule of these discussions is that when someone objects to non-cloud solutions on the grounds of staff costs, they have not actually calculated the staff costs.

I know currently our problem is finding proper staff and that is why we are considering moving to a cloud provider. Especially for database hosting, using a database as a service becomes very attractive from a staffing perspective just because we can't seem to find good candidates and we've been training younger people into being DBAs but they ultimately want to do something else after a while.

We use Amazon RDS. It's still 10x the price of bare metal, but that is acceptable and it's fast enough for most things. Plus they have automated backups and certified encryption.

RDS and the like still need about the same amount of DBA'ing, no?

I'm assuming it takes care of monitoring and managing space, disks, backups etc. I mean the true administration tasks, not helping with designing the database and queries.

As I said, we are considering but the analysis hasn't been done yet. If you have insight, please.

No.

I'd agree...

It's also for speed to market... Our Infra setup where I work is extremely slow and at times takes over 1 month to setup a database.... and a VM about the same.

Sure, because EC2 instances don't need any admin work. Unless you're doing serverless architectures, you'll still need sysadmins.

Engineers cost $130,000/month? Thats a fulltime salary for 14 network engineers (salary from Indeed).

There is no way you need that many for maintaining a fleet of 50 CPUs (chance is that in OPs case they got duel socket servers as well).

On the other hand, you also need to pay engineers, and rather senior ones, to monitor and upgrade cloud solutions. They are just less visible: typically an extra duty for "devops" programmers and project managers rather than dedicated server farm staff.

$130,000 a month is not 14 network engineers. Salary is not the full cost of an employee to the business.

Even fully loaded it should cover at least 8-9 engineers.