In 2015, I was able to get to 1 million read/write queries per second on only a couple nodes and tested this with multiple databases, it required (at the time) decent network tuning and node placement inside AWS but it cost me about 10 to 15 dollars per run if I recall correctly, obviously there is the matter of scaling such performance and so I want to recognize the engineering effort gone into this but this is too much money. This reminds of when one of my teams used Hadoop to process only a a few terabytes of offline data and were able to process the WHOLE THING in only a few hours. I did not have the heart or courage to tell them during the demo that this was overkill, but I did write a very simple (and small) piece of code that could extract all the signals from the offline files in mere seconds with careful network planning and storage optimization and invited them for a demo/lunch and learn next week.
> all the signals from the offline files in mere seconds
That sounds unlikely unless those machines had access to crazy disk I/O. A local RAID 5/10 with 8 drives would still take 30-50 minutes just to read that much data. Even with a mid-range SAN you still would have spent 15-25 minutes just reading data. This assumes 7.2K SAS/SATA since SSD/NVMe were not ubiquitous in 2015, but even with 2015-era SSDs you're still looking at half that much time spent reading.
Lots of EBS volumes mounted via 25GBPs or higher network, positioned carefully onto a single rack where possible and carefully tuned network/ip/os for both clients and servers. It is not feasible in most deployments as this would not have scaled to real production loads (RAID configurations requiring redundancy alone would slow you down, not to mention costs of using that many EBS volumes on extra high network/IO/provisioned IOPS nodes). This was only meant to prove what was possible in AWS at the time.
Your statement about SSD availability and performance does not fit with my personal recollections of history. Smoking fast SSDs were completely vanilla items you could order from anyone in 2015. For example, the Intel DC P35/6/700 series. Each of these was as good as 25 HDDs for linear reads and more like 5000 HDDs for random reads. The U.2 specification for putting SSDs in boxes was formalized in 2011. Outside of the mass market, cloud builders had private SSD designs long before 2015.
Less than 5% of commercial storage capacity was SSD of any flavor in 2015 (NVMe probably less than 1%), and the price-per-GB was anywhere from 5-15x. NVMe hardware RAID didn't happen until 2017, and AWS SSD via EBS was capped at 320 MB/s (~500 MB/s with striped EBS volumes). Before 2015 it was capped at 160 MB/s.
In practice even NVMe SSDs at the time were maybe 10x faster for sequential R/W and maybe only 200-500x for random I/O. Most databases in 2015 were limited by serial dependency chains like walking a B-tree to figure out what nodes to read next and couldn't max out what an NVMe was capable of. Not to mention we were stuck with libaio in 2015 that made O_DIRECT ops synchronous.
So the OPs claim of reading multiple terabytes of data in seconds on AWS in 2015 literally wasn't technically possible.
SSDs simply were not as ubiquitous in 2015 as they are today, and I didn't know anyone using them outside of mission critical systems were the cost made sense.
That's a good question, I was very high up in the leadership, couple levels below the CIOs and the biggest surprise for the team was that I could write even a single line of code. In large organizations, changing direction is not always easy so I could not get them off of using a data platform, which to be honest was acceptable because we could not ask the team to write such custom code for each job going forward, I did manage to shift them towards flink which I think was better suited to our actual real-time needs.
I see a few comments talking about the pain cassandra inflicts on ops, and that's fine, I agree with almost all of them. What the comments assume readers to know and understand is the absolute horror of foot guns, backup nightmares, data loss scenarios with seemingly safe choices around quorum and of course the performance purgatory lined with tombstones. Honestly, your workload is not big enough for postgres, trust me. It is like trying to do word count on a 5 TB file with flink, can you? should you though?
Yep, the pain is real. We don’t mention sstableloader in polite company. Add in kubernetes and getting your rf/racdc.properties
right in a 3 az cluster so everything doesn’t just die on a netsplit with local_quorum (harder than it sounds) the joys of the ‘repair’ cronjob, backups, and commitlogs only occasionally not corrupting to the point the 8th out of the 15 nodes in your kube sts can’t restart after a bad exit() (e.g during worker upgrades —- good luck with this one!) …. yeah. hate.
Postgres isn’t the answer for my workload tho we really tried, i even wrote custom sharding for our postgres and it couldn’t handle the writes the cassandra setup shrugs off, but for sure this isn’t a ‘normal’ requirement most applications have.
chatgpt just refused to tell me the first verse from a poem when I asked it by telling it the second verse and to remind me the first one, it complained about not being able to violate copyright laws! poetry by a dead poet is not something it could narrate, something a quick search on the internet returned immediately. I sound like an old man shouting at I don't even know what but come on!! Things are going bad and there is nobody in the drivers seat, speaking of which Tesla FSD has started driving like an actual drunk, moving the steering wheel right then left, then right then left on perfectly straight roads, making me dizzy as the driver, why? because neural network? what is happening with LLMs and AI feels like a very bad platuae of human existence.
This team does not report to me, I will ensure their demise and make sure their work is never adapted by anyone within my sphere of control. It is easy to justify such behavior behind snazzy terms but I have seen this so many times that it isn't funny. Sometimes leadership may make a decision you may not agree with or even understand but focusing on why it happened, what you can do to align yourself and how you can help product, customer and business succeed are more important than your walled garden of carefully controlled conway conventions. It is right there in your own terminology of tribes, how very tribal.
What exactly should the author have done differently? It's part of the leadership roles to understand the power structures within your organization. Reading between the lines, a new team was thrusted onto an arguably functioning sub-org to address concerns that they had not themselves raised. Then the expectation was for that sub-org to take a hit on their KPIs to onboard to the new teams platform.
It's not "tribal" to refuse to do something that is misaligned with all your explicit incentives. Otherwise we'd have to pay lip service to every internal tooling team just because they exist. It's the leadership team's job to keep pushing if they strongly believe the sub-org leader is acting in bad faith.
The best managers I've seen would turn this situation into a headcount request.
The problem is leadership has priorities 1-5. Your team works on 1-3, but the PM keeps getting hassled about 4 and 5, so they look for levers to get them to happen.
In this situation, the PM scrounged up headcount from elsewhere, but if you present the option of adding headcount to the existing team, then you create a more harmonious option of getting these lower priorities accomplished.
Of course, this guy was taken fully by surprise by the suggestion. It's much harder to present a better option after the fact, and I agree that letting leadership feel the consequences of its decisions is a reasonable thing to do in this case.
> What exactly should the author have done differently?
Work with the other engineering manager.
This person was angry about the team's appearance, but the other engineering manager who assembled the team is barely a side note in the document.
The way you deal with these situations is by working with the other managers involved and propose better solutions.
This is a weird blog post because it's supposedly from someone in a high up engineering manager position, but it's written with a political awareness that I'd expect from someone who had never managed before or who was a first-time manager without good mentorship.
They had better solution and used it. You cant "work with" people who are not working with you in the first place.
This was not a cooperative situation. Trying to cooperate in this situation will make you walked all over and also will make you the one to be blamed when it fails.
What about expecting them to suck up their pride, work with the other lead for a comprehensive plan that includes possible WONTFIX-es?
Strategy and leadership don’t come to exist on their own. It’s middle management that has the best operational and tactical view bear none. Use that to influence decision making instead of complaining. (Yes, this is a theme in my professional life. Our middle managers don’t know their own worth. Pretty please give me Plans about what you Want to Deliver. Those are so much better than general strategies.)
sports and competitive gaming can really highlight team attitude with this sort of thing.
if you are playing as a team, and someone on the team makes a committed decision then it no longer matters if it was the best decision or not given the info at hand - the team is committed. Everyone making an effort to make the best of ANY plan usually has way better results than a confused/sloppy execution of the best plan.
and with longer term plans, once everyone is moving together and following each others lead, you can quickly pivot into the right plan if something is very obviously wrong.
I have wanted to hold back from answering comments that ask for proof of real work/productivity gains because everyone works differently, has different skill levels and frankly not everyone is working on world changing stuff. I really liked a comment someone made a few of these posts ago, these models are amazing! amazing! if you don't actually need them, but if you actually do need them, you are going to find yourself in a world of hurt. I cannot agree more, I (believe) I am a good software engineer, I have developed some interesting pieces of software over the decades and usually when I got passionate about a project, I could do really interesting things within weeks, sometimes months. I will say this, I am working on some really cool stuff, stuff I cannot tell you about, or else. And my velocity is for what used to take months is days and hours for what used to take weeks. I still review everything, I understand all the gotchas of distributed systems, performance, latency/throughput, C, java, SQL, data and infra costs, I get all of it so I am able to catch these mofos when they are about to stab me in the back but man! my productivity is through the roof. And I am loving it. Just so I can avoid saying I cannot tell you I am working on, I will start something that I can share soon (as soon as decades of pent up work is done, its probably less than a few months away!). Take it with a grain of salt, and know this, these things are not your friends, they WILL stab you in the back when you least expect them, cut a corner, take a short cut, so you have to be the PHB (dilbert reference!) with actual experience to catch them slacking. Good luck.
I am good at software. It turns out that isn’t sufficient, or alternatively stated, you have to be good at a number of other things than just churning code, even good code. So to me, the combination of being good at software, understanding complexity and ability articulate it concisely and precisely, when combined with the latest and greatest LLMs, is magic. I know people want to examples of success, I wish I could share what we are working on, but it is unbelievable how much more productive our team is, and I promise, we are solving novel problems, some that have not been tackled yet, at least not in any meaningful way. And I am having time of my life doing what I love, coding. This is not to downplay folks who are having a hard time with LLMs or agents, I think, it’s a skill that you can learn, if you are already good at software and the adjacencies.
> This is not to downplay folks who are having a hard time with LLMs or agents, I think, it’s a skill that you can learn, if you are already good at software and the adjacencies.
Someone on the page already quoted Dijkstra, recommend reading that, he was correct.
Prompt engineering isn't engineering at all, it's throwing words at a wall to see which ones stick then declaring success if the outcome looks at all recognizable. That isn't actually success.
I like it, a test so bad, it just might work! I think the trick is not the equal sign, trick is to keep it so simple and small that most qualified people will not try to short circuit it.
