There are papers and Postgres projects that attempt this kind of learning-based optimization, with some success. None are in widespread use. (One part, but certainly not the entirety, of the problem is that it's not just A/B, it's an exponential number of options that all could seem close to each other.)
This isn't just theory either, for example: Revolut is a bank that does all its event persistence and streaming on top of postgres. No traditional message queues/brokers in their stack.
If you start here, with the "Postgres will take you wherever you need to go" meme, without thinking extremely deeply about your schema and how you expect to evolve it in the future, you can easily paint yourself into a very difficult and expensive corner.
It's easy to use Postgres poorly in ways that result in painful centralized bottlenecks.
(Obviously this is largely true for anything, but I think that in 2026, where there's also a lot of more-specialized/less-fleible but much-easier-to-scale well-supported mature alternatives, you should be VERY wary of making everything have a single central SPOF. What are your users going to expect in terms of maintenance windows, etc.)
I'd be cautious with articles that say things like "All cloud providers allow you to run (and scale!) PostgreSQL by clicking a single button." with no mention of how long that will take and what options should be set to make it faster, or the costs of those things.
A counter anecdata. We transitioned from a postgres job queue to Rabbit. We had never ending problems after that, many of them were misunderstandings, some where just wrong-fit. We migrated because we had some time on our hands and thought we would alleviate some high pressure jobs. Not only did it not solve the problem, but having written all the code that decides when to pull the next message and what to do with it, and how to dead-letter it - just worked great for us on Postgres. It was so easy to understand and doing things like reprocessing just using a standard postgres DB interface was much easier.
Ultimate the entire processing got removed from our team and no longer needs to do these deployments (acquisition transitions)...
I tried very hard to use postgres as a queue, it was robust but slow once I started to push from more than a few processes/servers. Moving to zeromq initially and sqs after solved all my perf issues, and was still solid.
I think that’s the way to go. Start with Postgres and only if there are problems, then think about something more specialized. Same for microservices. Start simple and introduce a service when really needed.
I hate it when people already start out with 10 or more different systems/services for a few messages per second.
from my experience they're actually much lower than people think, especially when using Postgres for things it's not well-suited for like queuing, and the problem is you rarely can just throw more hardware at it to solve the problem and it's really not a good place to be.
I think that using the right tool for the job is important and saves a lot of time in the long run. There are expensive headaches that we have to resolve.
Having done that, e.g. used rabbitmq plus postgres, honestly I wish I had just used postgresql for both messages and data. It would have been easier to manage by an order of magnitude, especially at scale and needing to satisfy enterprise requirements. Also the flexibility of postgres would have solved problems that we ran into because of limitations of rabbitmq.
Without knowing any specifics of your uses, my usual starting point on that sort of design is that "messages AND data" is it's own special little way of ending up with a hard-to-debug-and-operate system. ;)
It's very hard to best-of-both worlds event-driven system + RDBMS-storage, it's very easy to end up with worst-of-both-worlds. Hello distributed transactions!
Again, you just should think about all the ways you want to use it and the maintenance/uptime requirements your users are going to have in advance.
I think messages + database are extremely common in any sort of large application where you have data processing nodes. Postgresql actually has very good mechanisms to support message style communication, and as long as you design your message tables independently you shouldn't have horrid issues around locking and transactions. Message queues don't save you from thinking about that anyways, they just replace transactions with acknowledgements.
Trying to manage a highly available and durable rabbitmq or other message system that can also be recovered from backup to an offsite mirror infrastructure in the worst case is actually incredibly difficult. Usually these systems are designed with the assumption that you can just regenerate messages based on database state anyways in worst case scenarios.
In this use case your database already is highly available and can recover on an offsite backup if you have suitable wall shipping going on. So you've done all the hard work once, may as well reuse it unless you truly have some mind bogglingly large message throughput needs.
Finally, we had a need of a queue that was more than just first in first out. We wanted to fairly balance workloads across users and tenants. Whenever you have such a need postgresql lets you design this type of queue far easier than trying to do some elaborate multi-queue setup with a traditional queue.
It doesn't take very long (because compute and storage are separate in most of them) but good lord does it get expensive. Every time you click that upgrade button you are doubling your cost. It's really painful when you have a spiky workload that is performing fine like 95% of the time but you are watching the p99 and need to double the cost of a very expensive infra component, only to improve the experience of the heaviest 4% of your workload. This is to say nothing of the gambit you then have to play with reservations/prepays.
I haven't seen a way to get guarantees of upscaling operations under like 30 seconds (with Multi-AZ RDS) with well-supported RDS stuff (leaving out active-active setups with logical replication because that's a whole other can of worms).
If you know you're gonna be ok with that for a long time, go nuts. I'm just saying: think about it in advance!
The cost pain for spikes is also a thing - some of Aurora's billing models look potentially promising but I haven't used them in practice - though it's also somethings that's harder to avoid with alternatives. Distributed DBs aren't generally super friendly to dynamic scaling IME.
Sometimes to scale is "continue to satisfy SLAs as service usage increases" and other times to scale is "successfully evolve functional capabilities over time."
While the GP may have been referencing the former, embracing "PostgreSQL for Everything" often prohibits the latter.
It *can* do that. I wouldn't say it's necessarily "often". Again, it's hard to predict the future.
I think that the "use Postgres for everything" messaging was a necessity, even if it is overstated. Use it until you can demonstrate it doesn't meet your near term needs. When that happens, shift. It wasn't that many years ago when I'd enter situations where people were knee deep in FAANG level infrastructure when postgres on a relatively small instance would have more than been sufficient. I'd suggest they look at converting to postgres to save money & all the energy they spend maintaining their soup. "It won't scale the way we need it!". Sometimes they were demonstrably wrong. Other times they were half-right, in that the real problems was terrible decisions made at the software layer, leading to a situation that required heavier duty infra. Almost never were they actually right* though.
Might they have been right 5 years later? Perhaps. But I know for a fact that none of the ones I encountered were.
As the old saying goes; just because you can do something doesn't mean you should do it.
> I think that the "use Postgres for everything" messaging was a necessity, even if it is overstated. Use it until you can demonstrate it doesn't meet your near term needs. When that happens, shift.
The problem with this approach is, once "use PostgreSQL for everything" is identified as being no longer be feasible, it has already become an inextricable component underpinning system functionality. Thus making "[w]hen that happens, shift" extremely difficult.
Contrast the above with only using PostgreSQL (or any other RDBMS) to manage data and their relationships, eschewing stored procedures as well, and the "when <insert condition> happens, shift" decision becomes much more feasible to entertain.
Eh, the "it's impossible to change later!" line is the classic one for any of these types of conversations. My experience over the decades has been that while yes, this can happen, it's grossly outweighed by YAGNI-in-retrospect and the eventual shift turning out to not be nearly as painful as people think once all of the people & bureaucratic problems are out of the way.
I once needed to maintain an application written in everything Oracle. If I ever encounter the original author of that product: I have things to say to him.
We quickly replaced part by part by easier, less costly parts.
Software development is not just writing code; I think all HN users know that.
> If you start here, with the "Postgres will take you wherever you need to go" meme, without thinking extremely deeply about your schema and how you expect to evolve it in the future, you can easily paint yourself into a very difficult and expensive corner.
Yeah, backwards compatibility is not a thing for Java, Rust, C++, etc. :eye-roll:
Meanwhile in SQL if you need to make a backwards-incompatible change to your schema you can always use VIEWs and INSTEAD OF triggers to maintain backwards compatibility for code you've not fixed yet.
As SRE dealing with this at current company, a benefit of using well known software like Kafka is a lot of problems you will run into have solutions/guidance already available vs you having to explore solutions which a lot of time end with “Kafka could easily do this. “
There are two sizes of companies: those that can afford '1+ dedicated ____-person' and those that can't. Which should filter through to technology choices more than it does.
Or, you can do like one of my former bosses, and just rattle off a list of 60+ major projects which would require a team of 10 to make any reasonable progress on in the near future, then pin it on one underqualified person, refuse to provide a proper budget, and continually press them about why targeted deadlines are being missed.
You would think so! Perhaps in a more sane situation. We tried it and it did not help. Priorities shifted, facts on the ground changed, emergencies came up, and there was almost no chance to focus for long periods.
Part of it was me not handling certain kinds of stress well, but a different management approach absolutely could have made a difference. To cut them a bit of slack, this particular boss had come from a prior position where they were managing hundreds of roles... to a small shop where they managed just a handful (and only one other dedicated IT person). They had trouble breaking away from the "throw everything at the wall and go full steam on it all" approach and dealing with scarce resources more carefully.
One specialist and a group of generalists is often enough. Especially if you are allowed to contact the specialist outside normal working hours in rare emergencies.
This problem doesn't go away with postgres. It's totally anecdotal but this is one thing that I've noticed different in mysql shops and postgres shops - with mysql there is usually at least one person on staff who knows MySQL DBA and scaling pretty well, with postgres it's rarely the case to have someone who knows the internals well - like you said, the person capable of maintaining it when it goes wrong.
You could argue it's because postgres requires less poking though I would say you don't need the DBA for when things go right.
Of course most people are just handing the management off to the cloud and that's potentially why, but it doesn't cover everything
MySQL will generally run fairly well with default tuning, assuming you've sized the buffer pool well relative to the amount of RAM you have (cloud providers do this automatically, but it's also not that hard to calculate). There are some knobs you can turn to eke out more performance in certain situations, and there are some defaults that are truly terrible (lock_wait_timeout is set to 1 year...), but all in all, it doesn't take a lot of care and feeding to run reasonably well.
Postgres, on the other hand, has a million knobs, many of them interact, you'll find conflicting advice for some of them, and it can rapidly fall over if you aren't keeping a close eye on long-running transactions. It's also more performant than MySQL in _most_ situations (hello, clustered index), if you've tuned it correctly. It also of course has far more extensibility out of the box, with tons of index types that are extremely helpful, if you know how and when to use them.
This difference is why I'm always frustrated when people parrot "just use Postgres" as though that solves all problems. It's an extremely powerful tool that can replace most of your stack, yes, but it also would really, really like you to RTFM. Not random Medium blog posts, the canonical documentation.
I mean, with this custom thing, you have that question as well with downside is you cannot pick up the knowledge from off the street.
I became the Kafka guy at my current company, it took me about a week of reading and every time I had further question, I didn't have to bother anyone, I could Google and get data I needed.
When it's some NIH thing, you have to bother coworkers and knowledge is whatever is in YOUR company knowledge base with no ability to get knowledge from outside the company.
EDIT: You could also leverage contractors or outside support if not homegrown software.
Yup, and every time it breaks, you've got to pester someone whose job probably isn't maintaining that thing actively.
I worked at a startup with massive NIH syndrome, once. We even used our own in-house programming language, because it was "better than anything else out there on the market." It did have a lot of nifty features that others don't have: a pretty novel type system, programmatic macros, a built-in build system and other fun bells and whistles -- but also not-so-fun ones like having no syntax highlighter, LSP, or debugger, and having to constantly shuffle around your code to avoid ICEs in the compiler.
The compiler wasn't the product, but we found ourselves fighting that thing more actively than any of the real problems our custom programming language was supposed to solve. The CTO found himself spending all his nights and weekends mostly trying to get the compiler to not explode.
A few years later, after I had long left (for that reason, among many) I heard they switched to Python. Can't imagine how long it took them to get that all rewritten.
> e even used our own in-house programming language, because it was "better than anything else out there on the market." It did have a lot of nifty features that others don't have: a pretty novel type system, programmatic macros, a built-in build system and other fun bells and whistles
A DSL can work, but not for the features you list. Those features you already get from existing languages anyway!
If you need general programming language features like excellent type system, programmatic macros, a build system (doesn't need to be built into the language), etc... then use a general purpose programming language.
I have a DSL for backend/endpoints, and exactly none of those are in my feature list. What it has are things like easy way to specify access-control directives[1], the SQL query to execute, mapping request variables to SQL parameters, mapping SQL results-sets to response fields, etc.
I have another DSL for a test program. Both of those DSLs have specs that's literally 2x screens of bullet points and examples. LLMs can output those DSL programs because the spec for the DSL is so small.
For general purpose programming stuff (while loops, conditionals, etc) my DSLs break out to Python.
A good indicator that you shouldn't be creating a new language for production is when you find yourself implementing conditionals, loops, etc.
===========================
[1] Limit endpoint to specific roles, or members of the same team, or both, or even to the user itself - someone calling `/user/profile/update` should only be allowed if the profile they are updating is theirs, for example.
Ah, but you see, it was a general purpose programming language. A general purpose, functional, optionally-typed programming language, with its own optimizing compiler and build system. In fairness, it wasn't originally developed to be our in-house language, but it was the creation of the CTO.
It was fun while it lasted and I had a lot of fun working on it. But it was really not a good business fit. The programmatic macro system was supposed to allow us to build customer-facing DSLs on top of it, but everybody just wanted Python anyways.
During pgConf.eu in 2016(-ish, could have been one or two years later; I don't remember too well), a representative of payment processor Adyen told the audience that they were, essentially, one big postgres cluster in their backend, too ("cluster" used as per the postgres-native meaning of the term, as in, an installation on a single host with a data directory containing any number of databases).
I wonder if there's a more graph oriented score that could work well here - something pagerank ish so that a repo scores better if it has issues reported by users who themselves have a good score. So it's at least a little resilient to crude manipulation attempts
It would be more resilient indeed, I think. Definitely needs a way to figure out which users should have a good score, though - otherwise it's just shifting the problem somewhat. Perhaps it could be done with a reputation type of approach, where the initial reputation would be driven by a pool of "trusted" open source contributors from some major projects.
That said, I believe the core problem is that GitHub belongs to Microsoft, and so it will still go more towards operating like a social network than not - i.e. engagement matters. It will still take a good will to get rid of Social Network Disease at scale.
GitHub has all kinds of private internal metrics that could update the system to show a much higher signal/quality score. A score that is impervious to manipulation. And extremely well correlated with actual quality and popularity and value, not noise.
Two projects could look exactly the same from visible metrics, and one is complete shell and the other a great project.
But they choose not to publish it.
And those same private signals more effectively spot the signal-rich stargazers than PageRank.
It looks like they're pivoting into corporate finance tracking which seems like a good move but still a bit jarring as I was curious to give this a go!
Existing customers - what comms have you received?
I don't really want to self-host this even though I might. Before I commit, what are some alternatives? The UI seemed very compelling
Selegiline is also known as Emsam, the patches reportedly Sam Bankman-Fried was on, alongside ADHD meds. I recall reading how this may have been a large driver of his compulsive risk it all decision making that eventually brought it all down.
Delivery through transdermal patch (Emsam) is different than other presentations of Selegiline, and we don't know the doses he was on. I've heard a low dose of Emsam (6mg) gives a better, more stable dopamine baseline, without getting into MAO-A inhibition that happens at upper doses.
> I've heard a low dose of Emsam (6mg) gives a better, more stable dopamine baseline, without getting into MAO-A inhibition that happens at upper doses.
This is the broscience theory that you find on Internet forums, but it didn’t play out positively in the actual clinical trials. Getting into full MAOI inhibition was necessary for positive effects.
The “more dopamine” theory appeals to people who treat the brain as a simple machine where you’re adjusting levels of different chemicals, but doesn’t really work out in practice. Dopamine is the neurotransmitter du jour in these circles, but mood and mental health are more than one chemical.
Thanks for the info, I was missing an update on this space. If there's no solid, reproducible evidence of cognitive or energy benefits at this level -and full MAO‑A engagement seems necessary for those effects, then that doesn't look good.
About your second paragraph, I agree and I wasn't validating the bro part, just that I read about Emsam once. I guess it's mostly influenced by the popularity of performance enhancers in general, be it wakefulness enhancers, stimulants, etc. People are always trying to find "the edge", without the tradeoff, basically chasing a mirage.
Maybe a neglected topic in this theme has been the acetylcholine pathway. Cobenfy, for example, has been approved for treating negative symptoms of schizophrenia and it's not a classical drug in that space.
Try telling all this to someone who is effectively emotionally/physically disabled because their mind and body either enter a rollercoaster or just shutdown in response to anything vaguely reminiscent of something they experienced.
I think these treatments are better understood as methods to encourage one's mind and body to reprocess some experience so that it is "merely" a terrible thing that happened so they can live a stabler, more balanced, more normal life.
The tribe that you speak of is much much bigger than those who have an official diagnosis of PTSD via psychology. For example, addicts all have PTSD that is commonly related to past childhood/family abuse dynamics. They take drugs over it. So, as a matter of fact, that is exactly what is told to them. You can in fact hold your pain, and I believe it is righteous to do so.
It kind of reminds me of Eternal Sunshine of Spotless Mind. You cannot just erase something. Drugs effectively erase the bad feelings, but they don't last. Okay, so such a treatment being discussed is, I suppose, an everlasting solution. Why would I be against this? Because you cease being you in your entirety (which includes your trauma).
It's not an easy sell by any measure, especially for those who just go "well fuck that, I can erase this with some cocaine right now". Erasing it is a solution, for sure, but it's not a free solution. Something is lost in the process (you). Understanding that, stay with me here, that it's beautiful is part of it, that you have a beautiful part of you forever. It's completely didactic, you have to teach the person that they went through something and are PRETTIER in their surviving of it. Then it stops being traumatic.
If we zap them, we take away the pretty. If I zap you, and then say lets talk about it, and you say "well I don't feel any which way about it anymore", then it's lost. It's gone. So something is lost in the process. So yeah, this is exactly what is told to the most beaten down people in life.
There are "better" databases but they're better given some particular definition that may not be relevant to your needs. If SQL/the relational model and ACID semantics is what you need then postgres is simply the best in class. The fact it dates back to the 80s is probably an advantage (requirement?) when it comes to solving a problem really well
The Wikipedia page on this is sufficient. If F:X -> Y is a function between normed linear spaces then DF:X -> L(X,Y), where L(X,Y) is the vector space of linear operators from X to Y, satisfies F(x + h) = F(x) + DF(x)h + o(h). A function is differentiable if it can be locally approximated by a linear operator.
Some confusion arises from the difference between f:R -> R and f':R -> R. It's Fréchet derivative is Df:R -> L(R,R) where Df(x)h = f'(x)h. Row vectors and column vectors a just a clumsy way of thinking about this.
My best guess is a lack of willing participants combined with pressure from up top to move money to keep the operation going/meet targets.
We also don't know how much of that 500k is profit - maybe the margins in organised crime aren't what they used to be. Each $3,750 saved is money they can put straight into their pocket as their boss might never find out how many mules they ended up using.
I think it’s pretty simple. To many people, $3,750 is a huge amount of money. It’s easily several months salary.
I think a lot of people would be relatively unaware of how illegal a given thing is and would instead see the amount of money being offered as representative of the risk. Too much and they would get suspicious.
Even at effective minimum wage of 11.80 an hour[1], that’s only 2 months of salary. That’s not life altering in any way and wouldn’t be worth prison time to people in general. People have to have been tricked about the risks or were coerced.
People on minimum wage don’t get a chance to build savings are probably living month to month. Though it’s not an earth shattering of money it is meaningful to a lot of people.
I remember some very sad news about a drug smuggler, a poor woman from the Philippines, condemned to an insanely harsh sentence in Singapore (perhaps even death penalty?). At least, the UK justice system should be less cruel.
reply