We already have superintelligence though in some aspects like coding, AI is much smarter than any human because of its sheer speed and accuracy.
It can generate large amounts of essentially bug-free code. The only bugs are bugs in the spec.
It has essentially no common sense though so it often misinterprets the spec or leaves gaps and edge cases unhandled and that's where the human engineer is still essential.
The dangers of AI are much clearer today than they were a year ago. There is a certain precision and relentlessness which is going to be a problem. It's a matter of time before people unleash AI viruses which spread through computers and data centres and pursue various goals relentlessly with no off switch. Some of these viruses and their goals may be nefarious. Imagine an AI trying to disrupt a specific person's life relentlessly. Spreading from machine to machine, never stopping.
It just gets worse as your codebase grows with more code and more files especially. It's like a human developer who is under intense time pressure. If the codebase is large, it might miss an existing function and implement a second one... or it might miss the fact that a database table already exists to solve its current problem and create another one. The solution is to keep the codebase as simple and small as possible and name everything as intuitively as possible with proper separation of concerns. Same principles which were useful before AI are still useful with AI.
I don’t understand how it will ever overcome something like not replicating an existing function. It can’t possibly have full context on a massive codebase at all times so additional prompt runs checking for existing functionality would be necessary.
Even scarier is something like a legacy change made for a specific client, like “our biggest client needs checkboxes as the UI, do not switch it to a dropdown.” was in a commit message 2 years ago, AI will just overwrite that without thinking.
If your files are named well and codebase is of reasonable size, it will find them and re-use them. I haven't experienced the problem of duplicate functions at all on any of my personal projects. The trick is that my code is separated into distinct modules in a clear hierarchy with one-way dependency; no module has more than 5k lines total, no more than 20 files or so. Basically, I treat every part of my system as an independent, fully self-contained black-box product which could potentially be used by third-parties in different business domains. The interface of each module is generally quite simple and it doesn't expose any internal details.
My AI agent never has to work with more than 5k lines of code and like 20 files at a time; and most are much smaller than that; like 1k or 2k lines... Even though one of my projects is almost 30k lines, my AI agent never has to consider more than 5k.
It's a very different experience in my day job because the project has more lines, more files and all the files for the entire project are in the same repo and naming is not as clear. In this case, I have to point to the relevant files/context in every session.
There are solutions for this that those of us with the rare combination of experience, time and management buy-in are using - but they’re scary to people all-in on AI tools.
I’m talking about deterministic checkpoints and hooks. In every environment in which I need to use a coding agent I have had to essentially build a harness outside of the harness to be measurably productive. If you’re not using deterministic tools to box in what the agents are capable of doing, the problems you describe are ten times more rampant than anyone wants to admit.
Someone tried to hack my computer with a fake job interview once. I believe it was a real person because there wasn't much latency.
They tried to get me to git clone a repo, presumably to get me to install malicious npm dependencies.
I refused, then they wanted me to download Claude Code but I told them I already had Claude Code harness installed on my machine... But they said I should install a different version of it; they sent me a link... The landing page was the same kind of design as Anthropic uses but I didn't recognise the URL. I stopped the interview right there.
My main fear is that the integrity of everything will be compromised, but everyone will keep acting as if it's not. I feel like it's already somewhat the case but I'm worried about it getting worse.
My prediction is that some people will be able to use AI to create essentially infinite money inside their bank accounts (through hacking); the prices of assets will grow to ridiculous sums as the people with infinite money buy up assets, but yields on all assets would soon approach 0%; still, one asset is going to grow at a faster rate than all other assets (E.g. an AI company backed by blockchain/tokenized shares that are impossible to hack) and everyone is going to start piling their infinite money into that asset; every other asset will crash because of the rapid, exponential price acceleration of the top asset making every other asset unattractive by comparison. Then fiat money would cease to exist and shares (or crypto tokens) of the top company will be used as money instead. Then prices of assets would stabilize again; denominated in the new currency... Then after that, we will finally be able to start innovating with AI.
I also think so. The weakness of current popular cryptocurrencies though, is the complexity and size of their codebases. Many above 100k lines of code. That's risky. Though the benefit is that they are somewhat heterogeneous (different nodes running slightly different code/versions) so it may be hard to find a vulnerability which affects more than 50% of nodes; and this is a hard requirement for compromising the integrity of that system.
Traditional finance has the benefit that they can more easily reverse malicious transactions, but if hacks become too common, it will be impossible to correct fast enough and in a reliable way. With traditional banking, a hacker only needs to hack a few nodes to compromise the integrity of the entire system.
So yeah I also think crypto will win in the long run. I would bet on a tokenized stock with AI attached.
I think the top asset will combine the AI which is best at hacking with the blockchain which is hardest to hack.
Ability to corrupt others whilst maintaining perfect integrity yourself.
I enjoyed reading this article. I think it's an important article. Even though people might disagree about certain definitions like "maintainability," I liked what the author did by sticking to objective measurements which do correlate somewhat with the things we care about: Ability to implement the product fast, with few tokens and with as few lines of code as possible in the final product. These are well chosen metrics IMO.
Fewer lines of code (per piece of functionality) does, in fact, correlate strongly with maintainability. The downside of this metric to watch out for (as Goodhart's Law would have it) is to ensure that the framework isn't compromising (current or future) flexibility in an unacceptable way. I imagine if this kind of benchmark becomes popular, people would start coming up with low code frameworks which are extremely rigid and inflexible. Because rigidity is a good way to cut lines of code. There needs to be a way to ascertain the flexibility limits of the framework too.
Would be good to have a similar benchmark for a more complex application; one with authentication, access control (with individual ownership of records and group ownership).
Also would be good to compare the security of different frameworks in this way. See which frameworks leave security gaps. Would be good to set up a benchmark where an AI agent is made to impersonate different users (with different roles) and test access to all possible data types and views in an exhaustive way.
A while ago, I showed Claude a relatively complex application I had built with my framework https://saasufy.com/ and asked it to estimate how many lines of code and how long would be required to build the same application with a stack like React + Node.js + Postgres and it estimated that it would require 4 to 5 times the amount of code and amount of time so I'm keen to actually test it out with a benchmark like this.
I was not surprised that HTMX lead in a lot of the benchmarks. I suspect it would do even better with a more complex application. IMO, its design is closer in spirit to where the industry needs to go. There are a lot of maintainability and security gains to be had by standardizing frontend and backend interactions; though I believe there are other way to do this which can yield even more gains (though obviously with some tradeoffs depending on the use case).
This is true but its assessment did match my personal experience as a developer who has built quite a few apps with React + Node.js + Postgres combo. The project I had written with my framework was only about 1400 lines of code (mostly declarative HTML) and my experience is that applications of that complexity would normally approach or exceed 10k lines if well written with React/Node/Postgres. So its assessment even seemed conservative based on my own estimate.
I once also worked on a simpler application (feature-wise) for someone else which was approaching 100k lines but it was poorly architected with a lot of bad abstractions and over-engineered. In this case I could claim that my framework is cutting almost 99% of the code but that's comparing to the worst possible situation. Still, even that's not necessarily an unfair comparison because, in practice, a lot of real projects do end up bloated to such extent after a decade or so, likely even more so now with AI. Any framework which can prevent that bloat code from being generated is genuinely delivering long term benefits.
IMO, we'll need to rebuild all of crypto and all software from scratch using a minimalist approach. Minimize the amount of code, with proper modularization where each module has clear responsibilities and fully self-contained attack surface and exposes a well-defined set of features.
Yes but I agree with the previous comment that there is a bias towards maximizing complexity. My experience in this industry tells me that real dynamics tend to differ significantly from canned narratives such as "this tool can protect you from Y."
Sure, it could, if developers were perfectly rational entities and if hiring processes at big tech firms where optimal... But neither of these are even remotely true.
The psychological aspects transcend a tool's capabilities. A tool which aims to protect people may in fact cause more harm than it prevents.
Kind of like a gun; great tool for personal protection in the hands of decent people, but put enough of them in the hands of violent criminals and they become a net problem. Human psychology tends to subvert the intent of tools as well as that of laws and regulations, in my experience.
Except the internet has clear value, yet many remain unconvinced that cryptocurrency services any meaningful value to anyone but bad actors and oppressors. In fact, many would argue that it's a clear net negative and 5 billion USD going to North Korean oppressors would very much validate that.
That is a lot of tables for one schema. Maintaining so many tables seems like a security vulnerability in and of itself from risk PoV.
I think your prediction makes sense. There needs to be a lot of guardrails and you need a fully integrated platform/stack on the backend to protect the user from doing dangerous things because the AI ain't gonna tell them when something is dangerous or doesn't perform at the scale they need
But IMO, React didn't go as far as it could have; partly because it is backend-agnostic. You still have to re-implement a lot of the same wiring over and over and you keep running into the same issues and vulnerabilities. Also React offers too many ways to do the same thing; it just confuses the LLM as it mixes all different approaches, it lacks a clear philosophy to provide the guardrails you mentioned.
We should be able to just deal with pure UX issues/tradeoffs; we shouldn't be running into technical issues.
Frameworks matter, but they should be focused on minimizing the amount and complexity of code per feature.
The more feature-dense your code is, the less unnecessary hurdles and complexity, the more reliable your software is and the easier to extend and maintain. The more code can fit into the LLM's context window, the more complex your application can become whilst remaining reliable.
I built my own framework which is declarative HTML web components with a serverless backend (similar to Firebase but optimized for AI so that the AI can interact with the control panel, impersonate users and test access controls by itself) and it's so easy to create new features. It's just pure UX work. Just telling the AI what the problems are. Once fixed, it's fixed for good. By contrast, with a standard framework, I have to spend a lot of time debugging technical problems and keep running into regressions and it takes way longer to fix anything and requires coding knowledge.
LLMs work really well with custom frameworks provided that they're well documented via agent skills.
React doesn't provide optimal abstractions for safe and efficient LLM coding IMO; when building applications, React still makes you duplicate a lot of work and create a lot of custom components. A lot of the wiring has to be re-created over and over. If you really think about the essence of a web UI; it's just sections with lists of records, single records (loaded based on ID), single fields of specific records, forms... And each one requires access control rules which should be enforced by the backend; ideally in a declarative way so that it's easy to audit. IMO, the backend access control rules should be exhaustively auditable so that the AI can impersonate all the roles and access all possible resources in all possible states to verify that access control is enforced correctly in all possible situations. Anything less than that is not suitable for vibe-coding IMO. A user who doesn't even know what access control means cannot possibly build a secure application.
I was never interested in hacking but as a software engineer with 15 years of experience, I frequently encounter tricky situations in the code where I think to myself that it would present a perfect hacking opportunity and probably present in a large percentage of software.
Last time I tried my hand at whitehat hacking on HackerOne, it took me 30 minutes to find a major system crash/DoS vulnerability in a major platform. The company acknowledged that the issue was real but denied me the bounty payment because they said I would have to 'prove' that it leads to catastrophic failure. I had already done so in the sense that you could reliably infer it from the data I had provided, but it seemed like they were baiting me into committing a felony (DoS attack) to prove my point, which I wasn't prepared to do but I'm sure I could have done cheaply. So yeah, whitehat hacking seems to be a waste of time. Most software today is incredibly insecure.
It can generate large amounts of essentially bug-free code. The only bugs are bugs in the spec.
It has essentially no common sense though so it often misinterprets the spec or leaves gaps and edge cases unhandled and that's where the human engineer is still essential.
The dangers of AI are much clearer today than they were a year ago. There is a certain precision and relentlessness which is going to be a problem. It's a matter of time before people unleash AI viruses which spread through computers and data centres and pursue various goals relentlessly with no off switch. Some of these viruses and their goals may be nefarious. Imagine an AI trying to disrupt a specific person's life relentlessly. Spreading from machine to machine, never stopping.
reply