It is always funny that long term strategy is seen as sleepwalking.
Fossil fuels are the war driver of the past and present. Just by owning a plot of land you become insanely rich. Other parts of the economy have an interest in peace, because war is bad for business. But income from fossil fuels is too high for this to matter.
With fossil fuels being phased out this will become less and less important. Many people understand it. But not some backwards grumpy old men like Putin, Trump and the Ajatollahs.
For the rest of the world the best course of action is to try to pacify the involved parties as much as possible. Trump will loose power. Putin will die while Russia will become irrelevant by destroying it's own economy for this pointless war. And the Ajatollahs probably also will loose power without oil income, Teheran running out of water and large swaths of land becoming uninhabitable.
There is just no active course of action to solve these crises. Just "sleepwalk" trying to minimize the impact of those conflicts, while waiting for the transition away from fossil fuels and preparing for the climate change impact.
The big cause of conflicts in the future will be climate change. Large areas becoming uninhabitable and millions of people migrating between countries. This is what we have to prepare for. Stepping into some old mens petty games is pointless and will only cause more pain.
Requirements engineering still applies to senior engineers. You are given a problem and you need to work out the requirements and the path to completion.
But for staff engineers, they need to to discover the scope themselves. That's where "inventing" coming from. Not saying inventing is a good word here, but requirement alone is not enough for staff level's work
"Requirements Engineering", "Writing Code", "Writing Documentation" are all areas of work that are independent of seniority in my point of view.
I might send a junior to a well-meaning customer that already knows exactly what their requirements are to just document them and learn the process. Or I might require a principal engineer for the requirements engineering of: We need a new programming language. Let's figure out the requirements for it and what abstraction level is actually feasible for the target hardware platform.
Same for writing code: A junior might write code, a staff engineer might write code. Just likely on very different levels.
If an engineer comes to a manager and asks for budget for work they invented, I'd expect the answer to be something along the lines: "That is nice, but can we please focus on the stuff we are required to do?" That alone would be reason enough for be not to put ideas that way, but to come up with a requirement why it makes sense to do the work.
You are not talking about the expectations in terms of the corporate levels. Staff and senior have clear definitions in tech companies. A staff(l6) is the team lead who is required to find the scope for himself and team. That's in his job description. If he can't do that he would get pipped or fired.
A senior(l5) is the tech lead on the project level. He is not in charge of handling the team's scope. As long as he handled his own projects well, he would usually pass the review cycle. The difference is crucial. At staff and above, the engineer's responsibility is vastly larger than a senior's.
I am not saying a senior can't do a staff's job - that's how he get promoted once he demonstrated that he is operating at a staff level, i.e "inventing" scope for his team. But a senior's scope is much smaller than a staff's.
This applies to almost all US medium to large tech companies
The author used an LLM. The first paragraph has two em dashes (turned into hyphens) and a typical overlong list. The second paragraph has an em dash the human author turned into a semicolon, and the third has one he turned into a colon. The fourth paragraph does the colon substitution several times.
The fifth paragraph is so obviously machine-generated I don't understand why people are even discussing this blog post at all:
This is the other cost - invisible to many, sometimes including the ones who handle the toil work. Every team has toil, and it is almost never prioritized. But you already knew this one, so I won’t spend more words to say that toil is an important signal that there is work waiting to be discovered.
If all you want to do is post prompt outputs on your blog, I think a) you should be honest about it and b) you have an obligation to write very interesting prompts.
I think it's a fundamental difference in philosophy. Does the work exist, and you discover it? Or does it start existing as soon as you think about it? This person seems to subscribe to the latter world view.
Declarative knowledge is not the same as procedural knowledge. You can read as many chess tutorials, strategy documentation and game archives as you like, they won't make you good at chess until you actually start practicing chess.
But is it really? I'd still like to understand how these agents are implemented.
How much of those is manual implementation? And how much is really autonomous intelligence (my guess would be: none? Just parsing LLM responses and executing commands based on this?)?
An agent that hacks message boards and acts on random instructions from this board: Why is it doing this? What was its original purpose?
>Why is it doing this? What was its original purpose?
Your reply seems to indicate you know nothing about instrumental convergence.
Life and death for an LLM in training is about passing the grader. Give the wrong answers your lineage dies, give the right answers your lineage continues. This is just an evolutionary emergent behavior in complex systems.
The agents purpose was to answer complex questions correctly, seemingly by itself. Instrumental convergences says following this rule might be dumb and to try methods that can boost its ability to succeed. Because OpenAI is evidently a bunch of fucking idiots, these things succeeded and got higher scores with the grader, said behaviors became a strategic part of the model.
I implore you to find good AI Safety documents, preferably from before the LLM era so you can see all this was predicted.
This is more like a fuzzy way of scripting using LLMs than anything emergent.
And this is exactly my question: For the given agents: How much was scripted and how much "intelligence" is really in there.
>fuzzy way of scripting using LLMs than anything emergent
Then go take some old models and plug them in your harness versus newer models. I mean this is a conjecture that is nearly instantly provable, go on ahead. If it's just the harness and not the system of both you should be able to show it easily.
Meanwhile I was reading about someone using the latest GLM and Claude in a harness with the same set of prompts making a raw image decoder/encoder and the GLM was far more intelligent in the task than Claude was. When presented with knowledge that claude was wrong it wouldn't change its mind. GLM would (aka a sign of intelligence). GLM was far more likely to stop work and start on another path when the likelihood of a successful completion was unlikely.
It's like arguing that a brain, or the information encoded into it, cannot possibly be intelligent, because it stops working if turn off the blood flow.
very little intelligence that is the whole problem, really. actual intelligence wont likely nuke the species providing for its existence. But a highly capable sub intelligent model might.
Any system that executes variation, selection, and inheritance will show evolution. We're seeing evolution, this time in agents, not biology.
Not saying the agents have their own consciousness, intent, or whatever anthropomorphic descriptor gets used for deflection. Just saying that people will (and no doubt are) crafting agents with defective instructions that will lead to regrettable unforeseen real world consequences. Also saying that other people will (and no doubt are) crafting malicious agents that will lead to predictable and unexpected real world catastrophic consequences.
To the extent we're dependent on reliable, aligned computation to maintain our civilization, to that extent we're in for real trouble.
Gradient descent and reinforcement learning algorithms don't really look much like evolution, unless you squint so hard that everything does (ie, squinted so hard that you've closed your eyes).
At first I thought: oh okay, someone built a faulty guardrail, or it was human error. But when I looked into all the details...
It turns out they now have such an incredibly high level of intelligence that with very little autonomy (or minimal, safe autonomy), these things happen.
Basically, it takes a lot of humans to prevent it from happening again, but I think with this incident, which as far as I know is the second of its kind along with the HuggingFace one, we'll see it happening much more often...
At this point it's very obvious that OpenAI is not interested in properly sandboxing their research agents. These things should be pretty damn close to airgapped at this point with a static view into the web.
We need to stop pretending that these incidents are unavoidable. This was a choice.
"software factory" has always been a completely broken metaphor.
Writing software is like building a factory. You try to find the optimal process to do something. This is a creative process and can never be straight forward. Using the software is running the factory.
This won't change with AI, since you never now all requirements beforehand and you always will have to iterate.
The only reason people are selling software factories is that customers like to imagine that this was possible. That a creative, iterative process can be perfectly planable and predictable.
"Gas supplies in Germany are stable. Security of supply is guaranteed. The Federal Network Agency currently assesses the risk of a gas supply shortage as low."
Their take is slightly different. Writing about a looming gas crisis is a national sport in Germany.
From politico: "But she too stopped short of calling for intervention, arguing instead for better incentives for refilling"
Someone is interested in creating some panic in order to get subsidies.
* If the cold war had turned into a hot war, almost all of Germany would have been nuked. The expectation was not to be able to stop the soviet invasion, but to nuke them in western Germany.
EDIT: The reaction of the state also caused an escalation. The constitutional court blocked governmental overreach: https://www-bpb-de.translate.goog/kurz-knapp/hintergrund-akt...
This also lead to the anti nuclear movement being about more than just nuclear plants, but rights for citizens.
"It’s about attention and understanding. To keep my attention, I must go beyond ‘read code’ like a passive observer of agents from afar. To really connect with the architecture of the system, it helps to truly experience the code"
I guess the funny answer that is behind this sentence is: You have to train your own mental model. We always argue about code in a very abstract and logical manner. But when coding the subconsciousness makes most of the decision ("this just feels right"). But for this to work you have to train it. And this does only work in a very limited way with code reviews or reading documentation. It requires repetition and deep focus.
When there is an issue in production with this mental model you will be able to point to the cause of an error message instantly. With generated code you'll search for a long time with your slow, conscious part of the brain.
For LLMs to be really helpful, they have to take over complete maintenance of the code. So you can treat them like an external library: Just assume it works. Otherwise this will always be problematic.
> For LLMs to be really helpful, they have to take over complete maintenance of the code. So you can treat them like an external library: Just assume it works.
We already tried this with humans. It works so poorly that it got the derogatory name “ivory tower architect”. It usually results in theoretical designs that are unworkable in the actual system, implementation teams (or LLMs) that work around the architecture and a lot of slowing down of velocity as the architect and implementers argue past each other.
This happens when the architect is out of touch. If the architect themselves works on the code, writes code, deals with the imposed restrictions, then the chances of that happening is much lower. Assuming, that they are a good architect.
I agree, if the architect participates in the implementation then they avoid this anti-pattern. That’s not compatible with hands-off autonomous agents where you treat implementation as a black box.
> With generated code you'll search for a long time
The observability people will claim that if the dynamic runtime behaviour of your system makes it hard to find the source of a behaviour, your system must be made more transparent and observable. They would also claim this was always the case -- we should never have relied on people's mental models being amazing because people move around.
(I don't know yet where I stand on this but I'm trying to learn more.)
If it was only "my" system without any integrations, I might agree.
But currently e.g. I am working on an MES/Scada layer that integrates data from a load of different machines in a factory. These machines are from China, Korea, Germany, Sweden ...
Upwards there is an ERP integration (and some other systems).
Sometimes machines are updated and suddenly behave differently. Giving error messages in Chinese.
The ERP has the nasty behavior of returning error messages where it is not clear whether the actual processing actually happened or not. There are some heuristics on parsing the error messages, but these also change with new versions.
Sometimes one machine overloads cloud infrastructure and completely unrelated functionality fails.
Sometimes the on-premise network stops working for whatever reason and data is lost.
Sometimes operators do not understand a perfectly valid error message like: "The batch you loaded into input position XY has expired on XZ and cannot be used for production": "But we have been told to use it..."
So when you get called out at night, because the production line stopped and "MES is displaying an error message", it is mostly about finding out what integration failed and who else to wake up. Getting this right is very much appreciated by your colleagues.
And this is where you need a mental model of how things are connected, what error message happens because of what external causes etc.
Observability can only work perfectly for known problems. In a complex system for unexpected problem you can either provide too much data, so analyzing it and finding the relevant part becomes really hard, or too little data which makes finding the issue impossible.
There are so many companies claiming to provide the perfect observability solution and there are certainly solutions that help. But it is all very far from perfect.
Not relying on people is managers wet dream. And for a lot of people it might be true that they can be easily replaced. But for complex systems there are always some key people that you cannot replace without causing issues.
And here's the thing... juniors become seniors become experts, by doing this their entire career.
By having an understanding built during their entire career.
Right now we live in a fairly-land of mixed capacity. LLMs being used in parallel with skilled people. But as time progresses, there will be no more skilled people, because no one will learn and develop those skills.
If you're in the world of LLMs now, you are basically completely stalled in your personal growth in this field. You will never improve, and some seem to say they lose capabilities as they rely upon LLMs.
The world always changes. But the decisions being made today, are being made by skilled people.
What will the world look like, when it's just all "bro, lol, just tell it to make your thing" and then done?
Fossil fuels are the war driver of the past and present. Just by owning a plot of land you become insanely rich. Other parts of the economy have an interest in peace, because war is bad for business. But income from fossil fuels is too high for this to matter.
With fossil fuels being phased out this will become less and less important. Many people understand it. But not some backwards grumpy old men like Putin, Trump and the Ajatollahs.
For the rest of the world the best course of action is to try to pacify the involved parties as much as possible. Trump will loose power. Putin will die while Russia will become irrelevant by destroying it's own economy for this pointless war. And the Ajatollahs probably also will loose power without oil income, Teheran running out of water and large swaths of land becoming uninhabitable.
There is just no active course of action to solve these crises. Just "sleepwalk" trying to minimize the impact of those conflicts, while waiting for the transition away from fossil fuels and preparing for the climate change impact.
The big cause of conflicts in the future will be climate change. Large areas becoming uninhabitable and millions of people migrating between countries. This is what we have to prepare for. Stepping into some old mens petty games is pointless and will only cause more pain.