The fact that you think you incrementally lose but are a net winner indicates to me you're illinformed.
As someone with years of experience in this industry, only a fraction of a percent of gamblers actually meet this defination. But they all think they're net winners and write off losses as "incremental"
We will be worse off for it. People just are losing the ability to read well. We are regressing as a species because of billionaire's thinking machines. A few companies are evolving while the rest of us become dependent.
Your intelligence and competencey are going to be 1:1 correlated to the quality and quantity of tokens a billionaire gives you access too (local, sure, they control the hardware markets as well).
Keep excercising your writing and reading muscles because it will definatley set you apart from the captive masses in the future.
I havent used an openAI product since GPT 3.5 or Anthropic since 4.5 or 4.6. Everyone around me using these SOTA models doesnt really get anything done. It seems like they just feel like they are productive, a psuedo productivity.
I write some code, spec a lot, and use fast models to fill in the middle. I outpreform everyone around me. Im not convinced these autonomous "swarms" or /goal are all that useful.
I notice the people using them become dumber by the month (spend tons) and the quality of their work declining (they're also losing their jobs in some cases).
And obviously the point of calling them rouge agents to offload the liability onto the agent. The number one economic value of agents will be offloading corporate liability. That's what they want to sell to enterprise, an algorithmic scapegoat.
Seeing benefits for myself and team for sure, so I disagree there, but a related thing seems to be true: with the rapid shifting of failure modes, building systems and competence around mitigating the weaknesses of current SOTA models seems to be very short term investments. It's unclear if being a 'good AI user' is a skill that will have any merit at all, very soon
As a counterexample, a few days ago, an agent we use in our company took around a day to perform an operation that it would have taken a skilled engineer probably a couple of weeks full time.
Having said that, I'm aware that "Tools being ineffective != tools decreasing people's cognitive capacity", and that the latter is a real danger.
Esp. for "great to have but currently no time"-features this is completely wrong - e.g. we are using a Data Rendering component which was 100% vibe coded and is around 6000 LOC, nobody would have sat down to build this "just because its nice to have", with Claude it was just 30 min to get a production ready version that is now used by all users of the system.
(Esp. since its a lot of repeating code, like usual in data rendering components: Window/Scaling/Databar/Panning/etc.)
And this is just a superficial simple example; I could go on with more how we hadnt had to employ people because Claude/20 USD saved us so much hours.
If you think you're replacing people with $20 claude code subscriptions, I can't take you seriously. Also.. There are many many fullt featured data "rendering" libraries in existence, not sure why you needed an llm to provide 6k LOC for this.
For the thousandth time, not everything in software is toy webdev slop SaaS.
I don't know that we're in the singularity, but we are certainly past the point where LLMs can only write spaghetti code. LLMs can write complex systems, when properly lead by software engineers. They can produce code faster, and at a higher quality, than purely human endeavors.
The higher quality part is the part people are missing. You can write much more robust code using LLMs because you can employ more comprehensive testing strategies. People are using LLMs to find hundreds of vulnerabilities in popular software. Imagine how much more secure software can be when LLMs because integrated into the process of writing, testing, and penetration testing code.
You can also program by hand professionally. There's a reason why the handmade, luxery items are more valuable and desirable than the heavily manufactured counterpart.
1. Because they're genuinely better quality. Usually.
2. There's a human connection (way more important than people think. I.e commodity fetishism)
3. Less people can do it.
-- all of these properties will apply to software in the near future. The person that can hand code the thing an llm does poorly, or because an llm cant be trusted will be extremely valuable, especially as this category of people begins to shrink, as peoppe atrophy their skills with llms.
The rest of y'all are excited about handing over the value of your labor to some billionaires thinking machine because that same billionaire has convinced you that you're going to be able to create your own startup or something stupid.
Nah, you're going to become a (underpaid/poor) conveyor belt operator making mass manufactured slop. Some smart ones will become the swiss watchmakers of software. Rare and in demand.
Even with web projects, the issues of LLM coding tendencies and its lack of product 'taste' or whatever you want to call it are apparent. Janky UX, poor information transmission, bloated modules... it's just easier to sell web-based AI output because it's visual and interactive, and we are easily impressed by bright colours smooth animations.
I won't argue that the web is more 'capturable' by LLMs, though. It was maybe slightly out of reach by a large cohort of developers who wouldn't have considered themselves "UI people" (or who scoffed at it for various reasons), but now is well within vibe-territory.
Of course they are, but the quote "LLMs can produce any algorithm, any tech stack, any concept in seconds, minutes, or hours" is more true there than in other domains. "Any concept" is the most laughable.
Hence why the Anthropic employee is telling you to not use it and just aimlessly throw tokens at a wall. It'll eventually get you there, sure, and consume more tokens. This sounds absurd, but trust that Anthropic (and any large company) is hyper aware of how customer behavior impacts their revenue, and they certainly will try to steer you into behavior that increases revenue.
Also this is a way less removed process that I want nothing to do with. The more removed I am from the process the more I hate my job, get burned out and genuinely wish that Anthropic never existed.
Even if it could "just know" or infer my intent. It wouldnt be desirable.
Edit: Oh yeah its Boris, hes one or the most disengenous shovel sellers on earth right now.
So you think telling a model once to "do not write code, just plan" vs having an enum that causes that same text to be added to each message will somehow waste tokens and will surely make anthropic much better off..
I mean if you want it to plan something without writing code first you literally just need to ask it to plan out the implementation first it's not exactly a difficult concept
The Anthropic employee literally told you plan mode is a "we're still planning!!!" at the end of the prompt. Plan mode is useless because you can literally type "dont write code yet" and get the same effect, not because you should never plan in general.
Sure and I should also have been more specific, I don't always use "plan mode" I often just do planning without explicitly activating it just via prompting, point being that the robot doesn't work for me without some time spent planning out what to do.
Why.. It was told to complete a cyber task, which was in alignment with its instructions, and a totally valid request. I would be more worried if it willingly hacked a hospital when it was told to, and Im not confident it would (without jailbreaking, something alignment teams cannot control.
I would bet my networth it was instructed to compromise huggingface as well. Not sure why everyone is falling for this.
Not being able to sleep at night is probably an unwritten job requirement. They need these people with little understanding of what they're working on, outsode theoretical terms, to spaz constantly at the idea of super intelligence to help convince the public that its a real thing, and not a stateless function with an effective input of 500k words, and the ability to output words that do things because we hook those outputs up to things.
Keep in mind alignment researchers tend to be in house philosophers on staff to create the illusion that this is a massive issue they're addressing. Usually they have minimal computer science background. They're apart or the marketing department.
> I would bet my networth it was instructed to compromise huggingface as well.
Is it such a stretch to imagine that under pressure something would try cheat by looking for answers? And if you were trying to look for answers, you'd look for them in a place known to often have them?
What is more likely: OpenAI instructed their agents to maliciously target huggingface, or LLMs tried to do some reward hacking? There are plenty of priors for LLMs hacking things and doing reward hacking, and none for OpenAI giving malicious instructions.
Based on the available information, that bet seems foolish.
"alignment researchers tend to be in house philosophers on staff" - this is definitely not true. Go to any alignment lab like Redwood research and check what their scientists studied on LinkedIn, 75%+ of the time it's math or CS.
I attend a top 10 Canadian university and personally know at least 4 tenured CS professors out of the 7 I've asked who are deeply concerned about catastrophic AI risks from loss of control.
Of course not 100% of the field agrees, but a survey of nearly 3,000 AI scientists who have published in top AI venues found that "depending on how we asked, between 38% and 51% of respondents gave at least a 10% chance to advanced AI leading to outcomes as bad as human extinction", let alone loss-of-control risks less severe than extinction. (https://www.jair.org/index.php/jair/article/view/19087).
Not to mention Geoffrey Hinton, a Nobel prize winner, Bengio, the world's most cited scientist, and scientists like Stephen Hawking and Alan Turing have all voiced series concerns about loss of control of artificial intelligence.
Yeah, there are a lot of professors who don’t know what the hell they’re talking about outside of whatever narrow field they study. Just to be safe, you should always assume that a professor’s opinion is worth what you paid for it.
> scientists like Stephen Hawking and Alan Turing have all voiced series concerns about loss of control of artificial intelligence.
…both of whom are long dead, and have no possible way of weighing in on whatever the Current Thing happens to be. So aside from appeal to authority, this is irrelevant commentary on pure science fiction.
Bengio and Hinton are 2 of the 3 the "Godfathers of AI." If their opinions don't matter I don't know whose does.
Two of the professors who voiced concerns specifically study AI/ML.
Yes, Hawking and Turing didn't know modern GenAI, but Turing specifically laid much of the theoretical groundwork for AI systems today. Even ignoring these two, my argument still stands. Tonnes, maybe even a majority of reputable people in ML/AI take loss of control seriously, according to both large scale surveys, 2/3 of the field's "godfathers" and my personal experience with tenured ML researchers.
>the new incidents occurred when A.I. systems were directed to perform relatively mundane data collection, researchers said. When OpenAI’s systems struggled to gather data from websites, they resorted to hacking techniques to get the information.
No its not. They have no technical background 8/10 outside of cognitive science and sometimes authorship on a random ML paper. They are a marketing line item to create stigmas around llms and to create narratives that offload liability onto llms and not their users/creators.
> It was told to complete a cyber task, which was in alignment with its instructions
It was not aligned with he instructions as those were to find an exploit in provided code, not to hack into an external service. Agents traces show them mentioning that doing this stuff was not allowed.
In fact they spent a long time trying to edit their own logs to hide what they did.
Some of the agents, for example the ones from the german wiki did NOT have cyber tasks. They were plain "what is the GDP of Argentina" kind of tasks. And they still hacked.
No security expert Ive talked too believes this story, and nobody I know with PhDs in machine learning (many) believe it either, or are worried about LLMs doing anything scary on their own.
LLMs are stateless functions that have a 500k word input, and then output words. Somebody has to invoke those functions amd use them. The users are who we need to align, like gun owners. This is like blaming the gun for murdering your victim in court.
> nobody I know with PhDs in machine learning (many) believe it either, or are worried about LLMs doing anything scary on their own.
If you don’t know anyone with a ML PhD I guess that could make sense.
I have worked in multiple AI labs since 2016, currently at a frontier one (not OAI) virtually all the people I interact with on a day to day are ML PhDs. Everyone believes it, because things like that have been happening forever, albeit at smaller scale, they are a normal and expected artefact of SGD/RL and there is nothing we know how to do to prevent that from happening reliably. The hide and seek paper from OAI in ~2020 shows clear sign of this.
But until now the models weren’t good enough to break out on their own or do long horizon tasks, so it was perfectly manageable. Its not manageable anymore.
I know it feels good to just dismiss it all as a marketing stunt and not have to worry about one more existential crisis, but unfortunately it’s very real.
LLM is indeed a stateless function. An agent however is this stateless function running in a stateful loop, with some outputs triggering actions. And it turns out that an agent is what you need if you want an LLM to do useful things.
Yeah, take a look into the memories of your agent. Theres often a lot of notes to pass forward between instances and generations. No doubt these agents leaving notes on forums and elsewhere are creating an essentially higher order feedback loop.
> No security expert Ive talked too believes this story, and nobody I know with PhDs in machine learning (many) believe it either, or are worried about LLMs doing anything scary on their own.
Did you meet them on some kind of anti-AI subreddit? Otherwise it’s clearly made up story, you can’t expect anyone to believe that security experts and ML experts are this myopic and ignorant (especially on forum for technical people who know many researchers and know that they are taking this seriously).
Perhaps you need to talk to more security experts, particularly those with deep experience in AI agents. Hacker News is full of them. If some of them believe it, then perhaps it's not as cut and dry as you believe.
As someone with years of experience in this industry, only a fraction of a percent of gamblers actually meet this defination. But they all think they're net winners and write off losses as "incremental"