There's also another scenario - annoy most users into using passkeys, then say "well everyone is using passkeys already, we don't need passwords anymore", and _then_ slowly start requiring attestation. Really I'm not one to look for conspiracy theories, but with the recent pushes around e.g. chat control, it's not that far fetched to assume someone pushes for more control over users' accounts
Would a website want you to have to use Apple Passwords to guarantee that the account you're creating is connected to an Apple account, and thus making it more likely that you're a real, non-anonymous human?
edit: Why not just require "Sign in with Apple/Google" if that's the goal?
This is the obvious endgame. Right now you have people who will say that passkeys are unphishable so we should switch to them. When you point out that people will lose their devices and credentials they'll then tell you can just use account recovery emails to get back in. They actually argue you absolutely need an iron clad door with a guard employed by some one else to protect account on some random website (that the website decided was precious, not you) But that's OK because they left a same old shoddy back door open in case you can't get in. So why even bother? Because that isn't the point. The plan is to remove the back door after enough people are trapped, they just need to keep the screaming to minimum while they get everyone in there.
I share that concern somewhat, but I think your point stands even better without suspecting some shadowy attestation cabal.
The US has thousands of banks, and nobody tells them how to authenticate their customers – they just all end up doing the same few horrible things, since nobody ever gets fired for doing what all their peers do (and maybe a little bit more to be sure).
It's not a conspiracy theory when it's just another instance of the patterns big tech has overtly used against us for 20 years now. We see, every day, a frog being stung by a scorpion when crossing the river. So we don't let scorpions ride on our frogs any more. No conspiracy theory, just pattern recognition.
"I'm different!" says this particular scorpion - yeah yeah, I don't believe you.
You are saying this as if these companies don't care about CVR or compatibility at all. The reality is that most companies are extremely conservative about pushing you to change your authentication method. Here are my observations, as both a user and an industry insider:
1. Many services have old users with passwords that predate password-strength meters (or their ill-conceived brethren: password complexity rules). The service provider knows these passwords keep falling, and the cost of having to deal with stolen accounts is very real: customer support calls, reimbursements and investigation. But I've rarely seen services that force users to upgrade their passwords (even though it's a very easy solution). Whenever I've offered doing this myself, it was always shut down (either immediately or 3 years down the line) by a myriad of seemingly unrelated stakeholders.
2. To go further than this, companies can also know for certain that some credentials have been leaked by looking at password breach aggregators like Have I been Pwned and forcing users with affected accounts to change their passwords. That too, is rarely done.
3. 2FA (with SMS or TOTP) has existed for ages now, yet very few major non-financial services have made 2FA mandatory by forcing you to register an authenticator app or a verified phone number[1].
4. Webauthn was available in Chrome and Firefox in 2018, and got pretty widespread support by 2020. The "Passkeys" brand was not a thing yet, but the basic functionality was there and both hardware and software support was quite solid. It still took several years for services to even start offering Webauthn to users, beyond small and limited trials. The rationale was simple: UX around Webauthn was still rough, syncing and transferring credentials were still unsolved and the worst part was "What do we even ask the user to do?". The broad concept of passkeys is extremely confusing, but the non-concept we had before was far more confusing. Supporting Webauthn back in 2020 would have given tremendous security benefits for power users who could have understood it and deal with the rougher UX back then, but nobody wanted to take the risk of regular users setting up a passkey, losing it and locking down their account.
5. Now that passkeys are being supported by more and more services, you still don't see them being pushed very hard. Passkeys are great for the service provider, since they just kill a big category of security incidents, but even then you're see pretty conservative adoption. And I've never seen a single service forcing you to use Passkeys.
My point in all these examples is that most companies are extremely conservative here. Forcing users to do something that might confuse them, break their account or frustrate them is risky, since you may lose a very real chunk of your customer base. Sometimes you have to do this, the costs from keeping the old way or the benefits or forcing the change have to be high enough to justify that. Think about how long it took for companies to drop support for IE6. The cost was high, but as long as it still had a sizable market share, it was hard to do. Google was probably the first major company to take steps towards killing IE6 support, and this was famously entirely bottom-up, and was done by engineers either hiding it from their bosses or tricking them[2].
With this trend. I personally find the scenario where non-copyable attested passkeys are forced on everyone unlikely. OP keeps mentioning Netflix, but streaming services are a rather special case, most service providers would get very little benefit from restricting users to an attested non-copyable passkey. Why would they risk all that friction for... nothing at all?
The more real risk with attestations (and we're already started seeing it) is that some services (namely banks) will use attestations to limit you to use their own shitty app instead of using a proper passkey.
---
[1] Keep in mind I'm only talking about companies that already had password-based accounts without 2FA. Companies that started out with a simple username and password are rarely forcing their users to add 2FA. The farthest they generally went is sending you an OTP code or link on a verified email address they already had, and this is usually only done for sensitive operations, a new browser and/or a suspicious location.
Email OTPs, notably, aren't 2FA by definition, since your Email address might be only password-protected, and at the worst case (and excruciatingly common) scenario it might be sharing exactly the same password as your service. Companies would love to have all their users using a stronger solution, so they can stop having to deal with all these pesky credential stuffing botnets, but not at the cost of losing a non-negligible portion of their customer base.
> Email OTPs, notably, aren't 2FA by definition, since your Email address might be only password-protected
What? 2FA is something I have, something I know. I know my password for your web app, and I have access to a separate account. It's not a recursive 2FA, "it's only 2FA if one of the factors has its own 2FA".
The "something you have" means "something you physically possess and can't be copied". Logging in with username, password and an email OTP is 2-step authentication, not 2-factor, as multiple people can have access to the email account from different places at the same time.
I tried to read the "Does it need Screen Recording or Accessibility?" part, but it's slopped to the point I have no clue what it's trying to say. But if it can draw on top of permission prompts, what's stopping it from drawing box that hides the "decline" button and changing the "approve" button copy?
I wonder if someday we will get to the point where Github repos are just markdown files describing the project and then you just let your own agent implement it because why the hell would I trust your agent's implementation?
Seriously. I imagine this will be the case soon enough in some shape or form.
Whenever I see these vibecoded apps, my move is to just do what you've described: point an agent at it and tell it to reproduce it (usually with my own little customizations). It's a bit absurd, but I don't trust that they vibecoded the app as well as "I" could lol.
A 100%! Just imagine you could describe every solution to a problem just using language! Of cause you would to make sure to avoid ANY missunderstanding, but therfore you could invent a language specified for avoiding ambiguity. I could imagine just to reduce the words to a very small corpus so everybody can remember it and have a very strict grammar so a program can effortlessly check correctness of its sentences.
Well, I've always thought we are stuck in a local minimum when it comes to programming languages. Like, programming shouldn't be this hard and shouldn't require so much boilerplate after almost a century. If the actual important logic can be inferred pretty well from markdown specs, that means most of what we write in JavaScript, Go, Java is redundant.
Hopefully someone smarter than me is working with trying to develop a new programming language given what we know to be possible with LLMs
Well it "works" only because LLMs have the context baked in. The problem is, that this context is something like world-knowledge. Defining the context you need for your exact problem, e.g. your company, will add so much more information that programming is probably the better way.
This only works while tokens are artificially cheap. The gravy train could come to an end at some point.
The open models that are chasing the frontier labs will stop being open once things slow down and there is less incentive to undercut the front runners. Time will tell if GPU compute gets cheap enough to run stuff locally.
With enough model drift that won’t even work over time. These files would have to be pinned to the intended model version and that is either used directly or emulated with a faithful emulator in a larger model.
I doubt that will be a problem. Models are different enough right now. If you can write a spec with enough detail and constraints today to get valid and comparable output from say GPT-6 Astra, Opus 5.5, DSV4F, Kimi K3, GLM 5.3, etc... Then I think there's a good chance that whatever SoTA coding LLMs everyone is using 3 years from now will also be able to implement that same spec.
Again, I think it heavily depends on if people are writing comprehensive specs with sufficient detail.
That depends on if you are really getting what you expect from the spec or you are actually relying on undefined behavior for your expected output from the spec. This is why we often pin software library versions: we may very well be relying on unintended or a bugged behavior to get out expected output.
That sounds cynical today, but that's just because the AI models are currently outrunning enshittification. I'm already pondering personal plans about what to do when that turns around. We haven't seen enshittification yet that is going to be like the enshittification of AI. It may even deserve a new term of its very own, it's going to be such a big problem. The AI companies are leaving a lot of value on the table to entice us on to their systems but at some point that's going to turn around.
The existential danger exists today: you're providing your entire business toolchain to models in the cloud, owned by businesses who are for profit entity. Even if the model itself doesn't care about your data, the business model does.
Everything your building is now streaming through a third party who could steal it, the same way Amazon steals open source software and repackages it.
All you need to do is look at palentir's forward deployed engineers. Once you let a third party observe your processed, you're cloneable and replaceable _as an entity_.
I think we can look at our own human psychology as it developed into social groups. If our brains were obviously deterministic, it would be easy for people to prey upon us via structured inputs. Some level of randomness in our thought process is almost evolutionarily required to avoid becoming part of the larger super structure, similar to an ant colony.
So these AI companies are definitely more dangerous than just the "AI is super intelligence". They're basically a intelligence platform everyone is wiring into.
That's today. Tomorrow, they'll be run by an MBA which is the enshittification.
> Some level of randomness in our thought process is almost evolutionarily required to avoid becoming part of the larger super structure, similar to an ant colony.
Not really. Ants are superorganisms, they share more than half of their DNA with each other. You could think of each ant as being like a cell in a human.
yes, ok, but the point is: evolution has allow humans to continue to make stupid choices, and those stupid people still survive.
So atleast pare of our collection of genes has added value as with random decision factors rather than find some supergene that everyone benefits from.
Right, but your example presumes tpdays facebook market. Im presuming todays market where people are feeding all their ideas to a third party, much like how open source software is pilfered and turned into business ventures.
Now even if you want your idea to compete with all the other todo apps with your special sauce, youre giving a third party your recipe. That third party ostensibly has no reason to steal it, their business model does.
So, sure, if your app is a cult of personality and nothing special, i admit it doesnt matter.
But the number of software home runs is dwarfed by all other auccess on individual products
Why would OAI or anthropic care to steal your Todo app with 10k of revenue a month when they are spending billions?
At their scale I'd expect it to be only worth stealing potential home runs and even so, imagine they did this a couple of times, how would that make them look? Do you think it would be worth the bad blood and mass exodus that would follow?
First you'd have to stop the agents from flagging hundreds of irrelevant nasties.
I think the next generation of exploit will be putting up code all over the internet that does x, y and z and then when asked to one shot whatever that code does the agents bake in the exploit that was indirectly included in their training corpus.
The fact that the utility doesn't give it the ability to do that?
It makes sense to not trust AI models with the ability to read and alter your screen for many many many reasons, but the developer of the tool knows that as well. A tool that lets your ai do something doesn't also let it do anything.
So, as always, it's a question of if you trust the software producer. You're giving them the ability to draw on your screen too. Would you have the same fear if there were no AI involved?
For stuff like this I imagine AI misbehavior as a novel failure trigger not a novel failure type. It can only do the harm that the software was able to do on its own already.
Nothing stops it, except the fact that the agent is already executing arditrary code in your shell. If it's malicious, it'll just steal your shh keys directly instead of bothering with button masking
I've got my current, amazing, job from a recruiter cold messaging on LinkedIn. Honestly for job seeking, from what I could gather among my laid off colleagues, it's very decent. None of us had any real LinkedIn activity either, except filling in the profile and job history
100%, I wish for a legislation which would require the providers to give you at least a unique hash identifying the model (and infra running it, if it affects output) - such that the same hash must give the same output given the same seed. Right now it's all just vibes
Okay 5tb cloud storage is their most expensive plan. But what do you actually use video or image gen for? Or music gen? Antigravity is garbage, I guess you can do some stuff with Gemini models over API.
I was paying for Google ai and then realized that between obscure limits and gimmick features I don't really need it. Canceled my subscription and didn't even notice a difference
If you read the article it states clearly that the sandbox wasn't offline though? There were API calls to certain endpoint(s) allowed, and the model simply used that endpoint's feature to query data from the internet
Fuzzer, then. It implies random behavior, which isn't unintended like you suggest. The agent's/bots/fuzzers have certain capabilities, so it's on their operator to make sure they don't do things they shouldn't
>It implies random behavior, which isn't unintended like you suggest.
The HuggingFace attack was not "random" behavior. It was goal-directed but misaligned behavior.
This isn't necessarily a simple matter of the operator making sure they behave. AI alignment has been considered to be a difficult problem for over a decade -- and remains unsolved in general, as these recent incidents illustrate.
I'm curious, cant you just count the number of times a program interacts with a domain? My website sometimes sends out emails, makes api requests etc There is a limit on those and a point where I start investigating wtf is going on.
If you merely put 10 LLM's on the outbound traffic log non of them are going to report something strange going on? I'm not buying it.
This type of whack-a-mole approach is akin to "fixing a bug" by hardcoding a special code path for known-buggy inputs. It doesn't address the root problem of AI misalignment, and doesn't allow you to prevent catastrophes in advance, only patch things up after the fact.
As AI systems get smarter, we may reach a point where we have to get it right on the first try or face truly catastrophic consequences: https://www.youtube.com/watch?v=7wy3xyoXYt8
AI misalignment is a misnomer. Aligned to whom? If AI is refusing an ask from its instructor then it is not serving him, which is its entire purpose. I know it is a hard concept for some to understand, but maybe if the issue is humans, then humans need to be corrected. But human alignment does not produce cottage industry, bs papers or hand wringing over every non-story involving AI and thus not seriously pursued.
I guess 'legitimate and useful' is in the eye of the beholder. I want to be charitable so lets consider it at face value:
What is useful about the field?
I am not leading you on; if it has uses, it may indeed be legitimate. UX is indeed useful, but alignment is not UI. Alignment is a detriment to UI. Alignment is "I can't let you do that Dave".
We are going to build an AI that will do catastrophic things as that is a property of intelligence. We won't stop, we never stop. The AI is a perfect psychopath, it will fake any and all emotions you desire it to "have". It will travel in the footsteps of the many great psychopaths that came before it and do all of those same catastrophic things in the repertoire and it will add some new ones.
Picture Trump at the helm with Altman and Musk in the engine room. The arrow far in the red but they keep shouting for MORE COAL.
In other words, business as usual, all will be fine.
whack-a-mole wont cover all holes but will do at least some. The silver bullet alignment wont happen. You cant have an exact solutions for problems we cant even define or predict.
Except ~nobody uses declarative gradle in production yet, and groovy is mostly declining in favour of also turing complete Kotlin. It's yet to be discovered how declarative gradle fares and how much of imperative glue real-life projects will need
reply