> - Resource-based policies, such as bucket policies
> - AWS KMS key policies
> - Access across AWS accounts or across AWS Regions
> - Any other security control
> Some services in the emulator evaluate IAM policies for IAM users and roles that you create in the emulator, but not the same way as AWS. The emulator never denies a request signed with its own key. It doesn’t check AWS KMS key policies or Amazon S3 bucket policies. It also doesn’t support calls across accounts or Regions. So a request that AWS denies might succeed in the emulator. A request that AWS allows might fail.
This is a shame because I'd love for the official AWS product to be better than e.g. ministack and localstack on this, which will go some of the distance. I think because the agents were only building on the public API surface, it's hard to build this part. But they should've tried and let the agents read whatever policy engines AWS has. Or maybe it's a way to prevent someone from trivially running aws-emulator as AWSaaS?
I've implemented the AWS IAM engine and all policy types, conditions, etc in here if you want to try it out. It's the most thorough in terms of edge case coverage I'm aware of and I'm constantly adding enhancements and closing edge cases.
The current IAM engine and language “balsa”, is its own thing. Its vended as a library with large “there be dragons” disclaimers if you even think about going outside the public interfaces. There are also big opaque interactions between different components like STS, KMS, service principals, service-specific resource policy evaluations, etc. I doubt that can be reasonably open sourced or exposed.
There was an attempt at a public implementation of the “next gen” AWS IAM language/engine with “cedar.” However I think that got sidetracked as an authz product and didnt have any real penetration with AWS IAM.
I shudder at the thought that local emulation is somehow supposed to handle IAM conditions like source VPC endpoints, EKS Pod Identity, chained roles originating from web identity providers...
IAM itself is free. Most IAM conditions involve resources which are free (e.g. VPC gateway endpoints, web identity providers) or which are free additions on top of a cluster you have for other purposes (e.g. EKS Pod Identity). Presumably you already have an AWS account. What value does an emulator give here?
Yep, it's a complicated beast so I'd love to be able to mess around with it locally! I don't mind using an AWS account for a shared staging environment in the CI pipeline, but trying to figure out policies and roles during development, it can get a bit messy coordinating and handling the lifecycle of resources in a team's testing account (maybe this is easier if you get your own account?). And accounts aren't free when they require GuardDuty etc, unfortunately.
> accounts aren't free when they require GuardDuty etc, unfortunately
This is probably the root of your problems. In principle, non-production accounts should have vastly reduced security requirements. Personally, I can't understand how companies are OK with offering more-or-less unrestricted guest WiFi (which, of course, has no access to production systems, let alone internal systems), but somehow offering minimally restricted AWS accounts (basically, just slap billing alarms on, maybe common-sense SCP-based restrictions like no public IPv4) for development purposes is problematic?
I suppose it would have been extremely complex to replicate all of that. Maybe they thought it’s out of the scope to emulate all security patterns locally, they just wanted to focus into providing a native alternative to let users do integration tests.
Somewhat related question: what is it about READMEs that AI agents love to dump the most useless, hard to contextualise & comprehend, irrelevant rubbish into them that makes understanding a project and onboarding so hard?
It feels like like the AI agents can't help themselves sometimes, and the judgement exercised around what's included and omitted is baffling.
But maybe READMEs have always been this bad, and AI agents have raised the baseline?
That sounds pretty similar to what I get from LLMs regardless of what they're writing for. Any time I get more than a paragraph out from an LLM, it tends to have of a lot of irrelevant details or unnecessary hedging, and I have to either wade through it to find the actual answer or try to convince it to make it more direct.
As someone on the spectrum, this does happen to me with people sometimes too; I struggle when asking a question and getting an answer that doesn't fit the "shape" of what I expect (e.g. asking a yes or no question and getting a relatively long sentence in response that doesn't contain either "yes" or "no" in it, which means I need to do the equivalent of applying it as a diff to my mental model and seeing if there are conflicts). This happens less frequently with other humans though, and on average the amount of effort I need try to figure out what they're saying is a lot lower. This doesn't make it less frustrating when I get that kind of output from an LLM, but it doesn't surprise me all that much that it happens.
I'm sure I do this all the time to people too, though. One of the biggest lessons I've learned in the past several years is that I communicate in ways that I'd probably have trouble understanding in reverse a lot more frequently than I realized, even I still do the things I find confusing from others less often than most people I interact with. I'm sure a lot of people might find this comment to be pretty much exactly like what I'm complaining about even though I feel fairly confident at least in this moment that my point is clear.
> As someone on the spectrum, this does happen to me with people sometimes too; I struggle when asking a question and getting an answer that doesn't fit the "shape" of what I expect (e.g. asking a yes or no question and getting a relatively long sentence in response that doesn't contain either "yes" or "no" in it, which means I need to do the equivalent of applying it as a diff to my mental model and seeing if there are conflicts). This happens less frequently with other humans though, and on average the amount of effort I need try to figure out what they're saying is a lot lower. This doesn't make it less frustrating when I get that kind of output from an LLM, but it doesn't surprise me all that much that it happens.
I've have people in my life where I've got situation going to the point I got sick of it and just started responding 'That didn't answer my question. Do you remember what my question was?'. Turns out many people seem to invent out of whole cloth what your question is rather than actually responding to it. Seems like a whole lot of wasted effort. I mean, it's one thing to ask a yes or no question and getting 'it depends on X, Y, Z, blah blah blah', or just 'I'm not sure/don't know'. There have been far to many cases where I ask something and the response is functionally 'well you see, it all began in 1969, back when dinosaurs ruled the earth'.
LLMs seem to atleast actually respond to your question. The question might be phrased in a way which doesn't match what you intended, and the answer will be verbose and require you to wade through prose to actually find it, or it might just be wrong, but it will at least be an answer to the question you gave it.
My personal experience is that people who respond to my like this do tend to understand what I've asked but aren't particularly direct in the way they describe things, which both causes them to give answers hard for me to understand and would cause them to probably not do a great job trying to phrase my question in their own words (and very few people are memorizing the conversation so that they can recite parts of it back later on).
Some of the noise comes from writing for two audiences(humans and agents) in the same file. There are established conventions for separating the two, and research suggests that the “repository overview” documents agents tend to favor aren’t even helpful to agents themselves. https://arxiv.org/abs/2602.11988
They're pretty readable, I recommend that second one (though it leaves some questions I have unanswered). It's a good example of how latent issues with a large system can go unrecognised for a long time before a seeming unrelated change can trigger a cascading failure
One thing it doesn't make clear – the GPS card only supported the original L1 signal, which is what causes the GPS week rollover issue. The newer L2C signal has a much longer rollover period (157 years vs 19.6 years), which means no rollover until next century. If the GPS card had supported the newer L2C signal, then likely this would not have happened even if the other misconfigurations had still occurred.
It's a reference to this story where an OpenAI model broke out of its sandbox during cyber benchmarking and hacked into HuggingFace, in order to obtain test solutions: https://news.ycombinator.com/item?id=48997548
Is GitHub Copilot the best positioned platform for enterprise? They support Claude, GPT, Gemini, and now even open weight models. Larger orgs are paying at API rates anyway so it costs just as much as anywhere else. They have a pretty good agent CLI and SDK, and now a desktop app. They have hosted agents, and you can run their 'Agentic Workflows' in CI.
Has their reputation tanked so much that the alternatives get all the buzz? Or is it that non-enterprise users are priced out by the usage costs, so no free marketing?
We just cancelled everyone's plans and rolled liteLLM out internally. We kept it for the insanely cheap tokens, but now that they've switched to the new pricing, they're just like openrouter, just with far fewer models.
There is literally no more reason to use their service. I don't need a monthly expiring token pool drained at API pricing which I can get from any provider.
Must be the system prompts. Ask copilot to dump its system prompt, and compare the system prompt with claude. It is not accurate but handy. I bet they are quite different
Their harness is terrible compared to any of the other cli based harnesses I test against. Like shockingly bad.
This comes up all the time at work because the vendor management people don’t understand the llm ecosystem and think Claude through copilot is the same as Claude through Claude code.
A simple side by side comparison will show dramatic under performance 3 or 4 times out of five when I’m asked to explain the difference.
There won't be any new generation of models more powerful than Fable since the argument against Fable would apply even more. Opus 4.8 and GPT-5.5 is the best we'll ever see from this point forward. Soon low cost Chinese models will catch up to those thereby destroying Anthropic and OpenAI's pricing power which will mark the beginning of the end for them too.
They would have a golden opportunity to inflict damage to a geopolitical adversary. The US economy is being propped up by AI, I'm not sure they'd miss the chance to blow that bubble if they could.
there are ongoing tough competitions between China and America, that is for sure. however, a bold however, it is not in China's interest to see a crashed America. as an export oriented economy, China needs a stable and functioning America to maintain global order, that is how China got free lunch for the last 3 decades.
just imagine a world without the US acting as the world police, you'd be seeing armed conflicts in middle east, Africa, South East Asia, even in Europe and North East Asia. that would make China extremely hard to extract 1 trillion USD trade surplus a year, which is now required for China to maintain employment back at home.
without the US, even for a relatively stable global environment, trade won't be possible as most countries are not capable of providing goods and services wanted by China. Their currencies are literally junk (including Japanese Yen and Euro), Chinese are not going to take those junk in exchange for real goods. Trade is now possible because, by one way or another, those countries have USD to pay. those USD are backed by 300+ million highly productive Americans who repeatedly proved that they can create values in the scale of dozens of trillions a year.
the best part of this whole thing - America is singlehandedly footing the whole bill to provide such trade friendly environment for China for FREE. this is not cold war v2, back in the days of cold war, the US didn't help USSR to such extreme extent.
It depends on the end goal. Free good enough models are a way to drastically devalue Anthropic and OpenAI. A well timed release of a capable model that can run on obtainable hardware, so that a small/medium company can afford self hosting, has the potential to destroy one or both of these companies. This would narrow down the frontier model oligopoly and give the Chinese government a lot more power beyond its borders.
It really depends on whether the Chinese government wants to make good money or "win" the current AI bubbke.
> being open is not compatible with the Chinese culture.
Hardly, it's one of the least IP-law burdened places in the world. Ready access to media, yes, but also scientific papers, books, etc. No real restrictions on duping products, so execution often becomes the winning ticket. That's all pretty open and good for consumers.
You could argue they won't allow SOTA models to be exported but it doesn't really have anything to do with Chinese culture not being compatible with openness.
> Hardly, it's one of the least IP-law burdened places in the world.
that is one of the major reasons why companies there won't be open - they know full well that anything made publicly available would be cloned/copied within days.
it is not only a part of the competition common in all countries, there are unique reasons in China - millions of graduate engineers join workforce every single year, there are not that many projects they can work on. starting copying & cloning some existing stuff even at someone's own cost is a pretty effective way to get into the game.
> Ready access to media, yes, but also scientific papers, books, etc.
There is this old Chinese saying "Teach your apprentice and your own ruin follows" (教会徒弟饿死师傅), that has been telling a completely different story for thousands of years. When they don't even want to hand over tech know-hows to their own apprentice, why would anyone be expecting them to have the desire to share it publicly?
You can find Chinese sayings for almost any position. It's orientalism to reduce modern Chinese society/culture/economy to proverbs and sayings.
You say that you're Chinese so there's no such stereotyping involved, but actually Chinese people commit this sin against themselves all the time.
己欲立而立人，己欲达而达人
"wishing to stand, one helps others stand; wishing to succeed, one helps others succeed"
> that is one of the major reasons why companies there won't be open
But the AI labs _are_ often being open. And cloning stuff more generally doesn't really require OSS anyway. Product features are easily cloned in most cases, without any secret knowledge.
As OP says, it shines in constrained environments where the model is transforming user-owned data. Definitely less useful for anything more open-ended.
Yea I do not recommend treating chromes prompt API as a good example of local LLMs. It's fine and stuff but it's really weak. 8b models from a year ago are better in some ways. And a lot of the recent model drops are meaningfully better.
It's based on a Gemma 3n model, and yeah it's not the best. But if you have a use case that needs constrained JSON output for example, it's pretty neat.
Maybe it would do better with the new Gemma 4 models, which the Chrome devs have been hinting at moving to. And why the API doesn't let you introspect / pick the model, I'm still not sure.
> I've got some demos of what the new Prompt API can do:
> Use surrounding context to rewrite your ad copy:
Yup, that's the plan. No local model, no webpage; more, better and cheaper adtech extortion/surveillance for vendors while everyone else pays for the juice and hardware degradation.
So you're running an llm to do data transformation that deterministic processes would be much better suited for and running 1,000 watt power supply to do so. Wild.
> Don’t use the emulator to test the following:
> - IAM policies and permissions boundaries
> - Resource-based policies, such as bucket policies
> - AWS KMS key policies
> - Access across AWS accounts or across AWS Regions
> - Any other security control
> Some services in the emulator evaluate IAM policies for IAM users and roles that you create in the emulator, but not the same way as AWS. The emulator never denies a request signed with its own key. It doesn’t check AWS KMS key policies or Amazon S3 bucket policies. It also doesn’t support calls across accounts or Regions. So a request that AWS denies might succeed in the emulator. A request that AWS allows might fail.
This is a shame because I'd love for the official AWS product to be better than e.g. ministack and localstack on this, which will go some of the distance. I think because the agents were only building on the public API surface, it's hard to build this part. But they should've tried and let the agents read whatever policy engines AWS has. Or maybe it's a way to prevent someone from trivially running aws-emulator as AWSaaS?
[0] https://docs.aws.amazon.com/emulator/latest/userguide/securi...
reply