From my own experience, LLM generated code having more test cases is neither surprising nor an indication of improved quality. I've found LLMs are quite good at making lots of test but the tests are often not that useful. I would argue that total count remains a poor metric for quality.
Best metric is user satisfaction. Since that is ultimately what software is (or should be) about. Hence the leading header of "software is getting worse for obvious reason": we detached from user satisfaction as the metric to optimize for at least a decade ago.
Just simply ask for that, and they usually know what to do.
They should write a test and stash the fix they just wrote. In this state it should fail. Once they re-apply the patch, it should become green and this gave use enough information to deduce that the test does test something.
> Governments also get to hide real expenses that audited private enterprises never could.
You present this as something inherently true, but really there is no reason for this to be the case. We could make it so government entities are significantly more transparent than would be reasonable for a private entity. In practice, that is already the case and what you said really isn't true, though I'm always in favor of more transparency.
Yup, the input lag is not something I can stomach. Even in pretty optimal network conditions, it's very noticeable. Other people may be less sensitive to it than me, so they may find this to be nice. I do worry about a longer term trend of pushing cloud over local hardware for gaming, which was never something I seriously considered until recently with hardware prices being what they are.
> But if it truly was properly reported on prime-time news and so on, they would know, and they would care.
I don't know a single person that isn't aware of the issue, it's just a significant portion of them think it is outright fake or they just otherwise don't care because they see it as a lame thing to care about. How does one fix that side of things? It seems a major impediment to any progress.
Indeed, though this is platform dependent. Mainframes have strong hardware support for decimal fixed-point arithmetic, making it the preferred representation for fixed-scale business data, though conceptually it's very similar to binary fixed-point.
Decimal floating-point is also hardware-supported, so it doesn’t carry the same computational overhead it does on platforms where it must be implemented in software.
For many years, nobody has supported decimal numbers in hardware, except IBM (the deprecated instructions of Intel 8086 and 8087 do not count).
IBM has done this, despite it being an inferior technical solution, because it binds those who choose it to IBM hardware.
Programming currency operations with decimal floating-point numbers is easier for naive programmers.
Even on IBM mainframes, implementing currency operations using 64-bit or 128-bit integers is much faster and less resource-consuming than with decimal numbers, but the implementation of some of the operations can be a little tricky, when it must be guaranteed that no loss of precision may occur.
I think that I might have seen recently an announcement from someone else than IBM who has introduced hardware support for decimal floating-point numbers, perhaps from Fujitsu. In any case, whoever introduces such hardware support does it to lure some customers to migrate from IBM to them, and not because it were a good solution for implementing operations with money.
> What LLMs make possible is for me to say: find out all the ways this thing works.
They are not good at that. The space of possibilities can be massive and LLMs are terrible at exploring such space because they predict from the prior tokens they made. They are inherently bad at exploring new space because it is antithetical to how they work.
To me it's deeply concerning that so many people are getting fooled into thinking that LLMs are actually good at covering their bases like you're describing here. It's one of their weakest qualities.
> Analyze the different ways we can run this software, build a fuzzer, build property tests, and run this software in every scenario possible.
It usually does quite a bad job at this too, often the tests it wrote feel like that of a lazy student that didn't really want to do the task and just sort of cheats at it or does a really shallow job. It certainly cannot run the software in every scenario possible.
> Anyway all of this reads like someone who is not actually using LLMs to build software or hasn't tried them in a while
This is something that is said by everyone who contests anyone pointing out the risks in being overly trusting of AI or otherwise points out their flaws. I use the latest and greatest all the time and all the time I'll point out something that it totally overlooked and get hit with the classic "you're absolutely right". This occurs because I actually read the code and can see the myriad of blatant issues that still occur when using LLMs and know better than to trust them. You will find so many issue by delving into the details.
People ask AI to do things either they haven't done, cannot do, or don't want to.
So when the AI produces something that basically works and looks like a plausibly good attempt, they jump to the conclusion that AI must be excellent at that task.
Huh... not my experience with the "premium" license. The interface isn't as nice and definitely less powerful than something like codex/claude code/whatever, but I don't seem to have any issues with truncation or things like that. I'm usually using 5.6 thinking (not clear which 5.6 model exactly) when I'm using copilot at work. It seems... fine? I always see people here always complaining that it's nerfed, truncates things, etc etc. The only difference I see is that interface isn't as nice nor as powerful, but I don't notice the answers being noticeably worse or anything like that. Occasionally it messes up code block formatting, but that's the only thing I can think of that annoys me.
Now, the "copilots" integrated in certain Microsoft products can be really horrible. I assume those are some sort of custom models that are not very capable, most of them I consider effectively worthless. But the web chat interface where I can pick chatGPT 5.6 seems fine...
Given you're comparing to codex/claude code you probably arent talking about MS Copilot (e.g. the thing inside MS Teams but also available standalone as MS Copilot 365). Try it out and be convinced :-)
I don't think there is a misunderstanding. I don't recall ever having an issue with it either. Now I don't do anything extreme with it. Just questions, suggestions, or some basic task which are either Microsoft or basic business activity related. I haven't tried anything like coding, deep analysis, or anything complicated. It feels like it is for business tasks so I haven't really tried to push it to do anything outside of that.
I think I am! I use it in the browser instead of teams/the desktop app, but my understanding is that it should be the same thing. I do know most people don’t have the premium license for it at my company, not 100% sure what difference that makes but they say it’s better somehow and has access to better models, or at least did at one point.
But yeah, comparing to codex/claude code was a bit silly since those can do way more. It’s closer to the chatGPT. I’m pretty sure 5.6 Luna, not sure exactly since it doesn’t specify Luna/Terra/Sol
One difference is that the premium copilot has access to company files and those on your onedrive, whereas the non premium version requires you to upload files.
reply