In those situations you basically need to guide llm to do it properly. It rarely one shots complex problems like this, especially in non web dev, but could make it faster than doing it manually.
I've done this multiple times in various codebases, both medium sized personal ones (approx 50k lines for one project, and a smaller 20k line one earlier) and am currently in the process of doing a similar migration at work (~1.4 million lines, but we didn't migrate the whole thing, more like 300k of it).
I found success with it pretty easily for those smaller projects. They were gamedev projects, and the process was basically to generate a source of truth AST and diff it vs a target language AST, and then do some more verifier steps of comparing log output, screenshot output, and getting it to write integration tests. I wrote up a bit of a blog on it. I'm not sure if this will be of any use to you, maybe your case is more difficult, but anyway here you go: https://sigsegv.land/blog/migrating-typescript-to-csharp-acc...
For me it worked great, and I would (and am) using a similar method for more projects.
"I also wanted to build a LOT of unit tests, integration tests, and static validation. From a bit of prior experience I found that this is where AI tooling really shines, and it can write tests with far more patience that I ever could. This lets it build up a large hoard of regression and correctness tests that help when I want to implement more things later and the codebase grows."
The tests it writes in my experience are extremely terrible, even with verbose descriptions of what they should do. Every single test I've ever written with an LLM I've had to modify manually to adjust it or straight up redo it. This was as recent as a couple months ago for a C# MAUI project, doing playwright-style UI-based functionality testing.
I'm not sure your AST idea would work for my scenario. I'd be wanting to convert XNA game-play code to PhaserJS. It wouldn't even be close to 95% similar. Several things done manually in XNA would just be automated away with PhaserJS built-ins.
Ya I could see where framework patterns and stuff will need a lot of corrections in post after that type of migration. For mine it was the other direction and only the server portion (Express server written in typescript for a Phaser game, and porting to Kestrel on C#, which was able to use pretty much identical code and at the end after it was done I just switch and refactor ba few things to make it more idiomatic C#).
For the tests, I'm not sure why we have such different results but essentially it took a codebase I had no tests in, and in the port it one shot a ton of tests that have already helped me in adding new features. My game server for it runs in kubernetes and has a "auto-distribute" system that matches players to servers and redistributes them if one server is taken offline. The integration tests it wrote for testing that auto-distribute system found a legit race condition that was there in both the old and new code (it migrated it accurately enough that it had the same bugs) and as part of implementing that test it fixed the bug.
Of course I wouldn't use it if it wasn't a good tool but for me the difference between doing this port via this method versus doing it manually in prior massive projects was such an insane time save that I would have been crazy to do it any other way. I'm super happy with the new code and after also getting the test infra and stuff like that up it's honestly a huge upgrade from my original code that I thought I had so painstakingly crafted.
The only model that works well for complex things is Opus, and even then barely (but it does and you need to use api/token pricing if you want guarantee it’s the real thing).
Agree on the gap - in my own complex greenfield software dev spec test, opus 4.6 blows codex 5.3 out of the water, by wide margin, both in ui and backend.
that only works if you can oneshot. but nobody can oneshot.
iterating over work in excel and seeing it update correctly is exactly what people want. If they get it working in MSWord it will pick up even faster.
If the average office worker can get the benefit of AI by installing an add-on into the same office software they have been using since 2000 (the entire professional career of anyone under the age of 45), then they will do so. its also really easy to sell to companies because they dont have to redesign their teams or software stack, or even train people that much. the board can easily agree to budget $20 a head for claude pro
the other thing normies like is they can put in huge legacy spreadsheets and find all the errors
$150k is about 2-3 manufacturing floor salaries for one year. I am quite certain many companies would prefer to buy a compliant robot slave that will never ask for a raise, take sick leave, or get demoralized than to pay 2-3 fulltime employees the value of their labor, only to have them leave for a better job 6 months later and have to train new ones all over again.
When dot com crashed it was similar - compute use didn’t reduce, far from it, it’s just capex crashed to the ground and took 90% of Nasdaq value with it, for a few years.
Stop comparing to dotcom. There’s a big difference between the internet then and the internet now. Ignoring the astounding nature of the AI tech (dotcom isn’t even in the same ballpark), you also have to consider these two things:
1) We didn’t make the world digital native in that era
2) We didn’t connect the whole world to the internet
They won’t have any choice soon enough - all that wishy washy they are not our enemy bullshit goes out the window with the first missile/shell/drone flying over.
If you remember Putin is a spy by training, and a damn good one at that, you must consider spies really don’t want to change things when they are advantageous to them. Right now he knows very well what levers to pull to make things happen the way he wants. He won’t change that.
