A self hosted web archiving tool with support for extendible processing pipelines (eg. extract article -> translate -> summarize -> generate tags, download video -> split audio track -> transcribe -> summarize), which led me to make a managed chromium browser with extensions and warc support for archiving, and a RSS feed synthesizer (take random article listing page that doesn't have RSS and generate a feed for it) so that I can plug it into my archiver. An active learning loop for a model to clean up articles by removing junk like native ads and sponsored blocks.
A tabbed terminal with project management features like launching the database, app server, and claude code in different tabs with one click, and split browser/terminal panes (eg. opening a browser automatically at the correct URL when the terminal reads http://localhost:4000/).
A modular MCP server with a MCP proxy and OAuth2 dcr so that I can easily add new random ideas for MCP servers in a few minutes with Claude and deploy them such that it's available to Claude by refreshing the tool list.
A small tool to render Claude conversations so that I can link to them from my obsidian vault with something like convo://claude-code/-home-jfim-projects-foo/<guide>
And overall just deploying docker containers for my self hosted setup
Most of it is on GitHub, in various states of readiness.
The article makes a lot of points about cost viability, but says nothing about what happens at the end of life for space datacenters.
On Earth, the materials and equipment in the datacenter can be repurposed, recycled, or properly disposed of. In space, EOL'ed stuff either stays in orbit, burns in the atmosphere on reentry, or moved out of useful orbits.
I'm not sure I'm thrilled at the idea of more space junk in orbit or more aerosolized metals in the stratosphere.
Indeed. It's pretty interesting to realize after implementing GPT-2 that the frontier models are scaled up versions of that, with various tweaks to improve performance, model-wise.
The secret sauce though is all the datasets, RL training, knowledge of what works from doing all kinds of ablation experiments, and a massive compute moat.
The secret sauce is also having the necessary 'creativity' to not get ceased and desisted into oblivion and jail from all the copyrighted material you trained your model on. Btw, not making a moral judgement, [0] shows Michael and Dalton from YC discussing why Ilya Sutskever had to leave Google to pursue what's now ChatGPT
There is a whole moral judgement to be made here...lets hope Ilya wont get too pissed off if somebody leaks the work of his new initiative...information wants to be free and all that...
Also would love to know if the same Legal team advised on Gemini...
How do we know that today's frontier models are merely scaled up versions of that? Genuine question, since the labs have narrowed what they share over the years to now almost nothing, in terms of how the model was trained and how it works under the hood.
We know for sure the architecture of the open weights models since llama.cpp understands the architecture it needs to build to plug the weights into to run them. It's always possible that the latest closed model is doing something architecturally different than the open weights ones we know about, but judging by how close the large open weight models such as DeepSeek are to SOTA performance, this seems unlikely. When OpenAI first came out with their near-mythical "Strawberry" (aka "o1") thinking model there was all sorts of speculation that they had made some sort of architectural breakthough, but then DeepSeek replicated the capability and published how they did it, proving that it was just better training, not any architectural change.
There have been minor changes to the architecture over the years, but these are basically all efficiency tweaks such as various types of attention (some pioneered in the open by DeepSeek) that better scale to large context lengths, and the confusingly named "mixture of experts" architecture, but what's more notable really is how little the architecture has changed. The capability gains have been coming from better training and better data.
- R1 https://arxiv.org/abs/2501.12948 (RL applied to ML models was well-known beforehand, but they show it in the open, at scale, on big models)
Then, there's the incentive analysis. If you can see that these models empirically get better with scale, why would you swap the main architecture? Those events will be pretty rare. I'm not saying there's noone cooking a new architecture, just that it is a pretty rare event. And it would have to come from some researchers that would be happy to not publish their findings, which is not really what a sizable portion of elite researchers (obviously not all) are incentivized to do.
Of course, it's a bit of a verbal compression to claim simply 'scaled up'. They are recognisable scaled up transformers, but most new models come with a few tricks, but we're at the point where those usually are not an architectural rewrite and added to solve an explicit problem, like hallucination, not for big new capability gains.
No they are clearly not just scaled up versions of gpt 2; there are different LLM architectures like mixture of experts etc that appeared relatively recently. I am not an expert though, far from it.
separately trained experts can surpass performance in their activated regime and DOES result in a smarter model, the Claude system cards talk about this and eg there is https://openreview.net/forum?id=iydmH9boLb to read...
If you can make the existing model faster, you can then save your inference budget to then make your model bigger, which then makes it smarter.
A lot of how smart the models can be comes down to budget. If you can make your existing thing cheaper, you can instead make it bigger for the same price.
Not really “smarter” though? It’s just a big probability engine.
(Not trying to flame bait or anything. I just wouldn’t call LLM as exhibiting intelligence. It is great at making connections based on probability but doesn’t have a semantic understanding of what it is doing)
You do realize modern neuroscience considers the human brain as "just" a probability engine and that intelligence may well be the ability for an organism to predict well.
> doesn’t have a semantic understanding of what it is doing
I hope you realize this is an area of open, active research.
Didn't neuroscience some big scandals about bad statistics and overstating their findings (in addition to normal issues like replication)? Look up at least the "dead salmon study" (hint: it's related to fMRI, and you can probably guess its conclusions from its nickname).
The "Voodoo Correlations" and "Cluster Failure" papers are also a bit eye-opening.
In general we (humans) need to be humble about the limitations of our knowledge about how we function, it's an insanely complicated problem.
> The secret sauce though is all the datasets, RL training, knowledge of what works from doing all kinds of ablation experiments, and a massive compute moat.
ReAct loops and tool-calling are the critical development feature. They turn a model from something that generates text into something that can independently influence the world around them.
The big breakthrough is we can interact with the agents using natural language - because of the LLM.
It is the combination of LLM and agent-harnesses that make it look really smart. Agent-harness is a programmatic device that lets us tap into the vast knowledge in the LLM.
It is probabaly true that many TV-commentators fail to appreciate this fact and therefore think LLMs are super-intelligent. No, it is the combination of LLM and the programmatic agent-haness that is the breakthrough.
An interesting thought is that the LLM could in theory code the agent-harrness, start it running every time we interact with it. Currently the agent-harrness I think is pretty static I think. In theory it could be dynamically created for every task. Would that make it better don't know.
> The big breakthrough is we can interact with the agents using natural language - because of the LLM.
Without ReAct and tool calling, all you have is a chatbot. That's useful, but it's just a chatbot.
ReAct loops and tool calling is what unblocks high value usecases. It enables systems to actually address free-form problem statements, gather data that is not a part of their training set, inspect the current state of services,and trigger actions in external systems. This goes well beyond mere chatbots.
> It is the combination of LLM and agent-harnesses that make it look really smart.
It's really not about "smart". It's about autonomous systems, and being able to consume and analyze new data, and trigger actions in external systems.
It's not very novel, though, it's a fairly obvious step once you get something that can operate iteratively and largely independent, there were a ton of people trying to get LLMs to loop on their own even before deepseek r1.
And I remember talking about goal directed behavior (which what people are calling "agents" now don't seem to properly have) and autonomous operation decades ago in the intelligent agent course at uni, including react loops.
So no, the huge step with LLMs really was just that attention mechanism from that translation paper everyone forgot until Google brought its marketing to it, everything else is either just optimization/scaling, more money or old ideas suddenly relevant.
I completely disagree. The rollout of agentic tools, and even support for agent mode in IDEs, is the whole value proposition of AI code assistant services.
Otherwise you'd just have a glorified search engine in a chat window.
> (...) it's a fairly obvious step once you get something that can operate iteratively and largely independent,
There's some confusion in your reply. ReAct loops is exactly what this "operate iteratively and largely independently" represents.
The good thing is since the feature was cheap to implement, you can just say "this was a bad idea" and remove it, as long as adding that feature wasn't a one way decision. People are typically more reticent to remove things that were hard to implement, even if that's the right thing to do.
That's the problem, even with an LLM, removing a feature two weeks later can be a nightmare because things have grown to depend on it. In a way it's even harder because the velocity of stuff piling in is much greater.
> you can just say "this was a bad idea" and remove it
Have you ever actually done that in a "serious" product, or just made it up?
In any product with actual customers, especially those from other (big) companies, features don't just go away at the snap of a finger. Otherwise, have fun discovering users moving to your competitors.
Anecdotally, a product from another company had removed or made significant changes to important features our users rely on every day multiple times with short notice, several times. We didn't hesitate to migrate to another service, which completed within about a month.
> People are typically more reticent to remove things that were hard to implement, even if that's the right thing to do.
Careful. The sunk cost fallacy isn't just about time, it's also about money, and people may naturally be reluctant to remove bad features that cost them a lot of tokens, especially if the act of removal itself is going to cost even more tokens.
I would never trust Microsoft. Their next drama is revoking Office 2019 perpetual licenses https://www.youtube.com/watch?v=KRnno9VIZx0. It never ends with them because they know they have you by the balls.
Prototypes aren't only for UX though, sometimes they're for exploring whether something is technically possible, or what are the unknown unknowns in a particular area.
For example, for personal projects, I've been wondering if it's possible to automatically create RSS feeds for pages that don't have them (yes), what are the challenges when building an archive-style page dumping system (need to dump CSSOM alongside getOuterHTML, remove/rewrite remote content, walk iframes, automate Chrome, scroll to load lazily loaded content, etc.), and if training a model to remove native ads from markdown coming from readability is possible (no, at least not with my current approach, but using the dom might work).
A few reasons. Learning is one of them, since I don't normally deal much with browser and web related technologies, so it's a good way to learn more about them.
I also think there are a few interesting things you can explore that go beyond a simple carbon copy of what's on the Internet. Ideas that I've implemented are things like automatic extraction of audio tracks, transcription, and summarization, loading a page or podcast transcript into the context window of a LLM to discuss the arguments or factuality of the claims being made, automatically turning articles to reader view using readability/trafilatura, etc.
Directions I'd like to explore would be things like multimodal search ("that page I read six months ago about computer security with neon green text on a black background", or give me a list of fitness related pages I've read in the last twelve months), personal statistics (how is the mix of topics I've been reading about changing over time), annotating pages instead of just passively reading them, maybe even P2P archiving or discussions about pages, and all kinds of other things.
Archived version
This repository is a mirror of the version currently found on source forge, duplicated here in an effort to no help preserve this tool into the future. I, Julia Desmazes, have not contributed in any way to the development of this tool, all credit belongs to the original author.
In addition to being a command line tool, this tool also used to be available interactively though the now defuncts OutputLogic website.
It's bumping to manager level, except without the 1:1s, quarterly/yearly planning, headcount and budget reviews, org/reorg discussions, performance calibration, and OKR planning. No complaints about the last review cycle or about the upcoming one.
Totally!
But you know what? There are many, oh so many developers that are not ready, don't like and probably are not even cut for this kind of position.
A self hosted web archiving tool with support for extendible processing pipelines (eg. extract article -> translate -> summarize -> generate tags, download video -> split audio track -> transcribe -> summarize), which led me to make a managed chromium browser with extensions and warc support for archiving, and a RSS feed synthesizer (take random article listing page that doesn't have RSS and generate a feed for it) so that I can plug it into my archiver. An active learning loop for a model to clean up articles by removing junk like native ads and sponsored blocks.
A tabbed terminal with project management features like launching the database, app server, and claude code in different tabs with one click, and split browser/terminal panes (eg. opening a browser automatically at the correct URL when the terminal reads http://localhost:4000/).
A modular MCP server with a MCP proxy and OAuth2 dcr so that I can easily add new random ideas for MCP servers in a few minutes with Claude and deploy them such that it's available to Claude by refreshing the tool list.
A small tool to render Claude conversations so that I can link to them from my obsidian vault with something like convo://claude-code/-home-jfim-projects-foo/<guide>
And overall just deploying docker containers for my self hosted setup
Most of it is on GitHub, in various states of readiness.
reply