Fwiw, having taught online for well over a decade, I keep (kept?) trying to put people on to the marvel of a virtual machine for this (minus the sharing this offers, I suppose)
Just have a VM that you tell Zoom or whatnot is the "window to present" Now you can do anything, slides and more, and also that thing can be the only publicly seen thing, your computer stays private.
So there are a zillion of these, including ones that do what yours does. But that being said, I'm going to always upvote every single one of them because this is better and more important that what the big companies are doing across the board.
Thank you! Would love to hear which ones you've used. Working with local files without anything going to the cloud was the main thing I wanted. I tried Jan, OpenWork, Chatbox and Lobe, and couldn't get that working well in any of them. If there's one I missed it would be great to check it out!!
Have you tried Open WebUI? It doesn't determine which model works best for your hardware, but it otherwise has a very large feature set with a large community, and runs offline by default. Geared more for technical users, however.
It feels like the last mile is so long in this space. I'm a SWE with a ton of experience and I still haven't gotten over the decision paralysis of figuring out which model to run locally, and which inference platform should I use, and will my M1 Macbook Pro actually run anything well enough to be useful, and on and on.
It's kind of embarrassing to admit, but having something Just Work(tm) makes getting started so much easier.
It’s a huge new subject area and it’s completely exhausting to learn. I still feel like a total novice.
I have an M1 Max (more memory bandwidth) so I am not sure how much advice I can give about models, but I would suggest maybe trying the new LFM2.5-24B-A2B. It’s surprisingly good. If you want to mess about with multimodal work, Gemma 4 12B is really impressive. (I mostly use Muse Glimmer but it might be rather slow on an M1 Pro)
If you want to have the space to learn the terminology, tinker with what you can control etc., then you have three easy options: LM Studio, Unsloth Desktop or the llama.cpp native Mac app. But don’t get put off by the llama.cpp command line; most of these tools are running it behind the scenes, and most entries on Hugging Face have recommended parameters.
MLX is an option but so far I haven’t found it to have an edge over simple GGUF on Metal on the M1 at least; it seems it might have more advantages on the later M series chips, but it is usually slower for me.
I have explored running OpenCode inside a VM (you can “remote into it” from the OpenCode GUi desktop app). You might want to look at Paseo, which can do a similar thing but with many more coding agents.
I am quite intrigued by Osaurus; on macOS 26 and up it looks like it will manage a container for you.
You might also want to look at Orchard. Not done this myself yet but I can see the appeal.
I am an experienced web dev in my fifties. Six months ago everything I have written above would have been total gibberish to me. Learning some of the open weights stuff has made me feel much less helpless than I think I might if I just developed a Claude dependency. Nothing matters more to me than grasping the fundamentals.
I still don’t really use LLMs for hands-off coding. But I have found that a local LLM can be a useful sidekick; even at the most basic level it’s a better Google that remembers the context of queries.
Thanks for taking the time to write all this up, I'm checking out the projects you mentioned now.
I feel you on the tinkering-to-feel-less-overwhelmed front. I've been through a few ground shifts in my career so far, and even a little bit of "Oh, so that's how it works, ok..." can make it feel so much less daunting.
Mostly I think my own approach has been to try to combat my anxiety and exasperation about all of this by taking this measure of control.
While no really small local model comes super-close to a frontier model, the shape of the technology is settling and I think one can learn so much by approaching it the conventional nerdy way: what's a practical setup I can run on kit I already have? What can I build that helps me explore the problems?
I am quite cynical about AI still, including for generative coding, but I am not cynical about what I can run locally. There is no doubt that they are already helpful to me even in a minimal form.
And the really cool thing about being able to run them locally is to understand the tradeoffs.
Fwiw, I'm liking the koboldcpp "ecosystem", including koboldlite; basically it's letting me try a little of everything, though there may be something of a "jack of all trades master of none" thing going on. But it has these nice little presets for, e.g. llms, voice, music, images, a "chatgpt"-esque interface, etc.
It's interesting how I can be of two minds here, nearly simultaneously. Absent the horrific circumstances, my nerd brain says -- well, this is why phones should perhaps just be dumb devices that completely sync elsewhere such that physical possession is mostly meaningless. This also solves, e.g., border phone searches.
And also, yikes, don't actually want these type of horrific people doing this? It's difficult all around.
This is incorrect. Context and details matter in the law and we all agree that is a good thing. As an example - 1st degree murder, 2nd degree murder, and manslaughter are all different crimes even tho they’re all “killing a person” - the context matters.
Should the police wipe criminals phones? No.
Should the police wipe CSAM-adjacent data from phones to protect victims? Yes.
One is a consistent principal that is bad/cruel/overly broad. The other is a consistent principle that targets a specific harm.
Any sufficiently complex set of principles is going to have situations where two principles conflict with one another. This exact situation is one of them.
I mean, I was "there" for a lot of it in the modern-ish era, basically when air travel was like bus travel, buy ticket, get on plane.
FAR less distance and time between "car" and "plane." Closer parking, curbside service, but even if not, you're just physically closer to the gate. No TSA, etc.
This is a joke, right? "Flapping airplanes" is a great metaphor for AI as it exists today; e.g. we basically already have what we need and more is stupid.
Planes are awesome and important, and also building ones that actually flapped is technically possible and also wasteful and pointless.
For the record, I believe that the whole "building something as good as Photoshop" is BS, and a sort of retroactive justification of lock-in.
Yes, building something as complicated and locked in as Microsoft Word required a lot of work, and Microsoft has gained considerable money and power over it, but that doesn't mean that Word is necessarily good.
No different from "McDonald's hamburgers are objectively very good."
Outside of specialized niches, toll booths exist due to network effects.
Most of the revenue for Adobe is not due to individuals who need a competent standalone tool. It’s for organizations that need all of the usual qualities:
1) continuity with prior work (my old projects open properly, last year’s training is still relevant)
2) being able to hire/train employees on the tool
3) collaborating externally
4) interoperability across work products (eg vector, raster, video, etc)
You are not getting any of that from someone’s vibe coded weekend project.
Word is better than the alternatives. In Germany they keep going through the cycle of open source zealots mandating that government workers use off-brand office suites and then their unions going on strike to get Word back.
For a while I thought it was annoying that Word wanted to save my documents in the cloud but then I realized I could keep my lifestyle diary for my nutritionist in Word and update it on my iPad on the go and it works great.
I disagree with your use of the word zealot but I agree, MS Office is objectively the best office suite. I no longer use it daily, but anybody arguing otherwise has: never used it, never tried the alternatives to see how bad it can be, doesn't do collaborative work, or doesn't need any features beyond bold/italic. For example I can do most of my coding in Notepad, but it's still a very inferior experience. I feel the same way about MS Office vs LibreOffice.
That being said let's keep in mind that the latest wave of moving-away-from-MS-Office is using OnlyOffice this time and I believe it might work. OnlyOffice actually wants to entice MS Office users. And thus it is more compatible, has a much closer/better UX, and despite being a web app it is somehow more responsive than LibreOffice.
This is a pretty biased take. If there is anything the world is finally learning, it is that being completely dependent (not only for functionality but also availability) on a single vendor operating from a potentially uncooperative government, is not a great position to be in.
If there is anything else the world is learning, it’s that open source developers don’t have the same priorities as eneterprise or governments need for successful software use at scale.
I'm not talking about open source, I'm talking about the risk for non-US governments to be dependent on a sole supplier to be able to function, be it Microsoft or Starlink or any US-based corporation.
I'd say though that a lot of open source developers and technical people have a relationship with documents that has nothing to do with "word processing".
That is, a lot of us use plain text editors together with systems like markdown or restructured text or TeX. We never touch a WYSIWYG editor or if we do it is embedded in a web page.
If you look closely at how WYSIWYG editors behave (selections, cut-and-paste, etc.) you realize there is something conceptually wrong with the model behind them that causes them to misbehave, at worst they can seem like they are possessed by the devil. Word has problems but other editors, especially those ones embedded in web pages, are worse. I think plain-text editors in the open source world are fantastic but the WYSIWYG tools are awful... and plain-text tools are kind of a "blub" vs WYSIWYG tools, but you can live a great document life with them.
Great article -- and maybe this is just me -- but I do find the primacy of the online services interesting. Me and my 16gb vram card can get most of this done at home for "free."
Which the AI then passes on to a user who doesn't know any better and then you get recipes for mixing bleach and ammonia for curing asthma (extreme example).
If I write a story that includes a character mixing bleach and ammonia for curing asthma and publish it on my website, am I responsible for an LLM company ingesting my story without contact or my consent and then at some point in the future providing a user instructions to mix bleach and ammonia for curing asthma?
If you do this with the intent of LLMs scraping this and this ending up in the corpus used by LLMs, it is indeed very illegal, especially as LLMs become more load bearing for medicine, defense, etc.
very illegal? under what law? the one that's embedded in the us constitution that asserts that congress shall make no law affecting freedom of speech (and is incorporated against the states)? seems highly unlikely if so
On the contrary, I'm saying is that it's not okay that the world becomes worse-off, as long as it's not [the LLM companies'] fault. If it means they don't have a viable business model by only ingesting data that they have permission for and is verified, and by being held fully responsible for the content they serve their users - in order to prevent the world from becoming worse-off - then they should not operate.
This seems like a two wrongs don't make a right situation. Yes, the LLM providers are in the wrong for not vetting data. But in my mind, the person poisoning data is also partially responsible for negative downstream effects that are caused. What % of the blame is up for argument, but neither side is 0% in the wrong.
So the type and content of what fiction I write should be controlled so that LLM companies may choose for the sake of maximum profit to haphazardly collect any data they please and to neglect the safety and validity of the content they choose to serve users?
If it's meant to be fiction and not presented as a real-world fact, then I don't see any problem on your part. If you're presenting it as fact solely to trick an LLM into believing it, then yes I see a problem.
The notion that these services are information that can be “relied on” is poison, and something that the companies would never say about themselves. Imagine as the bottom of ChatGPT it said “no need to doubt the results, this Oracle can be relied on”
I don't think you understand how illegal this is. As these models start being used for autonomous decisionmaking in medicine, defense and whatnot, this will be considered industrial sabotage and if it ever harms someone you are so fucked.
I'm not even debating the morality of AI and LLM companies here, this is just fact. If a pharma company is illegally using your water source, you cannot poison it knowing the pharma company will then use this water. Well, not unless you like prison.
If writing harmful information online (including wikipedia) was something illegal, wouldn't there be just a ton of currently illegal shit online? Like anti-vax stuff?
At present, nobody does. You'd have to see what a court case says.
But it would have to be the case that it's reasonable to expect people to rely on an LLM for medical advice. Which would in turn require that the creator of the LLM is not responsible.
It should be illegal to make life and death decisions based on automated reasoning. I know it’s coming and I’m being a curmudgeon but I will hold fast, the liar is not responsible for being believed.
I like the Wikipedia comparison, forget about the automated reasoning. Wikipedia does not make any representations as to the truthfulness of its content. If you’re a medical doctor and believe some wiki graffiti, you are the responsible party, you are the one that’s supposed to know better.
If you want to rebut I’m most interested in knowing what law you believe would be cited in a data poisoning case.
One day the automated decisionmaking systems will be far better at making decisions than us. That day is not far off. At some point, we will turn to these systems for our most important life-and-death decisions. I don't think that's irrational.
In this era, intending to corrupt the automated decisionmaking systems we rely on, will certainly be sabotage (if done with that intent, as is the case with OP). However, I don't think any sabotage would actually work well in the era of superintelligence, so perhaps the point is moot.
Poison? Seriously? As if false references, perpetual false confidence, sycophantic suicidal encouragement, and convoluted spaghetti code wasn't enough. 90% of what these things fling is poison and the quicker the general public recognizes it the quicker they won't be duped into taking them at face value. The training sets are already going to be full of AI generated garbage. This just hastens the recognition.
And for this the company responsible for the AI disseminating this information in such a way that people are relying on it should be prosecuted and/or sued into oblivion.
The original poster of the bad information should perhaps as well, but only also if it was reasonable to rely on them for this.
(We have the legal principles, we just need to use them.)
Just have a VM that you tell Zoom or whatnot is the "window to present" Now you can do anything, slides and more, and also that thing can be the only publicly seen thing, your computer stays private.
reply