It doesn't need to be paid though, it's just a result of poor design and poor documentation. It's the only OS where I feel like I'm both 20 years in the future and 20 years in the past.
I think it's also an issue with the flexibility paradox. There isn't a good single way to package Python for Nix because before you even start there are several key questions which come up that most other distros can't even begin reasoning about:
- Are we packaging just one Python package for Nix, or a thing and all its dependencies?
- Is the package already on PyPI, and are we packaging the source from there, or is it the source from some upstream?
- Are any of the dependencies coming from source, either their upstreams or forks?
- For dependencies that already exist in the nixpkgs SHA that is supplying the Python interpreter, do we want to use those versions, or package newer versions? Is it the same decision for everything, or does it vary?
- Is there already a Python level dependency locking scheme in place such as poetry/pipenv/uv, or is it a plain setuptools package? Is it acceptable for Nix to become the locking mechanism, or will that mess things up for non-Nix consumers of this?
- Are we looking to supply a development environment here too, or is this purely about deployment?
To be clear, none of this is an excuse— it's horrifying that there can't be a single "Tool X is the singular Python-on-Nix story, follow the one page tutorial and you're all set". But I think the massive amount of choice and flexibility is the crux of why new methods and tools are still being rapidly invented and proposed.
For myself, I would choose poetry2nix as how I'd ship a Python project to Nix hosts, but that immediately implies some answers to a bunch of the above questions, mandates poetry for your top level project, and once you look closer there turn out to be some truly horrifying implementation details [1] that are what make poetry2nix appear as seamless and friendly as it does.
I can't think of a single external packaging system that actually does Python non-painfully. Then again I can't think of an internal Python packaging system (there are like 5 now) that does Python non-painfully. But since one of the selling points of Nix is "it's like virtualenv but for everything", it's a little disappointing that it doesn't mesh better.
I'm not a Python developer, though I use applications that require Python libraries. I don't want a separate numpy for every application I use tucked away in my home folder somewhere; I want a system numpy provided by my distro that can't get out of sync because there's only one copy of it anywhere.
I think the Debian model is okay, it just falls down in terms of being able to have any kind of meaningful cooperation between the debs and what pip does, and of course there's the massive velocity mismatch between what's on PyPI and what's in your distro.... and of course no way to install more than one instance of something so heaven help you if you have two down-tree packages that depend on a different version of a thing.
And this isn't new: it was true for gems with Ruby, it was true for pear with PHP, it was true for CPAN with perl. I don't do javascript at all but I assume it's doubly true for npm just because that place seems to make even pypi look conservative and glacial. I've mostly come around to the belief that if an piece of software has a package manager you should just use it instead of your system package manager. But man do I miss the days of CPAN packages that might update once a year.
Generally I'm in agreement, but the one place that starts to fall apart is with bindings— if there's a native code library, then I really would prefer that library come in from the native package manager rather than be smuggled in by pip... but that's not generally how it works these days, and outside of Nix there isn't a great way to resolve this case.
At the same time Nix implements python packaging particularly bad[1] when it does not have to. And that's on top of already insane state of python packaging. And poetry2nix becoming deprecated really bit me recently :/
Similar issues (minus insane packaging) is with Common Lisp packages, where there's annoying focus on compiling libraries into binaries (it would have been fine to have a bit of patching to provide things like correct paths for ASDF and CFFI search paths, but no....) which gives absolutely no benefit.
I will say, there is one element of Nix's core design that makes almost all of these integrations considerably worse than they have to be, and that's the whole IFD fiasco— if it were possible to ingest an ecosystem-specific lockfile, run a build on it, and evaluate that result, then a lot of the drama and workarounds would melt away. This is sort of supported today, but it's basically banned from nixpkgs because of how the current implementation pauses evaluation any time it needs to run a build that evaluation is dependent upon... and it can only run one such blocking build at a time.
The new rust implementation (tvix) is addressing this as one of its core design goals, allowing evaluation and builds to run in parallel. But even once it's complete and usable, it's unclear what the relationship will be between tvix and Eelco's Nix.
Another is that Nix as language is yet another of those "functional programming sounds fun, let's do bare minimal pure language and forget about all the niceties that proper ones figured out".
I think my argument is that most of those issues are relatively shallow— they're problems that could be addressed with tweaks to the syntax or additions to the standard library.
The IFD issue is fundamental. It's why poetry2nix has to be implemented entirely in Nix code in order to run at evaluation time without blocking, whereas in a world where eval and build could properly interleave, this would never be done— the core logic of poetry2nix would be implemented in something sane.
there is a very real sense in which nix has been riding on a good idea while ergonomics elsewhere in the field have advanced, yes. make no mistake, the technical work, getting programs to behave themselves in a fairly alien environment etc, has been impressive, but it's like in the marathon to get there, the idea of making that process somewhat nicer has been subject to constant procrastination. which sucks since in some sense the lay packager is working with the exact same tools as the people bringing the whole system together. not just nixpkgs dx has fallen down the list of priorities, but technical debt down to nix itself has accrued as well. the tvix effort emerged from dissatisfaction with instability (the same dissatisfaction that kept the actual version of nix used in a typical nixos install well behind master) well before any administrative/sponsorship struggle snuggle, and i still maintain that an effort more conservative in scope such as lix would have emerged politics aside, as again, what the end user sees switching to that is mostly "oh hey this random thing that repeated segfaults conditioned me out of attempting just... works now lol?" or various ux papercuts just ceasing to be. nix-at-large has a ways to go but i am optimistic
Thanks for the perspective. I mainly use NixOS to run my server's not personal machines. I can see why it can be a frustrating experience for a machine that you just want to run personal stuff on.
In my instance of creating server machines(cattle), the configurations are pretty light and what's important is the reproducibility aspect of it. If I need to take one down and rebuild another it takes about 10 minutes. All the upfront work of configuring Nix for that one machine has paid off.
I'm a big nixos fan myself, and I appreciate your post, I don't do any of the socials listed on your site and noticed the word 'serice' if you wanted some backseat editor. Apologies if hn comment in the wrong place for the feedback.
Yeah I agree it's manual but it takes about 5 minutes to SCP the password onto the machine.
I have some playbooks I setup to creating a new machine. All in all it takes about 10 min to get it up and running. Maybe not instant but at the moment I don't need anything else.
Mostly trying to get rid of longhorn. I've found it to be a continuous source of troubles w/ etcd sync & IO issues. Current iteration of cluster is all optane so might give longhorn another go, but still want to move storage off cluster.
Especially source repo needs to live on some sort of striped zfs array. Ordered one of these quad nvme NAS things [0] so that's probably going to be storage. Either nix or proxmox...not decided.
Also still a bit fuzzy on what best game plan on PVs is. Minio/s3 or nfs appear to be options.
DB...just the usual suspects...mainly postgres for gitea I think. Mongo for dev stuff. Used to vanilla proxmox/docker/lxc so this is all unchartered territory for me.
I could never get around on longhorn or any of the other "lightweight" K8 distros. I guess that's why I spent time setting up NixOS and writing this post.
Okay nice, seems like various storage services for a home lab setup. Seems like a cool project. Especially if you can distribute it across all those NAS's
To each their own. When K8's is managed it's awesome.
I would like to do without the headache of dealing with K8's installation or some orchestrator layer. I can "schedule" my on application instances at the size I'm working with.
Also, I run this setup on cloud vm(multiple actually). So it's not restricted to running on a single machine running in a closet.
I'm using Cap'N Proto in a message broker application(LcuidMQ) I'm building for serialization. It has allowed me to created client applications rather quickly. There are some quirks can be difficult to wrap your head around, but once you understand it is really solid.
There are some difference between the language libraries and documentation can be lacking around those language specific solutions. I'm hoping to add blog articles and or contribute back to the example of these repositories to help future users who want to dabble.
I've been working on something that may fit your criteria. The API is kafka-like and it's whole goal is to provide an easy to use local-first stream datastore/data-engine.
It currently only has support for Rust and Python clients but working on expanding to node(JS) and C/C++ in the future.
I'm a dev-ops engineer by day. I creating a simple to standup and use message queue(similar to kafka and rabbitmq) in Golang. As of right now, the project is solely to learn more about Go and internals of the commit log.
Interesting, I’m working on a kafka-like SaaS, in golang also. For the moment I’m keeping events/messages in postgres, but I have always targeted to have a real commit log per topic in some unspecified future. I will take a look at what you’ve got!
