This is a problem we, as a company, have thought about a lot, but we always concluded that Kubernetes is already the simplest abstraction of a distributed system that is feasible for the diverse needs that the biggest companies out there have.
We previously built a package manager for Kubernetes to abstract it in the simplest way possible `glasskube install app` but we failed because every abstraction needs to follow a "convention over configuration" pattern at some point. Also, we weren't able to monetize a package manager.
With Distr (https://github.com/distr-sh/distr), we have actually been able to help companies not only package but distribute and either manage or give their customers a way to self-manage applications. Our customers are able to land on-premises contracts at enterprises way faster than before, which is also a clear ROI for paying for Distr.
So, I don't think that you can get the flexibility of a distributed application orchestrator with a simple declarative YAML file if your target environments are diverse.
Same, I've tried three or four times to make it work, including one attempt that just translated compose.yaml into k8s yaml, and every time I came away thinking, "just use k8s". K8s yaml looks complex, and can start to feel very boilerplate, but attempts to hide the complexity often just lead to something not-flexible-enough because it encodes convention over configuration, and inevitably some project runs into limitations and pretty soon you've just built an abstraction layer that leaks or is equally complex/verbose and now you have to learn something new.
Just use k8s and follow similar patterns is the conclusion I've arrived at personally.
To be honest I never really understood the benefit of Docker (Compose) Secrets - which is different from Swarm Secrets. Imho there just plain host mounted volumes, which are hidden from inspect commands?
GP is talking about podman with generated systemd unit files (a.k.a. podman quadlet[0]), not the docker-compose-compatible podman-compose ...and I'd agree, systemd can manage services on a system just fine, and even better than any compose workload ever could.
journald will help with logs, and the pull policy[1] helps with mutable tags. What help do you need with "orphan containers"?
I'm not OP, but the whole podman compose topic gets quite confusing, as initially Podman didn't seem to know what they were trying to do. I've given some more context around it in previous comments.
You shouldn't be using podman compose. It's flimsy and doesn't work very well (at least it was last time I used it prior to Podman v3), and I'm pretty sure it doesn't have Red Hat's direct support.
Instead, activate Podman's Docker API compatibility socket, and simply set your `DOCKER_HOST` env var to that socket, and from there you can use your general docker client commands such as `docker`, `docker compose` and anything else that uses the Docker API. There are very few things that don't work with this, and the few things that don't are advanced setups.
For what it's worth, podman has also a thin wrapper (podman compose) which executes `docker-compose` or the old `podman-compose`. The docs should explain which it picks.
Note:
- `podman-compose` is an early attempt at remaking `docker-compose` v1 but for Podman. This used parsed the compose config and converts them to podman commands, and executes it.
- Later Podman wrote a Docker compatible socket instead, which can work with most docker clis that accept a `DOCKER_HOST` argument, including `docker` and `docker-compose` (both v1 and v2)
- `podman compose` is a thin wrapper that automatically selects `docker-compose` or `podman-compose` depending on which is installed.
Generally all you need is podman, docker-compose (the v2 binary), and that's it. From there you can use `podman` and/or `podman compose`.
One of the nastiest aspects of migrating from docker to podman really is "what to do about docker compose?" coz there are three wildly divergent ways to answer that all of which really suck under certain specific circumstances.
Im no fan of docker and podman by itself is a step up but orchestration headaches are enough to ruin that.
I don't understand what you're asking here. The answer to that is probably nothing. That is unless you want:
- systemd to manage your containers
- You want to use K8s primitives (which are mostly compatible)
I'm unsure what the 3rd method is you're talking about. The nice thing about Podman's compose API is you don't have to change anything (mostly). You can point all your docker tooling to Podman's socket, and it'll (mostly) magically work.
* use systemd, red hat's favorite kitchen sink for handling everything from setting up sound services to mounting your home dir to logging so why not this too i guess.
* docker compose where i have to run a whole separate podman service to lie to docker compose about not actually being docker.
* podman compose which would be the obvious solution if it didnt just plain suck.
> * use systemd, red hat's favorite kitchen sink for handling everything
Systemd is a tool for managing services. Containers are services. Why require an entirely separate bespoke service manager when you're already running one?
> * docker compose where i have to run a whole separate podman service to lie to docker compose about not actually being docker.
This is the same system state as using docker compose with docker: you have a client program speaking to a backing daemon. Only difference here is the Podman service, being daemonless, only runs when needed (assuming you're setting up things the documented way by enabling the podman socket).
> * podman compose which would be the obvious solution if it didnt just plain suck.
Yeah I haven't had the best luck with it either. But part of the reason it's languished is that it makes more sense to just reimplement the Compose spec on the backend rather than re-invent the wheel and create a new compose client as well.
There's also the fourth option of writing Kubernetes yaml and applying that with `podman kube play`. Honestly this is probably closer to being the podman equivalent of docker compose but since it involves writing The Bad YAML (kubernetes) rather than The Good YAML (compose) most people don't use it.
It's a tool for user age verification that happens to be something you can use to manage services.
Did you miss my point about it being a filthy kitchen sink?
>This is the same system state as using docker compose with docker
One of the major selling points of podman is that you dont need a daemon. except maybe yes you do because podman compose sucks so toss that selling point in the trash.
This shit is also incredibly fiddly. Ever had "docker compose" and "docker-compose" do subtly different things which drive your team mate to pull their hair out? I have.
Podman should stop trying to piggyback off docker if it's trying to be an alternative.
>Yeah I haven't had the best luck with it either. But part of the reason it's languished is that it makes more sense to just reimplement the Compose spec on the backend
Personally I suspect it languished because Red Hat simply cant abide the idea that somebody out there might avoid using systemd for something.
They happily built a docker compose to quadlets converter but they cant bring themselves to make podman compose not be a piece of shit even though it wouldnt be a lot of work.
> It's a tool for user age verification that happens to be something you can use to manage services.
Good talk buddy.
> Did you miss my point about it being a filthy kitchen sink?
I suspect there's not really a point in responding to this since you've already made up your mind.
Nevertheless, yes I am aware the systemd project contains many modular components. Some of which are good (systemd-the-service-manager that is what I was referring to), some of them are bad, and some of them are just odd (still haven't wrapped my head around systemd-homed's purpose). Podman integrates with the systemd service manager, not the rest of the project, so I'm really not concerned about that: there is no point where I am unable to use quadlets because I don't have, say `systemd-timesyncd` installed.
On the gripping hand, Quadlets are just a systemd-generator so there's nothing stopping you from getting that exact same benefits of Quadlets with some other service manager. You'd just have to write that implementation (and probably your own bespoke service manager) and will probably miss out on some of the niceties systemd provides to anything it manages.
> One of the major selling points of podman is that you dont need a daemon. except maybe yes you do because podman compose sucks so toss that selling point in the trash.
You skipped the second part of my sentence where I reminded you that Podman is daemonless. There is no long-running Podman daemon/service/etc, it is spun up on demand and then stops when the action is done. Having a second process instance is not a daemon, and I'm not sure how you would have expected this to work otherwise.
> Ever had "docker compose" and "docker-compose" do subtly different things which drive your team mate to pull their hair out? I have.
..Take this up with docker?
> Personally I suspect it languished because Red Hat simply cant abide the idea that somebody out there might avoid using systemd for something.
> They happily built a docker compose to quadlets converter but they cant bring themselves to make podman compose not be a piece of shit even though it wouldnt be a lot of work.
I don't think `podman-compose` was ever an official Red Hat project. I don't think there was every really much interest in ironing out all the corner cases, especially before compose was actually fully specced, and once Podman itself implemented the spec the interest has been drying up.
Assuming you're referring to podlet[0] for the latter, that was never a Red Hat project.
>I suspect there's not really a point in responding to this since you've already made up your mind.
I suspect you ignored the point because you didnt want to address the point.
My repeating it seems to only have highlighted your wish to continue avoiding it here.
>Nevertheless, yes I am aware the systemd project contains many modular components.
Another red herring. "Modular" really isnt the point here.
It's certainly one way to justify throwing even more shit in an already overloaded kitchen sink though.
>You skipped the second part of my sentence where I reminded you that Podman is daemonless.
No, you skipped the part where I acknowledged that it was daemonless-by-default but you actually DO need to run a podman daemon if you're using docker compose with podman.
>Take this up with docker?
They're not responsible for podman trying to piggyback on their tools.
This is what stopped me from picking up Podman more, all our devs use Docker and have been writing compose files for years now. When the response at the time was "you're using Podman wrong, Quadlets are the hot stuff now" it just felt like too big a risk and commitment to jump to at the time. Have things settled more? Getting away from Docker is a bigger priority nowadays for us.
Docker (Compose) has some quirks compared to Podman (Compose), e.g. when using gvisor or a lot of internal networks. Depending on what you do your milage will vary, though.
Agreed. I found compose overlay files merged list values differently between the Docker and Podman versions, which was a PITA in teams running Docker & Linux dev machines.
FWIW, most of these issues were recently fixed in podman-compose. I can now use the current git version of podman-compose interchangeably with docker-compose.
And one nice thing about podman-compose is that it's ONE PYTHON FILE. You can just copy it into your source tree.
Podman in general is much higher quality than Docker. For the last 5 years, Docker was throwing all kinds of stuff against the wall to see what sticks, neglecting foundational issues.
Moreover, podman-compose integrates really easily with systemd. You can create a service by just running "podman-compose systemd" and following the prompt! No quadlet nonsense required.
Then you learn podman can't even list containers for all users properly and it kind of starts smelling like the whole ip4 vs ip6 debacle: bunch of vocal proponents wanting you to subject yourself to endless torture for no discernible reason.
Congrats on building your custom delivery platform. We understand your pain ;-).
Any specific pain you want to share or feature you are still missing in your solution?
We’re happy to help you migrate to Distr if you decide you no longer want to maintain your solution or if you’d like to benefit from regular updates and continuous improvements.
Distr SaaS is configured for USD only (via Stripe) for simplicity, but EUR payments are definitely possible (either via Stripe, SEPA, etc.).
Its all bambi legs at the moment.. I wonder if we'll ever get past this stage.
Moving it all to a 'real' product is an option before it gets too serious. I'll bring this up with the group and lets see what happens.
Thanks for sticking around! Yes indeed, Distr already has a lot of features and can replace many other tools in your deployment stack (e.g., CI, log collection, alerts, licensing, secrets, OCI registry, customer & user management).
Users definitely also use us for internal deployments, but there are some trade-offs you’ll need to make compared to the zero-config setup you get when deploying your Next.js app to Vercel on the one hand, and the hardcore customization you get when running your own GitOps/Terraform setup on the other.
Is there any specific feature you’re missing, or is it just that our website doesn’t really highlight that use case?
Sure, happy to follow up with a detailed comparison.
TL;DR: Octopus Deploy has a strong focus on CD, providing a Cloud based framework to push your software to multiple targets. Distr also supports directly and continuously deploying your software to various deployment targets, but it additionally supports the pull approach, where the customer fully self-manages their infrastructure and simply fetches the application or artifacts from Distr.
Octopus Deploy also offers deep Argo CD integration (after they acquired Codefresh). On the other hand, Distr is completely open source and can be self-hosted if desired.
If you’re interested in specific features, I’m happy to go into more detail.
I feel you, but a huge percentage of recently funded companies are in the AI space. Software distribution for them is even more complex due to all the moving parts, and we want to make sure these companies know that our solution is a great fit for them.
Sure, multiple of our customers that distribute applications with a machine learning/AI component also need to distribute their models. They can use our OCI registry to distribute large images with huge layers. We specifically reworked our registry implementation to storing in-transit blobs on disk to save memory, ensuring the application doesn’t run out of memory [1].
Is registry OOM protection the only advantage your registry has for large layers? Robotics has a need for Docker tooling that handles large layers/images gracefully. Even if you've done the "right" thing and sideloaded your ML models with some other management system, CUDA layers and such are gigantic.
Edit: looking at this, this is very adjacent to some problems w/ robotics deployments. Fleet management, edge deployment, key management. Neat.
I'd be curious about the multi-artifact support. Can I declare a manifest that binds together multiple services (or a service and an ML model?) Do you support ML models as an artifact?
Feel free to reach out via our website[1]. Distr does not require an internet connection to keep your application running. Update commands are fetched directly from the agent and do not require any special connectivity.
Updates are pulled before the rollover to a new version is performed, so a poor internet connection may only affect the download speed of new updates. Distr is designed to operate even when no connection is available, or when connectivity is only allowed in short time slots.
We previously built a package manager for Kubernetes to abstract it in the simplest way possible `glasskube install app` but we failed because every abstraction needs to follow a "convention over configuration" pattern at some point. Also, we weren't able to monetize a package manager.
With Distr (https://github.com/distr-sh/distr), we have actually been able to help companies not only package but distribute and either manage or give their customers a way to self-manage applications. Our customers are able to land on-premises contracts at enterprises way faster than before, which is also a clear ROI for paying for Distr.
So, I don't think that you can get the flexibility of a distributed application orchestrator with a simple declarative YAML file if your target environments are diverse.