I'm very happy to see you better support for docker compose. I think about 50% of the time I find Kubernetes used in production/development when it could have just used Compose along with a simple Terraform or Pulumi deploy script.
Docker compose is extremely basic. Even on a single node Kubernetes is more capable than docker compose. Ingresses, Services, Deployments, Cronjobs, Statefulsets. Awesome operators. Cert-manager. There're so many things I could easily do with Kubernetes that would require countless unmaintainable bash scripts and ad-hoc approaches with docker-compose.
However with Kubernetes your infrastructure will be ready to scale. You need to expose both front-end and back-end services under the same host? No need to tinker with nginx configs, you just create two ingresses and do it in a standardized way. You need to run your service with two replicas and rolling zero-downtime update? Kubernetes has it out of the box. Good luck implementing all the necessary dances with custom docker-compose setup without dropped requests. You want to customize healthcheck? Kubernetes got you covered with all the imaginable scenarios.
The only missing block in Kubernetes is support for building containers. This is implemented with docker-compose extremely elegantly and simple. That I admit.
> Simple setups will be as simple with Kubernetes.
That is true if you already have Kubernetes. If you don't, then you still need to run and configure the Kubernetes' control plane (e.g. kube-apiserver, etcd, scheduler, etc). Doing that alone may exceed the complexity of a simple setup.
I say this as someone who has looked at Kubernetes a lot and wanted to use it, but could never justify it. I have concluded multiple times that docker compose is a better fit for my use case.
> However with Kubernetes your infrastructure will be ready to scale.
True, but for many purposes you aren't gonna need it.
kubeadm alone is maybe 40% of what a production Kubernetes platform is. Just running init and deploying your app is good for a toy project or really small scale cluster.
Every time you hear yourself saying Kubernetes, "only", and "just" on the same sentence, please pause for a moment.
How many lines of unnecessary code have been written to be “ready to scale” and that never happens? This is an anti pattern and I would caution anyone from falling into this mindset where they feel they need Kubernetes even for simple things. I urge people to start simple, focus on features, and deal with complexity when the need actually arises.
I'm just using docker alone sometimes w some AWS service set - agreed completely. So simple to iterate, test and deploy. The performance and even reliability of stuff now is impressive. One AMD 7950x3d box on a 5 gig att fiber connection. Price / perf / simplicity is amazing
Most people don't really need things like rolling zero-downtime updates though. If you deploy your service once per week, and it takes 5 seconds to restart, that's still five 9s of availability, which is more than enough for tons of applications. A single Scala+postgres service on my 7 year old quad core machine can handle 10s of thousands of HTTP requests per second, which is plenty of scalability for most applications.
> Simple setups will be as simple with Kubernetes.
"Simple setups" in Kubernetes will have massive amounts of complexity and overhead, they're just buried in a different abstraction layer.
If you have "free" k8s and don't need to touch it operationally? Sure, do everything with it!
But somebody is going to have to run that k8s environment. If it's you, and you're not a k8s expert, then you'd better buckle up, because all that hidden complexity is now your problem.
In the world where you need to deploy "an app" to "a VM", the simplicity of docker-compose is a sweet spot that any developer can grok without needing an encyclopedic knowledge of how to manage it.
I wonder if there are any plans to add a docker compose like api to kubernetes? I know, kubernetes should not be used for that but there must be a better way than to use docker compose or manually building containers? Don't know a lot about DevOps so I sometimes just build my containers manually with a docker file but it's not ideal for up to date dependencies.
If you can grasp the layers of abstraction it's a really really fast way to ship operational solutions. Yaml aside it's great for smaller teams and empowers them to do a lot more than they could traditionally do.
Lets us spin up a prometheus monitoring stack on our kubernetes cluster for monitoring metrics and scraping metrics from our services pretty much automatically.
- Thanos Operator
Spins up prometheus monitoring aggregations across multiple clusters with object storage backends, basically you can store metrics for years and query them with grafana.
- Strimzi Operator (Kafka)
Orchestrates a kafka cluster for kafka connect or even full blown kafka brokers with zookeeper.
- Istio Operator
Builds a service mesh on the cluster, offering MTLS and an envoy load balancer with some incredible flexibility
- Cert-manager operator
Lets us auto-generate certificates for resources via letsencrypt. This one is so nice for internal services and keeps certs valid without any hand holding
- Argo CD operator
We use this operator to deploy argocd pipelines and ship new services and updates via github actions in a very standardized way, and gives our users a UI to see what's going on.
- Actions runner controller
Using this actions runner controller we can offload github actions runners to self-hosted runners that automatically scale for pipeline workloads. Another set up and forget system and saves us from buying more github actions minutes.
Ultimately we use a ton of operators because it gives us an out of the box framework for tooling we need to deploy without reinventing the wheel. Of course some operators aren't worth using and it's definitely worth checking them out to see if they worth your time. A lot of them are really open to updates and changes also so you can help evolve them as you grow into other cases.
I will say, it's very important to understand what you are abstracting with these, you can absolutely blindly deploy services with operators and have it blow up in your face if you aren't sure what's going on behind the scenes or lack distributed service knowledge in production.
I'm running postgres database with two replicas and continous S3 backup. It works flawlessly and with zero effort. If I would need to implement this setup without operator, I'd spent few weeks or more. Operator makes it effortless.
Docker compose falls apart when you need something clustered across nodes or any other multi-host deployment. Once you have to rip it apart it becomes way less appealing. Though I haven't tried it in a while and perhaps the ergonomics have changed.
I've done some labbing with Swarm. Swarm is _just different enough_ to be a pain, but just similar enough to fool you into thinking you know what you're doing. Because swarm was also confused terminology (at one point there were two things with the name swarm that did things differently) the tech definitely got a bad rap for a while, but it also stagnated supporting tooling. You can use things like Portainer and Swarmpit to get some visibility into your swarm cluster members, deploy workloads, etc, but it definitely feels like a second class citizen.
Ultimately, I would love for something to exist out there that had the opinions and scope of deployment like Swarm in terms of simplicity, but didn't conflate other tools out there so that ergonomics and documentation were better and less confusing. Kubernetes gives you so much, but I do feel like it's a misstep that the only reasonable way we've simplified Kubernetes for production workflows is to pay a large PaaS to manage it for us.
Honestly with a github action and Argo CD workflow the kubernetes patterns aren't that scary. However maybe I'm just too comfortable in the ecosystem at this point.
I will say Kubernetes being as open as it is and all the overlapping tooling would seem incredibly overwhelming for someone trying to enter that space.
Swarm works, but has poor support for volumes - which means it's tricky to run legacy applications on swarm (which eg uploads files to local disk, not s3 - or keeps state/cache on disk, not a database).
Ingress is also more complicated/bespoke - the best I've found is traefik with labels for routing/config.
My advice today would be to scale Docker compose vertically (eg: on a dedicated server) - then move to Kubernetes.
The swarm middle ground isn't really worth it IMNHO.
> Swarm works, but has poor support for volumes - which means it's tricky to run legacy applications on swarm (which eg uploads files to local disk, not s3 - or keeps state/cache on disk, not a database).
One way round that is to use an NFS volume. However, I've hit problems with too many client NFS connections on a docker swarm and so found it better to mount the NFS volume on each host and use a bind mount instead.
My general feeling about adding an NFS depency to multinode swarm is that it's effectively adding a single point of failure to a system which is otherwise somewhat robust against single node failure...
Porting a ten year old app from local file storage to s3 might not be trivial.
For a new app, one generally should and can embrace 12-facors, and delegate state to stateful services (managed databases, key-value stores, s3 etc).
Do note that for simple services, local disk can be very hard to beat for low complexity, extremely high performance - with the caveat that horizontal scaling might be tricky.
Ed: also depending on privacy requirements - self- hosted s3 (eg minio) might be tricky for a production load. OTOH self-hosted Kubernetes is no walk in the park either!
the inevitable "you could have used something simpler than Kubernetes!" comments that appear every time it's mentioned neglect to note that you're more likely to find a Kubernetes example for whatever you're doing readily available in the wild.
Not sure if I follow. Let's say I have 2 kinds of clusters with these sets of containers:
A) 3 microserver containers on each machine
B) Postgres container and a logging container
The A and B group can be implemented with docker compose and deployed to each cluster with pulumi/Terraform. Two or a hundred, it doesn't really matter how many cluster groupings you have.
I guess they stopped doing this and now consider the product mature from looking at their website, but Docker Compose used to tell you that it wasn't suitable or intended for production use. That seemed like a pretty good reason not to do it.
Compose has its own footguns, but I've run a number of services with just Compose on a VM or direct on bare metal and the few little issues I've had were easy to debug and fix.
Very much unlike Kubernetes, where even a small deployment can sometimes be maddening to debug. Great for larger deployments where complexity is basically guaranteed, but anything that could run on one server with a hot spare is a great fit for Compose!
Surprised to hear you think debugging in kubernetes is maddening. It’s absolutely very different, but if I had to debug a system I knew nothing about, I’d rather debug a kubernetes based system rather than any other. Standards and all that. Anyways - for a single container or two obviously you’re right - I’m just not sure that’s so common outside of a side project anymore.
Especially because "production" implies an ingress controller, monitoring, etc. Does a Docker Compose-based production setup contain run its own Nginx with its own manually-wired rules? Its own Prometheus?
Compared to Compose on a single server, Kubernetes deployments can have multiple layers (control plane, containers, usually some sidecars) where little quirks occur.
It almost necessitates deeper monitoring and/or extra tools to give a dashboard/'single pane of glass', whereas I can run Compose and just log into the individual server and jump around logs.
Kubernetes control plane is the same as docker daemon.
Sidecars do not exist in kubernetes, unless you install something that adds them.
Containers are... containers? The same containers you'd see in docker.
Kubernetes does not need deeper monitoring compared to docker. You can jump around kubernetes logs just like you do with docker. Except you don't even need to log into individual server.
I don't use any kind of dashboard for my kubernetes cluster. I installed kubernetes-dashboard, so developers and managers can enjoy their dashboards but I've found no value with it. kubectl is more than enough.
If someone is working on complex kubernetes configuration, it's beneficial to have kubernetes in local dev environment, since it's trivial to test changes.
