Both great options, but I'm a local first kind of guy. If I can't get it running on a VM to experiment/create/destroy ad nauseum, I definitely am not gonna spin it up on a cloud versus a platform where I actually kind of know what I'm doing.
Why Ubuntu? Mostly because it just makes packaging and compatibility easier. There is a plugin system which creates tarballs out of overlays... so you can add additional software into the system and the building of the tarballs can include things like apt install. These tarballs get downloaded and installed early on by a hacked init script. This way, we can do things like install minimized downloads of GPU drivers.
There are some issues with this, but overall it works ok. For now, it is just what I ended up with. Total boot download is about 100megs, which isn't too bad... machine is up and running in 60s, which is fine for this workload right now. It does suck rebooting the whole datacenter at once though. ;-)
The lbu stuff looks interesting for sure. I'm not a huge fan of the additional NFS requirement, but I assume that can be worked around. That said, I'm open to changing stuff up and optimizing things further. This is some esoteric stuff and we are hiring. =)
My side project for the last while was pretty similar to what you're describing, and as a result I've become very familiar with this "esoteric stuff". If you're hiring, I'm interested :)
Interesting view.
How should i, as a sysadmin who just writes bash and powershell scripts, start to learn _serious_ programming? Is it still worth to force myself into OOP?
I argue that you should start simple with functions and IF you see a use case for classes use them.
To be able to make a reasonable choice, you must learn OOP at least the OOP machinery of your language.
A few hints:
a) Python's abc or Java Interfaces serves as template to implement multiple times the same behavior in different contexts
b) If you don't inhirit the class, the class is useless, except if you use the class as an interface (see a)
c) A class is more complex than a function ie. more guns to shot yourself in the foot.
d) think function pointer / first-class function. Many times, you can avoid a class by passing a function as parameter e.g. filter.
e) think function factory ie. a function that returns a function.
f) think pure function / side effect free functions. This is a BIG life savior. A function without side effects is much simpler to test that a function with side effects. Of course, you can not avoid all side effects (io et al.) or mutations, so build your abstraction so that the side-effects / mutation to happen in a well known place.
