I was doing something similar just the other day and came across sqlite-s3vfs[0]. It is likewise a SQLite VFS that translates IO to S3 api calls. However, that project is only for python and seemingly abandoned. Additionally, if you want more than one writer, you'd have to coordinate that yourself, afaik.
Temporal is an AMAZING piece of software, however I don't believe it's a replacement for something more simple like Celery. Even if you write helpers, the overhead to setting up workflows, invoking them, etc. is just too much for simple jobs like sending an email (imo). I would love to work in a codebase that had access to both, depending on the complexity of what you're trying to background.
It's okay till it's not. Everyone I know who had Celery in production was looking for a substitution (custom or third-party) on a regular basis. Too many moving pieces and nuances (config × logic × backend), too many unresolved problems deep in its core (we've seen some ghosts you can't debug), too much of a codebase to understand or hack. At some point we were able to stabilize it (a bunch of magic tricks and patches) and froze every related piece; it worked well under pressure (thanks, RabbitMQ).
Because it’s a seducer. It does what you need to do and you two are happy together. So you shower more tasks on Celery and it becomes cold and non-responsive at random times.
And debugging is a pain in the ass. Most places I’ve been that have it, I’ve tried to sell them on adding Flower to give better insight and everyone thinks that’s a very good idea but there isn’t time because we need to debug these inscrutable Celery issues.
Although we could say the same thing about Kafka, couldn't we? It's made for much higher throughput and has usually other use cases, but it's also great until it's not great.
At least the last time I used Kafka (which was several years ago so things might have changed) it wasn't at all easy to get started. It was a downright asshole in fact. If you pursue a relationship with an asshole, you shouldn't be surprised when they become cold to you
Celery is great and awful at the same time. In particular, because it is many Python folks' first introduction to distributed task processing and all the things that can go wrong with it. Not to mention, debugging can be a nightmare. Some examples:
- your function arguments aren't serializable
- your side effects (e.g. database writes) aren't idempotent
- discovering what backpressure is and that you need it
- losing queued tasks during deployment / non-compatible code changes
There's also some stuff particular to celery's runtime model that makes it incredibly prone to memory leaks and other fun stuff.
> your side effects (e.g. database writes) aren't idempotent
What does idempotent mean in this context, or did you mean atomic/rollback on error?
I'm confused because how could a database write be idempotent in Django? Maybe if it introduced a version on each entity and used that for crdt on writes? But that'd be a significant performance impact, as it couldn't just be a single write anymore, instead they'd have to do it via multiple round trips
In the context of background jobs idempotent means that if your job gets run for a second time (and it will get run for a second time at some point, they all do at-least-once delivery) there aren't any unfortunate side effects to that. Often that's just a case of checking if the relevant database updates have already been done, maybe not firing a push notification in cases of a repeated job.
If you need idempotent db writes, then use something like Temporal. You can't really blame Celery for not having that because that is not what Celery aims to be.
With Temporal, your activity logic still needs to ensure idempotency e.g. by checking if an event id / idempotency key exists in a table. It's still at-least-once delivery. Temporal does make it easy to mint an idempotency key by concatenating workflow run id and activity id, if you don't have a one provided client-side.
Temporal requires a lot more setup than setting up a Redis instance though. That's the only problem with it. And I find the Python API a bit more difficult to grasp. But otherwise a solid piece of technology.
In my experience async job idempotency is implemented as upserts. Insert all job outputs on the first run. Do (mostly) nothing on subsequent runs. Maybe increment a counter or timestamp.
When I went there in 2012, there was a fee each semester for dining hall silverware that goes missing. They assumed that you'd take $60 worth of silverware and plates.
Fun fact: replacing copper pipes with PEX during a renovation will often pay for tools and supplies needed for the replacement when the copper is taken for scrap.
Is that fun fact legitimate? I know that you can get a premium for copper (see: tweakers who rip out pipes in open buildings and sell it), but the ratio here seems improbable, at least.
According to copper.orgs fact page, the "average" single family home (whatever that means) has 151 pounds of copper pipe. The current scrap price for #1 copper where I live is $2.88. That comes out to around $446.
That would maybe cover the cost of the pex itself, OR the cost of fittings and tools, but probably not both.
I'd argue keep the copper in place as much as possible. PEX is plastic. I'm sure it contributes to all the microplastics/PFAS we see all over the place.
this is superstitious nonsense. pex contains no pfas and is the least likely kind of plastic to create microplastics
it's disturbing that people saying things like 'I would bet that running a single load of synthetic clothes in the dryer creates more microplastic than making 1000 cut and crimped connections of PEX tubing.' get flagged to death as if they were spamming penis enlargement pills; that comment was obviously correct
I would bet that running a single load of synthetic clothes in the dryer creates more microplastic than making 1000 cut and crimped connections of PEX tubing.
Honestly don't know which pipe is worse. I've had to crawl underneath my house 4 times to repair pinholes, but squirrels seem to think pex is made out of some sort of heroin candy. They can't get enough. Sadly, I've never gotten any video of what they think of the relatively high pressure water at the center of this candy... might've be some small consolation.
I think I heard he was actually addicted to alcohol, which is information that he withheld, and that some scientists in a European country tried unsuccessfully to recreate his results.
This might finally be the push for me to learn Rust. I'm primarily a Python developer working on Django, but I used to do embedded development in C, and Operating Systems with Remzi at UW Madison was my favorite class. It got me excited about CS. With Google and industry backing, it seems like Rust is here to stay.
Seconding this. I learned a little rust and really liked its philosophy. I was then working on a small project that needed to avoid the big data science python libraries because they're miserable in terms of portability to air-gapped networks.
We had re-implemented a few mathematical and curve finding functions and even with the newest python they were uncomfortably slow.
PyO3 made it so easy to implement a runtime switchable rust library exposed to python, it was almost unnerving. By writing those handful of functions in rust we got something like a 30x speedup.
Note this was a small project with limited funding for r&d so the level of effort for performance speedup was really nice.
needing to compile them on the airgapped machine introduces more dependencies. Things also get weird when your airgapped machine is a slight version off from your dev machine. There's a lot of ways around it but we've found for the size of this tool, it was preferable to just write our own handful of math functions over building out a deployment method for extra dependencies.
Yeah, PyO3 is great. I've tried to play around with releasing the GIL from rust in Python 3.12. I would enjoy writing a WSGI/ASGI server with a Celery runtime at some point too. Or contribute to Granian.
I think Rust is here to stay, but I wish folks had designed a systematic standard library (as Java has) rather than let a set of crates evolve uncontrolledly.
Hopefully, the language will develop further and then reach ISO standardization - which Java cannot as it remains proprietary.
Rust's std lib approach is the best way to approach a std lib (IMO). Even in java world, some dependencies end up being the defacto standard while others end up dying off.
The JDK has a bunch of garbage in the JDK that people shouldn't use. That stuff has to remain due to strong backwards compatibility guarantees.
Rust's approach means that non-standard stuff that we end up realizing is a mistake can quietly die off.
Now, should it be larger? Probably. I'd prefer if rustlang had a standard async/await implementation rather than leaving it up to the ecosystem. But I don't think rust needs, for example, a gui api (like swing or awt) in the standard library.
I'd prefer if rust was managed a bit more closely to the way java is managed today. Java doesn't pull in new apis willy nilly, but the ones they do pull in end up being things that have broad appeal and utility. Rust could take a look at common crates in the ecosystem and start pulling those in to the standard lib.
> I'd prefer if rustlang had a standard async/await implementation rather than leaving it up to the ecosystem
The "default" is tokio, and I think even the tokio authors would agree that it's neither suitable nor ready to be part of the stdlib.
OTOH, pollster[1] (or something like it) should be pulled into the standard library. Not particularly useful for anybody who wants async, but super useful for anybody who doesn't want async but wants to use a 3rd party crate that includes async.
On the async front, more work needs to be done to allow a more "plug and play" scenario for async runtimes. Standard APIs/traits for task spawning & mgmt, channels, locks, etc. Because right now it's not really feasible to write a library in an async-agnostic way, and tokio just owns the field even in places where it's not really appropriate or the best choice.
In Java land, I can just ignore dead stdlib stuff. In rust land I have to fight bugs when two crates depend on different async or applicator libraries.
Surely you can see how one is worse than the other?
> Now, should it be larger? Probably. I'd prefer if rustlang had a standard async/await implementation rather than leaving it up to the ecosystem.
What you are describing is a problem, no doubt about it, but it's not as if this isn't something that can't come up in javaland. For example, dealing with a project that uses both Gson and Jackson. Or dealing with a project that's mixed together Netty with Apache http.
And when you can ignore these dead libs very much depends on what the other parts of the lib you are dealing with. For example, you might never interact with `Enumeration` or `Vector` yet there are parts of the Swing api that expose those.
The massive feature set of the JDK bloats the size of every container shipping with the JVM. It pumps up the requirements for metaspace. And it negatively impacts JVM startup time.
I'm not saying the JVM needs to remove every dated API. But things like JNDI, for example, are not only dangerous to use (that was the root cause of log4j2's big vulnerability), they are massive feature sets that pretty much nobody wants to use.
Java was designed in an era where we thought having thinish clients streaming jars/classes from central network servers was probably a good idea. When we thought the JVM could be more than just a language VM, it could be an operating system. A lot of those concepts simply don't apply to modern jvm dev or even jvm dev that's happened in the last 20 years.
And, to be clear, the JDK was not wrong in bringing along all these libraries. After all, in the 90s it's not like we really had a great story around community package development. That was an era where devs routinely downloaded their dependencies manually. Clever devs even had curl scripts wired into ant or make to do that job.
Instead of a large std tied to the stability guarantees of the labguage, I would like to see something akin yo "stdlib distributions", where a versioned set of community libraries can be depended on to interoperate without duplicated deps in your tree. This decoupling would allow Rust to remain 1.X while the equivalent of the stdlib can bump their major version every, lets say, 3 years, without breaking older projects. You'll notice that this is almost the same thing we have today, and anyone could start it just by creating a crate called "my-std" with the list of dependencies they want to provide.
(have an upvote, no idea why you're in the negative)
I have been thinking about this for the past few years, and I think I'd go so far as to make the standard library "not special." That is, make it work similar to the edition system: cargo new would add a dependency for the latest std at the time, build-std would just be transparent, and you treat it like any other crate.
That said, I am sure there are a zillion issues with this today, especially in Rust itself, but if I were to make a Rust++, this is one of the things I'd consider trying to give a shot.
The issue with this approach lies with the current content of the stdlib: the stdlib is for vocabulary types, that is, types that are meant to be used as interoperability bricks in almost all programs (think Option, Result, Vec).
If you start versioning the stdlib now you need a bridge between the Option from std1.0 and the Option from std2.0. This will become confusing quickly for programs that use libraries that depend on different versions of the std.
We could forbid that situation, but then we have an ecosystem split
Still, what is blocked by lack of standardization or specification?
Many (all?) languages were successful before being fully specified, and many still are not (cough Python cough) yet that has not impacted their adoption, popularity, or effectiveness.
Many people misunderstand how software is written in regulated industries, and assume that a standard is necessary. In practice, this is not the case. Note that Ferrocene[1] had to produce a specification[2] in order to qualify the compiler. But there isn't a requirement that it must be a standard in any way, only that it describes how the Ferrocene compiler works. Nor that it be accepted by upstream.
Exactly. I've heard so much "you must follow MISRA!" but MISRA is not actually the requirement - it's a solution that satisfies the requirement, and there can be other way to satisfy the requirement of your industry's regulations.
I don't think GP described lack of the language specification as a blocker for future success? It's something nice-to-have when the ecosystem has enough resources, though I'm skeptical if this can be meaningfully kept up-to-date unless the core language team decides to actively maintain it. But this will definitely trade off the velocity to a certain degree.
Yeah, strongly agree. The lack of batteries in the Rust stdlib is by far the biggest mistake in the language.
The first reason is because the stdlib is the only thing you can always count on being there. People are often in situations where they can't download library packages due to security procedures, and have to rely on just the stdlib. People like to complain about urllib2 still being in Python even though it's not really used any more, but I've been in situations where urllib2 was the only thing available and I was damn glad it existed.
The second reason is because the way Rust does things is horribly confusing. What's the best crate to use for X? If you are a regular in the community you probably know, but a newbie is going to have no idea which of the many available options to pick. Whereas something within the stdlib is always a reasonable choice, even if it isn't the best choice.
I really hate that the Rust community in general is so dogmatic about this topic. Having so much functionality outside the stdlib makes the language worse, not better.
There are tradeoffs, but overall I prefer rust's "free market" solution over Pythons "batteries included". Crates can evolve, compete and innovate much more freely than the standard library can. Most importantly they can be deprecated or abandoned when better approaches are found, while anything in the standard library has to be supported forever. And when the dust settles and everyone agrees on one canonically best implementation, that sometimes does get incorporated into the standard library.
std promises stability to avoid a python2-3 situation. Which means any mistake we make will be with us for a long time. And we've already accumulated quite a bit of those mistakes. So non-trivial additions aren't made lightly.
Compare through how many breaking changes even high-quality ecosystems crates have gone through in the last few years.
Reading the first part of the book [0] featuring the concepts behind hypermedia systems has helped me to really understand the goals of HTMX. I would highly recommend it even if you don't experiment with HTMX yourself. I bought it as an ebook to support the author but it is also free to read.
My optometrist actually said that finding a prescription that works is more like a wine or chocolate tasting than a precise measurement and giving me a hard time for not being able to decide whether one lens or the other is better.
I'm attempting to call support and get them to give me this year's Premium subscription fee back. On the support page for deleting the account, there's a link to Contact Support. [1]
edit: A rep called right away. They will not give me my money back. It's in their terms and conditions that they won't even if there is a security breach and it's been longer than 30 days since I paid.
> Facebook, Google and Microsoft have cited Iowa's large inventory of available land, low energy costs, high portfolio of renewable energy and low risk of natural disasters as reasons for locating here.
I'm interested to hear how other states stack up against this and if this factors in at all.