Interestingly, I recently interviewed Samuel Colvin, Pydantic's author, and he said when designing his observability Saas called LogFire, he tried multiple backends, including ClickHouse.
But it didn't work out.
One of the reasons is LogFire allows the users to fetch the service data with arbitrary SQL queries.
So they had to build their own backend in rust, on top of DataFusion.
I used ClickHouse myself and it's been nice, but it's easy when you get to decide what schema you need yourself. For small to medium needs, this plus Grafana works well.
But I must admit that the plug and play aspect of great services like Sentry or LogFire make it so easy to setup it's tempting to skip the whole self hosting. They are not that expensive (unlike datadog), and maintaining your observability code is not free.
What kinds of SQL queries could ClickHouse not handle? Were the limitations about expressivity of queries, performance, or something else? I'm considering using CH for storing observability (particularly tracing) data, so I'm curious about any footguns or other reasons it wouldn't be a good fit.
I'm editing the transcript right now, and he says it's more about exposing a nice API to the user.
E.G: Clickhouse interval support, which is an important type for observability, was lacking. You couldn't subtract datetimes to get an interval. If you'd compared 2 milliseconds intervals to one second ones, it wouldn't look at the unit and would say 2 ms is bigger, etc. So he had to go to the dev team, and after enough back and forth, instead of fixing it, they decided to return an error and he had to insist for a long time until they actually implemented a proper solution.
Quoting him "But like these endless issues with ClickHouse's flavor of SQL were problematic."
Another problem seemed to be that to benefit from very big scaling with things like data in Parquet at rest + local cache meant basically leaking all your money to AWS because the self-hosted version didn't expose a way to do that yourself. Click house scales fine at my size, so I can only trust him on that front since I'm nowhere that big.
Funnily after that, they moved to TimeScale, and the perfs wouldn't work for their use case.
They landed on DataFusion after a lot of trials and errors.
But really interesting perspective on the whole thing, you can see he is kinda obsessed with the user experience. The guy wrote a popular marshmallow alternative, 2 popular celery alternative and one watchdog popular alternative, all FOSS.
These kind of people are the source of all imposter syndrome in the world.
I'll publish that video next week on Bite Code if I can. If I can't, it will have to wait 3 weeks cause I'm leaving for a bit. But Charlie Marsh's one (uv's author) is up, if you are into overachievers.
One of the devs working on Logfire here. Part of it was the level of support. Like Samuel said the ClickHouse folks were not receptive to bug reports. The Timescale team is leagues ahead in that sense, they’re super responsive and helpful. Ultimately one of the reasons for choosing DataFusion was that it’s much more approachable of a project and indeed we’ve already gotten tremendous bidirectional benefit: the DataFusion team has helped us figure out some complex bits and we’ve done significant upstream contributions. By the way, DataFusion is now the fastest single node query engine on ClickBench: https://datafusion.apache.org/blog/2024/11/18/datafusion-fas...
Another reason we use DataFusion is multi-tenancy: we found it was hard to use RLS and such to implement multi-tenancy. We’ve had much better luck with the extensibility of DataFusion.
I know it's not very nice to post that here, but given it's hard to get good info about python because the ecosystem is so huge, I'd rather give it straight:
Pex doesn't work on windows, is slower than shiv and shiv solves the resource files problem that you will encounter with dependencies like django. To achieve this, shiv unzips the whole zipapp in a cache on disk and create a transparent proxy to it.
Still, I wish zipapps had better tooling honestly. Using either pex or shiv is more complex than it needs to be.
I hope uv will eventually give you something like "uv zipapp" and make it good and simple.
Right now, to bundle all deps for shiv you have to download first all wheels, then figure out the proper incantation to include them. And if you use WSGI, you probably want some kind of dedicated entry point.
The tech is cool, but it deserves maturing to be a true ".war for python".
FWIW, pex now also has options to unzip the archive to a cache directory on startup (I believe this happens by default now, but am not at a computer to confirm), to side step the zipapp limitations that you reference.
While we're posting not-nice things, I'll throw my current favorite out: pipx ( https://github.com/pypa/pipx ). I use Python's Poetry to build a wheel and then pipx to install it. Super-easy, barely an inconvenience :)
I'm not a pipx expert (I'm a "I got it to work for my use case and then stopped" person :) ) but I was able to get my project working after installing a local wheel file (so i don't think you need access to the Internet to install things).
I thought the wheel file has all the dependencies since (my understanding of) pipx uses a venv for each individual wheel file/app. Certainly I'm using a bunch of packages that I needed to pip install into my development environment, but it's entirely possible that my pipx-managed wheels are using those installs, too.
I think the last two points are accurate, and I've never tried to use APIs like wsgi with it.
Also pipx isn't reproducible - it re-resolves dependencies so you may end up with different versions over time or in different places, eventually causing something to break.
If you have a shiv working it stays working, assuming you have a solution to distribute the required interpreter version.
In the early 2000’s open source hubris, next year was always the year of the Linux desktop. Since then consumer windows matured into a totally ok OS and one with the best support for graphics and C++ development at that.
Non-windows support nowadays is a fairly strong signal of a non-serious software offering if there is no obvious reason for it. And that’s totally fine, hobby tools developed by enthusiasts rock - but they are not industrial in scope as such.
A lot of serious software offerings are only concerned with the server use case, modern servers run Linux unless there's a good reason not to, and modern windows has more than one acceptable way to run Linux binaries if you absolutely have to.
> modern servers run Linux unless there's a good reason not to
I think many people on HN would be surprised at how many orgs are using Windows servers heavily, because of their familiarity and comfort with Microsoft, or because some application requires it.
Of course they are non-tech companies using the servers internally for enterprise applications, not web servers, but there is absolutely a lot of windows server usage in corporate environments.
> I think many people on HN would be surprised at how many orgs are using Windows servers heavily, because of their familiarity and comfort with Microsoft, or because some application requires it.
I think most HN readers are well aware that there are a lot of Windows servers out there, especially in the sorts of environments where it's "The Server".
That doesn't change the fact that there are orders of magnitude more Linux servers in the world, and as the post you replied to said Linux is the default assumption. Basically every container and the vast majority of VM guests are Linux. I'd be willing to bet that more Linux servers have been deployed in the time it took me to type this post than Windows servers will be deployed this week.
I'm not trying to make the point that there is a comparable number of Windows servers to Linux servers, and I believe that going off "number of servers" is of limited usefulness. There are of course going to be far more Linux servers because they are far easier to provision, and the people managing them are generally going to be much more apt at orchestrating large fleets of small servers than the people managing Windows servers, which will tend towards larger servers with multiple purposes.
I'm simply trying to say that Linux is not the default assumption in many contexts where a lot of money gets spent on licenses and services for server software.
There are of course large contexts (tech companies and web servers) where Linux is the default assumption.
It would today, because macos requires signing, and windows has "the mark of the web".
They don't apply to scripts, so Python is fine, but they do apply to executables.
Having to sign your zip every time you produce it would nullify the benefit of the concept. And because the checksum of the executable file change according to the content of the zip, you can't just sign the cosmopolitan part.
- The underlying python code cannot use open() to access its own resources, but must use pkg_resources, and most packages on pypi don't do that because most dev don't even know it exists or what problem is soves.
- _ _ file _ _ is not available, breaking a lot of things. zip_safe is a build option, but actual compliance to it is hard work and rare.
I was thinking more along the lines of using namespaces and FUSE. I'm aware that Python kinda treats them like folders transparently, but not completely for reasons (and more) that you described.
A 2016 comparison by the developer of WinFsp says WinFsp is several times faster in performance tests [1]. According to the readme for Donaky [2], since then Dokany got faster.
> you don't need a banner even of you do track users for anybody with DNT
This is not true. The specific text of the law requires that websites have to provide details about their cookies, and then document and store user preferences.
If you just honored the DNT, you would still be out of compliance.
I'm with @belval on this one, it's ok to prioritize github, but people that want the standard to implement an alternative should not feel like they are doing something that may not be supported.
Again, to be clear: the standard does not stipulate GitHub or any other specific identity providers. The plan is to enable GitLab and the other Trusted Publisher providers in short order.
This is exactly the same as Trusted Publishing, where people accused the feature of being a MSFT trojan horse because GitHub was enabled first. I think it would behoove everybody to assume the best intentions here and remember that the goal is to secure the most people by default.
It's said explicitly in the second sentence in the usage docs[1].
> Attestations are currently only supported when uploading with Trusted Publishing, and currently only with GitHub-based Trusted Publishers. Support for other Trusted Publishers is planned. See #17001 for additional information.
At this point it should be fairly obvious that if you have to defend the phrasing in multiple threads here on HN, get some folks to help rephrase the current document instead so you can comment with "we updated the text to make it clear this is a first pass and more options are getting added to the doc soon".
If you draw an ugly cat, and someone tells you it's ugly, it doesn't matter how much you insist that it isn't, and the same is true for docs. It doesn't matter what your intention was: if people keep falling over the same phrasing, just rephrase it. You're not your writing, it's just text to help support your product, and if that text is causing problems just change it (with the help of some reviewers, because it's clear you think this is phrased well enough, but you're not the target audience for this document, and the target audience is complaining).
Anyone can run an OIDC system if they want. But PyPI is not under an obligation to trust an OIDC provider running on a random rpi3 in your basement. More than that, GitHub is "trusted" because we can be pretty sure they have an on-call staff to handle incidents, that they can reliably say "This token was provided on behalf of this user at this time for this build", etc.
Even if you standardized the more technical parts like OIDC claim metadata (which is 100% provider specific), it wouldn't really change the thrust of any of this — PyPI is literally trusting the publisher in a social sense, not in some "They comply with RFC standards and therefore I can plug in my random favorite thing" sense.
This whole discussion is basically a non-issue, IMO. If you want to publish stuff from your super-duper-secret underground airgapped base buried a mile underneath the Himalayas, you can use an API token like you have been able to. It will be far less hassle than running your own OIDC solution for this stuff.
Please improve your reading comprehension. I swear, this website is embarassing sometimes. You can still do this with an API Token. You can upload from a C64 with an API token. What you cannot do is run some random OIDC provider on your random useless domain and have PyPI magically respect it and include as part of the Trusted Publishers program. There is no point in it, because the program itself is constrained by design because it only provides any benefit at "large scale." Your random dumb server providing a login for you alone does not provide any benefits over you just using an API Token.
Any pathway to provide trusted attestations for random individual Hacker News users like yourself will, in fact, require a different design.
I always ask the robocalls to say something specific.
It finally worked the other day! A robot repeated "chrysanthemum" back to me when asked. Sadly it seemed to be limited to this one single trick; as I continued my attempts to exploit it, it abruptly apologized and ended the call.
Cute, but one day I am going to get lucky and a scammer is going to get absolutely wrecked.
Chrysanthemum is also my go to. That or “You’re in a desert. You’re walking along in the sand when all of a sudden.. you look down and see a tortoise..”
It would seem to me that it's a lot easier to build an AI that acts as a non-tech-savvy granny to keep scammers busy, than one that actually successfully executes a complete scam from start to finish on a real life person.
I'm already surprised that they are not looking at the number, realizing that pilots cost much less to train than planes to build, and draw conclusions from there. The army has been more cynical in the past.