I don't get it, if it's running on the same (mentioning "local") machine, why does it even need the S3 API? Could just be plain IO on the local drive(s)
The app was already built against the S3 API when it used cloud storage. Keeping that interface means the code doesn't change - you just point it at a local S3-compatible gateway instead of AWS/DO. Makes it trivial to switch back or move providers if needed.
If the app was written using the S3 API, it would be much faster/cheaper to migrate to a local system the provides the same API. Switching to local IO would mean (probably) rewriting a lot of code.
Surely "read object" and "write object" are not hard to migrate to local file system. You can also use Apache OpenDAL which provide the same interface to both.
Yeah, unless you have the raw S3 API throughout your codebase you should be able to write a couple dozen lines of code (maximum) to introduce a shim that's trivial to replace with local file access. In fact, I've done this in most projects that work with S3 or similar APIs so I can test them locally without needing real S3!
seperate machine I think given the quoted point at the end:
> The costs have increased: renting an additional dedicated server costs more than storing ~100GB at a managed object storage service. But the improved performance and reliability are worth it.
What kind of vendor lock-in do you even talk about. Their API is public knowledge, AWS publishes the spec, there are multiple open source reference client implementations available on GitHub, there are multiple alternatives supporting the protocol, you can find writings from AWS people as high in hierarchy as Werner Vogels about internals. Maybe you could say that some s3 features with no alternative implementation in alternative products are a lock-in. I would consider it a „competitive advantage”. YMMV.
Apart from all these other products that implement s3? MinIO, Ceph (RGW), Garage, SeaweedFS, Zenko CloudServer, OpenIO, LakeFS, Versity, Storj, Riak CS,
JuiceFS, Rustfs, s3proxy.
Riak CS been dead for over a decade which makes me question the rest. Some of these also do not have the same behaviors when it comes to paths (MinIO is one of those IIRC).
Also, none of them implement full S3 API and features.
There's a difference between S3 API spec and what Amazon does with S3 - for isntance, the new CAS capabilities with Amazon are not part of the spec.
Ceph certainly implements the full API spec, though it may lag behind some changes.
It's mostly a question of engineering time available to the projects to keep up with changes.
Spec and features are intertwined. Customers who switch away from AWS S3 still want to use the same SDKs, libraries, etc that support S3 API. They don't want to rewrite their applications to use a new API. So then is it feature coverage or spec coverage?
It's both? Customer doesn't care if spec is 100% covered if feature that they are using in AWS S3 isn't supported.
Also, who is rewriting their application to change interaction with an object storage? People that directly use some S3 sdk all around the app should read a book on software engineering or at least a blog post.
Wrong again. Not only have I worked on DigitalOcean’s S3 implementation but I currently work on an open source product that targets S3 spec and can be used with any cloud provider and any other spec-compliant drop-in, like Garage.
> part of it is just to lock people into AWS once they start working with it.
This is some next-level conspiracy theory stuff. What exactly would the alternative have been in 2006? S3 is one of the most commonly implemented object storage APIs around, so if the goal is lock-in, they're really bad at it.
> What exactly would the alternative have been in 2006?
Well, WebDAV (Document Authoring and Versioning) had been around for 8 years when AWS decided they needed a custom API. And what service provider wasn't trying to lock you into a service by providing a custom API (especially pre-GPT) when one existed already? Assuming they made the choice for a business benefit doesn't require anything close to a conspiracy theory.
And it worked as a moat until other companies and open source projects started cloning the API. See also: Microsoft.
When I was in school, we had a SkunkDAV setup that department secretaries were supposed to use to update websites... supporting that was no fun at all. I'm not sure why it was so painful (was 25 years ago) but it left a bad taste in my mouth.
WebDAV is kinda bad, and back then it was a big deal that corporate proxies wouldn't forward custom HTTP methods. You could barely trust PUT to work, let alone PROPFIND.
yeah, sure, those 5-10 different API calls would surely be a huge toll to refactor... I'd rather run an additional service to reimplement the S3 API mapping to my local drive /s
Impressive, congrats! I‘m building https://sql-workbench.com which is similar, but focussed on DuckDB WASM instead of SQLite… Love that you offer different databases.
Your tool has been awesome! Seeing what it can do inspired me to write a POC that connected to an enterprise IBM application that I used to implement: https://github.com/karbasia/tririga-data-workbench (also uses DuckDB and Perspective with some additional hacks to make it work with IBM's tool).
Have a look at https://sql-workbench.com eventually. It runs DuckDB WASM in the browser, and with Perspective, which is used for data visualization, you can also visualize timeseries.
You can either drag & drop data, or use remote data sources via https
reply