When we first switched to ZFS, we had some ingestion issues. It turned out the problems were caused by poor architecture decisions we made and was merely exaggerated by ZFS. So yes, there was a noticeable increase in latency, but ZFS was only a tiny piece of it.
I'm not the author of the above comment, but here is my take..
I think one benefit of a curated course is that it includes materials you didn't even know exists. We can easily improve on our known unknowns - just pick up a book or google it - but unknowns unknowns are... well difficult to learn. I think going through a graduate program helps you get a better grasp of what you don't know AND what you didn't know you didn't know.
Yes, that's certainly an aspect of it. As someone with a lot of professional experience, it's easy to get into a rut with what you're required to do in your day job. I always tried to keep up with new techniques, frameworks, etc. But getting the degree forced me to learn more about the course subjects. Like in Computer Networking, most of my low-level networking knowledge was several years old. Being exposed to Software Defined Networking was very interesting, and I enjoyed experimenting with Pyretic to explore how it works.
I noticed that bigchainDB is written enitrely in python. I love python, but I'm surprised to see it used in the database space!
(if the author is reading this) - what were your interest in using python? Do you have any concern for it's performance/scalability as the service grows?
I've been using Python in enterprise-grade, large-scale production systems since the early 2000s. And it works, well. Most python libraries where performance matters have C under the hood. E.g. numpy matrix manipulation. Parallel computing got way better with the async support.
As for BigchainDB itself: it wraps RethinkDB (and soon MongoDB) which is where much of the compute intensive stuff happens. Also, we leverage these fast Python+C libraries, and parallelization. If/when needed, we can always convert some of our core stuff to C or another fast language.
"Hasselblad Acquired by DJI" is article author's speculation.
> Although the final conclusion has not yet been confirmed by Hasselblad or DJI, I am confident in my sources and believe that a formal announcement is forthcoming.
Are you serious? It's definitely possible, and almost simple, to build a limited scale service that is cheaper than S3. S3 has the Amazon footprint behind it, which you can't compete against, but the price is not hard to beat.
Yes, but who will be on call to manage, monitor, and fix it when it breaks? Answer: you! S3 is one of those things that's so cheap and works so well I'd never think of leaving it.
If it came down to it, you can lease dx lines from AWS to your datacenter switch and get 10 Gbs to S3 for pretty cheap.
But your company might have security requirements that prevent the use of S3.
For example, if you're a defense contractor that handles FOUO information, you can't just throw that on somebody else's cloud. You have to keep it entirely in-house.
Are you aware of a programming language/compiler that optimizes on recursively written code? I'm finding it difficult to imagine a PL where stack _isn't_ free except tail-recursive code.
My understanding is that it's the opposite -- function calls expend a cost to save (and later restore) the registers and push the locals onto the stack. Tail recursive functions on the other hand can be implemented as a jump to the top of the function, right after all the stack bookkeeping, and so saves on that expense.
Tail-merging doesn't only apply to recursive functions. As soon as you don't need to store intermediate result, you can jump to another function without allocating a frame.
Haskell (sometimes) does this kind of thing much more efficiently - it doesn't specify or always provide a stack, it models the program as a graph to be evaluated by reduction. And any language where you can implement continuations allows you to write this kind of thing efficiently (e.g. https://github.com/Atry/memcontinuationed ).
(Ultimately you can do "userspace continuations" in any programming language by explicitly transforming your code into CPS, using an explicit FSM if the language doesn't support tail-calls as such, the only price being developer sanity)
I do think there's a real gap in many languages between the natural way to express graph/tree transformations (recursively) and a cache-friendly/mechanical-sympathy-style runtime - see e.g. https://medium.com/@gkossakowski/kentucky-mule-limits-of-sca... . And really this gap shouldn't be there - the language implementation should be able to transform and fuse this kind of tree-walking operation into an efficient implementation - but at the same time it's hard to see how you'd ever make the language spec permit this without resorting to a full-Haskell "everything is lazy everywhere and the language implementation is allowed to evaluate whatever it likes however many times it likes (including 0) in whatever order".
Maybe an actual stack language like Forth could do so? The stack overhead of calling a word is a lot lower to begin with, and ColorForth for already has tail call optimisation[0]. More advanced concatenative languages like Factor[1] or Kitten[2] might be able to perform more interesting optimisations.
Likely because of it's nature as a network first language.
Go is largely speaking a language designed to push bytes over sockets to lots of concurrent clients.
There are two more things: 1. Go has high development efficiency. So we could build complex SQL logic easily. 2. Go is easy to write concurrent code. So we could enjoy the benefit of multi-core cpu. For example, we could use multiple goroutine to scan data, do join, do aggregation.
Yes, Go has higher development efficiency than Rust, especially for new comers. For a storage engine, the most important thing is stable. We focus on write right and stable codes. Rust's compiler is quite strict. Which is good for write right codes. We have many good programmers who is very good at writing Rust. So we do not suffer from the low development efficiency.
About the libraries, we could get most of we need. But some were missing when we begin to develop TiKV (GRPC/HTTP2 for example). But thanks to Rust Team/Community, they give us great supports. GRPC for Rust is available now. We also contribute back to the community. We develop prometheus client for Rust. The Rust community is active and helpful. So we do not worry about the lack of libraries.
I'm enjoying the new write-up on ownership/references/borrowing! It reads more like an fascinating reading on PL concept than a language-specific manual. Great work!
Did the use of ZFS in your system incur noticeable processing cost in your system? Any noticeable increase in latency on your system?