You assert that ECC RAM being necessary for ZFS is just a myth but provide no justification for why that is untrue.
Is it not the case that if you don't have ECC memory, ZFS could end up writing a checksum that does not match the data if you get a bitflip in just the right (wrong) spot?
Yes, indeed. ECC RAM is better than non-ECC RAM, also for ZFS.
The myth, popularized by a notorious thread on the TrueNAS forums [1], is specifically that ZFS requires ECC RAM, and will do worse than other filesystems without it, because scrubbing will multiply a single bitflip into a failed pool.
A ZFS core developer says that that isn't the case [2]. Here's some more reasoning [3], also about many other myths.
The only hangup with the myth "debunking" is that the point is that the corruption doesn't happen to a per-disk buffer, but to the in-flight data before it's persisted to your stripe.
Which means all copies of the data will be corrupted. This can be anything from an irrecoverable file to complete filesystem corruption.
But generally, yeah, not any more dangerous than any other filesystem, and ECC used to be cheap so it was a no-brainer, you should have backups anyway TBH if we're being honest about storage resiliency.
What has been debunked is the "scrub of death" issue, on a scrub a bad bit flip would cause an error, which would be copied over with good data -- well it was technically good before. It would be statistically difficult to have a fault on a read, then a clean read, then a second bit flip destroying the data.
Usually if you’re using a NAS you don’t want to lose data, ZFS is not significantly more sensitive than everything else.
But everything is actually quite sensitive.
We’ve accepted lack of ECC because Intel decided it would be a product line differentiator, and serious customers who didn’t want random crashes or to lose data would buy chips with ECC.
It’s actually less of an issue these days because DDR5 has (by spec) some in-line ECC; won’t help with multi-bit errors but its an improvement on what came before.
DDR5 on-die ECC is to achieve acceptable yields in the face of denser process nodes that decrease the reliability of RAM cells. It’s not clear how much of an improvement that is to what we had before, other than allowing for higher RAM speeds. It doesn’t replace side-band ECC.
> We’ve accepted lack of ECC because Intel decided it would be a product line differentiator, and serious customers who didn’t want random crashes or to lose data would buy chips with ECC.
AMD has been allowing ECC on lots of regular hardware for a long time.
People don't tend to buy ECC for desktop use because it costs significantly more (used server ram is/was often cheap... but it often doesn't work in desktop boards), and the performance specs are poor.
My home servers are mostly retired desktops, so they get my old desktop ram and I don't want to pay premium prices for jedec speed ecc ram on my desktops, thanks.
Since DDR5 doesn't include reporting on bit errors (afaik), it likely means much fewer single bit errors, but most experienced errors will be multi-bit. Although, I dunno what proportion of bit errors is on the ram chips and what's on the bus... there's no protection from bus errors.
If there were reporting, you could replace chips with high error rates, but without reporting you'll keep running them until they fail enough to notice.
> AMD has been allowing ECC on lots of regular hardware for a long time.
Often unsupported or untested by the motherboard manufacturer, because the precedent set quite a while ago has stood so strongly.
> because it costs significantly more [...] and the performance specs are poor
Because it's niche. If a large share of the desktop market was using ECC memory, the extra cost would have been 10-12%, and a typical kit would push the clocks and timings at least as much as non-ECC memory.
> Since DDR5 doesn't include reporting on bit errors (afaik), it likely means much fewer single bit errors, but most experienced errors will be multi-bit. Although, I dunno what proportion of bit errors is on the ram chips and what's on the bus... there's no protection from bus errors.
Yeah that sucks. I'm really hoping that DDR6 standardizes a light form of end to end ECC. (Bursts are 24 bits wide and 24 bits deep, so on top of the 512 bits of payload there's a bunch of extra bits that can be used for ECC. If memory chips would just store 4 extra bits per 128, that's enough for reasonable ECC on a per-burst basis.)
> the precedent set quite a while ago has stood so strongly.
I wasn't really around for it, but I appreciate the early penny pinchers that developed fake parity ram. The IBM PC required a parity bit per byte?, but you could add an xor chip to calculate it at read and save out on the expensive ram chip.
I wouldn't mind paying a little more for ECC, what with the extra chip and a little more circuitry, but desktop ECC often starts at 50% more, which I'm way too cheap to pay for.
My favorite parity trick was how the N64 was designed around ECC RAM chips but didn't bother using ECC, so it has a ninth bit at every address that the CPU ignores and the GPU uses as extra storage.
I know that LPDDR5 has ECC, and not just single bit AFAIK, but if you enable it you lose some memory capacity (the 128 bit bus minus 16 bit for ECC error correction, making it effectively 112 bits)
Had it happen to my PBS backups running on ZFS without ECC just the other day. Turned out to be heat related, memory was fine but during long backups heat was causing bit flips.
This seems to store the zfs send stream. That's a bad idea.
> Incremental ZFS send streams do not have any of these properties and full ZFS send streams only have a few of them. Neither full nor incremental streams have any resilience against damage to the stream; a stream is either entirely intact or it's useless. Neither has selective restores or readily available indexes. Incremental streams are completely useless without everything they're based on. All of these issues will sooner or later cause you pain if you use ZFS streams as a backup format.
If you’re looking to ZFS send streams for many of ZFS’s guarantees, absolutely.
However ZFS replication was originally designed with the assumption and use case in mind that organizations would store the send streams as opaque blobs on tape. This is, in part, why storing send streams as blobs is still a thing people do.
There are some use cases where this makes sense. I’ve stored full send streams of archived ZFS file systems in S3(-compatible services) where integrity is handled at the platform level. In that use case I didn’t benefit from having every copy of the filesystem in question running on live storage media, and incremental sends/snapshots weren’t on the table. (I also SHA checksummed the resulting files, and did restore tests.)
There is also a misconception that frequently gets brought up in the ZFS community that the send stream format isn’t stable between versions and cannot be relied upon in future, but it absolutely is stable. In fact the ZoL manpage for send explicitly states that it is. As with anything in ZFS though, you want to move versions forward or not at all, rather than backward.
we are writing blobs/ objects to zfs tape volumes. it gives us an extra layer of defense from ransom attacks and satisfies our 321 requirement. We make the blobs transparent with some metadata tags. The objects are recorded in the catalog and we can pull individual files out of the blob. deepspace storage manages the tape gateway and catalog for the objects. short answer yes storing send streams to tape is doable robust workflow.
Thank you! Yeah, a 7.4 is humongous. I did wonder about this because my wife's family reported quite intensive shaking and I've been in a 6-something before and that was quite disruptive. I can't imagine a 5.1 being that much.
One pitfall of this is that the decompressed contents of your initramfs must fit within half of your physical RAM since Linux decompresses it into a tmpfs.
Or if you set rootfstype=ramfs, then you can take up to all of physical RAM, but ramfs isn't swappable.
Wait I'm not following, why must the decompressed contents of the initramfs fit within half of physical ram?
Initramfs (and tmpfs) are swappable right? As you say, ramfs is the one that isn't swappable. So if your initramfs is "to big" can't it just be swapped out?
> Wait I'm not following, why must the decompressed contents of the initramfs fit within half of physical ram?
It makes sense if you replace "initramfs" with "initrd":
> One pitfall of this is that the decompressed contents of your ~initramfs~ _initrd_ must fit within half of your physical RAM since Linux decompresses it into a tmpfs.
> Or if you set rootfstype=ramfs, then you can take up to all of physical RAM
I'm very confused by your comment. The grandparent comment talks about using the gitrevisions syntax in a GitHub URL to search the reflog stored on GitHub. Nothing to do with your local clone of a repository.
Seriously, all the people of my parents generation have a physical book with account names and passwords in it. You keep it in a safe place, and nobody can hack it remotely.
Plus, should your heirs or caretakers need to access it, it's all there in plain text for them.
Other than a few web forums, I don't use my phone for anything that requires a password. Certainly not for email, banking, or anything that involves personal data, money, or business. A lot of stuff, I don't do on my desktop either.
Anything connected to the internet is insecure. No, it's not as convenient to do things by phone or paper mail, but I value security more than convenience.
Is it not the case that if you don't have ECC memory, ZFS could end up writing a checksum that does not match the data if you get a bitflip in just the right (wrong) spot?