Hacker Newsnew | past | comments | ask | show | jobs | submit | tandemstrong's commentslogin

Well, it will use much less than that. The real "memory cost" of decompression will be : + compressed size

and that's it. Decompression doesn't use any memory, only the I/O buffers.

On the compression side, the memory cost is : + buffer for compression (typically == size of data to compress) + 16 KB for tables

And that's it.

We are very very far from the 12 MB mentioned.


This benchmark is a fairly bad reference.

It measures memory usage from the lz4 command line utility, without mentioning it. This is completely different from lz4 algorithm.

Effectively, it measures input / output buffers, which are external to lz4. Plus it uses highest settings for such buffers, also without mentioning it. Using -B4 setting, for example, would reduce I/O buffer memory by a 50X factor.

lz4 itself uses only 16KB for compression.


LZ4 can do that.

There is a function, called LZ4_decompress_safe_usingDict() which seems to match your objectives.

In case of doubt, you should ask directly the author, at : https://groups.google.com/forum/#!forum/lz4c


The performance figures displayed on those pages look exceptional. Has anyone experimented with it already ?


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: