One can imagine that in a world that had taken a turn towards Pascal strings, we would have ended up in a world where string lengths were varint encoded -- 7 bits of string length in the first byte, plus a continuation bit. I'd imagine that the initial (pure software) implementation of this would have no specific length limit, but we'd be back in the < 1 MB system memory days; so, continuing the imagination, early 32-bit CISC processors would reasonably invest in a varint-to-int decoding probably limited to a 4-byte memory read (28 bit string length), which would be a significant accelerator and cover all sane use cases. In the mid 32 bit era the limit of 256 MB strings would become occasionally relevant, but assuming varint-to-int set an overflow flag if the continuation bit on the last byte was set, this could be mitigated sanely; and the late 32 bit era would give a 64-bit read variant (56 bit string length plus overflow flag), which would probably be cleaned up in the 64 bit era to drop the overflow flag due to lack of use once memory address sizes < 56 bit became standard.
The above is, of course, speculation on an alternate timeline -- but it's certainly consistent and believable. What's interesting is what impact such a path would have elsewhere. I'd expect two other consequences: First, our existing uses of varints (e.g. protobufs) would become faster and more efficient, and would probably appear earlier in the timeline. Second, UTF8 would likely become varint-representation-compatible, giving up the (valuable) ability to tell which byte of a codepoint is being decoded in isolation, in exchange for hardware acceleration of encoded<->codepoint conversion; and likely giving us 2^28 codepoints based on a 4-byte maximum length, rather than arbitrarily choosing a maximum length of three bytes to give 2^21 codepoints.
What else would be different? Hard to say… but I do think that size prefixes could have been an alternate route.
I cannot imagine varints working well for in-memory structures, especially in C. Iterating over string with x[i] is (was?) a very common operation, and with varint length there is an extra memory fetch, extra math and a possible jump. On later CPUs, there is also unaligned reads and possible pipeline-stalling data dependency. Smart optimizers can help, but when C was designed, the compilers were pretty simple. And there are other problems -- for example, a simple act of adding a character to string can cause shift-by-1 invalidating existing pointers.
Varints can be useful on disk/network streram, but I've never seen them used for actively modified in-memory storage.
Specifically in the case of string length encodings, these objections don't seem to apply. The memory fetch is from a cache line that is already needed for the (beginning of the) string data, so only adds ops in the case of pure substring access (in which case C patterns of holding offset/length out of band apply). Similarly you're going to be accessing a string -- unaligned reads are the least of your problem if you're doing byte-at-a-time ops; and you can always align the string to a word boundary to get the varint to start at a word boundary.
I totally agree that as a general tool, varints have failures. But for (a) encoding pascal-style string lengths; (b) encoding often-small numbers on the wire; and (c) encoding codepoints they seem to apply fine.
The above is, of course, speculation on an alternate timeline -- but it's certainly consistent and believable. What's interesting is what impact such a path would have elsewhere. I'd expect two other consequences: First, our existing uses of varints (e.g. protobufs) would become faster and more efficient, and would probably appear earlier in the timeline. Second, UTF8 would likely become varint-representation-compatible, giving up the (valuable) ability to tell which byte of a codepoint is being decoded in isolation, in exchange for hardware acceleration of encoded<->codepoint conversion; and likely giving us 2^28 codepoints based on a 4-byte maximum length, rather than arbitrarily choosing a maximum length of three bytes to give 2^21 codepoints.
What else would be different? Hard to say… but I do think that size prefixes could have been an alternate route.