Pointers can be easily mapped from and into integer indices or file offsets in-place, so there is no big difference really.

I think it is generally not a good idea when in-memory representation of objects is identical to the serialized representation. This approach tends to backfire in a number of ways, such as format incompatibility between software versions, or hardware architecture differences, so it is always better to assume that your serialized representation will be different from in-memory representation.