I wasn't really consider space overhead, which as you notice is small, the encoding just something I noticed. Just json+bloatified binary+compression seemed the epitome of solving problems with a hammer when you know the hammer really well.

But as commented to me, that was incorrect, the payload in the files I saw was actually binary-encoded metadata. The actual archives are stored only in extracted form in the client and .tar.gz in the registry. Not that putting binary metadata in JSON is really optimal to me either, but maybe there are terrific reasons for doing it. Just check /var/lib/docker/image/overlay2/layerdb/sha256/*/tar-split.json.tar.gz; the fragments seem to be straight from tar, so maybe tar is used to interpret them?

Certainly .tar.gz files are not the most optimal for the purpose of incremental updates either. It's difficult to say how much a more sophisticated transfer protocol/storage format would benefit registry servers, though some clients could benefit a lot from it.

If a space-efficient merkle-tree-hashed archive-format suitable for efficient incremental updates doesn't exist, then someone should make one ;-).