v.io/syncbase/x/ref/services/syncbase/localblobstore/chunkmap: Introduce chunkmap
The CL introduces chunkmap, which keeps a non-volatile mapping from chunk
content hashes to locations within blobs, and from blobs to lists of chunks
within them.
It will ultimately be integrated with fs_cablobstore, and use chunk data from
chunker.
chunkmap is not intended to be called directly by the client. Rather, the
client will call on the localblobstore interface, and fs_cablobstore will call
chunkmap on the client's behalf.
Expected uses:
- When a blob is added:
blobstore will use chunker to find the content hashes of the chunks,
and record them with chunkmap.AssociateChunkWithLocation().
- When a blob is deleted:
blobstore will call chunkmap.DeleteBlob()
- When the blopbstore is garbage collected:
blobstore will enumerate all blobs in the chunkmap,
and delete any that no longer appear in the store.
- When a device X wishes to transfer blob B to device Y.
X will get the list of chunk hashes for B using
chunkmap.NewBlobStream(), using its Advance()/Value() calls, and send
them to device B.
Y will callchunkmap.LookupChunk() on each chunk. For those that can be
found locally, it remembers the blob, offset and size data. For those
that cannot, it tells X it needs those chunks (perhaps by giving the
index).
X will get each chunk that Y needs, either by remembering the location
information from its earlier scan, or by retrieving it with
chunkmap.LookupChunk(). Y transfers those chunks to X.
Y writes the blob using the chunks it already has, plus those just
received from X.
chunkmap.AssociateChunkWithLocation() is used to add
the associatation between the chunks of the new blob and their
locations within the blob.
Change-Id: I59e57c2db14a86c75500b984a0477ce5f2cd80e2
2 files changed
