If we don't store every chunk hash to block hash, how can we download a model file. For example if a model does not have a key chunk, the shard info only contains "file hash -> list of chunks", the CAS only has "block hash -> block content", how can we know which block is needed ?
liming
largegpt
ยท
AI & ML interests
None yet
Recent Activity
commented on
an
article
3 days ago
From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub
commented on
an
article
4 days ago
From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub
Organizations
None yet
largegpt's activity
commented on
From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub
3 days ago
commented on
From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub
4 days ago
How can we find the chunk content using chunk hash? The CAS system only stores "block_hash -> block_content", Where does the map of chunk to block?
what does the shards store? Is it "file_name, shard_id, chunk_hash, block_hash"?