-
-
Notifications
You must be signed in to change notification settings - Fork 383
Description
Given some arbitrary key: value store (like the file system), in zarr we almost always want a small subset of that store. For example, when we work with a Zarr v3 array, we only want to expose the keys that are defined by that array (zarr.json and chunks). When we work with a group, we only want to expose the keys for metadata documents for sub-groups and the keys for arrays (see above).
Any other key (.ds_store, random jpgs) should probably not be visible to the array. This would allow a safe store-level "delete this array" operation that wouldn't risk hitting random objects that happen to share a prefix with the array. It would also allow us to safely expose arrays and groups over http without risk of accidentally leaking random user data.
This suggests a store_a -> store_b transformation that changes the keys of store_b relative to store_a, but keeps the same underlying kvstore backend. (this is actually supported today by a StorePath -> StorePath transformation -- the path attribute sets a lower bound in the 1D sequence of prefixes).
For arrays and groups this has to be dynamic -- for each key, we would need to evaluate a predicate against that key that checks if it's a metadata document or a chunk, based on the current zarr format and available metadata documents.