Skip to content

PoC: Introduce column update metadata#15445

Open
gaborkaszab wants to merge 1 commit intoapache:mainfrom
gaborkaszab:main_column_update_metadata
Open

PoC: Introduce column update metadata#15445
gaborkaszab wants to merge 1 commit intoapache:mainfrom
gaborkaszab:main_column_update_metadata

Conversation

@gaborkaszab
Copy link
Contributor

Content:

  • Column update table metadata
  • API to write column update metadata
  • Changes to read and write column update metadata to manifest entries
  • Bunch of TODOs :)

Details:

  • No overlap in field IDs across column updates. Overlapping field IDs removed from older column updates when committing a new one
  • Column updates covers entire data files (no partial column updates)
  • Follow the approach from the proposal where manifests are cloned with same content but new column update metadata
  • Other approach (delete manifest + new manifest file) depends on V4, not covered now
  • Integrated into V3 metadata with made up field IDs. Will have to adjust for V4 once available

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant