feat: expose snapshot load metrics in order to detect if log truncation was applied or not#7
Draft
feat: expose snapshot load metrics in order to detect if log truncation was applied or not#7
Conversation
…ma (due to schema evolution)
- The problem is visible with our serde modifications - We need to recreate an object store when trying to execute a delta scan that came over the wire in a different SessionContext - Initially, the object store is registered as part of the Delt logical plan - But, over the wire, there is no logical plan, so we need to register the object store inside the execute() function for DeltaScan - If the initial url ends with "/", and if there is no ensure_table_uri, or parse_table_uri for this initial url, there is an inconsistency between the DeltaWireScan object and the internal Delta Plan - The end effect is when we re-register the object store in the runtime map, inside DeltaScan::execute, we register it with a different url (without /), than what is used to search for the object store (with /) - This patch ensures that we ALWAYS pass the initial url to remove the final "/"
cdobre
requested review from
aditanase,
adragomir,
ccciudatu and
mihai-bleont
|
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
cdobre
changed the title
…: DeltaNextPhysicalCodec
ccciudatu
force-pushed
the
main
branch
2 times, most recently
from
b238c0b to
2f04aab
Compare
Signed-off-by: Adrian Tanase <atanase@adobe.com>
Signed-off-by: Adrian Tanase <atanase@adobe.com>
…r/de) Signed-off-by: Adrian Tanase <atanase@adobe.com>
Collaborator
|
I think this needs a rebase, it also contains Andrei's commit |
… fields nullable.
cdobre
force-pushed
the
log-size-limiter-metrics
branch
from
bc17a57 to
ccadfd9
Compare
cdobre
force-pushed
the
log-size-limiter-metrics
branch
from
24481ae to
cc5585c
Compare
cdobre
changed the title
adragomir
force-pushed
the
main
branch
2 times, most recently
from
0dae398 to
1c13763
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
There is no existing mechanism to detect if a table which was configured with a LogSizeLimiter policy has been actually truncated or not. The clients can see if the configuration contains LogSizeLimiter settings but not if the truncation actually happened.
This adds a new set of metrics on Snapshot structure which can be used by client to see if the truncation was applied and the effect of that truncation: