[fix](be) Materialize variant defaults when copying ranges#64085
Closed
eldenmoon wants to merge 1 commit into
Closed
[fix](be) Materialize variant defaults when copying ranges#64085eldenmoon wants to merge 1 commit into
eldenmoon wants to merge 1 commit into
Conversation
### What problem does this PR solve?
Issue Number: close #0
Related PR: #0
Problem Summary: Copying a VARIANT subcolumn range could skip the source subcolumn's pending default suffix. The destination variant kept the requested logical row count while its finalized root column could remain physically shorter than the copied range. Exchange and join paths that copy blocks containing such VARIANT columns could then read missing rows, fail, or return unstable results. The fix appends the remaining default rows after copied physical parts so the logical and physical row counts stay aligned.
### Release note
Fix an issue where queries using VARIANT columns through exchange or join paths could fail or return unstable results when copied VARIANT subcolumns contained pending default rows.
### Check List (For Author)
- Test
- [x] Unit Test: ./run-be-ut.sh --run --filter='ColumnVariantTest.insert_range_from_materializes_pending_default_suffix'
- [x] Build: ./build.sh --be
- [x] Manual test: attached local-shuffle LEFT ANTI query loop, 8 workers x 100 iterations, all 800 results were 0 with no ColumnVector or insert_range_from errors.
- [x] Manual test: constructed complex VARIANT hash join with local shuffle enabled and disabled returned identical hashes; local shuffle enabled loop was stable for 100 iterations.
- Behavior changed:
- [x] Yes. VARIANT range copies now materialize pending default rows so copied columns remain physically aligned with logical row counts.
- Does this need documentation?
- [x] No.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29172 ms |
Contributor
TPC-DS: Total hot run time: 169682 ms |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #0
Related PR: #0
Problem Summary: Copying a VARIANT subcolumn range could skip the source subcolumn's pending default suffix. The destination variant kept the requested logical row count while its finalized root column could remain physically shorter than the copied range. Exchange and join paths that copy blocks containing such VARIANT columns could then read missing rows, fail, or return unstable results. The fix appends the remaining default rows after copied physical parts so the logical and physical row counts stay aligned.
Release note
Fix an issue where queries using VARIANT columns through exchange or join paths could fail or return unstable results when copied VARIANT subcolumns contained pending default rows.
Check List (For Author)