fix(flow-php/parquet): both engines read and write logical types identically - #2659
Merged
Merged
Conversation
…tically - honour TIMESTAMP/TIME units, floor epoch splits, write isAdjustedToUTC=true - DATE written as the value's wall-clock day - decimals on INT32/INT64/FLBA/BYTE_ARRAY of any width, shortest repr rounded half away - ConvertedType-only columns read as temporal/decimal values - converters resolved per column via static forColumn() - arrow-ext: units, UTC flag, DATE days lane, Decimal256 read and schema
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 1.x #2659 +/- ##
============================================
+ Coverage 88.56% 88.65% +0.08%
- Complexity 26534 26551 +17
============================================
Files 2043 2044 +1
Lines 78096 78145 +49
============================================
+ Hits 69169 69281 +112
+ Misses 8927 8864 -63 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves: #xxx
Change Log
Added
flow-php/parquet- DECIMAL on INT32, INT64, BYTE_ARRAY and FLBA of any widthflow-php/parquet-TimeUnitenum with MILLIS, MICROS, NANOSflow-php/arrow-ext-unitandutcschema keys for TIMESTAMP and TIMEflow-php/arrow-ext- Decimal256 readingFixed
flow-php/parquet- TIMESTAMP and TIME honour MILLIS/NANOS unitsflow-php/parquet- DATE written as the value's wall-clock dayflow-php/parquet- ConvertedType-only columns read as temporal and decimal valuesflow-php/parquet- BYTE_ARRAY decimal statistics without length prefixflow-php/arrow-ext- pre-1970 timestamps with a fraction read correctlyflow-php/arrow-ext- declared TIMESTAMP/TIME unit written instead of MICROSflow-php/etl-adapter-parquet- legacy TIMESTAMP/TIME/DECIMAL columns read as datetime/time/floatChanged
flow-php/parquet- TIMESTAMP written with isAdjustedToUTC=trueflow-php/parquet-Converter::isFor()replaced by staticforColumn()flow-php/parquet-Timestamp/Timelogical types take aTimeUnitflow-php/parquet-encode_decimal()/decode_decimal()dropByteOrderflow-php/parquet- float decimals rounded half away from zero on the shortest reprflow-php/arrow-ext- DATEintlane is days since epochRemoved
flow-php/parquet-Int32DateTimeConverterflow-php/parquet-Option::ROUND_NANOSECONDSDeprecated
Security