diff --git a/documentation/components/cli/docs.md b/documentation/components/cli/docs.md index 143734fe21..506580ec8c 100644 --- a/documentation/components/cli/docs.md +++ b/documentation/components/cli/docs.md @@ -262,6 +262,7 @@ Options: --schema-union-by-name Union the column sets of every sniffed source instead of taking the first one. --stats-schema[=STATS-SCHEMA] Prints schema of executed data transformation pipeline. [default: false] --stats-columns[=STATS-COLUMNS] Prints number of rows in dataset. [default: false] + --stats-sources[=STATS-SOURCES] Prints the rows every source declared next to the rows it yielded. [default: false] -h, --help Display help for the given command. When no command is given display help for the list command --silent Do not output any message -q, --quiet Only errors are displayed. All other output is suppressed @@ -406,6 +407,7 @@ Options: --config=CONFIG Path to a local php file that MUST return instance of: Flow\ETL\Config --stats-schema[=STATS-SCHEMA] Prints schema of executed data transformation pipeline. [default: false] --stats-columns[=STATS-COLUMNS] Prints number of rows in dataset. [default: false] + --stats-sources[=STATS-SOURCES] Prints the rows every source declared next to the rows it yielded. [default: false] -h, --help Display help for the given command. When no command is given display help for the list command --silent Do not output any message -q, --quiet Only errors are displayed. All other output is suppressed diff --git a/documentation/components/core/core.md b/documentation/components/core/core.md index 4835c72fc9..e6467b93cd 100644 --- a/documentation/components/core/core.md +++ b/documentation/components/core/core.md @@ -48,7 +48,8 @@ tree read from the bottom up: a node's children are where its rows come from, an rows reach them - the source is `#1`. What a node does is listed under it. The verbs build a plan with no consumer on top; **the trigger adds the one it needs**, so `explain()` takes the -trigger it should print - `Trigger::rows` by default, `Trigger::run` for the plan `run()` executes. +trigger it should print - `Trigger::rows` by default, `Trigger::run` for the plan `run()` executes, `Trigger::count` +for the plan `count()` executes. ```php echo data_frame() @@ -154,6 +155,9 @@ step sits in is where rows stop flowing through, and every step lists the settin storage among them, which the logical stages cannot know because the planner picks it. ```php +$users = [['id' => 1, 'name' => 'Alice'], ['id' => 2, 'name' => 'Bob']]; +$emails = [['id' => 1, 'email' => 'alice@example.com']]; + echo data_frame() ->read(from_array($users)) ->join(data_frame()->read(from_array($emails)), join_on(['id' => 'id'], join_prefix: 'joined_'), Join::left) @@ -164,13 +168,14 @@ echo data_frame() ```text Physical plan -│ Schema: derived +│ Columns: id, name, joined_id, joined_email └─ Pipeline #1 │ Processor: CollectingProcessor │ Schema: declared │ Loader: StreamLoader └─ Pipeline #0 │ Extractor: ArrayExtractor + │ Statistics: rows exact 2 · size unknown │ Processor: HashJoinProcessor │ Join: left │ On: id = id @@ -180,9 +185,12 @@ Physical plan │ Batch: 1000 └─ Right side: Pipeline #0 Extractor: ArrayExtractor + Statistics: rows exact 1 · size unknown ``` A joined frame is planned apart, so its pipelines are numbered apart - `Right side:` says which plan they belong to. +Every source lists what it declares about itself before a row is read - `exact n`, `≤ n` (a guaranteed bound), +`~n ±e%` (an estimate) or `unknown`. A source that declares nothing lists no `Statistics:` line. Reaching this stage plans the frame, so a source that infers its schema by reading is read here; the logical stages never read a row. @@ -235,6 +243,7 @@ Before a frame runs, the optimizer rewrites its plan. `Optimizer::default()` run | `CombineSortAndLimit` | `sortBy()` followed by `limit()` keeps only the top rows instead of sorting all | | `PushLimitIntoSource` | the extractor stops reading once the limit (plus any `offset()`) is reached | | `PushFilterIntoSource` | a `filter()` on partition columns skips whole partition directories | +| `CountFromStatistics` | `count()` over a source that knows its rows exactly reads that number, no rows | Rules live in `Flow\ETL\Optimizer\Rule` and are configured through `config_builder()->optimizer()`: @@ -286,6 +295,7 @@ For detailed information about specific DataFrame operations, see the following - **[Partitioning](/documentation/components/core/partitioning.md)** - Data partitioning for efficient processing - **[Caching](/documentation/components/core/caching.md)** - Performance optimization through caching - **[Floe File Format](/documentation/components/core/floe.md)** - Flow's native self-describing binary row format +- **[Source Statistics](/documentation/components/core/statistics.md)** - What a source declares about its rows and bytes before it is read - **[Data Retrieval](/documentation/components/core/data-retrieval.md)** - Methods for getting processed data ### Data Quality & Validation diff --git a/documentation/components/core/data-retrieval.md b/documentation/components/core/data-retrieval.md index b321e488cc..3192a17f29 100644 --- a/documentation/components/core/data-retrieval.md +++ b/documentation/components/core/data-retrieval.md @@ -99,9 +99,17 @@ $totalCount = $dataFrame->count(); echo "Total rows: $totalCount\n"; ``` -> **Performance Warning**: The `count()` method must process the entire dataset to return the total count, which can -> be expensive for large datasets. Consider whether you actually need the exact count or if an approximation would -> suffice. +A frame that only reads one source is counted from the source's statistics when they are exact - a Parquet or Floe +file answers from its footer without reading a row (`explain(Trigger::count)` shows it): + +```php +read(from_parquet('orders.parquet'))->count(); +``` + +> **Performance Warning**: Any other frame - a filter, a limit, a new column, a join, an estimated source - is executed +> in full to count its rows, which can be expensive for large datasets. ## Iteration with Callback @@ -124,4 +132,34 @@ sinks and returns a `Report` when asked to analyze. $dataFrame->write(to_json('out.json'))->run(); $report = $dataFrame->write(to_json('out.json'))->run(analyze: analyze()->withSchema()); +``` + +`withSourceStatistics()` puts the rows every source declared next to the rows it actually yielded - the output row +count alone cannot tell them apart once a filter or a join sits in between: + +```php +read(from_parquet('orders/*.parquet')) + ->filter(ref('total')->greaterThan(lit(100))) + ->run(analyze: analyze()->withSourceStatistics()); + +foreach ($report->sources() as $source) { + echo $source->extractor; // ParquetExtractor + echo $source->declared->rows->estimate; // 20000 - extrapolated from the first footer + echo $source->rows; // 29000 - read + echo $source->rowsError(); // 0.31 - |declared - read| / read +} +``` + +`rowsError()` is null without an estimate, and for a read that did not cover the whole source - a pushed `limit()` or +partition filter, or a read stopped early - since the declaration describes the whole source: + +```php +sources() as $source) { + $source->complete; // false after a pushed limit() or a stopped read +} ``` \ No newline at end of file diff --git a/documentation/components/core/floe.md b/documentation/components/core/floe.md index 95e44491a7..9c2d84ca54 100644 --- a/documentation/components/core/floe.md +++ b/documentation/components/core/floe.md @@ -355,11 +355,15 @@ scanning rows**: "schema": { /* the file's single schema */ }, "sections": [ { "offset": 6, "partitionsId": 0, "rowCount": 2 } ], "partitions": [ { "country": "PL" } ], - "totalRows": 2, + "statistics": { "rows": 2, "byteSize": 118 }, "metadata": { /* typed key/value, Schema\Metadata */ } } ``` +- **`statistics`** holds the file's row count and the uncompressed bytes of its data frames (header and + footer excluded), so a reader can size the file before reading a row. The footer and its sections + ignore keys they do not know, so a field added later does not break an older reader. + - **`sections`** map a byte `offset` → `partitionsId` + `rowCount`, so a reader can skip whole sections (offset/limit pushdown) and know each section's partition combination up front. Sections bound partition combinations and appends only; every section shares the file's one schema. diff --git a/documentation/components/core/statistics.md b/documentation/components/core/statistics.md new file mode 100644 index 0000000000..c51dfec7d3 --- /dev/null +++ b/documentation/components/core/statistics.md @@ -0,0 +1,100 @@ +# Source Statistics + +[DOC_LINK:/documentation/components/core/core.md] + +[TOC] + +Every extractor says what it knows about its data before a row is read - how many rows, how many bytes: + +```php +statistics(); + +$statistics->rows; // Flow\ETL\Cardinality +$statistics->size; // Flow\ETL\Cardinality, bytes +``` + +The answer describes the whole source, with no `limit()` or partition filter applied. + +## Cardinality + +A `Cardinality` carries a guarantee and a guess side by side: + +```php +exactly(); // 1000 - null unless the count is exact +Cardinality::approximately(1_000)->confident(0.25); // null - the estimate is looser than 25% +Cardinality::unknown()->isUnknown(); // true +``` + +`atMost` is never exceeded. `estimate` may be wrong in either direction, by roughly `relativeError`. + +## What each source declares + +| Source | rows | size | +|------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|------------------------------| +| `from_parquet()`, `from_floe()` | exact from the footers; an estimate when a glob's footers were not all read | exact, or estimate on a glob | +| `from_csv()`, `from_json_lines()` | unknown before the schema is sniffed; exact when the sniff read every file whole; else an estimate from the mean row size | exact listed bytes | +| `from_json()`, `from_excel()` | unknown before the schema is sniffed; exact when the sniff read every file whole; else unknown | exact listed bytes | +| `from_text()`, `from_xml()` | unknown | exact listed bytes | +| `files()` | exact listed file count | exact listed bytes | +| `from_path_partitions()` | exact listed file count | unknown | +| `from_array()`, `from_rows()`, `from_sequence_*()` | exact | unknown | +| `from_cache()` | exact from the cache index | unknown | +| `from_pgsql_*()`, `from_dbal_*()` | estimate from `EXPLAIN` (never `ANALYZE`), bounded by `withMaximum()` | unknown | +| `from_google_sheet()` | bounded by the sheet's grid size once the schema is sniffed | unknown | +| `from_all()`, `batches()`, `batched_by()` | what the wrapped extractors declare, merged | same | +| `from_avro()`, `from_*_http_*()`, `from_memory()`, `from_data_frame()` | unknown | unknown | + +## Where statistics are used + +- `explain()->toString(Stage::physical)` lists them under every source: `Statistics: rows exact 1 000 · size exact 327 527 B`. +- `count()` over a source with exact rows reads the number instead of the rows - see + [Data Retrieval](/documentation/components/core/data-retrieval.md). +- `run(analyze: analyze()->withSourceStatistics())` puts the declared rows next to the rows each source yielded. + +## A custom extractor + +`statistics()` is part of the `Extractor` contract. A source that knows nothing returns `new Statistics()`: + +```php +statistics ??= new Statistics( + rows: Cardinality::approximately($this->api->totalHint(), Cardinality::DEFAULT_RELATIVE_ERROR), + ); + } + + // extract(), schema(), withSchema() +} +``` + +`statistics()` is called at most once per run, only when something needs the answer. Memoise what it reads. It +may spend only what the run spends anyway - a listing it must produce, metadata of files `schema()` already opens - +and never open an extra file or make an extra remote call. A database source may ask its planner (`EXPLAIN`), never +run the query. diff --git a/documentation/upgrading.md b/documentation/upgrading.md index dca41dedcf..fe2111ab3d 100644 --- a/documentation/upgrading.md +++ b/documentation/upgrading.md @@ -238,6 +238,46 @@ after it shift by one. `Stage::physical` is new - see the core documentation. | `CSVFileReader::samples()` yields `Generator`s | yields `CSVFileSample` (`IteratorAggregate`); `$unit->getIterator()` for the generator | | `CSVFileReader::sample($source)` | `new CSVFileSample($opener, $source)` | +### 24) `flow-php/etl` - Floe footer carries a statistics block, existing `.floe` files must be rewritten + +| Before | After | +|-------------------------------------------------|-----------------------------------------------------------------| +| footer key `totalRows` | `statistics.rows` | +| - | `statistics.byteSize` - uncompressed data bytes | +| footer/section parsers rejected any unknown key | unknown keys ignored, so later additions are not a break | +| reading a file written by 0.44.x | `Floe footer is malformed: ... "statistics"` - rewrite the file | + +Header version stays `0x02`: the change is in the footer, not the frame layout. Files written by 0.44.x +cannot be read. Regenerate them from their source, or export them with 0.44.x to another format before +upgrading. + +### 25) `flow-php/etl` - every `Extractor` declares `statistics()` + +| Before | After | +|---------------------------------------------------------|-------------------------------------------------------------------------------------------------| +| `Extractor`: `extract()`, `schema()`, `withSchema()` | `+ statistics(): Statistics` - a custom extractor that knows nothing returns `new Statistics()` | +| `SequenceGenerator`: `generate()` | `+ rows(): Cardinality` - how many items `generate()` yields | +| `SelfDescribingFile`: `close()`, `schema()`, `source()` | `+ statistics(): Statistics` - that one file's rows and bytes from its own metadata | +| `CacheIndex` rows: `key` | `key`, `rows` - an index written by 0.44.x reads fine, its row count is unknown | +| `JsonFileReader::samples()` yields `Generator`s | yields `JsonFileSample` (`IteratorAggregate`) | +| `WorkbookSampler::samples()` yields `Generator`s | yields `WorkbookSheetSample` (`IteratorAggregate`) | + +```php +final class MyExtractor implements Extractor +{ + public function statistics(): Statistics + { + return new Statistics(); + } +} +``` + +### 26) `flow-php/etl` - `Report` takes the source statistics of the run + +| Before | After | +|--------------------------------------------------|------------------------------------------------------------------------------------------------------| +| `new Report(?Schema $schema, Statistics $stats)` | `new Report(?Schema $schema, Statistics $stats, ?array $sources)` - `null` unless analyzed with them | + --- ## Upgrading from 0.43.x to 0.44.x diff --git a/src/adapter/etl-adapter-avro/src/Flow/ETL/Adapter/Avro/FlixTech/AvroExtractor.php b/src/adapter/etl-adapter-avro/src/Flow/ETL/Adapter/Avro/FlixTech/AvroExtractor.php index 5bc3921307..e1c982a389 100644 --- a/src/adapter/etl-adapter-avro/src/Flow/ETL/Adapter/Avro/FlixTech/AvroExtractor.php +++ b/src/adapter/etl-adapter-avro/src/Flow/ETL/Adapter/Avro/FlixTech/AvroExtractor.php @@ -9,6 +9,7 @@ use Flow\ETL\Extractor; use Flow\ETL\Extractor\FileExtractor; use Flow\ETL\Extractor\PathFiltering; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -65,4 +66,9 @@ public function withSchema(Schema $schema): static return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVExtractor.php b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVExtractor.php index 534ebc7af9..229bde0285 100644 --- a/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVExtractor.php +++ b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVExtractor.php @@ -4,6 +4,7 @@ namespace Flow\ETL\Adapter\CSV; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InferredSchemaException; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Extractor; @@ -12,9 +13,11 @@ use Flow\ETL\Extractor\FileExtractor; use Flow\ETL\Extractor\FileReading; use Flow\ETL\Extractor\InfersSchema; +use Flow\ETL\Extractor\ListedFiles; use Flow\ETL\Extractor\MetadataColumnsExtractor; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Row\RawRowValues; use Flow\ETL\Rows; @@ -52,10 +55,19 @@ final class CSVExtractor implements private CSVReadOptions $readOptions; + /** + * What the last schema inference sampled; null until one ran. + */ + private ?CSVSampledFiles $sampled = null; + private ?Schema $schema = null; + private ?Statistics $statistics = null; + private readonly Filesystem $filesystem; + private ?ListedFiles $listed = null; + public function __construct( private readonly Path $path, Filesystem $filesystem = new NativeLocalFilesystem(), @@ -99,11 +111,13 @@ public function extract(FlowContext $context, ?int $limit = null, Filter $pathFi $derived = $this->derivedSchema; if ($derived === null) { + $samples = new CSVSamples($reader->samples($this->inference->sampleSize)); $derived = $this->derivedSchema = (new SchemaInferrer( $this->inference, new StringTypeNarrower($this->inference->candidates()->toArray()), - ))->infer($reader->header()->names, $reader->samples($this->inference->sampleSize)); + ))->infer($reader->header()->names, $samples); + $this->sampled = $samples->sampledFiles(); } $base = $fileColumns->withoutTail($derived); @@ -184,6 +198,8 @@ public function inferSchema(SchemaInferenceBuilder $builder): static { $this->inference = $builder->build(); $this->derivedSchema = null; + $this->sampled = null; + $this->statistics = null; return $this; } @@ -205,11 +221,13 @@ public function schema(): Schema iterator_to_array($this->sourceFiles($this->filesystem, $this->path), false), ); + $samples = new CSVSamples($reader->samples($this->inference->sampleSize)); $derived = $this->derivedSchema = (new SchemaInferrer( $this->inference, new StringTypeNarrower($this->inference->candidates()->toArray()), - ))->infer($reader->header()->names, $reader->samples($this->inference->sampleSize)); + ))->infer($reader->header()->names, $samples); + $this->sampled = $samples->sampledFiles(); } return $fileColumns->declare($fileColumns->withoutTail($derived)); @@ -225,9 +243,34 @@ public function source(): Path return $this->path; } + /** + * Rows come from the sample schema inference already read; without one they are unknown, a sniff is never run + * just for them. + */ + public function statistics(): Statistics + { + if ($this->statistics !== null) { + return $this->statistics; + } + + $this->listed ??= ListedFiles::of($this->sourceFiles($this->filesystem, $this->path)); + $size = $this->listed->bytes; + + if ($this->sampled === null) { + return new Statistics(rows: Cardinality::unknown(), size: $size); + } + + return $this->statistics = new Statistics( + rows: $this->sampled->estimatedRows($this->listed->count, $size), + size: $size, + ); + } + public function withBOMRemoval(bool $removeBOM): self { $this->derivedSchema = null; + $this->sampled = null; + $this->statistics = null; $this->readOptions = $this->readOptions->withRemoveBOM($removeBOM); return $this; @@ -243,6 +286,8 @@ public function withCharactersReadInLine(int $charactersReadInLine): self } $this->derivedSchema = null; + $this->sampled = null; + $this->statistics = null; $this->readOptions = $this->readOptions->withCharactersReadInLine($charactersReadInLine); return $this; @@ -251,6 +296,8 @@ public function withCharactersReadInLine(int $charactersReadInLine): self public function withEmptyToNull(bool $emptyToNull): self { $this->derivedSchema = null; + $this->sampled = null; + $this->statistics = null; $this->readOptions = $this->readOptions->withEmptyToNull($emptyToNull); return $this; @@ -259,6 +306,8 @@ public function withEmptyToNull(bool $emptyToNull): self public function withEnclosure(string $enclosure): self { $this->derivedSchema = null; + $this->sampled = null; + $this->statistics = null; $this->readOptions = $this->readOptions->withEnclosure($enclosure); return $this; @@ -267,6 +316,8 @@ public function withEnclosure(string $enclosure): self public function withEscape(string $escape): self { $this->derivedSchema = null; + $this->sampled = null; + $this->statistics = null; $this->readOptions = $this->readOptions->withEscape($escape); return $this; @@ -275,6 +326,8 @@ public function withEscape(string $escape): self public function withHeader(bool $withHeader): self { $this->derivedSchema = null; + $this->sampled = null; + $this->statistics = null; $this->readOptions = $this->readOptions->withHeader($withHeader); return $this; @@ -290,6 +343,8 @@ public function withSchema(Schema $schema): static public function withSeparator(string $separator): self { $this->derivedSchema = null; + $this->sampled = null; + $this->statistics = null; $this->readOptions = $this->readOptions->withSeparator($separator); return $this; diff --git a/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVFileSample.php b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVFileSample.php index 10bb522b1c..719409c244 100644 --- a/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVFileSample.php +++ b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVFileSample.php @@ -16,11 +16,13 @@ /** * @implements IteratorAggregate */ -final readonly class CSVFileSample implements IteratorAggregate, SniffsColumnTypes +final class CSVFileSample implements IteratorAggregate, SniffsColumnTypes { + private ?CSVSampledRows $sampled = null; + public function __construct( - private CSVSourceOpener $opener, - private SourceFile $source, + private readonly CSVSourceOpener $opener, + private readonly SourceFile $source, ) {} /** @@ -48,9 +50,26 @@ public function sniffColumnTypes( $open = $this->opener->open($this->source); try { - return $open->sniff($names, $rowBudget, $inference, $typer); + $columnTypes = $open->sniff($names, $rowBudget, $inference, $typer); + + // the byte count stays out of ColumnTypes: SniffsColumnTypes is an inference contract, statistics are not + $this->sampled = new CSVSampledRows( + $open->producedRows(), + $open->producedBytes(), + $rowBudget === -1 || $columnTypes->rows() < $rowBudget, + ); + + return $columnTypes; } finally { $open->close(); } } + + /** + * What the last sniffColumnTypes() read; null until one ran. + */ + public function sampled(): ?CSVSampledRows + { + return $this->sampled; + } } diff --git a/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVLineReader.php b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVLineReader.php index 482134978c..c0441f8af4 100644 --- a/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVLineReader.php +++ b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVLineReader.php @@ -10,25 +10,43 @@ use function rtrim; use function str_contains; use function str_starts_with; +use function strlen; use function substr; -final readonly class CSVLineReader +final class CSVLineReader { - private CSVRecordBoundary $boundary; + private readonly CSVRecordBoundary $boundary; + + /** + * @var int<0, max> + */ + private int $lastRecordBytes = 0; /** * @param null|int<1, max> $charactersReadInLine */ public function __construct( - private string $enclosure, + private readonly string $enclosure, string $separator = ',', string $escape = '\\', - private ?int $charactersReadInLine = null, - private bool $removeBOM = true, + private readonly ?int $charactersReadInLine = null, + private readonly bool $removeBOM = true, ) { $this->boundary = new CSVRecordBoundary($enclosure, $separator, $escape); } + /** + * The bytes the record readLines() yielded last took in the stream, line endings and BOM included. + * SourceStream::readLines() cannot tell whether the last line ended with "\n", so a last record without one is + * counted one byte too long. + * + * @return int<0, max> + */ + public function lastRecordBytes(): int + { + return $this->lastRecordBytes; + } + /** * @return \Generator */ @@ -36,9 +54,12 @@ public function readLines(SourceStream $stream): Generator { $lineNumber = 0; $buffer = ''; + $bytes = 0; foreach ($stream->readLines(length: $this->charactersReadInLine) as $rawLine) { $buffer .= $rawLine; + $bytes += strlen($rawLine) + 1; + $this->lastRecordBytes = $bytes; if (!str_contains($buffer, $this->enclosure)) { yield $this->removeBOM && $lineNumber === 0 @@ -46,6 +67,7 @@ public function readLines(SourceStream $stream): Generator : rtrim($buffer, "\r\n"); $lineNumber++; $buffer = ''; + $bytes = 0; } else { if ($this->boundary->isComplete($buffer)) { yield $this->removeBOM && $lineNumber === 0 @@ -53,6 +75,7 @@ public function readLines(SourceStream $stream): Generator : rtrim($buffer, "\r\n"); $lineNumber++; $buffer = ''; + $bytes = 0; } else { $buffer .= "\n"; } diff --git a/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVOpenSource.php b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVOpenSource.php index e030ac48df..49b35a7200 100644 --- a/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVOpenSource.php +++ b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVOpenSource.php @@ -14,6 +14,20 @@ interface CSVOpenSource { public function close(): void; + /** + * The bytes, as read, of the rows producedRows() counts: line endings included, the header record excluded. + * + * @return int<0, max> + */ + public function producedBytes(): int; + + /** + * Rows read from the source so far by records() or sniff() - it may be more than a consumer took. + * + * @return int<0, max> + */ + public function producedRows(): int; + /** * This instance is consumed afterwards. * diff --git a/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVSampledFiles.php b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVSampledFiles.php new file mode 100644 index 0000000000..6e7062a08d --- /dev/null +++ b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVSampledFiles.php @@ -0,0 +1,44 @@ +whole && $this->files === $listedFiles) { + return Cardinality::exact($this->rows); + } + + if ($listedBytes->estimate === null || $this->rows === 0 || $this->bytes === 0) { + return Cardinality::unknown(); + } + + return Cardinality::approximately( + (int) round($listedBytes->estimate / ($this->bytes / $this->rows)), + Cardinality::DEFAULT_RELATIVE_ERROR, + ); + } +} diff --git a/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVSampledRows.php b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVSampledRows.php new file mode 100644 index 0000000000..312f05384b --- /dev/null +++ b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVSampledRows.php @@ -0,0 +1,40 @@ + + */ + public int $bytes; + + /** + * @var int<0, max> + */ + public int $rows; + + /** + * @param int $rows rows read while sniffing + * @param int $bytes bytes those rows took as read, line endings included + * @param bool $wholeFile the sniff reached the end of the file, so $rows is every row it holds + */ + public function __construct( + int $rows, + int $bytes, + public bool $wholeFile, + ) { + if ($rows < 0 || $bytes < 0) { + throw new InvalidArgumentException( + 'Sampled rows and bytes must not be negative, given: ' . $rows . ' rows, ' . $bytes . ' bytes', + ); + } + + $this->rows = $rows; + $this->bytes = $bytes; + } +} diff --git a/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVSamples.php b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVSamples.php new file mode 100644 index 0000000000..e6533e5215 --- /dev/null +++ b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/CSVSamples.php @@ -0,0 +1,63 @@ + + */ +final class CSVSamples implements IteratorAggregate +{ + /** + * @var list + */ + private array $pulled = []; + + /** + * @param iterable $samples + */ + public function __construct( + private readonly iterable $samples, + ) {} + + /** + * Keeps only the samples the consumer pulled: a listing the inference stopped short of costs nothing. + * + * @return Generator + */ + public function getIterator(): Generator + { + foreach ($this->samples as $sample) { + $this->pulled[] = $sample; + + yield $sample; + } + } + + public function sampledFiles(): CSVSampledFiles + { + $files = 0; + $rows = 0; + $bytes = 0; + $whole = true; + + foreach ($this->pulled as $sample) { + $sampled = $sample->sampled(); + + if ($sampled === null) { + continue; + } + + $files++; + $rows += $sampled->rows; + $bytes += $sampled->bytes; + $whole = $whole && $sampled->wholeFile; + } + + return new CSVSampledFiles($files, $rows, $bytes, $whole); + } +} diff --git a/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/NativeCSVOpenSource.php b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/NativeCSVOpenSource.php index ab3da4a2c1..12b826a463 100644 --- a/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/NativeCSVOpenSource.php +++ b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/NativeCSVOpenSource.php @@ -19,6 +19,7 @@ use function array_map; use function array_values; use function class_exists; +use function count; use function extension_loaded; use function Flow\Types\DSL\type_boolean; use function Flow\Types\DSL\type_date; @@ -32,7 +33,7 @@ use function Flow\Types\DSL\type_xml; use function max; -final readonly class NativeCSVOpenSource implements CSVOpenSource +final class NativeCSVOpenSource implements CSVOpenSource { /** * Each chunk is held as a PHP string, so it sets the path's peak memory. 32 KB measured 12.0 MB real peak - the PHP @@ -46,14 +47,19 @@ /** * @var int<1, max> */ - private int $chunkSize; + private readonly int $chunkSize; + + /** + * @var int<0, max> + */ + private int $producedRows = 0; /** * @param null|int<1, max> $charactersReadInLine */ public function __construct( - private SourceStream $stream, - private RustCSVReaderNative $reader, + private readonly SourceStream $stream, + private readonly RustCSVReaderNative $reader, ?int $charactersReadInLine = null, ) { // a local file ignores the read length, as NativeLocalSourceStream::readLines() already does on the PHP path; @@ -90,12 +96,24 @@ public function columns(): array return $this->reader->headers(); } + public function producedBytes(): int + { + return $this->reader->consumedBytes(); + } + + public function producedRows(): int + { + return $this->producedRows; + } + public function records(): Generator { foreach ($this->stream->iterate($this->chunkSize) as $chunk) { $this->reader->feed($chunk); while (($batch = $this->reader->next(self::BATCH)) !== []) { + $this->producedRows += count($batch); + foreach ($batch as $values) { yield $values; } @@ -105,6 +123,8 @@ public function records(): Generator $this->reader->finish(); while (($batch = $this->reader->next(self::BATCH)) !== []) { + $this->producedRows += count($batch); + foreach ($batch as $values) { yield $values; } @@ -137,7 +157,10 @@ public function sniff(array $names, int $rowBudget, SchemaInference $inference, foreach ($this->stream->iterate($this->chunkSize) as $chunk) { $this->reader->feed($chunk); - $this->reader->fold($fold, $rowBudget === -1 ? -1 : max(0, $rowBudget - $fold->rows())); + $this->producedRows += $this->reader->fold( + $fold, + $rowBudget === -1 ? -1 : max(0, $rowBudget - $fold->rows()), + ); if ($rowBudget !== -1 && $fold->rows() >= $rowBudget) { break; @@ -146,7 +169,10 @@ public function sniff(array $names, int $rowBudget, SchemaInference $inference, if ($rowBudget === -1 || $fold->rows() < $rowBudget) { $this->reader->finish(); - $this->reader->fold($fold, $rowBudget === -1 ? -1 : max(0, $rowBudget - $fold->rows())); + $this->producedRows += $this->reader->fold( + $fold, + $rowBudget === -1 ? -1 : max(0, $rowBudget - $fold->rows()), + ); } return ColumnTypes::fromColumnTypes( diff --git a/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/PhpCSVOpenSource.php b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/PhpCSVOpenSource.php index 5af14d0595..f352def658 100644 --- a/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/PhpCSVOpenSource.php +++ b/src/adapter/etl-adapter-csv/src/Flow/ETL/Adapter/CSV/PhpCSVOpenSource.php @@ -12,12 +12,22 @@ use Flow\Types\Type\TypeNarrower; use Generator; -final readonly class PhpCSVOpenSource implements CSVOpenSource +final class PhpCSVOpenSource implements CSVOpenSource { + /** + * @var int<0, max> + */ + private int $producedBytes = 0; + + /** + * @var int<0, max> + */ + private int $producedRows = 0; + public function __construct( - private SourceStream $stream, - private CSVEncoder $encoder, - private CSVLineReader $lineReader, + private readonly SourceStream $stream, + private readonly CSVEncoder $encoder, + private readonly CSVLineReader $lineReader, ) {} public function close(): void @@ -51,11 +61,24 @@ public function records(): Generator { foreach ($this->lineReader->readLines($this->stream) as $line) { foreach ($this->encoder->decode([$line]) as $values) { + $this->producedBytes += $this->lineReader->lastRecordBytes(); + $this->producedRows++; + yield $values; } } } + public function producedBytes(): int + { + return $this->producedBytes; + } + + public function producedRows(): int + { + return $this->producedRows; + } + public function sniff(array $names, int $rowBudget, SchemaInference $inference, TypeNarrower $typer): ColumnTypes { return (new SchemaInferrer($inference, $typer))->sniff($names, $this->records(), $rowBudget); diff --git a/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Context/CSVFixtureContext.php b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Context/CSVFixtureContext.php index c01f3cda5b..6f73770427 100644 --- a/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Context/CSVFixtureContext.php +++ b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Context/CSVFixtureContext.php @@ -38,7 +38,10 @@ use function Flow\Filesystem\DSL\memory_filesystem; use function Flow\Filesystem\DSL\path; use function Flow\Filesystem\DSL\path_real; +use function range; +use function sprintf; use function str_ends_with; +use function str_repeat; use function strlen; use function substr; @@ -243,6 +246,36 @@ public static function phpRecords(string $fixture): Generator } } + /** + * A 1000-row file whose first data row is 4 bytes and every other one 101 - 100 914 bytes in total. + */ + public static function narrowFirstRow(): string + { + $content = "id,payload\n1,a\n"; + + foreach (range(2, 1000) as $id) { + $content .= sprintf("%04d,%s\n", $id, str_repeat('x', 95)); + } + + return $content; + } + + /** + * An 8-byte header and $rows rows of exactly 10 bytes each. + * + * @param int<1, 999> $rows + */ + public static function tenByteRows(int $rows): string + { + $content = "id,name\n"; + + foreach (range(1, $rows) as $id) { + $content .= sprintf("%03d,abcde\n", $id); + } + + return $content; + } + public static function memory(string $content): MemoryFilesystem { $filesystem = memory_filesystem(); diff --git a/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/CSVFileSampleTest.php b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/CSVFileSampleTest.php index c20e5b80ba..6208c55566 100644 --- a/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/CSVFileSampleTest.php +++ b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/CSVFileSampleTest.php @@ -4,7 +4,9 @@ namespace Flow\ETL\Adapter\CSV\Tests\Integration; +use Flow\ETL\Adapter\CSV\CSVSampledRows; use Flow\ETL\Adapter\CSV\Tests\Context\CSVFixtureContext; +use Flow\ETL\Schema\Inference\SchemaInference; use Flow\ETL\Schema\Inference\TypeFloor; use Flow\ETL\Tests\Double\CountingFilesystem; use Flow\ETL\Tests\FlowTestCase; @@ -90,4 +92,36 @@ public function test_sniffing_narrows_with_the_narrower_it_is_handed(): void $sniffed->schema(new TypeFloor($inference->candidates())), ); } + + public function test_nothing_is_sampled_until_a_sniff_ran(): void + { + static::assertNull(CSVFixtureContext::sample('five_rows.csv')->sampled()); + } + + public function test_a_sniff_that_reached_the_end_sampled_the_whole_file(): void + { + $sample = CSVFixtureContext::sample('five_rows.csv'); + + $sample->sniffColumnTypes(['id', 'name'], 10, new SchemaInference(), new StringTypeNarrower()); + + static::assertEquals(new CSVSampledRows(5, 20, true), $sample->sampled()); + } + + public function test_a_sniff_without_a_budget_sampled_the_whole_file(): void + { + $sample = CSVFixtureContext::sample('five_rows.csv'); + + $sample->sniffColumnTypes(['id', 'name'], -1, new SchemaInference(), new StringTypeNarrower()); + + static::assertEquals(new CSVSampledRows(5, 20, true), $sample->sampled()); + } + + public function test_a_sniff_stopped_by_its_budget_did_not_sample_the_whole_file(): void + { + $sample = CSVFixtureContext::sample('five_rows.csv'); + + $sample->sniffColumnTypes(['id', 'name'], 5, new SchemaInference(), new StringTypeNarrower()); + + static::assertEquals(new CSVSampledRows(5, 20, false), $sample->sampled()); + } } diff --git a/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/CSVSamplesTest.php b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/CSVSamplesTest.php new file mode 100644 index 0000000000..87e2b6b233 --- /dev/null +++ b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/CSVSamplesTest.php @@ -0,0 +1,68 @@ +sniffColumnTypes(['id', 'name'], -1, new SchemaInference(), new StringTypeNarrower()); + } + + static::assertEquals(new CSVSampledFiles(2, 10, 40, true), $samples->sampledFiles()); + } + + public function test_samples_past_the_last_pulled_one_are_never_created(): void + { + $created = 0; + $samples = new CSVSamples( + ( + /** @return Generator */ + static function () use (&$created): Generator { + foreach (['five_rows.csv', 'five_rows.csv', 'five_rows.csv'] as $fixture) { + $created++; + + yield CSVFixtureContext::sample($fixture); + } + } + )(), + ); + + foreach ($samples as $sample) { + $sample->sniffColumnTypes(['id', 'name'], -1, new SchemaInference(), new StringTypeNarrower()); + + break; + } + + static::assertSame(1, $created); + static::assertEquals(new CSVSampledFiles(1, 5, 20, true), $samples->sampledFiles()); + } + + public function test_a_pulled_sample_that_was_never_sniffed_is_not_counted(): void + { + $samples = new CSVSamples([CSVFixtureContext::sample('five_rows.csv')]); + + iterator_to_array($samples, false); + + static::assertEquals(new CSVSampledFiles(0, 0, 0, true), $samples->sampledFiles()); + } +} diff --git a/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/NativeCSVOpenSourceTest.php b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/NativeCSVOpenSourceTest.php index 34bd1d36b6..6cae0ede10 100644 --- a/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/NativeCSVOpenSourceTest.php +++ b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/NativeCSVOpenSourceTest.php @@ -10,6 +10,7 @@ use Flow\ETL\Adapter\CSV\Tests\Context\CSVFixtureContext; use Flow\ETL\Adapter\CSV\Tests\Double\LengthCapturingSourceStream; use Flow\ETL\Schema\Inference\InferredTypes; +use Flow\ETL\Schema\Inference\SchemaInference; use Flow\ETL\Schema\Inference\SchemaInferrer; use Flow\ETL\Schema\Inference\TypeFloor; use Flow\ETL\Tests\Double\CountingFilesystem; @@ -332,4 +333,56 @@ public function test_columns_of_a_header_without_a_trailing_newline(): void )->columns(), ); } + + public function test_produced_rows_and_bytes_exclude_the_header(): void + { + if (!NativeCSVOpenSource::isSupported()) { + static::markTestSkipped('flow_php extension with RustCSVReaderNative is not loaded'); + } + + $open = CSVFixtureContext::openNative('five_rows.csv'); + + try { + iterator_to_array($open->records(), false); + + static::assertSame(5, $open->producedRows()); + static::assertSame(20, $open->producedBytes()); + } finally { + $open->close(); + } + } + + /** + * Samples that stop before EOF: SourceStream::readLines() hides whether the last line ended with "\n", so the PHP + * path counts a last record without one a byte longer than the native path does (CSVLineReader::lastRecordBytes()). + */ + #[TestWith(['annual-enterprise-survey-2019-financial-year-provisional-csv.csv'])] + #[TestWith(['orders_flow.csv'])] + #[TestWith(['five_rows.csv'])] + #[TestWith(['fold_traps.csv'])] + #[TestWith(['ragged.csv'])] + public function test_php_and_native_paths_agree_on_bytes_per_row(string $fixture): void + { + if (!NativeCSVOpenSource::isSupported()) { + static::markTestSkipped('flow_php extension with RustCSVReaderNative is not loaded'); + } + + $header = CSVFixtureContext::openPhp($fixture); + $names = $header->columns(); + $header->close(); + $php = CSVFixtureContext::openPhp($fixture); + $native = CSVFixtureContext::openNative($fixture); + $typer = new StringTypeNarrower(infer_schema()->build()->candidates()->toArray()); + + try { + $php->sniff($names, 2, new SchemaInference(), $typer); + $native->sniff($names, 2, new SchemaInference(), $typer); + + static::assertSame($php->producedRows(), $native->producedRows()); + static::assertSame($php->producedBytes(), $native->producedBytes()); + } finally { + $php->close(); + $native->close(); + } + } } diff --git a/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/PhpCSVOpenSourceTest.php b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/PhpCSVOpenSourceTest.php index debe8ca9b9..79f52fcd22 100644 --- a/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/PhpCSVOpenSourceTest.php +++ b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/PhpCSVOpenSourceTest.php @@ -101,4 +101,18 @@ public function test_records_yields_one_logical_record_per_iteration(): void $open->close(); } } + + public function test_produced_rows_and_bytes_exclude_the_header(): void + { + $open = CSVFixtureContext::openPhp('five_rows.csv'); + + try { + iterator_to_array($open->records(), false); + + static::assertSame(5, $open->producedRows()); + static::assertSame(20, $open->producedBytes()); + } finally { + $open->close(); + } + } } diff --git a/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Unit/CSVExtractorTest.php b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Unit/CSVExtractorTest.php new file mode 100644 index 0000000000..2f0bc0eca4 --- /dev/null +++ b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Unit/CSVExtractorTest.php @@ -0,0 +1,184 @@ + + */ + public static function sample_dropping_setters(): Generator + { + yield 'inferSchema' => [static fn(CSVExtractor $e): mixed => $e->inferSchema(infer_schema())]; + yield 'withBOMRemoval' => [static fn(CSVExtractor $e): mixed => $e->withBOMRemoval(true)]; + yield 'withCharactersReadInLine' => [static fn(CSVExtractor $e): mixed => $e->withCharactersReadInLine(1024)]; + yield 'withEmptyToNull' => [static fn(CSVExtractor $e): mixed => $e->withEmptyToNull(true)]; + yield 'withEnclosure' => [static fn(CSVExtractor $e): mixed => $e->withEnclosure('"')]; + yield 'withEscape' => [static fn(CSVExtractor $e): mixed => $e->withEscape('\\')]; + yield 'withHeader' => [static fn(CSVExtractor $e): mixed => $e->withHeader(true)]; + yield 'withSeparator' => [static fn(CSVExtractor $e): mixed => $e->withSeparator(',')]; + } + + #[DataProvider('sample_dropping_setters')] + public function test_a_changed_read_option_drops_the_sample(Closure $setter): void + { + $extractor = from_csv(CSVFixtureContext::path('five_rows.csv')); + $extractor->schema(); + + static::assertEquals(Cardinality::exact(5), $extractor->statistics()->rows); + + $setter($extractor); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + } + + public function test_a_declared_schema_declares_unknown_rows(): void + { + $extractor = from_csv( + CSVFixtureContext::path('five_rows.csv'), + schema: schema(int_schema('id'), str_schema('name')), + ); + $extractor->schema(); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + static::assertEquals(Cardinality::exact(28), $extractor->statistics()->size); + } + + public function test_a_file_shorter_than_the_sample_declares_an_exact_row_count(): void + { + $extractor = from_csv(CSVFixtureContext::path('five_rows.csv')); + $extractor->schema(); + + static::assertEquals(Cardinality::exact(5), $extractor->statistics()->rows); + } + + public function test_a_header_only_file_declares_zero_rows_exactly(): void + { + $extractor = from_csv(CSVFixtureContext::path('header_only.csv')); + $extractor->schema(); + + static::assertEquals(Cardinality::exact(0), $extractor->statistics()->rows); + static::assertEquals(Cardinality::exact(8), $extractor->statistics()->size); + } + + public function test_a_member_without_a_size_makes_both_facts_unknown(): void + { + $extractor = from_csv(CSVFixtureContext::path('glob_with_empty/*.csv')) + ->inferSchema(infer_schema()->sampleSize(1)); + $extractor->schema(); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->size); + } + + public function test_a_narrow_first_row_does_not_skew_the_estimate(): void + { + $extractor = from_csv( + path('memory://source.csv'), + filesystem: CSVFixtureContext::memory(CSVFixtureContext::narrowFirstRow()), + )->inferSchema(infer_schema()->sampleSize(20)); + $extractor->schema(); + + $rows = $extractor->statistics()->rows; + + static::assertEquals(Cardinality::approximately(1050, Cardinality::DEFAULT_RELATIVE_ERROR), $rows); + static::assertLessThanOrEqual($rows->relativeError, abs(1050 - 1000) / 1000); + } + + public function test_an_empty_file_declares_zero_rows_exactly(): void + { + $extractor = from_csv(CSVFixtureContext::path('empty.csv')); + $extractor->schema(); + + static::assertEquals(Cardinality::exact(0), $extractor->statistics()->rows); + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->size); + } + + public function test_extract_samples_like_schema_does(): void + { + $extractor = from_csv(CSVFixtureContext::path('five_rows.csv')); + + iterator_to_array($extractor->extract(flow_context()), false); + + static::assertEquals(Cardinality::exact(5), $extractor->statistics()->rows); + } + + public function test_it_declares_the_listed_byte_total_exactly(): void + { + $extractor = from_csv( + path('memory://source.csv'), + filesystem: CSVFixtureContext::memory(CSVFixtureContext::tenByteRows(100)), + ); + + static::assertEquals(Cardinality::exact(1008), $extractor->statistics()->size); + } + + public function test_it_estimates_rows_from_bytes_and_the_mean_sampled_row(): void + { + $extractor = from_csv( + path('memory://source.csv'), + filesystem: CSVFixtureContext::memory(CSVFixtureContext::tenByteRows(100)), + )->inferSchema(infer_schema()->sampleSize(10)); + $extractor->schema(); + + static::assertEquals( + Cardinality::approximately(101, Cardinality::DEFAULT_RELATIVE_ERROR), + $extractor->statistics()->rows, + ); + } + + public function test_statistics_are_computed_at_most_once(): void + { + $filesystem = new CountingFilesystem(new NativeLocalFilesystem()); + $extractor = from_csv(CSVFixtureContext::path('five_rows.csv'), filesystem: $filesystem); + $extractor->schema(); + $listCalls = $filesystem->listCalls; + + static::assertSame($extractor->statistics(), $extractor->statistics()); + static::assertSame($listCalls + 1, $filesystem->listCalls); + } + + public function test_statistics_before_schema_declares_unknown_rows(): void + { + $extractor = from_csv(CSVFixtureContext::path('five_rows.csv')); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + + $extractor->schema(); + + static::assertEquals(Cardinality::exact(5), $extractor->statistics()->rows); + } + + public function test_statistics_before_a_sample_list_the_source_once(): void + { + $filesystem = new CountingFilesystem(new NativeLocalFilesystem()); + $extractor = from_csv(CSVFixtureContext::path('five_rows.csv'), filesystem: $filesystem); + + $extractor->statistics(); + $extractor->statistics(); + + static::assertSame(1, $filesystem->listCalls); + } +} diff --git a/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Unit/CSVLineReaderTest.php b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Unit/CSVLineReaderTest.php index ae7675d4b2..3dfbec0e2c 100644 --- a/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Unit/CSVLineReaderTest.php +++ b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Unit/CSVLineReaderTest.php @@ -9,6 +9,7 @@ use Flow\Filesystem\Stream\MemorySourceStream; use Generator; use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestWith; use PHPUnit\Framework\TestCase; use function Flow\Filesystem\DSL\path; @@ -394,4 +395,28 @@ public function test_reading_simple_csv_without_quotes(): void static::assertSame('value1,value2,value3', $lines[1]); static::assertSame('value4,value5,value6', $lines[2]); } + + /** + * @param non-empty-string $content + * @param list $expected + */ + #[TestWith(["a,b\n1,2\n", [4, 4]])] + #[TestWith(["a,b\r\n1,2\r\n", [5, 5]])] + #[TestWith(["a,b\n\"x\ny\",1\n", [4, 8]])] + #[TestWith(["a,b\r\n\"x\r\ny\",1\r\n", [5, 10]])] + #[TestWith(["\xEF\xBB\xBFa,b\n1,2\n", [7, 4]])] + #[TestWith(["a,b\n1,2", [4, 4]])] + public function test_last_record_bytes_count_what_the_record_took_in_the_stream( + string $content, + array $expected, + ): void { + $reader = new CSVLineReader('"'); + $bytes = []; + + foreach ($reader->readLines(new MemorySourceStream($content)) as $_line) { + $bytes[] = $reader->lastRecordBytes(); + } + + static::assertSame($expected, $bytes); + } } diff --git a/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Unit/CSVSampledFilesTest.php b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Unit/CSVSampledFilesTest.php new file mode 100644 index 0000000000..4b7153af1f --- /dev/null +++ b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Unit/CSVSampledFilesTest.php @@ -0,0 +1,48 @@ +estimatedRows(1, Cardinality::exact(1000)), + ); + } + + public function test_every_listed_file_read_whole_is_exact(): void + { + static::assertEquals( + Cardinality::exact(10), + (new CSVSampledFiles(2, 10, 100, true))->estimatedRows(2, Cardinality::exact(100)), + ); + } + + public function test_a_listed_file_the_samples_never_read_makes_it_an_estimate(): void + { + static::assertEquals( + Cardinality::approximately(20), + (new CSVSampledFiles(1, 10, 100, true))->estimatedRows(2, Cardinality::exact(200)), + ); + } + + public function test_no_listed_bytes_or_no_sampled_rows_is_unknown(): void + { + static::assertEquals(Cardinality::unknown(), (new CSVSampledFiles(1, 10, 100, false))->estimatedRows( + 1, + Cardinality::unknown(), + )); + static::assertEquals(Cardinality::unknown(), (new CSVSampledFiles(1, 0, 0, false))->estimatedRows( + 1, + Cardinality::exact(100), + )); + } +} diff --git a/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Unit/CSVSampledRowsTest.php b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Unit/CSVSampledRowsTest.php new file mode 100644 index 0000000000..fe7037e74e --- /dev/null +++ b/src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Unit/CSVSampledRowsTest.php @@ -0,0 +1,34 @@ +rows); + static::assertSame(20, $sampled->bytes); + static::assertTrue($sampled->wholeFile); + } + + #[TestWith([-1, 0])] + #[TestWith([0, -1])] + public function test_negative_counts_are_rejected(int $rows, int $bytes): void + { + $this->expectException(InvalidArgumentException::class); + $this->expectExceptionMessage( + 'Sampled rows and bytes must not be negative, given: ' . $rows . ' rows, ' . $bytes . ' bytes', + ); + + new CSVSampledRows($rows, $bytes, false); + } +} diff --git a/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/DbalKeySetExtractor.php b/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/DbalKeySetExtractor.php index 208bdc3d78..2b5638b6b3 100644 --- a/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/DbalKeySetExtractor.php +++ b/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/DbalKeySetExtractor.php @@ -6,6 +6,7 @@ use Doctrine\DBAL\Connection; use Doctrine\DBAL\Query\QueryBuilder; +use Flow\ETL\Adapter\Doctrine\Explain\ExplainedRows; use Flow\ETL\Adapter\Doctrine\Pagination\Key; use Flow\ETL\Adapter\Doctrine\Pagination\KeySet; use Flow\ETL\Exception\InvalidArgumentException; @@ -16,6 +17,7 @@ use Flow\ETL\Extractor\Batches; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -50,6 +52,8 @@ final class DbalKeySetExtractor implements BatchableExtractor, Extractor, Rewind private ?SchemaNotDerivableException $refusal = null; + private ?Statistics $statistics = null; + public function __construct( private readonly Connection $connection, private readonly QueryBuilder $queryBuilder, @@ -212,6 +216,17 @@ public function schema(): Schema } } + public function statistics(): Statistics + { + return $this->statistics ??= new Statistics(rows: (new ExplainedRows())->of( + $this->connection, + $this->queryBuilder->getSQL(), + $this->queryBuilder->getParameters(), + $this->queryBuilder->getParameterTypes(), + $this->maximum, + )); + } + public function withKeyAliasSuffix(string $keyAliasSuffix): self { $this->keyAliasSuffix = $keyAliasSuffix; @@ -235,6 +250,7 @@ public function withMaximum(int $maximum): self } $this->maximum = $maximum; + $this->statistics = null; return $this; } diff --git a/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/DbalLimitOffsetExtractor.php b/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/DbalLimitOffsetExtractor.php index e48762273c..0394ed3701 100644 --- a/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/DbalLimitOffsetExtractor.php +++ b/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/DbalLimitOffsetExtractor.php @@ -6,6 +6,7 @@ use Doctrine\DBAL\Connection; use Doctrine\DBAL\Query\QueryBuilder; +use Flow\ETL\Adapter\Doctrine\Explain\ExplainedRows; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Exception\SchemaNotDerivableException; use Flow\ETL\Extractor; @@ -13,6 +14,7 @@ use Flow\ETL\Extractor\Batches; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -20,6 +22,7 @@ use function count; use function is_numeric; +use function max; use function min; final class DbalLimitOffsetExtractor implements BatchableExtractor, Extractor, RewindableExtractor @@ -36,6 +39,8 @@ final class DbalLimitOffsetExtractor implements BatchableExtractor, Extractor, R private ?SchemaNotDerivableException $refusal = null; + private ?Statistics $statistics = null; + public function __construct( private readonly Connection $connection, private readonly QueryBuilder $queryBuilder, @@ -183,6 +188,21 @@ public function schema(): Schema } } + /** + * The plan already applies the query builder's own limit and offset; withOffset() replaces that offset. + */ + public function statistics(): Statistics + { + return $this->statistics ??= new Statistics(rows: (new ExplainedRows())->of( + $this->connection, + $this->queryBuilder->getSQL(), + $this->queryBuilder->getParameters(), + $this->queryBuilder->getParameterTypes(), + $this->maximum ?? $this->queryBuilder->getMaxResults(), + $this->offset === 0 ? 0 : max(0, $this->offset - $this->queryBuilder->getFirstResult()), + )); + } + public function withMaximum(int $maximum): self { if ($maximum <= 0) { @@ -190,6 +210,7 @@ public function withMaximum(int $maximum): self } $this->maximum = $maximum; + $this->statistics = null; return $this; } @@ -201,6 +222,7 @@ public function withOffset(int $offset): self } $this->offset = $offset; + $this->statistics = null; return $this; } diff --git a/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/DbalQueryExtractor.php b/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/DbalQueryExtractor.php index 83728fd81c..686b22063c 100644 --- a/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/DbalQueryExtractor.php +++ b/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/DbalQueryExtractor.php @@ -8,18 +8,23 @@ use Doctrine\DBAL\Connection; use Doctrine\DBAL\ParameterType; use Doctrine\DBAL\Types\Type; +use Flow\ETL\Adapter\Doctrine\Explain\ExplainedRows; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\SchemaNotDerivableException; use Flow\ETL\Extractor; use Flow\ETL\Extractor\BatchableExtractor; use Flow\ETL\Extractor\Batches; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; use Generator; +use function array_slice; use function count; +use function round; /** * batchSize() is a yield cap, not a fetch size: fetchAllAssociative() materialises the whole result set per @@ -31,6 +36,8 @@ */ final class DbalQueryExtractor implements BatchableExtractor, Extractor, RewindableExtractor { + private const int EXPLAINED_PARAMETER_SETS = 10; + use Batches; private ParametersSet $parametersSet; @@ -41,6 +48,8 @@ final class DbalQueryExtractor implements BatchableExtractor, Extractor, Rewinda private ?SchemaNotDerivableException $refusal = null; + private ?Statistics $statistics = null; + /** * @var array|string, ArrayParameterType|ParameterType|string|Type> */ @@ -154,9 +163,45 @@ public function schema(): Schema } } + /** + * One plan per parameter set, summed; past EXPLAINED_PARAMETER_SETS the explained sets are scaled to all of them, + * so planning never pays a round trip per set. + */ + public function statistics(): Statistics + { + if ($this->statistics === null) { + $sets = $this->parametersSet->all(); + $explained = array_slice($sets, 0, self::EXPLAINED_PARAMETER_SETS); + $rows = Cardinality::exact(0); + + foreach ($explained as $parameters) { + $rows = $rows->merge((new ExplainedRows())->of( + $this->connection, + $this->query, + $parameters, + $this->types, + )); + } + + $this->statistics = new Statistics( + rows: count($sets) === count($explained) + ? $rows + : new Cardinality( + estimate: $rows->estimate === null + ? null + : (int) round(($rows->estimate * count($sets)) / count($explained)), + relativeError: Cardinality::DEFAULT_RELATIVE_ERROR, + ), + ); + } + + return $this->statistics; + } + public function withParameters(ParametersSet $parametersSet): self { $this->parametersSet = $parametersSet; + $this->statistics = null; return $this; } @@ -174,6 +219,7 @@ public function withSchema(Schema $schema): static public function withTypes(array $types): self { $this->types = $types; + $this->statistics = null; return $this; } diff --git a/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/Explain/ExplainedRows.php b/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/Explain/ExplainedRows.php new file mode 100644 index 0000000000..b769718d76 --- /dev/null +++ b/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/Explain/ExplainedRows.php @@ -0,0 +1,64 @@ +|list $parameters + * @param array|string, ArrayParameterType|ParameterType|string|Type> $types + */ + public function of( + Connection $connection, + string $sql, + array $parameters = [], + array $types = [], + ?int $maximum = null, + int $offset = 0, + ): Cardinality { + $platform = $connection->getDatabasePlatform(); + $planned = match (true) { + $platform instanceof PostgreSQLPlatform => (new PostgreSqlExplainedRows())->of( + $connection, + $sql, + $parameters, + $types, + ), + $platform instanceof MySQL80Platform => (new MySqlExplainedRows())->of( + $connection, + $sql, + $parameters, + $types, + ), + default => null, + }; + + if ($planned === null) { + return new Cardinality(atMost: $maximum); + } + + $estimate = max(0, $planned - $offset); + + return new Cardinality( + atMost: $maximum, + estimate: $maximum === null ? $estimate : min($estimate, $maximum), + relativeError: Cardinality::DEFAULT_RELATIVE_ERROR, + ); + } +} diff --git a/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/Explain/MySqlExplainedRows.php b/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/Explain/MySqlExplainedRows.php new file mode 100644 index 0000000000..dabfb9b165 --- /dev/null +++ b/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/Explain/MySqlExplainedRows.php @@ -0,0 +1,47 @@ + Filter: (t.g > 1) (cost=0.75 rows=2)". A root without an + * estimate (a GROUP BY read back from "") is unknown: a child's estimate counts other rows. + * + * @param array|list $parameters + * @param array|string, ArrayParameterType|ParameterType|string|Type> $types + */ + public function of(Connection $connection, string $sql, array $parameters = [], array $types = []): ?int + { + $plan = $connection->fetchOne('EXPLAIN FORMAT=TREE ' . $sql, $parameters, $types); + + if (!is_string($plan)) { + return null; + } + + $matches = []; + + if (preg_match('/\brows=(\d+(?:\.\d+)?(?:e[+-]?\d+)?)/i', explode("\n", $plan)[0], $matches) !== 1) { + return null; + } + + if (!is_numeric($matches[1])) { + return null; + } + + return (int) round((float) $matches[1]); + } +} diff --git a/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/Explain/PostgreSqlExplainedRows.php b/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/Explain/PostgreSqlExplainedRows.php new file mode 100644 index 0000000000..8cc00e8a86 --- /dev/null +++ b/src/adapter/etl-adapter-doctrine/src/Flow/ETL/Adapter/Doctrine/Explain/PostgreSqlExplainedRows.php @@ -0,0 +1,51 @@ +|list $parameters + * @param array|string, ArrayParameterType|ParameterType|string|Type> $types + */ + public function of(Connection $connection, string $sql, array $parameters = [], array $types = []): ?int + { + $plan = $connection->fetchOne('EXPLAIN (FORMAT JSON) ' . $sql, $parameters, $types); + + // @mago-expect analysis:mixed-assignment + $plan = is_string($plan) ? json_decode($plan, true) : $plan; + + if (!is_array($plan) || !array_key_exists(0, $plan) || !is_array($plan[0])) { + return null; + } + + // @mago-expect analysis:mixed-assignment + $root = $plan[0]['Plan'] ?? null; + + if (!is_array($root) || !array_key_exists('Plan Rows', $root)) { + return null; + } + + // @mago-expect analysis:mixed-assignment + $rows = $root['Plan Rows']; + + return is_int($rows) || is_float($rows) ? (int) round($rows) : null; + } +} diff --git a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Context/PlannedTable.php b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Context/PlannedTable.php new file mode 100644 index 0000000000..7e070d7b0d --- /dev/null +++ b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Context/PlannedTable.php @@ -0,0 +1,41 @@ +createTable( + (new Table($name, [ + new Column('id', Type::getType(Types::INTEGER), ['notnull' => true]), + new Column('grp', Type::getType(Types::INTEGER), ['notnull' => true]), + ]))->addPrimaryKeyConstraint(PrimaryKeyConstraint::editor()->setUnquotedColumnNames('id')->create()), + ); + + for ($id = 1; $id <= $rows; $id++) { + $context->insert($name, ['id' => $id, 'grp' => $id % 3]); + } + + // DBAL has no statement for refreshing planner statistics + $context + ->connection() + ->executeStatement( + $context->connection()->getDatabasePlatform() instanceof AbstractMySQLPlatform + ? 'ANALYZE TABLE ' . $name + : 'ANALYZE ' . $name, + ); + } +} diff --git a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Dialects/PosgreSQLDbalLimitOffsetExtractorTest.php b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Dialects/PosgreSQLDbalLimitOffsetExtractorTest.php index 0b71acca73..05a7c3d5b6 100644 --- a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Dialects/PosgreSQLDbalLimitOffsetExtractorTest.php +++ b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Dialects/PosgreSQLDbalLimitOffsetExtractorTest.php @@ -12,7 +12,9 @@ use Flow\ETL\Adapter\Doctrine\Order; use Flow\ETL\Adapter\Doctrine\OrderBy; use Flow\ETL\Adapter\Doctrine\Table as ETLTable; +use Flow\ETL\Adapter\Doctrine\Tests\Context\PlannedTable; use Flow\ETL\Adapter\Doctrine\Tests\IntegrationTestCase; +use Flow\ETL\Cardinality; use function Flow\ETL\Adapter\Doctrine\from_dbal_limit_offset; use function Flow\ETL\Adapter\Doctrine\from_dbal_limit_offset_qb; @@ -436,4 +438,25 @@ public function test_extracting_selected_columns(): void $data->toArray(), ); } + + public function test_statistics_apply_the_offset_and_the_query_builder_limit(): void + { + PlannedTable::create($this->pgsqlDatabaseContext, 'flow_doctrine_statistics_test', 100); + $queryBuilder = $this->pgsqlDatabaseContext + ->connection() + ->createQueryBuilder() + ->select('*') + ->from('flow_doctrine_statistics_test') + ->orderBy('id') + ->setMaxResults(20); + + static::assertEquals( + new Cardinality(atMost: 20, estimate: 10, relativeError: Cardinality::DEFAULT_RELATIVE_ERROR), + from_dbal_limit_offset_qb( + $this->pgsqlDatabaseContext->connection(), + $queryBuilder, + offset: 10, + )->statistics()->rows, + ); + } } diff --git a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Dialects/PostgreSQLDbalQueryExtractorTest.php b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Dialects/PostgreSQLDbalQueryExtractorTest.php index cba22d8321..a4042bc3cd 100644 --- a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Dialects/PostgreSQLDbalQueryExtractorTest.php +++ b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Dialects/PostgreSQLDbalQueryExtractorTest.php @@ -11,8 +11,11 @@ use Doctrine\DBAL\Types\Types; use Flow\ETL\Adapter\Doctrine\DbalLoader; use Flow\ETL\Adapter\Doctrine\ParametersSet; +use Flow\ETL\Adapter\Doctrine\Tests\Context\PlannedTable; use Flow\ETL\Adapter\Doctrine\Tests\IntegrationTestCase; +use Flow\ETL\Cardinality; +use function array_fill; use function Flow\ETL\Adapter\Doctrine\from_dbal_queries; use function Flow\ETL\Adapter\Doctrine\from_dbal_query; use function Flow\ETL\DSL\data_frame; @@ -193,4 +196,32 @@ public function test_extracting_multiple_rows_with_schema(): void $schema->normalize(), ); } + + public function test_statistics_sum_the_plan_of_every_parameter_set(): void + { + PlannedTable::create($this->pgsqlDatabaseContext, 'flow_doctrine_statistics_test', 99); + + static::assertEquals( + Cardinality::approximately(66), + from_dbal_queries( + $this->pgsqlDatabaseContext->connection(), + 'SELECT * FROM flow_doctrine_statistics_test WHERE grp = :grp', + new ParametersSet(['grp' => 0], ['grp' => 1]), + )->statistics()->rows, + ); + } + + public function test_statistics_scale_the_first_ten_parameter_sets_to_all_of_them(): void + { + PlannedTable::create($this->pgsqlDatabaseContext, 'flow_doctrine_statistics_test', 99); + + static::assertEquals( + Cardinality::approximately(396), + from_dbal_queries( + $this->pgsqlDatabaseContext->connection(), + 'SELECT * FROM flow_doctrine_statistics_test WHERE grp = :grp', + new ParametersSet(...array_fill(0, 10, ['grp' => 0]), ...array_fill(0, 2, ['grp' => 99])), + )->statistics()->rows, + ); + } } diff --git a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Dialects/PostgreSQLKeySetExtractorTest.php b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Dialects/PostgreSQLKeySetExtractorTest.php index a3fc228c78..a5ae126119 100644 --- a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Dialects/PostgreSQLKeySetExtractorTest.php +++ b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Dialects/PostgreSQLKeySetExtractorTest.php @@ -7,7 +7,9 @@ use DateTimeImmutable; use Flow\Clock\FakeClock; use Flow\ETL\Adapter\Doctrine\DbalMetadata; +use Flow\ETL\Adapter\Doctrine\Tests\Context\PlannedTable; use Flow\ETL\Adapter\Doctrine\Tests\IntegrationTestCase; +use Flow\ETL\Cardinality; use function Flow\ETL\Adapter\Doctrine\from_dbal_key_set_qb; use function Flow\ETL\Adapter\Doctrine\pagination_key_asc; @@ -404,4 +406,22 @@ public function test_extraction_when_key_is_ambiguous_column_with_custom_key_col $rows, ); } + + public function test_statistics_are_the_plan_of_the_base_query(): void + { + PlannedTable::create($this->pgsqlDatabaseContext, 'flow_doctrine_statistics_test', 100); + + static::assertEquals( + Cardinality::approximately(100), + from_dbal_key_set_qb( + $this->pgsqlDatabaseContext->connection(), + $this->pgsqlDatabaseContext + ->connection() + ->createQueryBuilder() + ->select('*') + ->from('flow_doctrine_statistics_test'), + pagination_key_set(pagination_key_asc('id')), + )->statistics()->rows, + ); + } } diff --git a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Explain/ExplainedRowsTest.php b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Explain/ExplainedRowsTest.php new file mode 100644 index 0000000000..edb08ed515 --- /dev/null +++ b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Explain/ExplainedRowsTest.php @@ -0,0 +1,77 @@ +pgsqlDatabaseContext, 'flow_explain_test', 5); + + static::assertEquals( + new Cardinality(atMost: 3, estimate: 3, relativeError: Cardinality::DEFAULT_RELATIVE_ERROR), + (new ExplainedRows())->of( + $this->pgsqlDatabaseContext->connection(), + 'SELECT * FROM flow_explain_test', + maximum: 3, + ), + ); + } + + public function test_an_offset_is_taken_off_the_estimate_and_never_below_zero(): void + { + PlannedTable::create($this->pgsqlDatabaseContext, 'flow_explain_test', 5); + + static::assertEquals( + Cardinality::approximately(1), + (new ExplainedRows())->of( + $this->pgsqlDatabaseContext->connection(), + 'SELECT * FROM flow_explain_test', + offset: 4, + ), + ); + static::assertEquals( + Cardinality::approximately(0), + (new ExplainedRows())->of( + $this->pgsqlDatabaseContext->connection(), + 'SELECT * FROM flow_explain_test', + offset: 9, + ), + ); + } + + public function test_mysql_plans_are_read(): void + { + PlannedTable::create($this->mysqlDatabaseContext, 'flow_explain_test', 5); + + static::assertEquals( + Cardinality::approximately(5), + (new ExplainedRows())->of($this->mysqlDatabaseContext->connection(), 'SELECT * FROM flow_explain_test'), + ); + } + + public function test_sqlite_declares_only_the_maximum(): void + { + PlannedTable::create($this->sqliteDatabaseContext, 'flow_explain_test', 5); + + static::assertEquals(Cardinality::unknown(), (new ExplainedRows())->of( + $this->sqliteDatabaseContext->connection(), + 'SELECT * FROM flow_explain_test', + )); + static::assertEquals( + Cardinality::atMost(3), + (new ExplainedRows())->of( + $this->sqliteDatabaseContext->connection(), + 'SELECT * FROM flow_explain_test', + maximum: 3, + ), + ); + } +} diff --git a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Explain/MySqlExplainedRowsTest.php b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Explain/MySqlExplainedRowsTest.php new file mode 100644 index 0000000000..d53ed380f4 --- /dev/null +++ b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Explain/MySqlExplainedRowsTest.php @@ -0,0 +1,43 @@ +mysqlDatabaseContext, 'flow_explain_test', 5); + + static::assertNull((new MySqlExplainedRows())->of( + $this->mysqlDatabaseContext->connection(), + 'SELECT grp, COUNT(*) FROM flow_explain_test GROUP BY grp', + )); + } + + public function test_a_parameterised_query_is_planned(): void + { + PlannedTable::create($this->mysqlDatabaseContext, 'flow_explain_test', 100); + + static::assertSame(10, (new MySqlExplainedRows())->of( + $this->mysqlDatabaseContext->connection(), + 'SELECT * FROM flow_explain_test WHERE id > ?', + [90], + )); + } + + public function test_the_root_line_rows_are_the_estimate(): void + { + PlannedTable::create($this->mysqlDatabaseContext, 'flow_explain_test', 5); + + static::assertSame(5, (new MySqlExplainedRows())->of( + $this->mysqlDatabaseContext->connection(), + 'SELECT * FROM flow_explain_test', + )); + } +} diff --git a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Explain/PostgreSqlExplainedRowsTest.php b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Explain/PostgreSqlExplainedRowsTest.php new file mode 100644 index 0000000000..1262e6188f --- /dev/null +++ b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Integration/Explain/PostgreSqlExplainedRowsTest.php @@ -0,0 +1,33 @@ +pgsqlDatabaseContext, 'flow_explain_test', 100); + + static::assertSame(10, (new PostgreSqlExplainedRows())->of( + $this->pgsqlDatabaseContext->connection(), + 'SELECT * FROM flow_explain_test WHERE id > ?', + [90], + )); + } + + public function test_the_root_plan_rows_are_the_estimate(): void + { + PlannedTable::create($this->pgsqlDatabaseContext, 'flow_explain_test', 5); + + static::assertSame(5, (new PostgreSqlExplainedRows())->of( + $this->pgsqlDatabaseContext->connection(), + 'SELECT * FROM flow_explain_test', + )); + } +} diff --git a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Unit/DbalKeySetExtractorTest.php b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Unit/DbalKeySetExtractorTest.php index 7076e5bf52..ef3b1fb4b3 100644 --- a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Unit/DbalKeySetExtractorTest.php +++ b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Unit/DbalKeySetExtractorTest.php @@ -9,6 +9,7 @@ use Flow\ETL\Adapter\Doctrine\Tests\Context\InMemorySqlite; use Flow\ETL\Adapter\Doctrine\Tests\Context\SelectQueryCounter; use Flow\ETL\Adapter\Doctrine\Tests\Double\NativeHandleStub; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\SchemaNotDerivableException; use Flow\ETL\Tests\FlowTestCase; use stdClass; @@ -122,4 +123,21 @@ public function test_is_repeatable(): void ))->isRepeatable(), ); } + + public function test_a_new_maximum_drops_the_statistics(): void + { + $connection = InMemorySqlite::withUsers(InMemorySqlite::connection(), 3); + $extractor = (new DbalKeySetExtractor( + $connection, + $connection->createQueryBuilder()->select('*')->from('users'), + pagination_key_set(pagination_key_asc('id')), + ))->withMaximum(2); + + static::assertSame($extractor->statistics(), $extractor->statistics()); + static::assertEquals(Cardinality::atMost(2), $extractor->statistics()->rows); + + $extractor->withMaximum(5); + + static::assertEquals(Cardinality::atMost(5), $extractor->statistics()->rows); + } } diff --git a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Unit/DbalLimitOffsetExtractorTest.php b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Unit/DbalLimitOffsetExtractorTest.php index 68e4d9bfbe..86de2399b0 100644 --- a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Unit/DbalLimitOffsetExtractorTest.php +++ b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Unit/DbalLimitOffsetExtractorTest.php @@ -9,6 +9,7 @@ use Flow\ETL\Adapter\Doctrine\Tests\Context\InMemorySqlite; use Flow\ETL\Adapter\Doctrine\Tests\Context\SelectQueryCounter; use Flow\ETL\Adapter\Doctrine\Tests\Double\NativeHandleStub; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\SchemaNotDerivableException; use Flow\ETL\Rows; use Flow\ETL\Tests\FlowTestCase; @@ -243,4 +244,47 @@ public function test_is_repeatable(): void ))->isRepeatable(), ); } + + public function test_a_new_maximum_drops_the_statistics(): void + { + $connection = InMemorySqlite::withUsers(InMemorySqlite::connection(), 3); + $extractor = (new DbalLimitOffsetExtractor( + $connection, + $connection->createQueryBuilder()->select('*')->from('users')->orderBy('id'), + ))->withMaximum(2); + + static::assertSame($extractor->statistics(), $extractor->statistics()); + static::assertEquals(Cardinality::atMost(2), $extractor->statistics()->rows); + + $extractor->withMaximum(5); + + static::assertEquals(Cardinality::atMost(5), $extractor->statistics()->rows); + } + + public function test_a_new_offset_drops_the_statistics(): void + { + $connection = InMemorySqlite::withUsers(InMemorySqlite::connection(), 3); + $extractor = new DbalLimitOffsetExtractor( + $connection, + $connection->createQueryBuilder()->select('*')->from('users')->orderBy('id'), + ); + $before = $extractor->statistics(); + + $extractor->withOffset(1); + + static::assertNotSame($before, $extractor->statistics()); + } + + public function test_the_query_builder_limit_bounds_the_rows(): void + { + $connection = InMemorySqlite::withUsers(InMemorySqlite::connection(), 3); + + static::assertEquals( + Cardinality::atMost(2), + (new DbalLimitOffsetExtractor( + $connection, + $connection->createQueryBuilder()->select('*')->from('users')->orderBy('id')->setMaxResults(2), + ))->statistics()->rows, + ); + } } diff --git a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Unit/DbalQueryExtractorTest.php b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Unit/DbalQueryExtractorTest.php index 18257d0bb7..905908cd2b 100644 --- a/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Unit/DbalQueryExtractorTest.php +++ b/src/adapter/etl-adapter-doctrine/tests/Flow/ETL/Adapter/Doctrine/Tests/Unit/DbalQueryExtractorTest.php @@ -10,6 +10,7 @@ use Flow\ETL\Adapter\Doctrine\Tests\Context\InMemorySqlite; use Flow\ETL\Adapter\Doctrine\Tests\Context\SelectQueryCounter; use Flow\ETL\Adapter\Doctrine\Tests\Double\NativeHandleStub; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\SchemaNotDerivableException; use Flow\ETL\Tests\FlowTestCase; use PHPUnit\Framework\Attributes\TestWith; @@ -133,4 +134,32 @@ public function test_is_repeatable(): void (new DbalQueryExtractor(InMemorySqlite::connection(), 'SELECT * FROM users'))->isRepeatable(), ); } + + public function test_new_parameters_drop_the_statistics(): void + { + $extractor = new DbalQueryExtractor( + InMemorySqlite::withUsers(InMemorySqlite::connection(), 3), + 'SELECT * FROM users', + ); + + static::assertSame($extractor->statistics(), $extractor->statistics()); + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + + $extractor->withParameters(new ParametersSet()); + + static::assertEquals(Cardinality::exact(0), $extractor->statistics()->rows); + } + + public function test_new_types_drop_the_statistics(): void + { + $extractor = new DbalQueryExtractor( + InMemorySqlite::withUsers(InMemorySqlite::connection(), 3), + 'SELECT * FROM users', + ); + $before = $extractor->statistics(); + + $extractor->withTypes([]); + + static::assertNotSame($before, $extractor->statistics()); + } } diff --git a/src/adapter/etl-adapter-excel/src/Flow/ETL/Adapter/Excel/ExcelExtractor.php b/src/adapter/etl-adapter-excel/src/Flow/ETL/Adapter/Excel/ExcelExtractor.php index 95d71f3cc7..858b0a539b 100644 --- a/src/adapter/etl-adapter-excel/src/Flow/ETL/Adapter/Excel/ExcelExtractor.php +++ b/src/adapter/etl-adapter-excel/src/Flow/ETL/Adapter/Excel/ExcelExtractor.php @@ -4,6 +4,7 @@ namespace Flow\ETL\Adapter\Excel; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InferredSchemaException; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Extractor; @@ -12,9 +13,11 @@ use Flow\ETL\Extractor\FileExtractor; use Flow\ETL\Extractor\FileReading; use Flow\ETL\Extractor\InfersSchema; +use Flow\ETL\Extractor\ListedFiles; use Flow\ETL\Extractor\MetadataColumnsExtractor; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Row\RawRowValues; use Flow\ETL\Rows; @@ -57,12 +60,23 @@ final class ExcelExtractor implements */ private ?WorkbookSampler $sampled = null; + private int $sampledFiles = 0; + + /** + * What the last inference's samples read; null until one ran. Kept apart from $sampled, which extract() takes. + */ + private ?Cardinality $sampledRows = null; + private ?Schema $schema = null; + private ?Statistics $statistics = null; + private string $inferredFrom = ''; private readonly Filesystem $filesystem; + private ?ListedFiles $listed = null; + public function __construct( private readonly Path $path, Filesystem $filesystem = new NativeLocalFilesystem(), @@ -140,6 +154,8 @@ public function extract(FlowContext $context, ?int $limit = null, Filter $pathFi $this->inference, new CellTypeNarrower($this->inference->candidates()), ))->infer($header->names, $sampler->samples($this->inference->sampleSize)); + $this->sampledRows = $sampler->sampledRows(); + $this->sampledFiles = count($sources); } catch (Throwable $failure) { $sampler->close(); @@ -251,9 +267,10 @@ public function schema(): Schema $derived = $this->derivedSchema; if ($derived === null) { + $sources = iterator_to_array($this->sourceFiles($this->filesystem, $this->path), false); $sampler = new WorkbookSampler( new WorkbookReader($this->readOptions, new ExcelFormatDetector($this->filesystem)), - iterator_to_array($this->sourceFiles($this->filesystem, $this->path), false), + $sources, ); try { @@ -266,6 +283,8 @@ public function schema(): Schema $this->inference, new CellTypeNarrower($this->inference->candidates()), ))->infer($header->names, $sampler->samples($this->inference->sampleSize)); + $this->sampledRows = $sampler->sampledRows(); + $this->sampledFiles = count($sources); } catch (Throwable $failure) { $sampler->close(); @@ -289,6 +308,30 @@ public function source(): Path return $this->path; } + /** + * The size is the listed, compressed byte count. Rows are known only when the sample schema inference already + * read every row of every file: OpenSpout discards the row bound of a sheet's . + */ + public function statistics(): Statistics + { + if ($this->statistics !== null) { + return $this->statistics; + } + + $this->listed ??= ListedFiles::of($this->sourceFiles($this->filesystem, $this->path)); + $size = $this->listed->bytes; + + if ($this->sampledRows === null) { + return new Statistics(rows: Cardinality::unknown(), size: $size); + } + + // an extract() under a pushed path filter sampled only the files the filter kept + return $this->statistics = new Statistics( + rows: $this->sampledFiles === $this->listed->count ? $this->sampledRows : Cardinality::unknown(), + size: $size, + ); + } + public function withConvertEmptyToNull(bool $convertEmptyToNull): self { $this->readOptions = $this->readOptions->withConvertEmptyToNull($convertEmptyToNull); @@ -345,5 +388,8 @@ private function forgetInference(): void $this->inferredFrom = ''; $this->sampled?->close(); $this->sampled = null; + $this->sampledRows = null; + $this->sampledFiles = 0; + $this->statistics = null; } } diff --git a/src/adapter/etl-adapter-excel/src/Flow/ETL/Adapter/Excel/WorkbookSampler.php b/src/adapter/etl-adapter-excel/src/Flow/ETL/Adapter/Excel/WorkbookSampler.php index 633656f8d7..fb2e0bf2ed 100644 --- a/src/adapter/etl-adapter-excel/src/Flow/ETL/Adapter/Excel/WorkbookSampler.php +++ b/src/adapter/etl-adapter-excel/src/Flow/ETL/Adapter/Excel/WorkbookSampler.php @@ -4,12 +4,13 @@ namespace Flow\ETL\Adapter\Excel; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor\SourceFile; -use Flow\ETL\Row\RawRowValues; use Flow\ETL\Schema\Inference\SchemaSampler; use Generator; use function array_key_exists; +use function count; final class WorkbookSampler implements SchemaSampler { @@ -18,6 +19,11 @@ final class WorkbookSampler implements SchemaSampler */ private array $sheets = []; + /** + * @var array + */ + private array $samples = []; + /** * @param list $files materialised, not a Generator: columns() and samples() both walk it */ @@ -50,17 +56,37 @@ public function header(): ExcelHeader * A bounded sample keeps what it parsed for take(); an unbounded one parses every row, so it streams and closes * as it goes. Either way SchemaInferrer stops advancing a sheet once its budget is spent. * - * @return Generator> + * @return Generator */ public function samples(int $rowBudget): iterable { foreach ($this->files as $index => $file) { $sheet = $this->sheets[$index] ??= $this->workbook->sheet($file); - yield $rowBudget === -1 ? $sheet->rows() : $sheet->sample(); + yield $this->samples[$index] = new WorkbookSheetSample( + $rowBudget === -1 ? $sheet->rows() : $sheet->sample(), + ); } } + /** + * Exact when the samples read every row of every file, unknown otherwise. + */ + public function sampledRows(): Cardinality + { + $rows = 0; + + foreach ($this->samples as $sample) { + if (!$sample->wholeSheet()) { + return Cardinality::unknown(); + } + + $rows += $sample->rows(); + } + + return count($this->samples) === count($this->files) ? Cardinality::exact($rows) : Cardinality::unknown(); + } + /** * The sheet this sampler read $file through, handed over to the caller to read on and close; null when the * sample never reached the file. diff --git a/src/adapter/etl-adapter-excel/src/Flow/ETL/Adapter/Excel/WorkbookSheetSample.php b/src/adapter/etl-adapter-excel/src/Flow/ETL/Adapter/Excel/WorkbookSheetSample.php new file mode 100644 index 0000000000..e176188f72 --- /dev/null +++ b/src/adapter/etl-adapter-excel/src/Flow/ETL/Adapter/Excel/WorkbookSheetSample.php @@ -0,0 +1,56 @@ + + */ +final class WorkbookSheetSample implements IteratorAggregate +{ + /** + * @var int<0, max> + */ + private int $rows = 0; + + private bool $wholeSheet = false; + + /** + * @param Generator $sheetRows + */ + public function __construct( + private readonly Generator $sheetRows, + ) {} + + /** + * @return Generator + */ + public function getIterator(): Generator + { + foreach ($this->sheetRows as $rowValues) { + $this->rows++; + + yield $rowValues; + } + + $this->wholeSheet = true; + } + + /** + * @return int<0, max> + */ + public function rows(): int + { + return $this->rows; + } + + public function wholeSheet(): bool + { + return $this->wholeSheet; + } +} diff --git a/src/adapter/etl-adapter-excel/tests/Flow/ETL/Adapter/Excel/Tests/Integration/ExcelExtractorTest.php b/src/adapter/etl-adapter-excel/tests/Flow/ETL/Adapter/Excel/Tests/Integration/ExcelExtractorTest.php index c4b2e5e64f..cacb94e1f7 100644 --- a/src/adapter/etl-adapter-excel/tests/Flow/ETL/Adapter/Excel/Tests/Integration/ExcelExtractorTest.php +++ b/src/adapter/etl-adapter-excel/tests/Flow/ETL/Adapter/Excel/Tests/Integration/ExcelExtractorTest.php @@ -9,6 +9,7 @@ use Flow\ETL\Adapter\Excel\ExcelExtractor; use Flow\ETL\Adapter\Excel\ExcelReader; use Flow\ETL\Adapter\Excel\Tests\Context\ExcelFixtureContext; +use Flow\ETL\Cardinality; use Flow\ETL\Config; use Flow\ETL\Exception\InferredSchemaException; use Flow\ETL\Exception\InvalidArgumentException; @@ -18,6 +19,8 @@ use Flow\ETL\Rows; use Flow\ETL\Tests\Context\ExtractedRows; use Flow\ETL\Tests\Double\CountingFilesystem; +use Flow\ETL\Tests\Double\KeepPaths; +use Flow\ETL\Tests\Double\UnsizedFilesystem; use Flow\ETL\Tests\FlowTestCase; use Generator; use PHPUnit\Framework\Attributes\DataProvider; @@ -776,4 +779,125 @@ public function test_limit_reached_on_the_first_file_tail_batch_skips_the_remain static::assertCount(2, ExtractedRows::of($extractor, limit: 2)); } + + /** + * @return Generator + */ + public static function sample_dropping_setters(): Generator + { + yield 'inferSchema' => [static fn(ExcelExtractor $e): mixed => $e->inferSchema(infer_schema())]; + yield 'withConvertEmptyToNull' => [static fn(ExcelExtractor $e): mixed => $e->withConvertEmptyToNull(false)]; + yield 'withHeader' => [static fn(ExcelExtractor $e): mixed => $e->withHeader(true)]; + yield 'withOffset' => [static fn(ExcelExtractor $e): mixed => $e->withOffset(1)]; + yield 'withReader' => [static fn(ExcelExtractor $e): mixed => $e->withReader(ExcelReader::XLSX)]; + yield 'withSheetName' => [static fn(ExcelExtractor $e): mixed => $e->withSheetName('Sheet1')]; + } + + #[DataProvider('sample_dropping_setters')] + public function test_a_changed_read_option_drops_the_sampled_rows(Closure $setter): void + { + $extractor = from_excel(ExcelFixtureContext::path('sniff/a')); + $extractor->schema(); + + static::assertEquals(Cardinality::exact(10), $extractor->statistics()->rows); + + $setter($extractor); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + } + + public function test_a_declared_schema_declares_unknown_rows(): void + { + $extractor = from_excel(ExcelFixtureContext::path('sniff/a'))->withSchema(schema(int_schema('id'))); + $extractor->schema(); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + } + + public function test_a_member_without_a_size_makes_the_size_unknown(): void + { + $extractor = from_excel( + ExcelFixtureContext::path('sniff/*'), + new UnsizedFilesystem(native_local_filesystem(), [ExcelFixtureContext::path('sniff/b')->uri()]), + ); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->size); + } + + public function test_a_sample_that_read_every_sheet_declares_exact_rows(): void + { + $extractor = from_excel(ExcelFixtureContext::path('sniff/*')); + $extractor->schema(); + + static::assertEquals(Cardinality::exact(20), $extractor->statistics()->rows); + } + + public function test_a_sample_that_stopped_early_declares_unknown_rows(): void + { + $extractor = from_excel(ExcelFixtureContext::path('sniff/*'))->inferSchema(infer_schema()->sampleSize(3)); + $extractor->schema(); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + } + + public function test_extract_samples_like_schema_does(): void + { + $extractor = from_excel(ExcelFixtureContext::path('sniff/a')); + + iterator_to_array($extractor->extract(flow_context()), false); + + static::assertEquals(Cardinality::exact(10), $extractor->statistics()->rows); + } + + public function test_it_declares_the_listed_byte_total_exactly(): void + { + static::assertEquals( + Cardinality::exact(17_504), + from_excel(ExcelFixtureContext::path('sniff/*'))->statistics()->size, + ); + } + + public function test_statistics_before_schema_declares_unknown_rows(): void + { + $extractor = from_excel(ExcelFixtureContext::path('sniff/a')); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + + $extractor->schema(); + + static::assertEquals(Cardinality::exact(10), $extractor->statistics()->rows); + } + + public function test_statistics_are_computed_at_most_once(): void + { + $filesystem = new CountingFilesystem(native_local_filesystem()); + $extractor = from_excel(ExcelFixtureContext::path('sniff/a'), $filesystem); + $extractor->schema(); + $listCalls = $filesystem->listCalls; + + static::assertSame($extractor->statistics(), $extractor->statistics()); + static::assertSame($listCalls + 1, $filesystem->listCalls); + } + + public function test_the_listing_is_read_once(): void + { + $filesystem = new CountingFilesystem(native_local_filesystem()); + $extractor = from_excel(ExcelFixtureContext::path('sniff/a'), $filesystem); + + $extractor->statistics(); + $extractor->statistics(); + + static::assertSame(1, $filesystem->listCalls); + } + + public function test_a_filtered_extract_declares_unknown_rows(): void + { + $extractor = from_excel(ExcelFixtureContext::path('sniff/*')); + + iterator_to_array($extractor->extract(flow_context(), null, new KeepPaths([ExcelFixtureContext::path( + 'sniff/a', + )])), false); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + } } diff --git a/src/adapter/etl-adapter-excel/tests/Flow/ETL/Adapter/Excel/Tests/Integration/WorkbookSamplerTest.php b/src/adapter/etl-adapter-excel/tests/Flow/ETL/Adapter/Excel/Tests/Integration/WorkbookSamplerTest.php index 2f968fed9a..bee1633f24 100644 --- a/src/adapter/etl-adapter-excel/tests/Flow/ETL/Adapter/Excel/Tests/Integration/WorkbookSamplerTest.php +++ b/src/adapter/etl-adapter-excel/tests/Flow/ETL/Adapter/Excel/Tests/Integration/WorkbookSamplerTest.php @@ -6,9 +6,10 @@ use Flow\ETL\Adapter\Excel\ExcelReadOptions; use Flow\ETL\Adapter\Excel\Tests\Context\ExcelFixtureContext; +use Flow\ETL\Adapter\Excel\WorkbookSheetSample; +use Flow\ETL\Cardinality; use Flow\ETL\Tests\Double\CountingFilesystem; use Flow\ETL\Tests\FlowTestCase; -use Generator; use function Flow\ETL\DSL\bool_schema; use function Flow\ETL\DSL\date_schema; @@ -233,7 +234,7 @@ public function test_samples_does_not_ration_the_row_budget(): void } } - public function test_samples_yields_one_unstarted_generator_per_sheet(): void + public function test_samples_yields_one_unopened_sample_per_sheet(): void { $filesystem = new CountingFilesystem(native_local_filesystem()); $sampler = ExcelFixtureContext::globSampler('sniff/*', $filesystem); @@ -243,11 +244,11 @@ public function test_samples_yields_one_unstarted_generator_per_sheet(): void $names = $sampler->header()->names; $afterHeader = $filesystem->readFromCalls; - // the first inner generator is the sheet header() read from, so its header row is already spent + // the first sample is the sheet header() read from, so its header row is already spent $firstSheet = iterator_to_array($samples[0], false); static::assertCount(2, $samples); - static::assertContainsOnlyInstancesOf(Generator::class, $samples); + static::assertContainsOnlyInstancesOf(WorkbookSheetSample::class, $samples); static::assertSame(0, $beforeAnyAdvance); static::assertSame(['id', 'name', 'email'], $names); static::assertSame(1, $afterHeader); @@ -341,4 +342,47 @@ public function test_typed_cells_keep_their_types(): void ExcelFixtureContext::infer(ExcelFixtureContext::sampler('int_float_bool.xlsx')), ); } + + public function test_nothing_sampled_declares_unknown_rows(): void + { + static::assertEquals(Cardinality::unknown(), ExcelFixtureContext::sampler('sniff/a')->sampledRows()); + } + + public function test_a_file_the_samples_never_reached_declares_unknown_rows(): void + { + $sampler = ExcelFixtureContext::globSampler('sniff/*'); + $sampler->header(); + + foreach ($sampler->samples(20_480) as $sample) { + iterator_to_array($sample, false); + + break; + } + + static::assertEquals(Cardinality::unknown(), $sampler->sampledRows()); + } + + public function test_samples_that_read_every_sheet_declare_exact_rows(): void + { + $sampler = ExcelFixtureContext::globSampler('sniff/*'); + $sampler->header(); + + foreach ($sampler->samples(20_480) as $sample) { + iterator_to_array($sample, false); + } + + static::assertEquals(Cardinality::exact(20), $sampler->sampledRows()); + } + + public function test_an_unbounded_sample_counts_the_rows_it_streamed(): void + { + $sampler = ExcelFixtureContext::sampler('sniff/a'); + $sampler->header(); + + foreach ($sampler->samples(-1) as $sample) { + iterator_to_array($sample, false); + } + + static::assertEquals(Cardinality::exact(10), $sampler->sampledRows()); + } } diff --git a/src/adapter/etl-adapter-excel/tests/Flow/ETL/Adapter/Excel/Tests/Unit/WorkbookSheetSampleTest.php b/src/adapter/etl-adapter-excel/tests/Flow/ETL/Adapter/Excel/Tests/Unit/WorkbookSheetSampleTest.php new file mode 100644 index 0000000000..45d87eb0a2 --- /dev/null +++ b/src/adapter/etl-adapter-excel/tests/Flow/ETL/Adapter/Excel/Tests/Unit/WorkbookSheetSampleTest.php @@ -0,0 +1,58 @@ + 1]); + yield new RawRowValues(['id' => 2]); + })(), + ); + + foreach ($sample as $_rowValues) { + break; + } + + static::assertSame(1, $sample->rows()); + static::assertFalse($sample->wholeSheet()); + } + + public function test_a_sample_read_to_the_end_is_whole(): void + { + $sample = new WorkbookSheetSample( + (static function () { + yield new RawRowValues(['id' => 1]); + yield new RawRowValues(['id' => 2]); + })(), + ); + + iterator_to_array($sample, false); + + static::assertSame(2, $sample->rows()); + static::assertTrue($sample->wholeSheet()); + } + + public function test_nothing_is_counted_before_iteration(): void + { + $sample = new WorkbookSheetSample( + (static function () { + yield new RawRowValues(['id' => 1]); + })(), + ); + + static::assertSame(0, $sample->rows()); + static::assertFalse($sample->wholeSheet()); + } +} diff --git a/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetExtractor.php b/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetExtractor.php index b2711f71e3..2472e454e5 100644 --- a/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetExtractor.php +++ b/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetExtractor.php @@ -4,6 +4,7 @@ namespace Flow\ETL\Adapter\GoogleSheet; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InferredSchemaException; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Extractor; @@ -14,6 +15,7 @@ use Flow\ETL\Extractor\MetadataColumnsExtractor; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Row\RawRowValues; use Flow\ETL\Rows; @@ -58,8 +60,15 @@ final class GoogleSheetExtractor implements private int $rowsPerPage = 1000; + /** + * The grid row count the last schema inference fetched; null until one ran. + */ + private ?int $gridRowCount = null; + private ?Schema $schema = null; + private ?Statistics $statistics = null; + public function __construct( private readonly Sheets $service, private readonly string $spreadsheetId, @@ -87,6 +96,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator $this->inference, $this->readOptions->typer($this->inference->candidates()), ))->infer($sampler->header(), $sampler->samples($this->inference->sampleSize))); + + if ($sampledNow) { + $this->gridRowCount = $sampler->gridRowCount(); + } + $schema = $this->addMetadataColumns ? $base->add(str_schema('_spread_sheet_id'), str_schema('_sheet_name')) : $base; @@ -163,6 +177,8 @@ public function inferSchema(SchemaInferenceBuilder $builder): static { $this->inference = $builder->build(); $this->derivedSchema = null; + $this->gridRowCount = null; + $this->statistics = null; return $this; } @@ -173,20 +189,41 @@ public function schema(): Schema new GoogleSheetReader($this->service, $this->spreadsheetId, $this->columnRange, $this->readOptions), $this->inference->sampleSize, ); + $sampledNow = $this->schema === null && $this->derivedSchema === null; $base = $this->schema ?? ($this->derivedSchema ??= (new SchemaInferrer( $this->inference, $this->readOptions->typer($this->inference->candidates()), ))->infer($sampler->header(), $sampler->samples($this->inference->sampleSize))); + if ($sampledNow) { + $this->gridRowCount = $sampler->gridRowCount(); + } + return $this->addMetadataColumns ? $base->add(str_schema('_spread_sheet_id'), str_schema('_sheet_name')) : $base; } + /** + * The grid's allocated rows bound the populated ones; they come from the sample schema inference already + * fetched, and without one nothing is declared - no request is made just for them. The Sheets API reports no + * byte size. + */ + public function statistics(): Statistics + { + if ($this->gridRowCount === null) { + return new Statistics(); + } + + return $this->statistics ??= new Statistics(rows: Cardinality::atMost($this->gridRowCount)); + } + public function withDropExtraColumns(bool $dropExtraColumns): self { $this->readOptions = $this->readOptions->withDropExtraColumns($dropExtraColumns); $this->derivedSchema = null; + $this->gridRowCount = null; + $this->statistics = null; return $this; } @@ -195,6 +232,8 @@ public function withEmptyToNull(bool $emptyToNull): self { $this->readOptions = $this->readOptions->withEmptyToNull($emptyToNull); $this->derivedSchema = null; + $this->gridRowCount = null; + $this->statistics = null; return $this; } @@ -203,6 +242,8 @@ public function withHeader(bool $withHeader): self { $this->readOptions = $this->readOptions->withHeader($withHeader); $this->derivedSchema = null; + $this->gridRowCount = null; + $this->statistics = null; return $this; } @@ -214,6 +255,8 @@ public function withOptions(array $options): self { $this->readOptions = $this->readOptions->withOptions($options); $this->derivedSchema = null; + $this->gridRowCount = null; + $this->statistics = null; return $this; } diff --git a/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetReader.php b/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetReader.php index eb07c04d44..c4eae47aac 100644 --- a/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetReader.php +++ b/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetReader.php @@ -95,6 +95,6 @@ public function sample(int $rows): GoogleSheetSample $values->get($this->spreadsheetId, $range->toString(), $this->readOptions->options)->getValues() ?? [], )); - return new GoogleSheetSample($encoder->headers(), $decoded, $range->endRow >= $rowCount); + return new GoogleSheetSample($encoder->headers(), $decoded, $range->endRow >= $rowCount, $rowCount); } } diff --git a/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetSample.php b/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetSample.php index e05ecdb94f..77a7380ed4 100644 --- a/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetSample.php +++ b/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetSample.php @@ -13,10 +13,12 @@ * @param list $rows * @param bool $wholeSheet the range reached the last row of the grid, so $rows IS the sheet and a read can be * served from it instead of fetching the same rows again + * @param int $gridRowCount the rows the grid allocates - an upper bound on the rows it holds */ public function __construct( public array $names, public array $rows, public bool $wholeSheet, + public int $gridRowCount, ) {} } diff --git a/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetSampler.php b/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetSampler.php index 5e7222ed15..362ad86c77 100644 --- a/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetSampler.php +++ b/src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/GoogleSheetSampler.php @@ -54,6 +54,11 @@ public function batches(int $batchSize): ?Generator })(); } + public function gridRowCount(): int + { + return $this->sample()->gridRowCount; + } + /** * @return list */ diff --git a/src/adapter/etl-adapter-google-sheet/tests/Flow/ETL/Adapter/GoogleSheet/Tests/Unit/GoogleSheetExtractorTest.php b/src/adapter/etl-adapter-google-sheet/tests/Flow/ETL/Adapter/GoogleSheet/Tests/Unit/GoogleSheetExtractorTest.php index 96c83a550c..a9f72d5cea 100644 --- a/src/adapter/etl-adapter-google-sheet/tests/Flow/ETL/Adapter/GoogleSheet/Tests/Unit/GoogleSheetExtractorTest.php +++ b/src/adapter/etl-adapter-google-sheet/tests/Flow/ETL/Adapter/GoogleSheet/Tests/Unit/GoogleSheetExtractorTest.php @@ -9,9 +9,11 @@ use Flow\ETL\Adapter\GoogleSheet\Tests\Context\GoogleSheetFixtureContext; use Flow\ETL\Adapter\GoogleSheet\Tests\Double\StubSpreadsheetsResource; use Flow\ETL\Adapter\GoogleSheet\Tests\Mother\SheetValuesMother; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InferredSchemaException; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Exception\SchemaMismatchException; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\Rows; use Flow\ETL\Tests\Context\ExtractedRows; use Flow\ETL\Tests\FlowTestCase; @@ -544,4 +546,75 @@ public function test_is_repeatable(): void { static::assertTrue(GoogleSheetFixtureContext::extractor(new Sheets())->isRepeatable()); } + + public function test_it_declares_the_grid_row_count_as_an_upper_bound(): void + { + $values = GoogleSheetFixtureContext::values(SheetValuesMother::range([['id'], ['1']])); + $extractor = GoogleSheetFixtureContext::extractor(GoogleSheetFixtureContext::service(100, $values)); + $extractor->schema(); + + static::assertEquals(Cardinality::atMost(100), $extractor->statistics()->rows); + } + + public function test_it_declares_no_byte_size(): void + { + $values = GoogleSheetFixtureContext::values(SheetValuesMother::range([['id'], ['1']])); + $extractor = GoogleSheetFixtureContext::extractor(GoogleSheetFixtureContext::service(100, $values)); + $extractor->schema(); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->size); + } + + public function test_the_grid_row_count_costs_no_extra_request(): void + { + $values = GoogleSheetFixtureContext::values(SheetValuesMother::range([['id'], ['1']])); + $service = GoogleSheetFixtureContext::service(100, $values); + $extractor = GoogleSheetFixtureContext::extractor($service); + + $extractor->schema(); + $statistics = $extractor->statistics(); + + static::assertSame($statistics, $extractor->statistics()); + static::assertCount(1, $values->getCalls); + + /** @var StubSpreadsheetsResource $spreadsheets */ + $spreadsheets = $service->spreadsheets; + static::assertSame(1, $spreadsheets->calls); + } + + public function test_statistics_without_a_sample_declare_nothing_and_request_nothing(): void + { + $values = GoogleSheetFixtureContext::values(); + $service = GoogleSheetFixtureContext::service(100, $values); + + static::assertEquals(new Statistics(), GoogleSheetFixtureContext::extractor($service)->statistics()); + static::assertSame([], $values->getCalls); + + /** @var StubSpreadsheetsResource $spreadsheets */ + $spreadsheets = $service->spreadsheets; + static::assertSame(0, $spreadsheets->calls); + } + + public function test_a_read_samples_the_grid_row_count_too(): void + { + $values = GoogleSheetFixtureContext::valuesAndBatches([SheetValuesMother::range([['id'], ['1'], ['2']])], []); + $extractor = GoogleSheetFixtureContext::extractor(GoogleSheetFixtureContext::service(3, $values)); + + iterator_to_array($extractor->extract(flow_context()), false); + + static::assertEquals(Cardinality::atMost(3), $extractor->statistics()->rows); + } + + #[DataProvider('memo_resetting_setters')] + public function test_shape_changing_setters_drop_the_sampled_statistics(Closure $setter): void + { + $values = GoogleSheetFixtureContext::values(SheetValuesMother::range([['id'], ['1']])); + $extractor = GoogleSheetFixtureContext::extractor(GoogleSheetFixtureContext::service(100, $values)); + + $extractor->schema(); + $extractor->statistics(); + $setter($extractor); + + static::assertEquals(new Statistics(), $extractor->statistics()); + } } diff --git a/src/adapter/etl-adapter-google-sheet/tests/Flow/ETL/Adapter/GoogleSheet/Tests/Unit/GoogleSheetSampleTest.php b/src/adapter/etl-adapter-google-sheet/tests/Flow/ETL/Adapter/GoogleSheet/Tests/Unit/GoogleSheetSampleTest.php index 740fbb5874..3cabc1ae69 100644 --- a/src/adapter/etl-adapter-google-sheet/tests/Flow/ETL/Adapter/GoogleSheet/Tests/Unit/GoogleSheetSampleTest.php +++ b/src/adapter/etl-adapter-google-sheet/tests/Flow/ETL/Adapter/GoogleSheet/Tests/Unit/GoogleSheetSampleTest.php @@ -12,10 +12,11 @@ final class GoogleSheetSampleTest extends FlowTestCase { public function test_carries_names_and_rows(): void { - $sample = new GoogleSheetSample(['a'], [new RawRowValues(['a' => 1])], wholeSheet: true); + $sample = new GoogleSheetSample(['a'], [new RawRowValues(['a' => 1])], wholeSheet: true, gridRowCount: 1000); static::assertSame(['a'], $sample->names); static::assertSame([['a' => 1]], [$sample->rows[0]->values]); static::assertTrue($sample->wholeSheet); + static::assertSame(1000, $sample->gridRowCount); } } diff --git a/src/adapter/etl-adapter-google-sheet/tests/Flow/ETL/Adapter/GoogleSheet/Tests/Unit/GoogleSheetSamplerTest.php b/src/adapter/etl-adapter-google-sheet/tests/Flow/ETL/Adapter/GoogleSheet/Tests/Unit/GoogleSheetSamplerTest.php index be2405a929..32780a6304 100644 --- a/src/adapter/etl-adapter-google-sheet/tests/Flow/ETL/Adapter/GoogleSheet/Tests/Unit/GoogleSheetSamplerTest.php +++ b/src/adapter/etl-adapter-google-sheet/tests/Flow/ETL/Adapter/GoogleSheet/Tests/Unit/GoogleSheetSamplerTest.php @@ -68,4 +68,14 @@ public function test_header_and_samples_share_one_request(): void static::assertCount(1, $values->getCalls); } + + public function test_grid_row_count_comes_from_the_sample_request(): void + { + $values = GoogleSheetFixtureContext::values(SheetValuesMother::range([['id'], ['1']])); + $sampler = GoogleSheetFixtureContext::sampler($values, 2, rowCount: 500); + + static::assertSame(500, $sampler->gridRowCount()); + $sampler->header(); + static::assertCount(1, $values->getCalls); + } } diff --git a/src/adapter/etl-adapter-http/src/Flow/ETL/Adapter/Http/PsrHttpClientDynamicExtractor.php b/src/adapter/etl-adapter-http/src/Flow/ETL/Adapter/Http/PsrHttpClientDynamicExtractor.php index 9f38ffddf5..863b36448a 100644 --- a/src/adapter/etl-adapter-http/src/Flow/ETL/Adapter/Http/PsrHttpClientDynamicExtractor.php +++ b/src/adapter/etl-adapter-http/src/Flow/ETL/Adapter/Http/PsrHttpClientDynamicExtractor.php @@ -7,6 +7,7 @@ use Flow\ETL\Adapter\Http\DynamicExtractor\NextRequestFactory; use Flow\ETL\Extractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -126,4 +127,9 @@ public function withSchema(Schema $schema): static return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/adapter/etl-adapter-http/src/Flow/ETL/Adapter/Http/PsrHttpClientPaginatedExtractor.php b/src/adapter/etl-adapter-http/src/Flow/ETL/Adapter/Http/PsrHttpClientPaginatedExtractor.php index 3b51857e37..481cc8dac3 100644 --- a/src/adapter/etl-adapter-http/src/Flow/ETL/Adapter/Http/PsrHttpClientPaginatedExtractor.php +++ b/src/adapter/etl-adapter-http/src/Flow/ETL/Adapter/Http/PsrHttpClientPaginatedExtractor.php @@ -8,6 +8,7 @@ use Flow\ETL\Adapter\Http\Pagination\Paginator; use Flow\ETL\Extractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -130,4 +131,9 @@ public function withSchema(Schema $schema): static return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/adapter/etl-adapter-http/src/Flow/ETL/Adapter/Http/PsrHttpClientStaticExtractor.php b/src/adapter/etl-adapter-http/src/Flow/ETL/Adapter/Http/PsrHttpClientStaticExtractor.php index 1c1f880c36..97210c3fc3 100644 --- a/src/adapter/etl-adapter-http/src/Flow/ETL/Adapter/Http/PsrHttpClientStaticExtractor.php +++ b/src/adapter/etl-adapter-http/src/Flow/ETL/Adapter/Http/PsrHttpClientStaticExtractor.php @@ -6,6 +6,7 @@ use Flow\ETL\Extractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -121,4 +122,9 @@ public function withSchema(Schema $schema): static return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonExtractor.php b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonExtractor.php index 23af3904af..fe306d8b1f 100644 --- a/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonExtractor.php +++ b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonExtractor.php @@ -4,6 +4,7 @@ namespace Flow\ETL\Adapter\JSON\JSONMachine; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Extractor; use Flow\ETL\Extractor\BatchableExtractor; @@ -11,9 +12,11 @@ use Flow\ETL\Extractor\FileExtractor; use Flow\ETL\Extractor\FileReading; use Flow\ETL\Extractor\InfersSchema; +use Flow\ETL\Extractor\ListedFiles; use Flow\ETL\Extractor\MetadataColumnsExtractor; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Row\RawRowValues; use Flow\ETL\Rows; @@ -49,10 +52,19 @@ final class JsonExtractor implements private bool $pointerToEntryName = false; + /** + * What the last schema inference sampled; null until one ran. + */ + private ?JsonSampledFiles $sampled = null; + private ?Schema $schema = null; + private ?Statistics $statistics = null; + private readonly Filesystem $filesystem; + private ?ListedFiles $listed = null; + public function __construct( private readonly Path $path, Filesystem $filesystem = new NativeLocalFilesystem(), @@ -101,11 +113,13 @@ public function extract(FlowContext $context, ?int $limit = null, Filter $pathFi $derived = $this->derivedSchema; if ($derived === null) { + $samples = new JsonSamples($reader->samples($this->inference->sampleSize)); $derived = $this->derivedSchema = (new SchemaInferrer($this->inference, new InstanceOfTypeNarrower()))->infer( [], - $reader->samples($this->inference->sampleSize), + $samples, ); + $this->sampled = $samples->sampledFiles(); } $base = $fileColumns->withoutTail($derived); @@ -145,6 +159,8 @@ public function inferSchema(SchemaInferenceBuilder $builder): static { $this->inference = $builder->build(); $this->derivedSchema = null; + $this->sampled = null; + $this->statistics = null; return $this; } @@ -169,11 +185,13 @@ public function schema(): Schema iterator_to_array($this->sourceFiles($this->filesystem, $this->path), false), ); + $samples = new JsonSamples($reader->samples($this->inference->sampleSize)); $derived = $this->derivedSchema = (new SchemaInferrer($this->inference, new InstanceOfTypeNarrower()))->infer( [], - $reader->samples($this->inference->sampleSize), + $samples, ); + $this->sampled = $samples->sampledFiles(); } return $fileColumns->declare($fileColumns->withoutTail($derived)); @@ -189,6 +207,26 @@ public function source(): Path return $this->path; } + /** + * A document has no bytes-per-row unit, so rows are known only when the sample schema inference already read + * every element of every file. + */ + public function statistics(): Statistics + { + if ($this->statistics !== null) { + return $this->statistics; + } + + $this->listed ??= ListedFiles::of($this->sourceFiles($this->filesystem, $this->path)); + $size = $this->listed->bytes; + + if ($this->sampled === null) { + return new Statistics(rows: Cardinality::unknown(), size: $size); + } + + return $this->statistics = new Statistics(rows: $this->sampled->exactRows($this->listed->count), size: $size); + } + /** * @param string $pointer * @param bool $pointerToEntryName - when true pointer will be used as entry name for extracted data @@ -196,6 +234,8 @@ public function source(): Path public function withPointer(string $pointer, bool $pointerToEntryName = false): self { $this->derivedSchema = null; + $this->sampled = null; + $this->statistics = null; $this->pointer = $pointer; $this->pointerToEntryName = $pointerToEntryName; diff --git a/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonFileReader.php b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonFileReader.php index 3237d46534..89cfbbf2c9 100644 --- a/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonFileReader.php +++ b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonFileReader.php @@ -18,6 +18,7 @@ use function is_array; use function iterator_to_array; use function json_decode; +use function strlen; use function trim; /** @@ -127,9 +128,12 @@ public function documentItems(SourceStream $stream): Generator * * @return Generator */ - public function lineItems(SourceStream $stream): Generator + public function lineItems(SourceStream $stream, JsonReadBytes $read = new JsonReadBytes()): Generator { foreach ($stream->readLines() as $line) { + // readLines() strips the "\n" and hides whether the last line had one, so that line counts 1 byte more + $read->add(strlen($line) + 1); + if (trim($line, self::C_ISSPACE) === '') { continue; } @@ -155,14 +159,14 @@ public function lineItems(SourceStream $stream): Generator * * @return Generator */ - public function sample(SourceFile $source): Generator + public function sample(SourceFile $source, JsonReadBytes $read = new JsonReadBytes()): Generator { $stream = $this->filesystem->readFrom($source->path); try { $items = match ($this->format) { JsonFormat::Document => $this->documentItems($stream), - JsonFormat::Lines => $this->lineItems($stream), + JsonFormat::Lines => $this->lineItems($stream, $read), }; /** @var array $row */ @@ -186,12 +190,12 @@ public function sample(SourceFile $source): Generator /** * $rowBudget is deliberately unused: sample() is lazy and SchemaInferrer stops advancing it. * - * @return Generator> + * @return Generator */ public function samples(int $rowBudget): iterable { foreach ($this->sources as $source) { - yield $this->sample($source); + yield new JsonFileSample($this, $source); } } } diff --git a/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonFileSample.php b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonFileSample.php new file mode 100644 index 0000000000..817cda9469 --- /dev/null +++ b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonFileSample.php @@ -0,0 +1,55 @@ + + */ +final class JsonFileSample implements IteratorAggregate +{ + private ?JsonSampledRows $sampled = null; + + public function __construct( + private readonly JsonFileReader $reader, + private readonly SourceFile $source, + ) {} + + /** + * Records what it read when the consumer stops or the file ends, whichever comes first. + * + * @return Generator + */ + public function getIterator(): Generator + { + $read = new JsonReadBytes(); + $rows = 0; + $wholeFile = false; + + try { + foreach ($this->reader->sample($this->source, $read) as $row) { + $rows++; + + yield $row; + } + + $wholeFile = true; + } finally { + $this->sampled = new JsonSampledRows($rows, $read->total(), $wholeFile); + } + } + + /** + * Null until the sample was iterated. + */ + public function sampled(): ?JsonSampledRows + { + return $this->sampled; + } +} diff --git a/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonLinesExtractor.php b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonLinesExtractor.php index 768b3a295f..249feaac05 100644 --- a/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonLinesExtractor.php +++ b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonLinesExtractor.php @@ -4,6 +4,7 @@ namespace Flow\ETL\Adapter\JSON\JSONMachine; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Extractor; use Flow\ETL\Extractor\BatchableExtractor; @@ -11,9 +12,11 @@ use Flow\ETL\Extractor\FileExtractor; use Flow\ETL\Extractor\FileReading; use Flow\ETL\Extractor\InfersSchema; +use Flow\ETL\Extractor\ListedFiles; use Flow\ETL\Extractor\MetadataColumnsExtractor; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Row\RawRowValues; use Flow\ETL\Rows; @@ -49,10 +52,19 @@ final class JsonLinesExtractor implements private bool $pointerToEntryName = false; + /** + * What the last schema inference sampled; null until one ran. + */ + private ?JsonSampledFiles $sampled = null; + private ?Schema $schema = null; + private ?Statistics $statistics = null; + private readonly Filesystem $filesystem; + private ?ListedFiles $listed = null; + public function __construct( private readonly Path $path, Filesystem $filesystem = new NativeLocalFilesystem(), @@ -101,11 +113,13 @@ public function extract(FlowContext $context, ?int $limit = null, Filter $pathFi $derived = $this->derivedSchema; if ($derived === null) { + $samples = new JsonSamples($reader->samples($this->inference->sampleSize)); $derived = $this->derivedSchema = (new SchemaInferrer($this->inference, new InstanceOfTypeNarrower()))->infer( [], - $reader->samples($this->inference->sampleSize), + $samples, ); + $this->sampled = $samples->sampledFiles(); } $base = $fileColumns->withoutTail($derived); @@ -145,6 +159,8 @@ public function inferSchema(SchemaInferenceBuilder $builder): static { $this->inference = $builder->build(); $this->derivedSchema = null; + $this->sampled = null; + $this->statistics = null; return $this; } @@ -169,11 +185,13 @@ public function schema(): Schema iterator_to_array($this->sourceFiles($this->filesystem, $this->path), false), ); + $samples = new JsonSamples($reader->samples($this->inference->sampleSize)); $derived = $this->derivedSchema = (new SchemaInferrer($this->inference, new InstanceOfTypeNarrower()))->infer( [], - $reader->samples($this->inference->sampleSize), + $samples, ); + $this->sampled = $samples->sampledFiles(); } return $fileColumns->declare($fileColumns->withoutTail($derived)); @@ -189,6 +207,29 @@ public function source(): Path return $this->path; } + /** + * Rows come from the sample schema inference already read; without one they are unknown, a sample is never read + * just for them. + */ + public function statistics(): Statistics + { + if ($this->statistics !== null) { + return $this->statistics; + } + + $this->listed ??= ListedFiles::of($this->sourceFiles($this->filesystem, $this->path)); + $size = $this->listed->bytes; + + if ($this->sampled === null) { + return new Statistics(rows: Cardinality::unknown(), size: $size); + } + + return $this->statistics = new Statistics( + rows: $this->sampled->estimatedRows($this->listed->count, $size), + size: $size, + ); + } + /** * @param string $pointer * @param bool $pointerToEntryName - when true pointer will be used as entry name for extracted data @@ -196,6 +237,8 @@ public function source(): Path public function withPointer(string $pointer, bool $pointerToEntryName = false): self { $this->derivedSchema = null; + $this->sampled = null; + $this->statistics = null; $this->pointer = $pointer; $this->pointerToEntryName = $pointerToEntryName; diff --git a/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonReadBytes.php b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonReadBytes.php new file mode 100644 index 0000000000..65d8fdb6d6 --- /dev/null +++ b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonReadBytes.php @@ -0,0 +1,29 @@ + + */ + private int $total = 0; + + /** + * @param int<0, max> $bytes + */ + public function add(int $bytes): void + { + $this->total += $bytes; + } + + /** + * @return int<0, max> + */ + public function total(): int + { + return $this->total; + } +} diff --git a/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonSampledFiles.php b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonSampledFiles.php new file mode 100644 index 0000000000..457a6eccea --- /dev/null +++ b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonSampledFiles.php @@ -0,0 +1,52 @@ +whole && $this->files === $listedFiles ? Cardinality::exact($this->rows) : Cardinality::unknown(); + } + + /** + * Listed bytes over the mean bytes of the sampled rows; exact when the samples read every row of every listed file. + */ + public function estimatedRows(int $listedFiles, Cardinality $listedBytes): Cardinality + { + if ($this->whole && $this->files === $listedFiles) { + return Cardinality::exact($this->rows); + } + + if ($listedBytes->estimate === null || $this->rows === 0 || $this->bytes === 0) { + return Cardinality::unknown(); + } + + return Cardinality::approximately( + (int) round($listedBytes->estimate / ($this->bytes / $this->rows)), + Cardinality::DEFAULT_RELATIVE_ERROR, + ); + } +} diff --git a/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonSampledRows.php b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonSampledRows.php new file mode 100644 index 0000000000..8d0c68509c --- /dev/null +++ b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonSampledRows.php @@ -0,0 +1,40 @@ + + */ + public int $bytes; + + /** + * @var int<0, max> + */ + public int $rows; + + /** + * @param int $rows rows read while sampling + * @param int $bytes bytes of the lines those rows came from, line endings and blank lines included; 0 for a document + * @param bool $wholeFile the sample reached the end of the file, so $rows is every row it holds + */ + public function __construct( + int $rows, + int $bytes, + public bool $wholeFile, + ) { + if ($rows < 0 || $bytes < 0) { + throw new InvalidArgumentException( + 'Sampled rows and bytes must not be negative, given: ' . $rows . ' rows, ' . $bytes . ' bytes', + ); + } + + $this->rows = $rows; + $this->bytes = $bytes; + } +} diff --git a/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonSamples.php b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonSamples.php new file mode 100644 index 0000000000..0de9d8a3bb --- /dev/null +++ b/src/adapter/etl-adapter-json/src/Flow/ETL/Adapter/JSON/JSONMachine/JsonSamples.php @@ -0,0 +1,63 @@ + + */ +final class JsonSamples implements IteratorAggregate +{ + /** + * @var list + */ + private array $pulled = []; + + /** + * @param iterable $samples + */ + public function __construct( + private readonly iterable $samples, + ) {} + + /** + * Keeps only the samples the consumer pulled: a listing the inference stopped short of costs nothing. + * + * @return Generator + */ + public function getIterator(): Generator + { + foreach ($this->samples as $sample) { + $this->pulled[] = $sample; + + yield $sample; + } + } + + public function sampledFiles(): JsonSampledFiles + { + $files = 0; + $rows = 0; + $bytes = 0; + $whole = true; + + foreach ($this->pulled as $sample) { + $sampled = $sample->sampled(); + + if ($sampled === null) { + continue; + } + + $files++; + $rows += $sampled->rows; + $bytes += $sampled->bytes; + $whole = $whole && $sampled->wholeFile; + } + + return new JsonSampledFiles($files, $rows, $bytes, $whole); + } +} diff --git a/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonExtractorTest.php b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonExtractorTest.php index b867f3fc00..fe1ed868bf 100644 --- a/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonExtractorTest.php +++ b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonExtractorTest.php @@ -7,12 +7,14 @@ use Closure; use Flow\ETL\Adapter\JSON\JSONMachine\JsonExtractor; use Flow\ETL\Adapter\JSON\Tests\Context\JsonFixtureContext; +use Flow\ETL\Cardinality; use Flow\ETL\Config; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Exception\SchemaMismatchException; use Flow\ETL\Extractor\Signal; use Flow\ETL\Schema\Definition\StringDefinition; use Flow\ETL\Tests\Context\ExtractedRows; +use Flow\ETL\Tests\Context\MemoryFiles; use Flow\ETL\Tests\Double\CountingFilesystem; use Flow\ETL\Tests\FlowTestCase; use Flow\Filesystem\Local\NativeLocalFilesystem; @@ -581,4 +583,108 @@ public function test_is_repeatable(): void { static::assertTrue(from_json(__DIR__ . '/../../Fixtures/timezones.json')->isRepeatable()); } + + /** + * @param Closure(JsonExtractor): void $setter + */ + #[DataProvider('shapeChangingSetters')] + public function test_a_shape_changing_setter_drops_the_sample(Closure $setter): void + { + $extractor = from_json(JsonFixtureContext::path('five_rows.json')); + $extractor->schema(); + + static::assertEquals(Cardinality::exact(5), $extractor->statistics()->rows); + + $setter($extractor); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + } + + public function test_a_declared_schema_declares_unknown_rows(): void + { + $extractor = from_json(JsonFixtureContext::path('five_rows.json'), schema: schema(int_schema('id'))); + $extractor->schema(); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + static::assertEquals(Cardinality::exact(46), $extractor->statistics()->size); + } + + public function test_a_document_the_sample_read_whole_declares_an_exact_row_count(): void + { + $extractor = from_json(JsonFixtureContext::path('five_rows.json')); + $extractor->schema(); + + static::assertEquals(Cardinality::exact(5), $extractor->statistics()->rows); + } + + public function test_a_document_the_sample_did_not_finish_declares_unknown_rows(): void + { + $extractor = from_json(JsonFixtureContext::path('five_rows.json'))->inferSchema(infer_schema()->sampleSize(2)); + $extractor->schema(); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + } + + public function test_a_file_the_sample_never_opened_declares_unknown_rows(): void + { + $extractor = from_json(JsonFixtureContext::path('glob_with_empty/*.json')) + ->inferSchema(infer_schema()->filesToSniff(1)); + $extractor->schema(); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + } + + public function test_a_member_without_a_size_makes_the_size_unknown(): void + { + static::assertEquals( + Cardinality::unknown(), + from_json(JsonFixtureContext::path('glob_with_empty/*.json'))->statistics()->size, + ); + } + + public function test_it_declares_the_listed_byte_total_exactly(): void + { + $extractor = from_json( + path('memory://dir/*.json'), + filesystem: MemoryFiles::with([ + 'memory://dir/a.json' => '[{"id":1}]', + 'memory://dir/b.json' => '[{"id":2},{"id":3}]', + ]), + ); + + static::assertEquals(Cardinality::exact(29), $extractor->statistics()->size); + } + + public function test_statistics_before_schema_declares_unknown_rows(): void + { + $extractor = from_json(JsonFixtureContext::path('five_rows.json')); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + + $extractor->schema(); + + static::assertEquals(Cardinality::exact(5), $extractor->statistics()->rows); + } + + public function test_statistics_are_computed_at_most_once(): void + { + $filesystem = new CountingFilesystem(new NativeLocalFilesystem()); + $extractor = from_json(JsonFixtureContext::path('five_rows.json'), filesystem: $filesystem); + $extractor->schema(); + $listCalls = $filesystem->listCalls; + + static::assertSame($extractor->statistics(), $extractor->statistics()); + static::assertSame($listCalls + 1, $filesystem->listCalls); + } + + public function test_the_listing_is_read_once(): void + { + $filesystem = new CountingFilesystem(new NativeLocalFilesystem()); + $extractor = from_json(JsonFixtureContext::path('five_rows.json'), filesystem: $filesystem); + + $extractor->statistics(); + $extractor->statistics(); + + static::assertSame(1, $filesystem->listCalls); + } } diff --git a/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonFileReaderTest.php b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonFileReaderTest.php index 1491a63b58..443fef7467 100644 --- a/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonFileReaderTest.php +++ b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonFileReaderTest.php @@ -4,13 +4,13 @@ namespace Flow\ETL\Adapter\JSON\Tests\Integration\JSONMachine; +use Flow\ETL\Adapter\JSON\JSONMachine\JsonFileSample; use Flow\ETL\Adapter\JSON\JSONMachine\JsonFormat; use Flow\ETL\Adapter\JSON\Tests\Context\JsonFixtureContext; use Flow\ETL\Row\RawRowValues; use Flow\ETL\Tests\Double\CountingFilesystem; use Flow\ETL\Tests\FlowTestCase; use Flow\Filesystem\Local\NativeLocalFilesystem; -use Generator; use JsonMachine\Exception\SyntaxErrorException; use JsonMachine\Items; use JsonMachine\JsonDecoder\ExtJsonDecoder; @@ -268,7 +268,7 @@ public function test_a_glob_reader_samples_every_listed_file_in_listing_order(): static::assertSame(2, $counting->closedStreams()); } - public function test_samples_yields_one_unstarted_generator_per_source_in_listing_order(): void + public function test_samples_yields_one_unopened_sample_per_source_in_listing_order(): void { $counting = new CountingFilesystem(new NativeLocalFilesystem()); $reader = JsonFixtureContext::reader( @@ -280,12 +280,12 @@ public function test_samples_yields_one_unstarted_generator_per_source_in_listin $samples = iterator_to_array($reader->samples(10), false); $beforeAdvance = $counting->readFromCalls; - $first = $samples[0]->current(); + $first = $samples[0]->getIterator()->current(); $afterAdvance = $counting->readFromCalls; static::assertCount(2, $samples); - static::assertContainsOnlyInstancesOf(Generator::class, $samples); - static::assertSame(0, $beforeAdvance, 'samples() opens nothing until an inner generator is advanced'); + static::assertContainsOnlyInstancesOf(JsonFileSample::class, $samples); + static::assertSame(0, $beforeAdvance, 'samples() opens nothing until a sample is iterated'); static::assertSame(1, $afterAdvance, 'advancing the first sample opens exactly the first source'); static::assertNotNull($first); static::assertSame(['id' => 1], $first->values); diff --git a/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonFileSampleTest.php b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonFileSampleTest.php new file mode 100644 index 0000000000..624c03290e --- /dev/null +++ b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonFileSampleTest.php @@ -0,0 +1,90 @@ +sampled()); + } + + public function test_a_document_counts_rows_and_no_bytes(): void + { + $sample = new JsonFileSample(JsonFixtureContext::reader(), JsonFixtureContext::source('five_rows.json')); + + iterator_to_array($sample, false); + + static::assertEquals(new JsonSampledRows(5, 0, true), $sample->sampled()); + } + + public function test_a_last_line_without_a_line_ending_counts_one_byte_more(): void + { + $sample = new JsonFileSample( + JsonFixtureContext::reader(JsonFormat::Lines, MemoryFiles::with(['memory://source.jsonl' => '{"id":1}'])), + new SourceFile(path('memory://source.jsonl')), + ); + + iterator_to_array($sample, false); + + static::assertEquals(new JsonSampledRows(1, 9, true), $sample->sampled()); + } + + public function test_a_sample_stopped_early_records_what_it_read(): void + { + $sample = new JsonFileSample( + JsonFixtureContext::reader(JsonFormat::Lines), + JsonFixtureContext::source('five_rows.jsonl'), + ); + + foreach ($sample as $index => $_row) { + if ($index === 1) { + break; + } + } + + static::assertEquals(new JsonSampledRows(2, 18, false), $sample->sampled()); + } + + public function test_a_whole_file_records_every_row_and_byte(): void + { + $sample = new JsonFileSample( + JsonFixtureContext::reader(JsonFormat::Lines), + JsonFixtureContext::source('five_rows.jsonl'), + ); + + iterator_to_array($sample, false); + + static::assertEquals(new JsonSampledRows(5, 45, true), $sample->sampled()); + } + + public function test_nothing_is_recorded_before_the_sample_is_iterated(): void + { + static::assertNull( + (new JsonFileSample( + JsonFixtureContext::reader(JsonFormat::Lines), + JsonFixtureContext::source('five_rows.jsonl'), + ))->sampled(), + ); + } +} diff --git a/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonLinesExtractorTest.php b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonLinesExtractorTest.php index b3bb2c4d44..dc9a26c557 100644 --- a/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonLinesExtractorTest.php +++ b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonLinesExtractorTest.php @@ -7,12 +7,14 @@ use Closure; use Flow\ETL\Adapter\JSON\JSONMachine\JsonLinesExtractor; use Flow\ETL\Adapter\JSON\Tests\Context\JsonFixtureContext; +use Flow\ETL\Cardinality; use Flow\ETL\Config; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Exception\SchemaMismatchException; use Flow\ETL\Extractor\Signal; use Flow\ETL\Schema\Definition\StringDefinition; use Flow\ETL\Tests\Context\ExtractedRows; +use Flow\ETL\Tests\Context\MemoryFiles; use Flow\ETL\Tests\Double\CountingFilesystem; use Flow\ETL\Tests\FlowTestCase; use Flow\Filesystem\Local\NativeLocalFilesystem; @@ -20,6 +22,7 @@ use PHPUnit\Framework\Attributes\DataProvider; use PHPUnit\Framework\Attributes\TestWith; +use function abs; use function array_keys; use function array_sum; use function Flow\ETL\Adapter\JSON\from_json_lines; @@ -40,6 +43,8 @@ use function Flow\Types\DSL\type_integer; use function iterator_to_array; use function max; +use function sprintf; +use function str_repeat; final class JsonLinesExtractorTest extends FlowTestCase { @@ -598,4 +603,140 @@ public function test_is_repeatable(): void { static::assertTrue(from_json_lines(__DIR__ . '/../../Fixtures/timezones.jsonl')->isRepeatable()); } + + /** + * @param Closure(JsonLinesExtractor): void $setter + */ + #[DataProvider('shapeChangingSetters')] + public function test_a_shape_changing_setter_drops_the_sample(Closure $setter): void + { + $extractor = from_json_lines(JsonFixtureContext::path('five_rows.jsonl')); + $extractor->schema(); + + static::assertEquals(Cardinality::exact(5), $extractor->statistics()->rows); + + $setter($extractor); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + } + + public function test_a_declared_schema_declares_unknown_rows(): void + { + $extractor = from_json_lines(JsonFixtureContext::path('five_rows.jsonl'))->withSchema(schema(int_schema('id'))); + $extractor->schema(); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + static::assertEquals(Cardinality::exact(45), $extractor->statistics()->size); + } + + public function test_a_file_shorter_than_the_sample_declares_an_exact_row_count(): void + { + $extractor = from_json_lines(JsonFixtureContext::path('five_rows.jsonl')); + $extractor->schema(); + + static::assertEquals(Cardinality::exact(5), $extractor->statistics()->rows); + } + + public function test_a_member_without_a_size_makes_the_size_unknown(): void + { + $extractor = from_json_lines(JsonFixtureContext::path('glob_with_empty/*.jsonl')) + ->inferSchema(infer_schema()->sampleSize(1)); + $extractor->schema(); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->size); + } + + public function test_a_narrow_first_row_does_not_skew_the_estimate(): void + { + $lines = "{\"a\":1}\n"; + + for ($i = 0; $i < 99; $i++) { + $lines .= sprintf("{\"a\":\"%012d\"}\n", $i); + } + + $extractor = from_json_lines( + path('memory://source.jsonl'), + MemoryFiles::with([ + 'memory://source.jsonl' => $lines, + ]), + )->inferSchema(infer_schema()->sampleSize(20)); + $extractor->schema(); + + $rows = $extractor->statistics()->rows; + + static::assertEquals(Cardinality::approximately(103, Cardinality::DEFAULT_RELATIVE_ERROR), $rows); + static::assertLessThanOrEqual($rows->relativeError, abs(103 - 100) / 100); + } + + public function test_extract_samples_like_schema_does(): void + { + $extractor = from_json_lines(JsonFixtureContext::path('five_rows.jsonl')); + + iterator_to_array($extractor->extract(flow_context()), false); + + static::assertEquals(Cardinality::exact(5), $extractor->statistics()->rows); + } + + public function test_it_declares_the_listed_byte_total_exactly(): void + { + $extractor = from_json_lines( + path('memory://dir/*.jsonl'), + MemoryFiles::with([ + 'memory://dir/a.jsonl' => "{\"id\":1}\n", + 'memory://dir/b.jsonl' => "{\"id\":2}\n{\"id\":3}\n", + ]), + ); + + static::assertEquals(Cardinality::exact(27), $extractor->statistics()->size); + } + + public function test_it_estimates_rows_from_bytes_and_the_mean_sampled_row(): void + { + $extractor = from_json_lines( + path('memory://source.jsonl'), + MemoryFiles::with([ + 'memory://source.jsonl' => str_repeat("{\"i\":1234}\n", 100), + ]), + )->inferSchema(infer_schema()->sampleSize(10)); + $extractor->schema(); + + static::assertEquals( + Cardinality::approximately(100, Cardinality::DEFAULT_RELATIVE_ERROR), + $extractor->statistics()->rows, + ); + } + + public function test_statistics_are_computed_at_most_once(): void + { + $filesystem = new CountingFilesystem(new NativeLocalFilesystem()); + $extractor = from_json_lines(JsonFixtureContext::path('five_rows.jsonl'), filesystem: $filesystem); + $extractor->schema(); + $listCalls = $filesystem->listCalls; + + static::assertSame($extractor->statistics(), $extractor->statistics()); + static::assertSame($listCalls + 1, $filesystem->listCalls); + } + + public function test_statistics_before_schema_declares_unknown_rows(): void + { + $extractor = from_json_lines(JsonFixtureContext::path('five_rows.jsonl')); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->rows); + + $extractor->schema(); + + static::assertEquals(Cardinality::exact(5), $extractor->statistics()->rows); + } + + public function test_the_listing_is_read_once(): void + { + $filesystem = new CountingFilesystem(new NativeLocalFilesystem()); + $extractor = from_json_lines(JsonFixtureContext::path('five_rows.jsonl'), filesystem: $filesystem); + + $extractor->statistics(); + $extractor->statistics(); + + static::assertSame(1, $filesystem->listCalls); + } } diff --git a/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonSamplesTest.php b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonSamplesTest.php new file mode 100644 index 0000000000..c6d3ac3b80 --- /dev/null +++ b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Integration/JSONMachine/JsonSamplesTest.php @@ -0,0 +1,81 @@ +sampledFiles()); + } + + public function test_samples_past_the_last_pulled_one_are_never_created(): void + { + $created = 0; + $samples = new JsonSamples( + ( + /** @return Generator */ + static function () use (&$created): Generator { + foreach (['five_rows.jsonl', 'five_rows.jsonl'] as $fixture) { + $created++; + + yield new JsonFileSample( + JsonFixtureContext::reader(JsonFormat::Lines), + JsonFixtureContext::source($fixture), + ); + } + } + )(), + ); + + foreach ($samples as $sample) { + iterator_to_array($sample, false); + + break; + } + + static::assertSame(1, $created); + static::assertEquals(new JsonSampledFiles(1, 5, 45, true), $samples->sampledFiles()); + } + + public function test_a_pulled_sample_that_was_never_iterated_is_not_counted(): void + { + $samples = new JsonSamples([ + new JsonFileSample( + JsonFixtureContext::reader(JsonFormat::Lines), + JsonFixtureContext::source('five_rows.jsonl'), + ), + ]); + + iterator_to_array($samples, false); + + static::assertEquals(new JsonSampledFiles(0, 0, 0, true), $samples->sampledFiles()); + } +} diff --git a/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Unit/JSONMachine/JsonReadBytesTest.php b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Unit/JSONMachine/JsonReadBytesTest.php new file mode 100644 index 0000000000..7d3d862790 --- /dev/null +++ b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Unit/JSONMachine/JsonReadBytesTest.php @@ -0,0 +1,25 @@ +total()); + } + + public function test_it_sums_what_was_added(): void + { + $read = new JsonReadBytes(); + $read->add(9); + $read->add(10); + + static::assertSame(19, $read->total()); + } +} diff --git a/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Unit/JSONMachine/JsonSampledFilesTest.php b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Unit/JSONMachine/JsonSampledFilesTest.php new file mode 100644 index 0000000000..5c90dc5398 --- /dev/null +++ b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Unit/JSONMachine/JsonSampledFilesTest.php @@ -0,0 +1,53 @@ +estimatedRows(1, Cardinality::exact(1000)), + ); + } + + public function test_every_listed_file_read_whole_is_exact(): void + { + $sampled = new JsonSampledFiles(2, 10, 100, true); + + static::assertEquals(Cardinality::exact(10), $sampled->estimatedRows(2, Cardinality::exact(100))); + static::assertEquals(Cardinality::exact(10), $sampled->exactRows(2)); + } + + public function test_a_listed_file_the_samples_never_read_is_not_exact(): void + { + $sampled = new JsonSampledFiles(1, 10, 100, true); + + static::assertEquals(Cardinality::approximately(20), $sampled->estimatedRows(2, Cardinality::exact(200))); + static::assertEquals(Cardinality::unknown(), $sampled->exactRows(2)); + } + + public function test_a_partial_document_sample_is_unknown(): void + { + static::assertEquals(Cardinality::unknown(), (new JsonSampledFiles(1, 10, 0, false))->exactRows(1)); + } + + public function test_no_listed_bytes_or_no_sampled_bytes_is_unknown(): void + { + static::assertEquals(Cardinality::unknown(), (new JsonSampledFiles(1, 10, 100, false))->estimatedRows( + 1, + Cardinality::unknown(), + )); + static::assertEquals(Cardinality::unknown(), (new JsonSampledFiles(1, 10, 0, false))->estimatedRows( + 1, + Cardinality::exact(100), + )); + } +} diff --git a/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Unit/JSONMachine/JsonSampledRowsTest.php b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Unit/JSONMachine/JsonSampledRowsTest.php new file mode 100644 index 0000000000..0359a02575 --- /dev/null +++ b/src/adapter/etl-adapter-json/tests/Flow/ETL/Adapter/JSON/Tests/Unit/JSONMachine/JsonSampledRowsTest.php @@ -0,0 +1,34 @@ +expectException(InvalidArgumentException::class); + $this->expectExceptionMessage( + 'Sampled rows and bytes must not be negative, given: ' . $rows . ' rows, ' . $bytes . ' bytes', + ); + + new JsonSampledRows($rows, $bytes, false); + } + + public function test_it_keeps_what_it_was_given(): void + { + $sampled = new JsonSampledRows(3, 27, true); + + static::assertSame(3, $sampled->rows); + static::assertSame(27, $sampled->bytes); + static::assertTrue($sampled->wholeFile); + } +} diff --git a/src/adapter/etl-adapter-parquet/src/Flow/ETL/Adapter/Parquet/ParquetExtractor.php b/src/adapter/etl-adapter-parquet/src/Flow/ETL/Adapter/Parquet/ParquetExtractor.php index a43c8f7fcd..fd53ee644d 100644 --- a/src/adapter/etl-adapter-parquet/src/Flow/ETL/Adapter/Parquet/ParquetExtractor.php +++ b/src/adapter/etl-adapter-parquet/src/Flow/ETL/Adapter/Parquet/ParquetExtractor.php @@ -14,6 +14,7 @@ use Flow\ETL\Extractor\MetadataColumnsExtractor; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -31,6 +32,7 @@ use function array_values; use function count; +use function iterator_count; use function max; use function sprintf; @@ -63,6 +65,8 @@ final class ParquetExtractor implements private SchemaConverter $schemaConverter; + private ?Statistics $statistics = null; + private readonly Filesystem $filesystem; /** @@ -206,6 +210,11 @@ public function schema(): Schema ); } + public function statistics(): Statistics + { + return $this->statistics ??= $this->declare(); + } + /** * Reconcile every listed file's schema instead of trusting the first one. */ @@ -213,6 +222,7 @@ public function unionByName(bool $union = true): self { $this->unionByName = $union; $this->derivedSchema = null; + $this->statistics = null; return $this; } @@ -231,6 +241,7 @@ public function withByteOrder(ByteOrder $byteOrder): self { $this->byteOrder = $byteOrder; $this->derivedSchema = null; + $this->statistics = null; return $this; } @@ -242,6 +253,7 @@ public function withColumns(array $columns): self { $this->columns = $columns; $this->derivedSchema = null; + $this->statistics = null; return $this; } @@ -250,6 +262,7 @@ public function withEngine(?ParquetEngine $engine): self { $this->engine = $engine; $this->derivedSchema = null; + $this->statistics = null; return $this; } @@ -261,6 +274,7 @@ public function withOffset(int $offset): self } $this->offset = $offset; + $this->statistics = null; return $this; } @@ -269,10 +283,24 @@ public function withOptions(Options $options): self { $this->options = $options; $this->derivedSchema = null; + $this->statistics = null; return $this; } + /** + * The footers schema() reads - the first file's, or every file's under unionByName - read once whichever of the + * two asks first, and scaled to the listing when only the first was read. + */ + private function declare(): Statistics + { + $footers = $this->schema === null + ? $this->derivedFooters($this->files(), $this->unionByName) + : $this->footers($this->files(), $this->unionByName); + + return $footers->of(iterator_count($this->sourceFiles($this->filesystem, $this->path)), $this->offset ?? 0); + } + /** * @return Generator */ diff --git a/src/adapter/etl-adapter-parquet/src/Flow/ETL/Adapter/Parquet/ParquetSourceFile.php b/src/adapter/etl-adapter-parquet/src/Flow/ETL/Adapter/Parquet/ParquetSourceFile.php index f518270f70..6ee4a400a4 100644 --- a/src/adapter/etl-adapter-parquet/src/Flow/ETL/Adapter/Parquet/ParquetSourceFile.php +++ b/src/adapter/etl-adapter-parquet/src/Flow/ETL/Adapter/Parquet/ParquetSourceFile.php @@ -4,8 +4,10 @@ namespace Flow\ETL\Adapter\Parquet; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor\SelfDescribingFile; use Flow\ETL\Extractor\SourceFile; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\Schema; use Flow\Filesystem\SourceStream; use Flow\Parquet\ParquetFile; @@ -44,4 +46,19 @@ public function source(): SourceFile { return $this->source; } + + /** + * Uncompressed row-group bytes, as the format reports them. + */ + public function statistics(): Statistics + { + $metadata = $this->file->metadata(); + $bytes = 0; + + foreach ($metadata->rowGroups()->all() as $rowGroup) { + $bytes += $rowGroup->totalByteSize(); + } + + return new Statistics(rows: Cardinality::exact($metadata->rowsNumber()), size: Cardinality::exact($bytes)); + } } diff --git a/src/adapter/etl-adapter-parquet/tests/Flow/ETL/Adapter/Parquet/Tests/Integration/ParquetExtractorTest.php b/src/adapter/etl-adapter-parquet/tests/Flow/ETL/Adapter/Parquet/Tests/Integration/ParquetExtractorTest.php index ef49fcc324..8cb86d2065 100644 --- a/src/adapter/etl-adapter-parquet/tests/Flow/ETL/Adapter/Parquet/Tests/Integration/ParquetExtractorTest.php +++ b/src/adapter/etl-adapter-parquet/tests/Flow/ETL/Adapter/Parquet/Tests/Integration/ParquetExtractorTest.php @@ -6,8 +6,10 @@ use DateTimeImmutable; use Flow\ETL\Adapter\Parquet\Tests\Context\ParquetFilesContext; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InferredSchemaException; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Plan\Stage; use Flow\ETL\Tests\Context\ExtractedRows; use Flow\ETL\Tests\Double\CountingFilesystem; use Flow\ETL\Tests\FlowTestCase; @@ -20,9 +22,12 @@ use function array_keys; use function Flow\ETL\Adapter\Parquet\from_parquet; +use function Flow\ETL\Adapter\Parquet\to_parquet; +use function Flow\ETL\DSL\analyze; use function Flow\ETL\DSL\config; use function Flow\ETL\DSL\df; use function Flow\ETL\DSL\flow_context; +use function Flow\ETL\DSL\from_sequence_number; use function Flow\ETL\DSL\int_schema; use function Flow\ETL\DSL\partition_types; use function Flow\ETL\DSL\row; @@ -233,6 +238,64 @@ public function test_extract_closes_the_reader_when_the_generator_is_abandoned() static::assertSame($filesystem->readFromCalls, $filesystem->closedStreams()); } + public function test_explain_lists_the_statistics_the_file_declares(): void + { + static::assertStringContainsString( + <<<'PLAN' + Extractor: ParquetExtractor + Statistics: rows exact 1 000 · size exact 327 527 B + PLAN, + df() + ->read(from_parquet(path(__DIR__ . '/Fixtures/orders_1k.parquet'))) + ->explain() + ->toString(Stage::physical), + ); + } + + public function test_analyze_reports_the_extrapolated_rows_next_to_the_rows_read(): void + { + $memory = memory_filesystem(); + df() + ->read(from_sequence_number('id', 1, 10_000)) + ->write(to_parquet(path('memory://glob/a.parquet'), filesystem: $memory)) + ->run(); + df() + ->read(from_sequence_number('id', 1, 19_000)) + ->write(to_parquet(path('memory://glob/b.parquet'), filesystem: $memory)) + ->run(); + + $report = df()->read(from_parquet(path('memory://glob/*.parquet'), filesystem: $memory))->run( + analyze()->withSourceStatistics(), + ); + + static::assertNotNull($report); + $sources = $report->sources(); + static::assertNotNull($sources); + static::assertCount(1, $sources); + static::assertSame('ParquetExtractor', $sources[0]->extractor); + static::assertEquals(Cardinality::approximately(20_000), $sources[0]->declared->rows); + static::assertSame(29_000, $sources[0]->rows); + static::assertEqualsWithDelta(9_000 / 29_000, $sources[0]->rowsError(), 0.000_001); + } + + public function test_count_answers_from_the_footer_without_reading_the_rows(): void + { + $filesystem = new CountingFilesystem(new NativeLocalFilesystem()); + + $counted = df() + ->read(from_parquet(path(__DIR__ . '/Fixtures/orders_1k.parquet'), filesystem: $filesystem)) + ->count(); + + static::assertSame( + df() + ->read(from_parquet(path(__DIR__ . '/Fixtures/orders_1k.parquet'))) + ->fetch() + ->count(), + $counted, + ); + static::assertSame(1, $filesystem->readFromCalls); + } + public function test_extract_closes_every_reader_it_opens(): void { $filesystem = new CountingFilesystem(new NativeLocalFilesystem()); diff --git a/src/adapter/etl-adapter-parquet/tests/Flow/ETL/Adapter/Parquet/Tests/Unit/ParquetExtractorTest.php b/src/adapter/etl-adapter-parquet/tests/Flow/ETL/Adapter/Parquet/Tests/Unit/ParquetExtractorTest.php new file mode 100644 index 0000000000..928019bc73 --- /dev/null +++ b/src/adapter/etl-adapter-parquet/tests/Flow/ETL/Adapter/Parquet/Tests/Unit/ParquetExtractorTest.php @@ -0,0 +1,174 @@ +statistics(); + + static::assertEquals( + Cardinality::approximately(5 * 1000, Cardinality::DEFAULT_RELATIVE_ERROR), + $statistics->rows, + ); + static::assertEquals( + Cardinality::approximately(5 * 15934, Cardinality::DEFAULT_RELATIVE_ERROR), + $statistics->size, + ); + } + + public function test_a_single_file_declares_the_footer_row_count(): void + { + static::assertEquals( + Cardinality::exact(1000), + from_parquet(__DIR__ . '/../Integration/Fixtures/Pagination/01_1000.parquet')->statistics()->rows, + ); + } + + public function test_a_single_file_declares_the_uncompressed_byte_size(): void + { + static::assertEquals( + Cardinality::exact(15934), + from_parquet(__DIR__ . '/../Integration/Fixtures/Pagination/01_1000.parquet')->statistics()->size, + ); + } + + public function test_an_offset_is_subtracted_from_the_row_count(): void + { + static::assertEquals( + Cardinality::exact(900), + from_parquet( + __DIR__ . '/../Integration/Fixtures/Pagination/01_1000.parquet', + offset: 100, + )->statistics()->rows, + ); + } + + public function test_an_offset_larger_than_the_file_declares_zero_rows(): void + { + static::assertEquals( + Cardinality::exact(0), + from_parquet( + __DIR__ . '/../Integration/Fixtures/Pagination/01_1000.parquet', + offset: 1_000_000_000, + )->statistics()->rows, + ); + } + + public function test_changing_the_offset_invalidates_the_memo(): void + { + $extractor = from_parquet(__DIR__ . '/../Integration/Fixtures/Pagination/01_1000.parquet'); + + static::assertEquals(Cardinality::exact(1000), $extractor->statistics()->rows); + static::assertEquals(Cardinality::exact(750), $extractor->withOffset(250)->statistics()->rows); + } + + public function test_statistics_are_computed_at_most_once(): void + { + $filesystem = new CountingFilesystem(new NativeLocalFilesystem()); + $extractor = from_parquet( + __DIR__ . '/../Integration/Fixtures/Pagination/01_1000.parquet', + filesystem: $filesystem, + ); + + static::assertSame($extractor->statistics(), $extractor->statistics()); + static::assertSame(1, $filesystem->readFromCalls); + static::assertSame(1, $filesystem->closedStreams()); + } + + public function test_union_by_name_sums_every_footer_exactly(): void + { + $statistics = from_parquet(__DIR__ . '/../Integration/Fixtures/Pagination/*.parquet') + ->unionByName() + ->statistics(); + + static::assertEquals(Cardinality::exact(1000 + 500 + 350 + 2000 + 15), $statistics->rows); + static::assertEquals(Cardinality::exact(15934 + 7932 + 5532 + 32935 + 252), $statistics->size); + } + + /** + * @return Generator + */ + public static function memo_resetting_setters(): Generator + { + yield 'unionByName' => [static fn(ParquetExtractor $e): mixed => $e->unionByName()]; + yield 'withByteOrder' => [static fn(ParquetExtractor $e): mixed => $e->withByteOrder(ByteOrder::LITTLE_ENDIAN)]; + yield 'withColumns' => [static fn(ParquetExtractor $e): mixed => $e->withColumns(['id'])]; + yield 'withEngine' => [static fn(ParquetExtractor $e): mixed => $e->withEngine(null)]; + yield 'withOptions' => [static fn(ParquetExtractor $e): mixed => $e->withOptions(Options::default())]; + } + + #[DataProvider('memo_resetting_setters')] + public function test_a_setter_reads_the_footer_again(Closure $setter): void + { + $filesystem = new CountingFilesystem(new NativeLocalFilesystem()); + $extractor = from_parquet( + __DIR__ . '/../Integration/Fixtures/Pagination/01_1000.parquet', + filesystem: $filesystem, + ); + + $extractor->statistics(); + $setter($extractor); + $extractor->statistics(); + + static::assertSame(2, $filesystem->readFromCalls); + } + + public function test_a_new_offset_reuses_the_footer_already_read(): void + { + $filesystem = new CountingFilesystem(new NativeLocalFilesystem()); + $extractor = from_parquet( + __DIR__ . '/../Integration/Fixtures/Pagination/01_1000.parquet', + filesystem: $filesystem, + ); + + $extractor->statistics(); + $extractor->withOffset(100); + + static::assertEquals(Cardinality::exact(900), $extractor->statistics()->rows); + static::assertSame(1, $filesystem->readFromCalls); + } + + public function test_schema_then_statistics_reads_the_footer_once(): void + { + $filesystem = new CountingFilesystem(new NativeLocalFilesystem()); + $extractor = from_parquet( + __DIR__ . '/../Integration/Fixtures/Pagination/01_1000.parquet', + filesystem: $filesystem, + ); + + $extractor->schema(); + $extractor->statistics(); + + static::assertSame(1, $filesystem->readFromCalls); + } + + public function test_a_declared_schema_still_reads_the_footer_for_statistics(): void + { + static::assertEquals( + Cardinality::exact(1000), + from_parquet(__DIR__ . '/../Integration/Fixtures/Pagination/01_1000.parquet') + ->withSchema(schema(int_schema('id'))) + ->statistics() + ->rows, + ); + } +} diff --git a/src/adapter/etl-adapter-postgresql/src/Flow/ETL/Adapter/PostgreSql/ExplainedRows.php b/src/adapter/etl-adapter-postgresql/src/Flow/ETL/Adapter/PostgreSql/ExplainedRows.php new file mode 100644 index 0000000000..4237c74c1b --- /dev/null +++ b/src/adapter/etl-adapter-postgresql/src/Flow/ETL/Adapter/PostgreSql/ExplainedRows.php @@ -0,0 +1,32 @@ + $parameters + */ + public function of(Client $client, string|Sql $query, array $parameters = [], ?int $maximum = null): Cardinality + { + $estimate = $client->explain($query, $parameters, ExplainConfig::forEstimate())->rootNode()->estimatedRows(); + + return new Cardinality( + atMost: $maximum, + estimate: $maximum === null ? $estimate : min($estimate, $maximum), + relativeError: Cardinality::DEFAULT_RELATIVE_ERROR, + ); + } +} diff --git a/src/adapter/etl-adapter-postgresql/src/Flow/ETL/Adapter/PostgreSql/PostgreSqlCursorExtractor.php b/src/adapter/etl-adapter-postgresql/src/Flow/ETL/Adapter/PostgreSql/PostgreSqlCursorExtractor.php index c8efec2ff5..6575ba692a 100644 --- a/src/adapter/etl-adapter-postgresql/src/Flow/ETL/Adapter/PostgreSql/PostgreSqlCursorExtractor.php +++ b/src/adapter/etl-adapter-postgresql/src/Flow/ETL/Adapter/PostgreSql/PostgreSqlCursorExtractor.php @@ -11,6 +11,7 @@ use Flow\ETL\Extractor\Batches; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -50,6 +51,8 @@ final class PostgreSqlCursorExtractor implements BatchableExtractor, Extractor, private ?Schema $schema = null; + private ?Statistics $statistics = null; + /** * @param list $parameters */ @@ -186,6 +189,16 @@ public function schema(): Schema } } + public function statistics(): Statistics + { + return $this->statistics ??= new Statistics(rows: (new ExplainedRows())->of( + $this->client, + $this->query, + $this->parameters, + $this->maximum, + )); + } + public function withCursorName(string $cursorName): self { $this->cursorName = $cursorName; @@ -200,6 +213,7 @@ public function withMaximum(int $maximum): self } $this->maximum = $maximum; + $this->statistics = null; return $this; } diff --git a/src/adapter/etl-adapter-postgresql/src/Flow/ETL/Adapter/PostgreSql/PostgreSqlKeySetExtractor.php b/src/adapter/etl-adapter-postgresql/src/Flow/ETL/Adapter/PostgreSql/PostgreSqlKeySetExtractor.php index bb15917b57..037cb6b9d2 100644 --- a/src/adapter/etl-adapter-postgresql/src/Flow/ETL/Adapter/PostgreSql/PostgreSqlKeySetExtractor.php +++ b/src/adapter/etl-adapter-postgresql/src/Flow/ETL/Adapter/PostgreSql/PostgreSqlKeySetExtractor.php @@ -14,6 +14,7 @@ use Flow\ETL\Extractor\Batches; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -47,6 +48,8 @@ final class PostgreSqlKeySetExtractor implements BatchableExtractor, Extractor, private ?Schema $schema = null; + private ?Statistics $statistics = null; + /** * @param list $parameters */ @@ -154,6 +157,16 @@ public function schema(): Schema } } + public function statistics(): Statistics + { + return $this->statistics ??= new Statistics(rows: (new ExplainedRows())->of( + $this->client, + $this->query, + $this->parameters, + $this->maximum, + )); + } + public function withMaximum(int $maximum): self { if ($maximum <= 0) { @@ -161,6 +174,7 @@ public function withMaximum(int $maximum): self } $this->maximum = $maximum; + $this->statistics = null; return $this; } diff --git a/src/adapter/etl-adapter-postgresql/src/Flow/ETL/Adapter/PostgreSql/PostgreSqlLimitOffsetExtractor.php b/src/adapter/etl-adapter-postgresql/src/Flow/ETL/Adapter/PostgreSql/PostgreSqlLimitOffsetExtractor.php index b89e26476c..8d3ea01930 100644 --- a/src/adapter/etl-adapter-postgresql/src/Flow/ETL/Adapter/PostgreSql/PostgreSqlLimitOffsetExtractor.php +++ b/src/adapter/etl-adapter-postgresql/src/Flow/ETL/Adapter/PostgreSql/PostgreSqlLimitOffsetExtractor.php @@ -11,6 +11,7 @@ use Flow\ETL\Extractor\Batches; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -36,6 +37,8 @@ final class PostgreSqlLimitOffsetExtractor implements BatchableExtractor, Extrac private ?Schema $schema = null; + private ?Statistics $statistics = null; + /** * @param list $parameters */ @@ -143,6 +146,16 @@ public function schema(): Schema } } + public function statistics(): Statistics + { + return $this->statistics ??= new Statistics(rows: (new ExplainedRows())->of( + $this->client, + $this->query, + $this->parameters, + $this->maximum, + )); + } + public function withMaximum(int $maximum): self { if ($maximum <= 0) { @@ -150,6 +163,7 @@ public function withMaximum(int $maximum): self } $this->maximum = $maximum; + $this->statistics = null; return $this; } diff --git a/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Double/SpyClient.php b/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Double/SpyClient.php index c6cd29919c..e240caa745 100644 --- a/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Double/SpyClient.php +++ b/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Double/SpyClient.php @@ -13,7 +13,10 @@ use Flow\PostgreSql\Client\Notification; use Flow\PostgreSql\Client\RowMapper; use Flow\PostgreSql\Client\Types\ValueConverters; +use Flow\PostgreSql\Explain\Plan\Cost; use Flow\PostgreSql\Explain\Plan\Plan; +use Flow\PostgreSql\Explain\Plan\PlanNode; +use Flow\PostgreSql\Explain\Plan\PlanNodeType; use Flow\PostgreSql\QueryBuilder\Schema\ColumnType; use Flow\PostgreSql\QueryBuilder\Sql; use RuntimeException; @@ -50,6 +53,11 @@ final class SpyClient implements Client private int $scalarIntAnswer = 0; + private int $explainAnswer = 0; + + /** @var list, config: null|ExplainConfig}> */ + public array $explained = []; + /** @var array */ private array $executeFailures = []; @@ -81,6 +89,16 @@ public function willFailRollBack(Throwable $failure): self return $this; } + /** + * explain() answers with a root plan node estimating $rows. + */ + public function willExplain(int $rows): self + { + $this->explainAnswer = $rows; + + return $this; + } + public function willCountTotal(int $total): self { $this->scalarIntAnswer = $total; @@ -198,7 +216,14 @@ public function execute(Sql|string $sql, array|ConvertedParameters $parameters = public function explain(Sql|string $sql, array $parameters = [], ?ExplainConfig $config = null): Plan { - throw new RuntimeException('SpyClient does not implement ' . __FUNCTION__); + $this->calls[] = 'explain'; + $this->explained[] = [ + 'sql' => $sql instanceof Sql ? $sql->toSql() : $sql, + 'parameters' => $parameters, + 'config' => $config, + ]; + + return new Plan(new PlanNode(PlanNodeType::SEQ_SCAN, new Cost(0.0, 10.0), $this->explainAnswer, 8)); } public function fetch(Sql|string $sql, array $parameters = []): ?array diff --git a/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Integration/PostgreSqlCursorExtractorIntegrationTest.php b/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Integration/PostgreSqlCursorExtractorIntegrationTest.php index 96ab6deee7..b6cba927f5 100644 --- a/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Integration/PostgreSqlCursorExtractorIntegrationTest.php +++ b/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Integration/PostgreSqlCursorExtractorIntegrationTest.php @@ -5,6 +5,7 @@ namespace Flow\ETL\Adapter\PostgreSql\Tests\Integration; use Flow\ETL\Adapter\PostgreSql\Tests\IntegrationTestCase; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Exception\SchemaNotDerivableException; use Flow\PostgreSql\Client\Exception\QueryException; @@ -16,6 +17,7 @@ use function Flow\ETL\DSL\from_array; use function Flow\ETL\DSL\int_schema; use function Flow\ETL\DSL\schema; +use function Flow\PostgreSql\DSL\analyze; use function Flow\PostgreSql\DSL\asc; use function Flow\PostgreSql\DSL\binary_expr; use function Flow\PostgreSql\DSL\col; @@ -27,8 +29,10 @@ use function Flow\PostgreSql\DSL\create; use function Flow\PostgreSql\DSL\delete; use function Flow\PostgreSql\DSL\func; +use function Flow\PostgreSql\DSL\gt; use function Flow\PostgreSql\DSL\insert; use function Flow\PostgreSql\DSL\literal; +use function Flow\PostgreSql\DSL\param; use function Flow\PostgreSql\DSL\select; use function Flow\PostgreSql\DSL\star; use function Flow\PostgreSql\DSL\table; @@ -381,4 +385,18 @@ private function insertTestData(int $count): void $this->client->execute($insert); } + + public function test_statistics_are_planned_without_reading_a_row(): void + { + $this->client->execute(analyze()->table($this->tableName)); + + static::assertEquals( + Cardinality::approximately(5), + from_pgsql_cursor( + $this->client, + select(star())->from(table($this->tableName))->where(gt(col('id'), param(1))), + [20], + )->statistics()->rows, + ); + } } diff --git a/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Unit/ExplainedRowsTest.php b/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Unit/ExplainedRowsTest.php new file mode 100644 index 0000000000..0daa9bc922 --- /dev/null +++ b/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Unit/ExplainedRowsTest.php @@ -0,0 +1,41 @@ +of((new SpyClient())->willExplain(7), 'SELECT 1', maximum: 50), + ); + } + + public function test_a_maximum_below_the_estimate_caps_it(): void + { + static::assertEquals( + new Cardinality(atMost: 5, estimate: 5, relativeError: Cardinality::DEFAULT_RELATIVE_ERROR), + (new ExplainedRows())->of((new SpyClient())->willExplain(7), 'SELECT 1', maximum: 5), + ); + } + + public function test_the_plan_is_asked_for_an_estimate_only(): void + { + $client = (new SpyClient())->willExplain(7); + + static::assertEquals(Cardinality::approximately(7), (new ExplainedRows())->of($client, 'SELECT $1', [3])); + static::assertEquals( + [['sql' => 'SELECT $1', 'parameters' => [3], 'config' => ExplainConfig::forEstimate()]], + $client->explained, + ); + } +} diff --git a/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Unit/PostgreSqlCursorExtractorTest.php b/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Unit/PostgreSqlCursorExtractorTest.php index d3ecbeba17..90d4cc7ec9 100644 --- a/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Unit/PostgreSqlCursorExtractorTest.php +++ b/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Unit/PostgreSqlCursorExtractorTest.php @@ -7,10 +7,12 @@ use Flow\ETL\Adapter\PostgreSql\Tests\Double\SpyClient; use Flow\ETL\Adapter\PostgreSql\Tests\Double\StubCursor; use Flow\ETL\Adapter\PostgreSql\Tests\Mother\ColumnMother; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Exception\SchemaMismatchException; use Flow\ETL\Exception\SchemaNotDerivableException; use Flow\ETL\Tests\FlowTestCase; +use Flow\PostgreSql\AST\Transformers\ExplainConfig; use Flow\PostgreSql\Client\Exception\PostgreSqlError; use Flow\PostgreSql\Client\Exception\QueryException; use RuntimeException; @@ -411,4 +413,45 @@ public function test_is_repeatable(): void { static::assertTrue(from_pgsql_cursor(new SpyClient(), 'SELECT id FROM t')->isRepeatable()); } + + public function test_statistics_are_the_planner_estimate_without_analyze(): void + { + $client = (new SpyClient())->willExplain(42); + + static::assertEquals( + Cardinality::approximately(42), + from_pgsql_cursor($client, 'SELECT * FROM t WHERE id > $1', [1])->statistics()->rows, + ); + static::assertEquals( + [['sql' => 'SELECT * FROM t WHERE id > $1', 'parameters' => [1], 'config' => ExplainConfig::forEstimate()]], + $client->explained, + ); + } + + public function test_statistics_are_explained_once(): void + { + $client = (new SpyClient())->willExplain(42); + $extractor = from_pgsql_cursor($client, 'SELECT * FROM t WHERE id > $1', [1]); + + static::assertSame($extractor->statistics(), $extractor->statistics()); + static::assertSame(1, $client->callsTo('explain')); + } + + public function test_a_maximum_bounds_and_caps_the_estimate_and_a_new_one_drops_the_statistics(): void + { + $client = (new SpyClient())->willExplain(42); + $extractor = from_pgsql_cursor($client, 'SELECT * FROM t WHERE id > $1', [1])->withMaximum(10); + + static::assertEquals( + new Cardinality(atMost: 10, estimate: 10, relativeError: Cardinality::DEFAULT_RELATIVE_ERROR), + $extractor->statistics()->rows, + ); + + $extractor->withMaximum(100); + + static::assertEquals( + new Cardinality(atMost: 100, estimate: 42, relativeError: Cardinality::DEFAULT_RELATIVE_ERROR), + $extractor->statistics()->rows, + ); + } } diff --git a/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Unit/PostgreSqlKeySetExtractorTest.php b/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Unit/PostgreSqlKeySetExtractorTest.php index 05b1d7e9f2..9876dd9593 100644 --- a/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Unit/PostgreSqlKeySetExtractorTest.php +++ b/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Unit/PostgreSqlKeySetExtractorTest.php @@ -7,9 +7,11 @@ use Flow\ETL\Adapter\PostgreSql\Tests\Double\SpyClient; use Flow\ETL\Adapter\PostgreSql\Tests\Double\StubCursor; use Flow\ETL\Adapter\PostgreSql\Tests\Mother\ColumnMother; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Exception\SchemaNotDerivableException; use Flow\ETL\Tests\FlowTestCase; +use Flow\PostgreSql\AST\Transformers\ExplainConfig; use Flow\PostgreSql\Client\Exception\PostgreSqlError; use Flow\PostgreSql\Client\Exception\QueryException; @@ -337,4 +339,60 @@ public function test_is_repeatable(): void )->isRepeatable(), ); } + + public function test_statistics_are_the_planner_estimate_without_analyze(): void + { + $client = (new SpyClient())->willExplain(42); + + static::assertEquals( + Cardinality::approximately(42), + from_pgsql_key_set( + $client, + 'SELECT * FROM t WHERE id > $1', + pgsql_pagination_key_set(pgsql_pagination_key_asc('id')), + [1], + )->statistics()->rows, + ); + static::assertEquals( + [['sql' => 'SELECT * FROM t WHERE id > $1', 'parameters' => [1], 'config' => ExplainConfig::forEstimate()]], + $client->explained, + ); + } + + public function test_statistics_are_explained_once(): void + { + $client = (new SpyClient())->willExplain(42); + $extractor = from_pgsql_key_set( + $client, + 'SELECT * FROM t WHERE id > $1', + pgsql_pagination_key_set(pgsql_pagination_key_asc('id')), + [1], + ); + + static::assertSame($extractor->statistics(), $extractor->statistics()); + static::assertSame(1, $client->callsTo('explain')); + } + + public function test_a_maximum_bounds_and_caps_the_estimate_and_a_new_one_drops_the_statistics(): void + { + $client = (new SpyClient())->willExplain(42); + $extractor = from_pgsql_key_set( + $client, + 'SELECT * FROM t WHERE id > $1', + pgsql_pagination_key_set(pgsql_pagination_key_asc('id')), + [1], + )->withMaximum(10); + + static::assertEquals( + new Cardinality(atMost: 10, estimate: 10, relativeError: Cardinality::DEFAULT_RELATIVE_ERROR), + $extractor->statistics()->rows, + ); + + $extractor->withMaximum(100); + + static::assertEquals( + new Cardinality(atMost: 100, estimate: 42, relativeError: Cardinality::DEFAULT_RELATIVE_ERROR), + $extractor->statistics()->rows, + ); + } } diff --git a/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Unit/PostgreSqlLimitOffsetExtractorTest.php b/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Unit/PostgreSqlLimitOffsetExtractorTest.php index e68e4ae5f1..7d1909e5ae 100644 --- a/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Unit/PostgreSqlLimitOffsetExtractorTest.php +++ b/src/adapter/etl-adapter-postgresql/tests/Flow/ETL/Adapter/PostgreSql/Tests/Unit/PostgreSqlLimitOffsetExtractorTest.php @@ -7,10 +7,12 @@ use Flow\ETL\Adapter\PostgreSql\Tests\Double\SpyClient; use Flow\ETL\Adapter\PostgreSql\Tests\Double\StubCursor; use Flow\ETL\Adapter\PostgreSql\Tests\Mother\ColumnMother; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Exception\SchemaNotDerivableException; use Flow\ETL\Rows; use Flow\ETL\Tests\FlowTestCase; +use Flow\PostgreSql\AST\Transformers\ExplainConfig; use Flow\PostgreSql\Client\Exception\PostgreSqlError; use Flow\PostgreSql\Client\Exception\QueryException; @@ -322,4 +324,45 @@ public function test_is_repeatable(): void { static::assertTrue(from_pgsql_limit_offset(new SpyClient(), 'SELECT id FROM t ORDER BY id')->isRepeatable()); } + + public function test_statistics_are_the_planner_estimate_without_analyze(): void + { + $client = (new SpyClient())->willExplain(42); + + static::assertEquals( + Cardinality::approximately(42), + from_pgsql_limit_offset($client, 'SELECT * FROM t WHERE id > $1', [1])->statistics()->rows, + ); + static::assertEquals( + [['sql' => 'SELECT * FROM t WHERE id > $1', 'parameters' => [1], 'config' => ExplainConfig::forEstimate()]], + $client->explained, + ); + } + + public function test_statistics_are_explained_once(): void + { + $client = (new SpyClient())->willExplain(42); + $extractor = from_pgsql_limit_offset($client, 'SELECT * FROM t WHERE id > $1', [1]); + + static::assertSame($extractor->statistics(), $extractor->statistics()); + static::assertSame(1, $client->callsTo('explain')); + } + + public function test_a_maximum_bounds_and_caps_the_estimate_and_a_new_one_drops_the_statistics(): void + { + $client = (new SpyClient())->willExplain(42); + $extractor = from_pgsql_limit_offset($client, 'SELECT * FROM t WHERE id > $1', [1])->withMaximum(10); + + static::assertEquals( + new Cardinality(atMost: 10, estimate: 10, relativeError: Cardinality::DEFAULT_RELATIVE_ERROR), + $extractor->statistics()->rows, + ); + + $extractor->withMaximum(100); + + static::assertEquals( + new Cardinality(atMost: 100, estimate: 42, relativeError: Cardinality::DEFAULT_RELATIVE_ERROR), + $extractor->statistics()->rows, + ); + } } diff --git a/src/adapter/etl-adapter-text/src/Flow/ETL/Adapter/Text/TextExtractor.php b/src/adapter/etl-adapter-text/src/Flow/ETL/Adapter/Text/TextExtractor.php index abd8604eec..dbe45a2bff 100644 --- a/src/adapter/etl-adapter-text/src/Flow/ETL/Adapter/Text/TextExtractor.php +++ b/src/adapter/etl-adapter-text/src/Flow/ETL/Adapter/Text/TextExtractor.php @@ -10,9 +10,11 @@ use Flow\ETL\Extractor\Batches; use Flow\ETL\Extractor\FileExtractor; use Flow\ETL\Extractor\FileReading; +use Flow\ETL\Extractor\ListedFiles; use Flow\ETL\Extractor\MetadataColumnsExtractor; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Row\RawRowValues; use Flow\ETL\Rows; @@ -43,6 +45,8 @@ final class TextExtractor implements private readonly Filesystem $filesystem; + private ?ListedFiles $listed = null; + public function __construct( private readonly Path $path, Filesystem $filesystem = new NativeLocalFilesystem(), @@ -158,6 +162,13 @@ public function source(): Path return $this->path; } + public function statistics(): Statistics + { + $this->listed ??= ListedFiles::of($this->sourceFiles($this->filesystem, $this->path)); + + return new Statistics(size: $this->listed->bytes); + } + public function withSchema(Schema $schema): static { $this->schema = $schema; diff --git a/src/adapter/etl-adapter-text/tests/Flow/ETL/Adapter/Text/Tests/Integration/TextExtractorTest.php b/src/adapter/etl-adapter-text/tests/Flow/ETL/Adapter/Text/Tests/Integration/TextExtractorTest.php index 517321fd33..f350ac449f 100644 --- a/src/adapter/etl-adapter-text/tests/Flow/ETL/Adapter/Text/Tests/Integration/TextExtractorTest.php +++ b/src/adapter/etl-adapter-text/tests/Flow/ETL/Adapter/Text/Tests/Integration/TextExtractorTest.php @@ -5,9 +5,13 @@ namespace Flow\ETL\Adapter\Text\Tests\Integration; use Flow\ETL\Adapter\Text\TextExtractor; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor\Signal; use Flow\ETL\Tests\Context\ExtractedRows; +use Flow\ETL\Tests\Context\MemoryFiles; +use Flow\ETL\Tests\Double\CountingFilesystem; use Flow\ETL\Tests\Double\RecordingFilesystem; +use Flow\ETL\Tests\Double\UnsizedFilesystem; use Flow\ETL\Tests\FlowTestCase; use function Flow\ETL\Adapter\Text\from_text; @@ -17,6 +21,7 @@ use function Flow\ETL\DSL\schema; use function Flow\ETL\DSL\str_schema; use function Flow\Filesystem\DSL\native_local_filesystem; +use function Flow\Filesystem\DSL\path; use function Flow\Filesystem\DSL\path_real; use function Flow\Types\DSL\type_string; use function iterator_to_array; @@ -143,4 +148,42 @@ public function test_a_stopped_read_closes_its_stream(): void static::assertContains('closeSource', $filesystem->calls); } + + public function test_it_declares_the_listed_byte_total_exactly(): void + { + $statistics = from_text( + path('memory://dir/*.txt'), + MemoryFiles::with(['memory://dir/a.txt' => 'abcdefghij', 'memory://dir/b.txt' => 'abcde']), + )->statistics(); + + static::assertEquals(Cardinality::exact(15), $statistics->size); + static::assertEquals(Cardinality::unknown(), $statistics->rows); + } + + public function test_a_member_without_a_size_makes_the_size_unknown(): void + { + $statistics = from_text( + path('memory://dir/*.txt'), + new UnsizedFilesystem(MemoryFiles::with([ + 'memory://dir/a.txt' => 'abcdefghij', + 'memory://dir/b.txt' => 'abcde', + ]), ['memory://dir/b.txt']), + )->statistics(); + + static::assertEquals(Cardinality::unknown(), $statistics->size); + } + + public function test_the_listing_is_read_once(): void + { + $filesystem = new CountingFilesystem(MemoryFiles::with([ + 'memory://dir/a.txt' => 'abcdefghij', + 'memory://dir/b.txt' => 'abcde', + ])); + $extractor = from_text(path('memory://dir/*.txt'), $filesystem); + + $extractor->statistics(); + $extractor->statistics(); + + static::assertSame(1, $filesystem->listCalls); + } } diff --git a/src/adapter/etl-adapter-xml/src/Flow/ETL/Adapter/XML/XMLParserExtractor.php b/src/adapter/etl-adapter-xml/src/Flow/ETL/Adapter/XML/XMLParserExtractor.php index 952b94f36b..36f7a8ce3f 100644 --- a/src/adapter/etl-adapter-xml/src/Flow/ETL/Adapter/XML/XMLParserExtractor.php +++ b/src/adapter/etl-adapter-xml/src/Flow/ETL/Adapter/XML/XMLParserExtractor.php @@ -10,9 +10,11 @@ use Flow\ETL\Extractor\Batches; use Flow\ETL\Extractor\FileExtractor; use Flow\ETL\Extractor\FileReading; +use Flow\ETL\Extractor\ListedFiles; use Flow\ETL\Extractor\MetadataColumnsExtractor; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Row\RawRowValues; use Flow\ETL\Rows; @@ -50,6 +52,8 @@ final class XMLParserExtractor implements private readonly Filesystem $filesystem; + private ?ListedFiles $listed = null; + /** * To iterate only over nodes, use `$loader->withXMLNodePath('root/elements/element')`. * @@ -168,6 +172,13 @@ public function source(): Path return $this->path; } + public function statistics(): Statistics + { + $this->listed ??= ListedFiles::of($this->sourceFiles($this->filesystem, $this->path)); + + return new Statistics(size: $this->listed->bytes); + } + /** * @param int<1, max> $bufferSize $bufferSize - size of the chunks to read from the xml file. Bigger chunks means faster reading but more memory usage. */ diff --git a/src/adapter/etl-adapter-xml/src/Flow/ETL/Adapter/XML/XMLReaderExtractor.php b/src/adapter/etl-adapter-xml/src/Flow/ETL/Adapter/XML/XMLReaderExtractor.php index 45b0e0bc74..f41e7b0ef1 100644 --- a/src/adapter/etl-adapter-xml/src/Flow/ETL/Adapter/XML/XMLReaderExtractor.php +++ b/src/adapter/etl-adapter-xml/src/Flow/ETL/Adapter/XML/XMLReaderExtractor.php @@ -11,9 +11,11 @@ use Flow\ETL\Extractor\Batches; use Flow\ETL\Extractor\FileExtractor; use Flow\ETL\Extractor\FileReading; +use Flow\ETL\Extractor\ListedFiles; use Flow\ETL\Extractor\MetadataColumnsExtractor; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Row\RawRowValues; use Flow\ETL\Rows; @@ -50,6 +52,8 @@ final class XMLReaderExtractor implements private readonly Filesystem $filesystem; + private ?ListedFiles $listed = null; + /** * In order to iterate only over nodes us root/elements/element. * @@ -218,6 +222,13 @@ public function source(): Path return $this->path; } + public function statistics(): Statistics + { + $this->listed ??= ListedFiles::of($this->sourceFiles($this->filesystem, $this->path)); + + return new Statistics(size: $this->listed->bytes); + } + public function withSchema(Schema $schema): static { $this->schema = $schema; diff --git a/src/adapter/etl-adapter-xml/tests/Flow/ETL/Adapter/XML/Tests/Fixtures/listed/a.xml b/src/adapter/etl-adapter-xml/tests/Flow/ETL/Adapter/XML/Tests/Fixtures/listed/a.xml new file mode 100644 index 0000000000..f3f286eafc --- /dev/null +++ b/src/adapter/etl-adapter-xml/tests/Flow/ETL/Adapter/XML/Tests/Fixtures/listed/a.xml @@ -0,0 +1 @@ + diff --git a/src/adapter/etl-adapter-xml/tests/Flow/ETL/Adapter/XML/Tests/Fixtures/listed/b.xml b/src/adapter/etl-adapter-xml/tests/Flow/ETL/Adapter/XML/Tests/Fixtures/listed/b.xml new file mode 100644 index 0000000000..80eac2ed95 --- /dev/null +++ b/src/adapter/etl-adapter-xml/tests/Flow/ETL/Adapter/XML/Tests/Fixtures/listed/b.xml @@ -0,0 +1 @@ + diff --git a/src/adapter/etl-adapter-xml/tests/Flow/ETL/Adapter/XML/Tests/Integration/XMLParserExtractorTest.php b/src/adapter/etl-adapter-xml/tests/Flow/ETL/Adapter/XML/Tests/Integration/XMLParserExtractorTest.php index 2cf17beecb..262d9ad7c7 100644 --- a/src/adapter/etl-adapter-xml/tests/Flow/ETL/Adapter/XML/Tests/Integration/XMLParserExtractorTest.php +++ b/src/adapter/etl-adapter-xml/tests/Flow/ETL/Adapter/XML/Tests/Integration/XMLParserExtractorTest.php @@ -5,12 +5,16 @@ namespace Flow\ETL\Adapter\XML\Tests\Integration; use Flow\ETL\Adapter\XML\XMLParserExtractor; +use Flow\ETL\Cardinality; use Flow\ETL\Config; use Flow\ETL\Exception\RuntimeException; use Flow\ETL\Exception\SchemaMismatchException; use Flow\ETL\Extractor\Signal; use Flow\ETL\Tests\Context\ExtractedRows; +use Flow\ETL\Tests\Context\MemoryFiles; +use Flow\ETL\Tests\Double\CountingFilesystem; use Flow\ETL\Tests\Double\RecordingFilesystem; +use Flow\ETL\Tests\Double\UnsizedFilesystem; use Flow\ETL\Tests\FlowIntegrationTestCase; use function array_keys; @@ -400,4 +404,42 @@ public function test_a_stopped_read_closes_its_stream(): void static::assertContains('closeSource', $filesystem->calls); } + + public function test_it_declares_the_listed_byte_total_exactly(): void + { + $statistics = from_xml( + path('memory://dir/*.xml'), + filesystem: MemoryFiles::with(['memory://dir/a.xml' => 'abcdefghij', 'memory://dir/b.xml' => 'abcde']), + )->statistics(); + + static::assertEquals(Cardinality::exact(15), $statistics->size); + static::assertEquals(Cardinality::unknown(), $statistics->rows); + } + + public function test_a_member_without_a_size_makes_the_size_unknown(): void + { + $statistics = from_xml( + path('memory://dir/*.xml'), + filesystem: new UnsizedFilesystem(MemoryFiles::with([ + 'memory://dir/a.xml' => 'abcdefghij', + 'memory://dir/b.xml' => 'abcde', + ]), ['memory://dir/b.xml']), + )->statistics(); + + static::assertEquals(Cardinality::unknown(), $statistics->size); + } + + public function test_the_listing_is_read_once(): void + { + $filesystem = new CountingFilesystem(MemoryFiles::with([ + 'memory://dir/a.xml' => 'abcdefghij', + 'memory://dir/b.xml' => 'abcde', + ])); + $extractor = from_xml(path('memory://dir/*.xml'), filesystem: $filesystem); + + $extractor->statistics(); + $extractor->statistics(); + + static::assertSame(1, $filesystem->listCalls); + } } diff --git a/src/adapter/etl-adapter-xml/tests/Flow/ETL/Adapter/XML/Tests/Integration/XMLReaderExtractorTest.php b/src/adapter/etl-adapter-xml/tests/Flow/ETL/Adapter/XML/Tests/Integration/XMLReaderExtractorTest.php index e8f03eaea1..ee4304e0ee 100644 --- a/src/adapter/etl-adapter-xml/tests/Flow/ETL/Adapter/XML/Tests/Integration/XMLReaderExtractorTest.php +++ b/src/adapter/etl-adapter-xml/tests/Flow/ETL/Adapter/XML/Tests/Integration/XMLReaderExtractorTest.php @@ -6,8 +6,11 @@ use DOMDocument; use Flow\ETL\Adapter\XML\XMLReaderExtractor; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor\Signal; use Flow\ETL\Tests\Context\ExtractedRows; +use Flow\ETL\Tests\Double\CountingFilesystem; +use Flow\ETL\Tests\Double\UnsizedFilesystem; use Flow\ETL\Tests\FlowIntegrationTestCase; use function array_keys; @@ -17,6 +20,7 @@ use function Flow\ETL\DSL\schema; use function Flow\ETL\DSL\str_schema; use function Flow\ETL\DSL\xml_schema; +use function Flow\Filesystem\DSL\native_local_filesystem; use function Flow\Filesystem\DSL\path; use function Flow\Filesystem\DSL\path_real; use function Flow\Types\DSL\type_string; @@ -193,4 +197,37 @@ public function test_is_repeatable(): void (new XMLReaderExtractor(path_real(__DIR__ . '/../Fixtures/flow_orders.xml'), 'root/row'))->isRepeatable(), ); } + + public function test_it_declares_the_listed_byte_total_exactly(): void + { + // @mago-ignore analysis:deprecated-class + $statistics = (new XMLReaderExtractor(path_real(__DIR__ . '/../Fixtures/listed/*.xml')))->statistics(); + + static::assertEquals(Cardinality::exact(13), $statistics->size); + static::assertEquals(Cardinality::unknown(), $statistics->rows); + } + + public function test_a_member_without_a_size_makes_the_size_unknown(): void + { + $filesystem = new UnsizedFilesystem(native_local_filesystem(), [ + path_real(__DIR__ . '/../Fixtures/listed/b.xml')->uri(), + ]); + + // @mago-ignore analysis:deprecated-class + $extractor = new XMLReaderExtractor(path_real(__DIR__ . '/../Fixtures/listed/*.xml'), '', $filesystem); + + static::assertEquals(Cardinality::unknown(), $extractor->statistics()->size); + } + + public function test_the_listing_is_read_once(): void + { + $filesystem = new CountingFilesystem(native_local_filesystem()); + // @mago-ignore analysis:deprecated-class + $extractor = new XMLReaderExtractor(path_real(__DIR__ . '/../Fixtures/listed/*.xml'), '', $filesystem); + + $extractor->statistics(); + $extractor->statistics(); + + static::assertSame(1, $filesystem->listCalls); + } } diff --git a/src/cli/src/Flow/CLI/Command/FileAnalyzeCommand.php b/src/cli/src/Flow/CLI/Command/FileAnalyzeCommand.php index dbdc07576d..bb5bbb48ff 100644 --- a/src/cli/src/Flow/CLI/Command/FileAnalyzeCommand.php +++ b/src/cli/src/Flow/CLI/Command/FileAnalyzeCommand.php @@ -157,6 +157,10 @@ protected function execute(InputInterface $input, OutputInterface $output): int $analyze->withSchema()->withColumnStatistics(); } + if (option_bool('stats-sources', $input)) { + $analyze->withSourceStatistics(); + } + $report = $df->write(new ProgressBarLoader($progress))->run(analyze: $analyze); $progress->finish(); diff --git a/src/cli/src/Flow/CLI/Command/PipelineRunCommand.php b/src/cli/src/Flow/CLI/Command/PipelineRunCommand.php index 1d8fda5dfc..9bff0a522e 100644 --- a/src/cli/src/Flow/CLI/Command/PipelineRunCommand.php +++ b/src/cli/src/Flow/CLI/Command/PipelineRunCommand.php @@ -82,6 +82,10 @@ public function execute(InputInterface $input, OutputInterface $output): int $analyze->withColumnStatistics(); } + if ($analyze && option_bool('stats-sources', $input)) { + $analyze->withSourceStatistics(); + } + try { ob_start(); $extension = $this->pipelinePath->extension(); diff --git a/src/cli/src/Flow/CLI/Command/Traits/StatisticsOptions.php b/src/cli/src/Flow/CLI/Command/Traits/StatisticsOptions.php index 814375acb2..5d15887f5d 100644 --- a/src/cli/src/Flow/CLI/Command/Traits/StatisticsOptions.php +++ b/src/cli/src/Flow/CLI/Command/Traits/StatisticsOptions.php @@ -11,12 +11,21 @@ trait StatisticsOptions { private function addStatisticsOptions(Command $command): void { - $command->addOption( - 'stats-schema', - null, - InputOption::VALUE_OPTIONAL, - 'Prints schema of executed data transformation pipeline.', - false, - )->addOption('stats-columns', null, InputOption::VALUE_OPTIONAL, 'Prints number of rows in dataset.', false); + $command + ->addOption( + 'stats-schema', + null, + InputOption::VALUE_OPTIONAL, + 'Prints schema of executed data transformation pipeline.', + false, + ) + ->addOption('stats-columns', null, InputOption::VALUE_OPTIONAL, 'Prints number of rows in dataset.', false) + ->addOption( + 'stats-sources', + null, + InputOption::VALUE_OPTIONAL, + 'Prints the rows every source declared next to the rows it yielded.', + false, + ); } } diff --git a/src/cli/src/Flow/CLI/Formatter/PipelineReportFormatter.php b/src/cli/src/Flow/CLI/Formatter/PipelineReportFormatter.php index ecf78dc1be..3dd059b031 100644 --- a/src/cli/src/Flow/CLI/Formatter/PipelineReportFormatter.php +++ b/src/cli/src/Flow/CLI/Formatter/PipelineReportFormatter.php @@ -5,6 +5,7 @@ namespace Flow\CLI\Formatter; use Flow\ETL\Dataset\Report; +use Flow\ETL\Plan\Explain\StatisticsLine; use Symfony\Component\Console\Helper\TableSeparator; use Symfony\Component\Console\Input\InputInterface; use Symfony\Component\Console\Style\SymfonyStyle; @@ -91,6 +92,32 @@ public function format(): void ->render(); } + $sources = $this->report->sources(); + + if (option_bool('stats-sources', $this->input) && $sources !== null) { + $statisticsLine = new StatisticsLine(); + $normalizedSources = []; + + foreach ($sources as $source) { + $normalizedSources[] = [ + 'extractor' => $source->extractor, + 'declared' => $statisticsLine->cardinality($source->declared->rows), + 'measured' => $statisticsLine->number($source->rows), + 'error' => $statisticsLine->percent($source->rowsError()), + ]; + } + + $this->style->newLine(); + $this->style->section('Sources'); + + $this->style + ->createTable() + ->setHeaders(['Extractor', 'Declared Rows', 'Measured Rows', 'Rows Error']) + ->setRows($normalizedSources) + ->setStyle('box') + ->render(); + } + $this->style->newLine(); $this->style->definitionList( 'Statistics', diff --git a/src/cli/tests/Flow/CLI/Tests/Integration/FileAnalyzeCommandTest.php b/src/cli/tests/Flow/CLI/Tests/Integration/FileAnalyzeCommandTest.php index e7832e3a64..e228ee133a 100644 --- a/src/cli/tests/Flow/CLI/Tests/Integration/FileAnalyzeCommandTest.php +++ b/src/cli/tests/Flow/CLI/Tests/Integration/FileAnalyzeCommandTest.php @@ -137,6 +137,26 @@ public function test_read_rows_csv_without_schema(): void self::assertCommandOutputContains('Execution Time', $tester->getDisplay()); } + public function test_read_rows_csv_with_source_statistics(): void + { + $tester = new CommandTester(new FileAnalyzeCommand('file:analyze')); + + $tester->execute([ + 'input-file' => __DIR__ . '/Fixtures/orders.csv', + '--stats-sources' => true, + ]); + + $tester->assertCommandIsSuccessful(); + + self::assertCommandOutputContains(<<<'OUTPUT' + ┌──────────────┬───────────────┬───────────────┬────────────┐ + │ Extractor │ Declared Rows │ Measured Rows │ Rows Error │ + ├──────────────┼───────────────┼───────────────┼────────────┤ + │ CSVExtractor │ exact 43 │ 43 │ 0% │ + └──────────────┴───────────────┴───────────────┴────────────┘ + OUTPUT, $tester->getDisplay()); + } + public function test_file_analyze_command_registers_the_analyze_alias(): void { $application = new Application(); diff --git a/src/cli/tests/Flow/CLI/Tests/Integration/PipelineRunCommandTest.php b/src/cli/tests/Flow/CLI/Tests/Integration/PipelineRunCommandTest.php index 0ade486cfe..51e8433970 100644 --- a/src/cli/tests/Flow/CLI/Tests/Integration/PipelineRunCommandTest.php +++ b/src/cli/tests/Flow/CLI/Tests/Integration/PipelineRunCommandTest.php @@ -64,6 +64,30 @@ public function test_run_and_analyze_command(): void OUTPUT, $tester->getDisplay()); } + public function test_run_and_analyze_command_lists_declared_and_measured_source_rows(): void + { + $tester = new CommandTester(new PipelineRunCommand('run')); + + $tester->execute([ + 'pipeline-file' => __DIR__ . '/Fixtures/pipeline.php', + '--analyze' => true, + '--stats-sources' => true, + ]); + + $tester->assertCommandIsSuccessful(); + + self::assertCommandOutputContains(<<<'OUTPUT' + Sources + ------- + + ┌────────────────┬───────────────┬───────────────┬────────────┐ + │ Extractor │ Declared Rows │ Measured Rows │ Rows Error │ + ├────────────────┼───────────────┼───────────────┼────────────┤ + │ ArrayExtractor │ exact 3 │ 3 │ 0% │ + └────────────────┴───────────────┴───────────────┴────────────┘ + OUTPUT, $tester->getDisplay()); + } + public function test_run_command(): void { $tester = new CommandTester(new PipelineRunCommand('run')); diff --git a/src/core/etl/src/Flow/ETL/Analyze.php b/src/core/etl/src/Flow/ETL/Analyze.php index 229f7a80de..2c16108003 100644 --- a/src/core/etl/src/Flow/ETL/Analyze.php +++ b/src/core/etl/src/Flow/ETL/Analyze.php @@ -10,6 +10,8 @@ final class Analyze private bool $collectSchema = false; + private bool $collectSourceStatistics = false; + public function __construct() {} public function collectColumnStatistics(): bool @@ -22,6 +24,11 @@ public function collectSchema(): bool return $this->collectSchema; } + public function collectSourceStatistics(): bool + { + return $this->collectSourceStatistics; + } + public function withColumnStatistics(): self { $this->collectColumnStatistics = true; @@ -35,4 +42,11 @@ public function withSchema(): self return $this; } + + public function withSourceStatistics(): self + { + $this->collectSourceStatistics = true; + + return $this; + } } diff --git a/src/core/etl/src/Flow/ETL/Cache/CacheIndex.php b/src/core/etl/src/Flow/ETL/Cache/CacheIndex.php index 7cb8ccbe82..81c5aa46f2 100644 --- a/src/core/etl/src/Flow/ETL/Cache/CacheIndex.php +++ b/src/core/etl/src/Flow/ETL/Cache/CacheIndex.php @@ -4,25 +4,28 @@ namespace Flow\ETL\Cache; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Row; use Flow\ETL\Rows; use function array_map; +use function Flow\ETL\DSL\int_schema; use function Flow\ETL\DSL\row; use function Flow\ETL\DSL\rows; use function Flow\ETL\DSL\schema; use function Flow\ETL\DSL\str_schema; use function get_debug_type; +use function is_int; use function is_string; use function sprintf; final class CacheIndex { /** - * @var array + * @var list */ - private array $index = []; + private array $chunks = []; public function __construct( public readonly string $key, @@ -45,30 +48,55 @@ public static function fromRows(string $key, Rows $rows): self )); } - $index->add($value); + // an index written by 0.44.x has no "rows" column + $chunkRows = $row->has('rows') ? $row->get('rows') : null; + + if ($chunkRows !== null && !is_int($chunkRows)) { + throw new InvalidArgumentException(sprintf( + 'CacheIndex expects rows with an integer "rows" entry, got: %s', + get_debug_type($chunkRows), + )); + } + + $index->add($value, $chunkRows); } return $index; } - public function add(string $value): void + public function add(string $value, ?int $rows = null): void { - $this->index[] = $value; + $this->chunks[] = ['key' => $value, 'rows' => $rows]; + } + + public function rows(): Cardinality + { + $total = 0; + + foreach ($this->chunks as $chunk) { + if ($chunk['rows'] === null) { + return Cardinality::unknown(); + } + + $total += $chunk['rows']; + } + + return Cardinality::exact($total); } public function toRows(): Rows { return rows( - schema(str_schema('key')), - ...array_map(static fn(string $value): Row => row(['key' => $value]), $this->index), + schema(str_schema('key'), int_schema('rows', nullable: true)), + ...array_map(static fn(array $chunk): Row => row($chunk), $this->chunks), ); } /** - * @return array + * @return list */ public function values(): array { - return $this->index; + return array_map(static fn(array $chunk): string => $chunk['key'], $this->chunks); } } diff --git a/src/core/etl/src/Flow/ETL/Cardinality.php b/src/core/etl/src/Flow/ETL/Cardinality.php new file mode 100644 index 0000000000..c3e9b4ba1e --- /dev/null +++ b/src/core/etl/src/Flow/ETL/Cardinality.php @@ -0,0 +1,108 @@ + + */ + public ?int $atMost; + + /** + * @var null|int<0, max> + */ + public ?int $estimate; + + public float $relativeError; + + /** + * @param null|int $atMost a GUARANTEED upper bound; null when nothing is guaranteed + * @param null|int $estimate a best guess that may be wrong in either direction + * @param float $relativeError rough 1-sigma relative error of $estimate; meaningless without one + */ + public function __construct( + ?int $atMost = null, + ?int $estimate = null, + float $relativeError = self::DEFAULT_RELATIVE_ERROR, + ) { + if ($atMost !== null && $atMost < 0) { + throw new InvalidArgumentException('Cardinality upper bound must not be negative, given: ' . $atMost); + } + + if ($estimate !== null && $estimate < 0) { + throw new InvalidArgumentException('Cardinality estimate must not be negative, given: ' . $estimate); + } + + if ($relativeError < 0.0) { + throw new InvalidArgumentException('Relative error must not be negative, given: ' . $relativeError); + } + + $this->atMost = $atMost; + $this->estimate = $estimate; + $this->relativeError = $relativeError; + } + + public static function unknown(): self + { + return new self(); + } + + public static function exact(int $rows): self + { + return new self($rows, $rows, 0.0); + } + + public static function atMost(int $rows): self + { + return new self(atMost: $rows); + } + + public static function approximately(int $rows, float $relativeError = self::DEFAULT_RELATIVE_ERROR): self + { + return new self(estimate: $rows, relativeError: $relativeError); + } + + /** + * The estimate when it is known to at least $maxRelativeError, null when it is not. + * A caller that needs a guarantee reads $atMost instead. + * + * @return null|int<0, max> + */ + public function confident(float $maxRelativeError): ?int + { + return $this->estimate !== null && $this->relativeError <= $maxRelativeError ? $this->estimate : null; + } + + /** + * The count when it is guaranteed and known exactly, null otherwise. + * + * @return null|int<0, max> + */ + public function exactly(): ?int + { + return $this->atMost !== null && $this->atMost === $this->estimate && $this->relativeError === 0.0 + ? $this->atMost + : null; + } + + public function isUnknown(): bool + { + return $this->atMost === null && $this->estimate === null; + } + + public function merge(self $other): self + { + return new self( + atMost: $this->atMost !== null && $other->atMost !== null ? $this->atMost + $other->atMost : null, + estimate: $this->estimate !== null && $other->estimate !== null ? $this->estimate + $other->estimate : null, + relativeError: max($this->relativeError, $other->relativeError), + ); + } +} diff --git a/src/core/etl/src/Flow/ETL/DataFrame.php b/src/core/etl/src/Flow/ETL/DataFrame.php index 327c3bd11b..168075b601 100644 --- a/src/core/etl/src/Flow/ETL/DataFrame.php +++ b/src/core/etl/src/Flow/ETL/DataFrame.php @@ -38,6 +38,7 @@ use function count; use function Flow\ETL\DSL\refs; use function Flow\ETL\DSL\to_output; +use function Flow\Types\DSL\type_integer; use function is_string; /** @@ -202,18 +203,19 @@ public function constrain(Constraint $constraint, Constraint ...$constraints): s */ public function count(): int { - $total = 0; - - foreach ($this->context - ->config - ->executor() - ->execute( - $this->context->config->planner()->plan(Trigger::rows->plan($this->root, $this->sinks), $this->context), - ) as $rows) { - $total += $rows->count(); - } - - return $total; + return type_integer()->assert( + $this->context + ->config + ->executor() + ->fetch( + $this->context + ->config + ->planner() + ->plan(Trigger::count->plan($this->root, $this->sinks), $this->context), + ) + ->first() + ->get('count'), + ); } /** @@ -715,7 +717,10 @@ public function run(bool|Analyze $analyze = false): ?Report ->config ->executor() ->execute( - $this->context->config->planner()->plan(Trigger::run->plan($this->root, $this->sinks), $this->context), + $this->context + ->config + ->planner() + ->plan(Trigger::run->plan($this->root, $this->sinks), $this->context, $collector->sources()), ) as $rows) { $collector->capture($rows); } diff --git a/src/core/etl/src/Flow/ETL/Dataset/Report.php b/src/core/etl/src/Flow/ETL/Dataset/Report.php index 1cebf4a142..40fbd90c8f 100644 --- a/src/core/etl/src/Flow/ETL/Dataset/Report.php +++ b/src/core/etl/src/Flow/ETL/Dataset/Report.php @@ -8,11 +8,23 @@ final readonly class Report { + /** + * @param null|list $sources null when the run was not analyzed with source statistics + */ public function __construct( private ?Schema $schema, private Statistics $statistics, + private ?array $sources, ) {} + /** + * @return null|list + */ + public function sources(): ?array + { + return $this->sources; + } + public function schema(): ?Schema { return $this->schema; diff --git a/src/core/etl/src/Flow/ETL/Dataset/SourceStatistics.php b/src/core/etl/src/Flow/ETL/Dataset/SourceStatistics.php new file mode 100644 index 0000000000..ac78dbdb8f --- /dev/null +++ b/src/core/etl/src/Flow/ETL/Dataset/SourceStatistics.php @@ -0,0 +1,45 @@ +declared->rows->estimate; + + if ($estimate === null || !$this->complete) { + return null; + } + + if ($this->rows === 0) { + return $estimate === 0 ? 0.0 : INF; + } + + return abs($estimate - $this->rows) / $this->rows; + } +} diff --git a/src/core/etl/src/Flow/ETL/Executor.php b/src/core/etl/src/Flow/ETL/Executor.php index 84a8975b9e..cf51e2bbea 100644 --- a/src/core/etl/src/Flow/ETL/Executor.php +++ b/src/core/etl/src/Flow/ETL/Executor.php @@ -9,6 +9,7 @@ use Flow\ETL\Executor\PhysicalPlan; use Flow\ETL\Executor\Pipeline; use Flow\ETL\Extractor\FileExtractor; +use Flow\Filesystem\Path\Filter\OnlyFiles; use Generator; use Throwable; use WeakMap; @@ -133,6 +134,15 @@ public function executePipeline(Pipeline $pipeline): Generator $generator = $source instanceof FileExtractor ? $source->extract($leaf->context(), $leaf->limit(), $leaf->pathFilter()) : $source->extract($leaf->context(), $leaf->limit()); + $sources = $leaf->sources(); + + if ($sources !== null) { + $generator = $sources->count( + $source, + $generator, + $leaf->limit() !== null || !$leaf->pathFilter() instanceof OnlyFiles, + ); + } foreach ($chain as $stage) { foreach ($stage->segments()->all() as $segment) { diff --git a/src/core/etl/src/Flow/ETL/Executor/Pipeline.php b/src/core/etl/src/Flow/ETL/Executor/Pipeline.php index 59711d6f11..534927d41b 100644 --- a/src/core/etl/src/Flow/ETL/Executor/Pipeline.php +++ b/src/core/etl/src/Flow/ETL/Executor/Pipeline.php @@ -22,6 +22,8 @@ * behind an input edge * @param Filter $pathFilter handed to a FileExtractor source when this pipeline reads it; meaningless * behind an input edge + * @param null|SourceRows $sources counts the rows of the source when this pipeline reads it and the run is + * analyzed with source statistics; null otherwise */ public function __construct( public int $id, @@ -30,6 +32,7 @@ public function __construct( private ?self $input = null, private ?int $limit = null, private Filter $pathFilter = new OnlyFiles(), + private ?SourceRows $sources = null, ) {} public function segments(): Segments @@ -55,6 +58,11 @@ public function pathFilter(): Filter return $this->pathFilter; } + public function sources(): ?SourceRows + { + return $this->sources; + } + /** * The upstream stage, cut off after a blocking node. The Executor flattens the chain. */ diff --git a/src/core/etl/src/Flow/ETL/Executor/SourceRead.php b/src/core/etl/src/Flow/ETL/Executor/SourceRead.php new file mode 100644 index 0000000000..602a908493 --- /dev/null +++ b/src/core/etl/src/Flow/ETL/Executor/SourceRead.php @@ -0,0 +1,43 @@ +open++; + $this->narrowed = $this->narrowed || $narrowed; + } + + public function counted(int $rows): void + { + $this->rows += $rows; + } + + public function closed(): void + { + $this->open--; + } + + public function rows(): int + { + return $this->rows; + } + + /** + * Every read ran to its end and none had a limit or a partition filter pushed into it. + */ + public function isComplete(): bool + { + return $this->open === 0 && !$this->narrowed; + } +} diff --git a/src/core/etl/src/Flow/ETL/Executor/SourceRows.php b/src/core/etl/src/Flow/ETL/Executor/SourceRows.php new file mode 100644 index 0000000000..093aa8475c --- /dev/null +++ b/src/core/etl/src/Flow/ETL/Executor/SourceRows.php @@ -0,0 +1,80 @@ + + */ + private SplObjectStorage $reads; + + public function __construct() + { + /** @var SplObjectStorage $reads */ + $reads = new SplObjectStorage(); + $this->reads = $reads; + } + + /** + * Passes $batches through unchanged. A stop is forwarded and ends the count: the consumer that sent it reads no + * further batch, so that read never closes. + * + * @param Generator $batches what $extractor yields + * @param bool $narrowed a limit or a partition filter was pushed into this read + * + * @return Generator + */ + public function count(Extractor $extractor, Generator $batches, bool $narrowed): Generator + { + if (!$this->reads->offsetExists($extractor)) { + $this->reads[$extractor] = new SourceRead(); + } + + $read = $this->reads[$extractor]; + $read->opened($narrowed); + + foreach ($batches as $batch) { + $read->counted($batch->count()); + + $signal = yield $batch; + + if ($signal === Signal::STOP) { + $batches->send(Signal::STOP); + + return; + } + } + + $read->closed(); + } + + /** + * @return list in the order the sources were first read + */ + public function statistics(): array + { + $statistics = []; + + foreach ($this->reads as $extractor) { + $statistics[] = new SourceStatistics( + (new ReflectionClass($extractor))->getShortName(), + $extractor->statistics(), + $this->reads[$extractor]->rows(), + $this->reads[$extractor]->isComplete(), + ); + } + + return $statistics; + } +} diff --git a/src/core/etl/src/Flow/ETL/Executor/StatisticsCollector.php b/src/core/etl/src/Flow/ETL/Executor/StatisticsCollector.php index b354ceeb52..ea9d8ad556 100644 --- a/src/core/etl/src/Flow/ETL/Executor/StatisticsCollector.php +++ b/src/core/etl/src/Flow/ETL/Executor/StatisticsCollector.php @@ -31,6 +31,8 @@ final class StatisticsCollector private ?Schema $schema = null; + private ?SourceRows $sources = null; + private readonly DateTimeImmutable $startedAt; private readonly HighResolutionTime $startTime; @@ -55,6 +57,7 @@ public function __construct( if ($this->analyze !== null) { $this->columnStatistics = $this->analyze->collectColumnStatistics() ? new Columns() : null; $this->schema = $this->analyze->collectSchema() ? new Schema() : null; + $this->sources = $this->analyze->collectSourceStatistics() ? new SourceRows() : null; } gc_collect_cycles(); @@ -92,6 +95,14 @@ public function capture(Rows $rows): void } } + /** + * Handed to the planner, so the pipelines that read a source count its rows. + */ + public function sources(): ?SourceRows + { + return $this->sources; + } + /** * @return (T is Analyze|true ? Report : null) */ @@ -112,6 +123,7 @@ public function report(): ?Report $this->memory, $this->columnStatistics, ), + $this->sources?->statistics(), ); } } diff --git a/src/core/etl/src/Flow/ETL/Extractor.php b/src/core/etl/src/Flow/ETL/Extractor.php index af514a1d7f..2cb20bb0be 100644 --- a/src/core/etl/src/Flow/ETL/Extractor.php +++ b/src/core/etl/src/Flow/ETL/Extractor.php @@ -6,6 +6,7 @@ use Flow\ETL\Exception\SchemaNotDerivableException; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Generator; interface Extractor @@ -38,6 +39,21 @@ public function extract(FlowContext $context, ?int $limit = null): Generator; */ public function schema(): Schema; + /** + * What the source knows about the data before a single row is read: how many rows, how many bytes. A source + * that knows nothing returns `new Statistics()`. Describes the source with NO pushdown applied - a limit or a + * partition filter the plan pushed is the plan's to account for. + * + * Called AT MOST ONCE per run, and only when something needs the answer - so it must be lazy, never computed in + * the constructor - and it MUST be idempotent and cheap on repeat: memoise what it reads, the way schema() does. + * + * A file source may only spend what the run already spends: it may complete a listing it must produce anyway and + * read metadata from the files schema() already opens, and it may extrapolate from those. It must not open an + * extra file or make an extra remote call. A database source may ask the planner for an estimate (EXPLAIN without + * ANALYZE), never run the query. + */ + public function statistics(): Statistics; + /** * Declares the shape every yielded batch must carry. A source that describes itself uses this * instead of its own description, and values are cast to fit. diff --git a/src/core/etl/src/Flow/ETL/Extractor/ArrayExtractor.php b/src/core/etl/src/Flow/ETL/Extractor/ArrayExtractor.php index 43b97b2610..4e44843471 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/ArrayExtractor.php +++ b/src/core/etl/src/Flow/ETL/Extractor/ArrayExtractor.php @@ -4,6 +4,7 @@ namespace Flow\ETL\Extractor; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InvalidLogicException; use Flow\ETL\Extractor; use Flow\ETL\FlowContext; @@ -39,6 +40,8 @@ final class ArrayExtractor implements BatchableExtractor, Extractor, InfersSchem private readonly ?Path $spillRoot; + private ?Statistics $statistics = null; + /** * @param iterable> $dataset an array is described in place; anything else is read * exactly once, spilled, and replayed @@ -157,6 +160,14 @@ public function schema(): Schema ))->infer([], $this->source->samples(-1)); } + public function statistics(): Statistics + { + // a generator is one-shot and spilled, so its length is unknown until it has been read + return $this->statistics ??= new Statistics( + rows: is_array($this->dataset) ? Cardinality::exact(count($this->dataset)) : Cardinality::unknown(), + ); + } + public function withSchema(Schema $schema): static { $this->schema = $schema; diff --git a/src/core/etl/src/Flow/ETL/Extractor/BatchByExtractor.php b/src/core/etl/src/Flow/ETL/Extractor/BatchByExtractor.php index 55dc468c9b..81ae95d6f2 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/BatchByExtractor.php +++ b/src/core/etl/src/Flow/ETL/Extractor/BatchByExtractor.php @@ -97,6 +97,11 @@ public function schema(): Schema return $this->extractor->schema(); } + public function statistics(): Statistics + { + return $this->extractor->statistics(); + } + public function withSchema(Schema $schema): static { $this->schema = $schema; diff --git a/src/core/etl/src/Flow/ETL/Extractor/BatchExtractor.php b/src/core/etl/src/Flow/ETL/Extractor/BatchExtractor.php index 35b6f9dfd3..ccf19c498b 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/BatchExtractor.php +++ b/src/core/etl/src/Flow/ETL/Extractor/BatchExtractor.php @@ -79,6 +79,11 @@ public function schema(): Schema return $this->extractor->schema(); } + public function statistics(): Statistics + { + return $this->extractor->statistics(); + } + public function withSchema(Schema $schema): static { $this->schema = $schema; diff --git a/src/core/etl/src/Flow/ETL/Extractor/CacheExtractor.php b/src/core/etl/src/Flow/ETL/Extractor/CacheExtractor.php index 681b6d595f..488727bbd4 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/CacheExtractor.php +++ b/src/core/etl/src/Flow/ETL/Extractor/CacheExtractor.php @@ -6,6 +6,7 @@ use Flow\ETL\Cache; use Flow\ETL\Cache\CacheIndex; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor; use Flow\ETL\FlowContext; use Flow\ETL\Rows; @@ -20,6 +21,13 @@ final class CacheExtractor implements Extractor, RewindableExtractor private ?Extractor $fallbackExtractor = null; + /** + * The index schema() and statistics() read from the constructor's cache, parsed once. + */ + private ?CacheIndex $index = null; + + private ?Statistics $statistics = null; + public function __construct( private readonly string $id, private readonly ?Cache $cache = null, @@ -99,7 +107,9 @@ public function schema(): Schema if ($this->cache !== null && $this->cache->has($this->id)) { $schema = new Schema(); - foreach (CacheIndex::fromRows($this->id, $this->cache->get($this->id))->values() as $cacheKey) { + $this->index ??= CacheIndex::fromRows($this->id, $this->cache->get($this->id)); + + foreach ($this->index->values() as $cacheKey) { $schema = $schema->merge($this->cache->schema($cacheKey)); } @@ -116,6 +126,33 @@ public function schema(): Schema return new Schema(); } + /** + * Answers from the arm extract() will take, and only when a cache was handed to the constructor: one taken from + * the FlowContext may hold the entry this cache lacks. + */ + public function statistics(): Statistics + { + if ($this->statistics !== null) { + return $this->statistics; + } + + if ($this->cache === null) { + return new Statistics(); + } + + if ($this->cache->has($this->id)) { + return $this->statistics = new Statistics( + rows: ($this->index ??= CacheIndex::fromRows($this->id, $this->cache->get($this->id)))->rows(), + ); + } + + if ($this->fallbackExtractor === null) { + return new Statistics(rows: Cardinality::exact(0)); + } + + return $this->fallbackExtractor->statistics(); + } + public function withSchema(Schema $schema): static { $this->schema = $schema; diff --git a/src/core/etl/src/Flow/ETL/Extractor/ChainExtractor.php b/src/core/etl/src/Flow/ETL/Extractor/ChainExtractor.php index 6648e8bde5..6f4a444d11 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/ChainExtractor.php +++ b/src/core/etl/src/Flow/ETL/Extractor/ChainExtractor.php @@ -4,12 +4,15 @@ namespace Flow\ETL\Extractor; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; use Generator; +use function array_reduce; + final class ChainExtractor implements Extractor, OverridingExtractor, RewindableExtractor { private ?Schema $schema = null; @@ -71,6 +74,17 @@ public function schema(): Schema return $schema; } + public function statistics(): Statistics + { + return array_reduce( + $this->extractors, + static fn(Statistics $statistics, Extractor $extractor): Statistics => $statistics->merge( + $extractor->statistics(), + ), + new Statistics(rows: Cardinality::exact(0), size: Cardinality::exact(0)), + ); + } + public function withSchema(Schema $schema): static { $this->schema = $schema; diff --git a/src/core/etl/src/Flow/ETL/Extractor/CollectingExtractor.php b/src/core/etl/src/Flow/ETL/Extractor/CollectingExtractor.php index 35d5593ec6..159e1c1602 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/CollectingExtractor.php +++ b/src/core/etl/src/Flow/ETL/Extractor/CollectingExtractor.php @@ -54,6 +54,11 @@ public function schema(): Schema return $this->extractor->schema(); } + public function statistics(): Statistics + { + return $this->extractor->statistics(); + } + public function withSchema(Schema $schema): static { $this->schema = $schema; diff --git a/src/core/etl/src/Flow/ETL/Extractor/DataFrameExtractor.php b/src/core/etl/src/Flow/ETL/Extractor/DataFrameExtractor.php index 049478c3e1..cf3c7a86ea 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/DataFrameExtractor.php +++ b/src/core/etl/src/Flow/ETL/Extractor/DataFrameExtractor.php @@ -85,4 +85,9 @@ public function withSchema(Schema $schema): static return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/src/Flow/ETL/Extractor/FeedExtractor.php b/src/core/etl/src/Flow/ETL/Extractor/FeedExtractor.php index 79048d0016..e242faafd5 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/FeedExtractor.php +++ b/src/core/etl/src/Flow/ETL/Extractor/FeedExtractor.php @@ -89,4 +89,9 @@ public function withSchema(Schema $schema): static return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/src/Flow/ETL/Extractor/FileReading.php b/src/core/etl/src/Flow/ETL/Extractor/FileReading.php index 370c6706b3..bd57eb3735 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/FileReading.php +++ b/src/core/etl/src/Flow/ETL/Extractor/FileReading.php @@ -35,7 +35,7 @@ private function fileColumns(Filesystem $filesystem, Path $path): FileColumns private function sourceFiles(Filesystem $filesystem, Path $path, Filter $pathFilter = new OnlyFiles()): Generator { foreach ((new FileListing($filesystem))->list($path, $pathFilter) as $status) { - yield new SourceFile($status->path); + yield new SourceFile($status->path, $status->size); } } } diff --git a/src/core/etl/src/Flow/ETL/Extractor/FilesExtractor.php b/src/core/etl/src/Flow/ETL/Extractor/FilesExtractor.php index c4f52896b6..3f4383ab28 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/FilesExtractor.php +++ b/src/core/etl/src/Flow/ETL/Extractor/FilesExtractor.php @@ -4,6 +4,7 @@ namespace Flow\ETL\Extractor; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Extractor; use Flow\ETL\FlowContext; @@ -27,6 +28,8 @@ final class FilesExtractor implements BatchableExtractor, Extractor, FileExtract { private ?Schema $schema = null; + private ?Statistics $statistics = null; + use Batches; use ListingPartitions; @@ -66,7 +69,7 @@ public function extract(FlowContext $context, ?int $limit = null, Filter $pathFi $yielded = 0; foreach ((new FileListing($this->filesystem))->list($this->path, $pathFilter) as $fileStatus) { - $constants = $fileColumns->forFile(new SourceFile($fileStatus->path), $schema); + $constants = $fileColumns->forFile(new SourceFile($fileStatus->path, $fileStatus->size), $schema); $extension = $fileStatus->path->extension(); $buffer[] = $constants->fill([ @@ -134,6 +137,16 @@ public function schema(): Schema )); } + public function statistics(): Statistics + { + if ($this->statistics === null) { + $listed = ListedFiles::of((new FileListing($this->filesystem))->list($this->path, new OnlyFiles())); + $this->statistics = new Statistics(rows: Cardinality::exact($listed->count), size: $listed->bytes); + } + + return $this->statistics; + } + public function withSchema(Schema $schema): static { $this->schema = $schema; diff --git a/src/core/etl/src/Flow/ETL/Extractor/FooterStatistics.php b/src/core/etl/src/Flow/ETL/Extractor/FooterStatistics.php new file mode 100644 index 0000000000..ecfb822ab0 --- /dev/null +++ b/src/core/etl/src/Flow/ETL/Extractor/FooterStatistics.php @@ -0,0 +1,59 @@ + + */ + public int $files; + + /** + * @param Statistics $statistics what the read footers declare, merged + * @param int $files how many footers were read + */ + public function __construct( + public Statistics $statistics, + int $files, + ) { + if ($files < 0) { + throw new InvalidArgumentException('Read footer count must not be negative, given: ' . $files); + } + + $this->files = $files; + } + + /** + * Exact when every listed file's footer was read; otherwise the read footers are scaled to the listing + * (DuckDB's rule, parquet_extension.cpp:866-877). The offset is taken off the rows, never below zero. + */ + public function of(int $listedFiles, int $offset): Statistics + { + $rows = $this->statistics->rows->estimate ?? 0; + $bytes = $this->statistics->size->estimate ?? 0; + + if ($this->files === 0 || $this->files >= $listedFiles) { + return new Statistics(rows: Cardinality::exact(max(0, $rows - $offset)), size: Cardinality::exact($bytes)); + } + + return new Statistics( + rows: Cardinality::approximately( + max(0, (int) round(($rows * $listedFiles) / $this->files) - $offset), + Cardinality::DEFAULT_RELATIVE_ERROR, + ), + size: Cardinality::approximately( + (int) round(($bytes * $listedFiles) / $this->files), + Cardinality::DEFAULT_RELATIVE_ERROR, + ), + ); + } +} diff --git a/src/core/etl/src/Flow/ETL/Extractor/ListedFiles.php b/src/core/etl/src/Flow/ETL/Extractor/ListedFiles.php new file mode 100644 index 0000000000..c6c7c6e9bd --- /dev/null +++ b/src/core/etl/src/Flow/ETL/Extractor/ListedFiles.php @@ -0,0 +1,58 @@ + + */ + public int $count; + + /** + * @param Cardinality $bytes total listed bytes, unknown when ANY member did not report a size + */ + public function __construct( + int $count, + public Cardinality $bytes, + ) { + if ($count < 0) { + throw new InvalidArgumentException('Listed file count must not be negative, given: ' . $count); + } + + $this->count = $count; + } + + /** + * One pass, nothing kept but the two totals. NativeLocalFilesystem::statFor() is `filesize($p) ?: null`, so it + * reports a zero-byte file as null. + * + * @param iterable $files + */ + public static function of(iterable $files): self + { + $count = 0; + $total = 0; + $sized = true; + + foreach ($files as $file) { + $count++; + + if ($file->size === null) { + $sized = false; + + continue; + } + + $total += $file->size; + } + + return new self($count, $sized ? Cardinality::exact($total) : Cardinality::unknown()); + } +} diff --git a/src/core/etl/src/Flow/ETL/Extractor/MemoryExtractor.php b/src/core/etl/src/Flow/ETL/Extractor/MemoryExtractor.php index 51cf095249..2c62187509 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/MemoryExtractor.php +++ b/src/core/etl/src/Flow/ETL/Extractor/MemoryExtractor.php @@ -107,4 +107,12 @@ public function withSchema(Schema $schema): static return $this; } + + /** + * Declares nothing: ArrayMemory::save() is array_merge, so the row count can grow mid-run. + */ + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/src/Flow/ETL/Extractor/PathFiltering.php b/src/core/etl/src/Flow/ETL/Extractor/PathFiltering.php index 23e13d22a5..27772816f5 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/PathFiltering.php +++ b/src/core/etl/src/Flow/ETL/Extractor/PathFiltering.php @@ -4,6 +4,7 @@ namespace Flow\ETL\Extractor; +use Flow\ETL\Cardinality; use Flow\ETL\Schema; use Flow\Filesystem\Path; use Flow\Filesystem\Path\Filter\OnlyFiles; @@ -18,6 +19,11 @@ trait PathFiltering private string $derivedFrom = ''; + /** + * The footers the fold read while deriving the schema; set together with $derivedSchema. + */ + private ?FooterStatistics $derivedFooters = null; + /** * One listing per extractor instance, however often schema() is asked. * @@ -38,6 +44,8 @@ private function derivedSchema(Generator $files, bool $unionByName): Schema } $schema = new Schema(); + $statistics = new Statistics(rows: Cardinality::exact(0), size: Cardinality::exact(0)); + $read = 0; $this->derivedFrom = ''; foreach ($files as $file) { @@ -47,6 +55,8 @@ private function derivedSchema(Generator $files, bool $unionByName): Schema } $schema = $schema->merge($file->schema()); + $statistics = $statistics->merge($file->statistics()); + $read++; } finally { $file->close(); } @@ -56,9 +66,49 @@ private function derivedSchema(Generator $files, bool $unionByName): Schema } } + $this->derivedFooters = new FooterStatistics($statistics, $read); + return $this->derivedSchema = $schema; } + /** + * The footers the schema fold reads, derived once: statistics() never opens a footer schema() already did. + * + * @param Generator $files an unstarted generator + */ + private function derivedFooters(Generator $files, bool $unionByName): FooterStatistics + { + $this->derivedSchema($files, $unionByName); + + return $this->derivedFooters ?? new FooterStatistics(new Statistics(), 0); + } + + /** + * The same footers without deriving a schema, for a read whose schema was declared: nothing else opens them. + * + * @param Generator $files an unstarted generator + */ + private function footers(Generator $files, bool $unionByName): FooterStatistics + { + $statistics = new Statistics(rows: Cardinality::exact(0), size: Cardinality::exact(0)); + $read = 0; + + foreach ($files as $file) { + try { + $statistics = $statistics->merge($file->statistics()); + $read++; + } finally { + $file->close(); + } + + if (!$unionByName) { + break; + } + } + + return new FooterStatistics($statistics, $read); + } + /** * @return array */ diff --git a/src/core/etl/src/Flow/ETL/Extractor/PathPartitionsExtractor.php b/src/core/etl/src/Flow/ETL/Extractor/PathPartitionsExtractor.php index a655bdc357..567f4a823b 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/PathPartitionsExtractor.php +++ b/src/core/etl/src/Flow/ETL/Extractor/PathPartitionsExtractor.php @@ -4,6 +4,7 @@ namespace Flow\ETL\Extractor; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Extractor; use Flow\ETL\FlowContext; @@ -27,12 +28,15 @@ use function Flow\ETL\DSL\str_schema; use function Flow\Types\DSL\type_map; use function Flow\Types\DSL\type_string; +use function iterator_count; use function sprintf; final class PathPartitionsExtractor implements BatchableExtractor, Extractor, FileExtractor, RewindableExtractor { private ?Schema $schema = null; + private ?Statistics $statistics = null; + use Batches; use ListingPartitions; @@ -72,7 +76,7 @@ public function extract(FlowContext $context, ?int $limit = null, Filter $pathFi $yielded = 0; foreach ((new FileListing($this->filesystem))->list($this->path, $pathFilter) as $fileStatus) { - $constants = $fileColumns->forFile(new SourceFile($fileStatus->path), $schema); + $constants = $fileColumns->forFile(new SourceFile($fileStatus->path, $fileStatus->size), $schema); $buffer[] = $constants->fill([ 'path' => $fileStatus->path->uri(), 'partitions' => array_merge(...array_values(array_map(static fn(Partition $p) => [ @@ -128,6 +132,15 @@ public function schema(): Schema )); } + public function statistics(): Statistics + { + return $this->statistics ??= + new Statistics(rows: Cardinality::exact(iterator_count((new FileListing($this->filesystem))->list( + $this->path, + new OnlyFiles(), + )))); + } + public function withSchema(Schema $schema): static { $this->schema = $schema; diff --git a/src/core/etl/src/Flow/ETL/Extractor/RowsExtractor.php b/src/core/etl/src/Flow/ETL/Extractor/RowsExtractor.php index 1d83eef516..a21dcb9dd2 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/RowsExtractor.php +++ b/src/core/etl/src/Flow/ETL/Extractor/RowsExtractor.php @@ -4,6 +4,7 @@ namespace Flow\ETL\Extractor; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor; use Flow\ETL\FlowContext; use Flow\ETL\Rows; @@ -14,6 +15,8 @@ final class RowsExtractor implements Extractor, RewindableExtractor { private ?Schema $schema = null; + private ?Statistics $statistics = null; + /** * @var array */ @@ -60,6 +63,21 @@ public function schema(): Schema return $schema; } + public function statistics(): Statistics + { + if ($this->statistics === null) { + $count = 0; + + foreach ($this->rows as $rows) { + $count += $rows->count(); + } + + $this->statistics = new Statistics(rows: Cardinality::exact($count)); + } + + return $this->statistics; + } + public function withSchema(Schema $schema): static { $this->schema = $schema; diff --git a/src/core/etl/src/Flow/ETL/Extractor/SelfDescribingFile.php b/src/core/etl/src/Flow/ETL/Extractor/SelfDescribingFile.php index 12d7a97272..65fcc12a3e 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/SelfDescribingFile.php +++ b/src/core/etl/src/Flow/ETL/Extractor/SelfDescribingFile.php @@ -13,4 +13,9 @@ public function close(): void; public function schema(): Schema; public function source(): SourceFile; + + /** + * What this one file's own metadata declares, read from what schema() already opened. + */ + public function statistics(): Statistics; } diff --git a/src/core/etl/src/Flow/ETL/Extractor/SequenceExtractor.php b/src/core/etl/src/Flow/ETL/Extractor/SequenceExtractor.php index 39985cb7d1..80b299ef89 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/SequenceExtractor.php +++ b/src/core/etl/src/Flow/ETL/Extractor/SequenceExtractor.php @@ -27,6 +27,8 @@ final class SequenceExtractor implements BatchableExtractor, Extractor, InfersSc private ?Schema $schema = null; + private ?Statistics $statistics = null; + public function __construct( private readonly SequenceGenerator $generator, private readonly string $entryName = 'entry', @@ -108,6 +110,11 @@ public function schema(): Schema ); } + public function statistics(): Statistics + { + return $this->statistics ??= new Statistics(rows: $this->generator->rows()); + } + public function withSchema(Schema $schema): static { $this->schema = $schema; diff --git a/src/core/etl/src/Flow/ETL/Extractor/SequenceGenerator/DatePeriodSequenceGenerator.php b/src/core/etl/src/Flow/ETL/Extractor/SequenceGenerator/DatePeriodSequenceGenerator.php index 0e98a60a2e..c9d2059984 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/SequenceGenerator/DatePeriodSequenceGenerator.php +++ b/src/core/etl/src/Flow/ETL/Extractor/SequenceGenerator/DatePeriodSequenceGenerator.php @@ -5,8 +5,11 @@ namespace Flow\ETL\Extractor\SequenceGenerator; use DatePeriod; +use Flow\ETL\Cardinality; use Generator; +use function iterator_count; + final readonly class DatePeriodSequenceGenerator implements SequenceGenerator { /** @@ -23,4 +26,9 @@ public function generate(): Generator yield $item; } } + + public function rows(): Cardinality + { + return Cardinality::exact(iterator_count($this->period)); + } } diff --git a/src/core/etl/src/Flow/ETL/Extractor/SequenceGenerator/NumberSequenceGenerator.php b/src/core/etl/src/Flow/ETL/Extractor/SequenceGenerator/NumberSequenceGenerator.php index 4a9b15a42c..e9f50f6563 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/SequenceGenerator/NumberSequenceGenerator.php +++ b/src/core/etl/src/Flow/ETL/Extractor/SequenceGenerator/NumberSequenceGenerator.php @@ -4,8 +4,13 @@ namespace Flow\ETL\Extractor\SequenceGenerator; +use Flow\ETL\Cardinality; use Generator; +use function abs; +use function count; +use function intdiv; +use function is_int; use function range; final readonly class NumberSequenceGenerator implements SequenceGenerator @@ -22,4 +27,24 @@ public function generate(): Generator yield $item; } } + + /** + * Integer bounds with a positive step no wider than the span are counted arithmetically; everything else counts + * the same range() generate() walks, because PHP rounds float ranges its own way and throws on a step that does + * not fit - a formula would diverge from what extract() yields. + */ + public function rows(): Cardinality + { + if ( + is_int($this->start) + && is_int($this->end) + && is_int($this->step) + && $this->step > 0 + && abs($this->end - $this->start) >= $this->step + ) { + return Cardinality::exact(intdiv(abs($this->end - $this->start), $this->step) + 1); + } + + return Cardinality::exact(count(range($this->start, $this->end, $this->step))); + } } diff --git a/src/core/etl/src/Flow/ETL/Extractor/SequenceGenerator/SequenceGenerator.php b/src/core/etl/src/Flow/ETL/Extractor/SequenceGenerator/SequenceGenerator.php index c23e62c813..a84446ea2f 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/SequenceGenerator/SequenceGenerator.php +++ b/src/core/etl/src/Flow/ETL/Extractor/SequenceGenerator/SequenceGenerator.php @@ -4,6 +4,7 @@ namespace Flow\ETL\Extractor\SequenceGenerator; +use Flow\ETL\Cardinality; use Generator; interface SequenceGenerator @@ -12,4 +13,6 @@ interface SequenceGenerator * @return \Generator */ public function generate(): Generator; + + public function rows(): Cardinality; } diff --git a/src/core/etl/src/Flow/ETL/Extractor/SourceFile.php b/src/core/etl/src/Flow/ETL/Extractor/SourceFile.php index eac798b388..136f2cade7 100644 --- a/src/core/etl/src/Flow/ETL/Extractor/SourceFile.php +++ b/src/core/etl/src/Flow/ETL/Extractor/SourceFile.php @@ -18,6 +18,7 @@ public function __construct( public Path $path, + public ?int $size = null, ) { $values = []; diff --git a/src/core/etl/src/Flow/ETL/Extractor/Statistics.php b/src/core/etl/src/Flow/ETL/Extractor/Statistics.php new file mode 100644 index 0000000000..0b2079d608 --- /dev/null +++ b/src/core/etl/src/Flow/ETL/Extractor/Statistics.php @@ -0,0 +1,20 @@ +rows->merge($other->rows), $this->size->merge($other->size)); + } +} diff --git a/src/core/etl/src/Flow/ETL/Optimizer.php b/src/core/etl/src/Flow/ETL/Optimizer.php index 6095cde1d0..507fc28e28 100644 --- a/src/core/etl/src/Flow/ETL/Optimizer.php +++ b/src/core/etl/src/Flow/ETL/Optimizer.php @@ -9,6 +9,7 @@ use Flow\ETL\Optimizer\Rule; use Flow\ETL\Optimizer\Rule\CombineLimits; use Flow\ETL\Optimizer\Rule\CombineSortAndLimit; +use Flow\ETL\Optimizer\Rule\CountFromStatistics; use Flow\ETL\Optimizer\Rule\PushFilterIntoSource; use Flow\ETL\Optimizer\Rule\PushLimitIntoSource; use Flow\ETL\Plan\LogicalPlan; @@ -37,6 +38,7 @@ public static function default(): self new CombineSortAndLimit(), new PushLimitIntoSource(), new PushFilterIntoSource(), + new CountFromStatistics(), ); } diff --git a/src/core/etl/src/Flow/ETL/Optimizer/Rule/CountFromStatistics.php b/src/core/etl/src/Flow/ETL/Optimizer/Rule/CountFromStatistics.php new file mode 100644 index 0000000000..1ab40fce68 --- /dev/null +++ b/src/core/etl/src/Flow/ETL/Optimizer/Rule/CountFromStatistics.php @@ -0,0 +1,40 @@ +root instanceof Result ? $plan->root->children()[0] : null; + $read = $count instanceof Count ? $count->children()[0] : null; + + if (!$read instanceof Read || $read->limit() !== null || !$read->pathFilter() instanceof OnlyFiles) { + return $plan; + } + + $rows = $read->extractor()->statistics()->rows->exactly(); + + return $rows === null + ? $plan + : new LogicalPlan(new Result(new Read(from_rows((new CountingProcessor())->rows($rows))))); + } +} diff --git a/src/core/etl/src/Flow/ETL/Plan/Explain/StatisticsLine.php b/src/core/etl/src/Flow/ETL/Plan/Explain/StatisticsLine.php new file mode 100644 index 0000000000..4c158f0cb8 --- /dev/null +++ b/src/core/etl/src/Flow/ETL/Plan/Explain/StatisticsLine.php @@ -0,0 +1,62 @@ +rows->isUnknown() && $statistics->size->isUnknown()) { + return null; + } + + return ( + 'rows ' . $this->cardinality($statistics->rows) . ' · size ' . $this->cardinality($statistics->size, ' B') + ); + } + + public function cardinality(Cardinality $cardinality, string $unit = ''): string + { + $atMost = $cardinality->atMost; + $estimate = $cardinality->estimate; + $exactly = $cardinality->exactly(); + + if ($exactly !== null) { + return 'exact ' . $this->number($exactly) . $unit; + } + + if ($estimate === null) { + return $atMost === null ? 'unknown' : '≤ ' . $this->number($atMost) . $unit; + } + + $approximately = '~' . $this->number($estimate) . $unit . ' ±' . $this->percent($cardinality->relativeError); + + return $atMost === null ? $approximately : $approximately . ' (≤ ' . $this->number($atMost) . $unit . ')'; + } + + public function percent(?float $error): string + { + return match (true) { + $error === null => 'unknown', + is_infinite($error) => '∞', + default => round($error * 100, 2) . '%', + }; + } + + public function number(int $number): string + { + return number_format($number, 0, '', ' '); + } +} diff --git a/src/core/etl/src/Flow/ETL/Plan/Explain/StepDetails.php b/src/core/etl/src/Flow/ETL/Plan/Explain/StepDetails.php index 7738770a65..26ad2c6fd6 100644 --- a/src/core/etl/src/Flow/ETL/Plan/Explain/StepDetails.php +++ b/src/core/etl/src/Flow/ETL/Plan/Explain/StepDetails.php @@ -39,6 +39,7 @@ public function __construct( private Details $details = new Details(), private Condition $condition = new Condition(), + private StatisticsLine $statistics = new StatisticsLine(), ) {} /** @@ -105,10 +106,21 @@ public function settings(Extractor|Loader|Processor|Transformer $step): array : ['Constraints: ' . implode(', ', array_map($this->details->name(...), $step->constraints))], $step instanceof CollectingProcessor => $step->declared === null ? [] : ['Schema: declared'], $step instanceof OffsetProcessor => ['Skip: ' . $step->offset], + $step instanceof Extractor => $this->extractor($step), default => [], }; } + /** + * @return list + */ + public function extractor(Extractor $extractor): array + { + $statistics = $this->statistics->of($extractor->statistics()); + + return $statistics === null ? [] : ['Statistics: ' . $statistics]; + } + /** * @return list */ diff --git a/src/core/etl/src/Flow/ETL/Plan/Node/Count.php b/src/core/etl/src/Flow/ETL/Plan/Node/Count.php new file mode 100644 index 0000000000..eb2daa625e --- /dev/null +++ b/src/core/etl/src/Flow/ETL/Plan/Node/Count.php @@ -0,0 +1,51 @@ + + */ + public function children(): array + { + return [$this->input]; + } + + public function withChildren(array $children): self + { + return $children[0] === $this->input ? $this : new self($children[0]); + } + + public function rowCount(): RowCount + { + return RowCount::reducing; + } + + public function transparency(): Transparency + { + return Transparency::opaque; + } + + public function materialization(): Materialization + { + return Materialization::blocking; + } + + public function redefines(): Redefined + { + return Redefined::unknown(); + } +} diff --git a/src/core/etl/src/Flow/ETL/Plan/Trigger.php b/src/core/etl/src/Flow/ETL/Plan/Trigger.php index f724ce89e4..1cb3c46060 100644 --- a/src/core/etl/src/Flow/ETL/Plan/Trigger.php +++ b/src/core/etl/src/Flow/ETL/Plan/Trigger.php @@ -15,12 +15,15 @@ enum Trigger case run; + case count; + public function plan(Node $root, Sinks $sinks = new Sinks()): LogicalPlan { $all = $sinks->all(); $consumers = match ($this) { self::rows => [new Node\Result($root), ...$all], + self::count => [new Node\Result(new Node\Count($root)), ...$all], self::run => ($all[0] ?? null) instanceof Node\Write && $all[0]->children()[0] === $root ? $all : [new Node\Result($root), ...$all], diff --git a/src/core/etl/src/Flow/ETL/Planner.php b/src/core/etl/src/Flow/ETL/Planner.php index 29141168ef..d195a44a15 100644 --- a/src/core/etl/src/Flow/ETL/Planner.php +++ b/src/core/etl/src/Flow/ETL/Planner.php @@ -6,6 +6,7 @@ use Flow\ETL\Exception\SchemaNotDerivableException; use Flow\ETL\Executor\PhysicalPlan; +use Flow\ETL\Executor\SourceRows; use Flow\ETL\Plan\LogicalPlan; use Flow\ETL\Plan\Node; use Flow\ETL\Planner\NodeTranslator; @@ -24,17 +25,19 @@ public function __construct( * The optimizer rewrites the plan first, so the one walk that translates and binds sees the final tree. A frame * this one joined is part of the plan: nothing is planned with another frame's planner. * A failure is reported as a started and failed DataFrame span of $context, then rethrown. + * + * @param null|SourceRows $sources counts the rows every source yields, a joined frame's included */ - public function plan(LogicalPlan $logical, FlowContext $context): PhysicalPlan + public function plan(LogicalPlan $logical, FlowContext $context, ?SourceRows $sources = null): PhysicalPlan { $planned = new PlannedNodes(); try { $logical = $this->optimizer->optimize($logical, $context); - $this->node($logical->root, $context, $planned); + $this->node($logical->root, $context, $planned, $sources); - return (new PipelineSplit())->of($logical, $planned, $context); + return (new PipelineSplit())->of($logical, $planned, $context, $sources); } catch (Throwable $e) { // a failure is never reported without a start $context->telemetry()->dataFrameStarted($context); @@ -48,8 +51,12 @@ public function plan(LogicalPlan $logical, FlowContext $context): PhysicalPlan * Plans $node and everything under it, each node once by identity: translated to steps, then bound to the * schema of its input. A prefix several consumers share is planned once; a joined frame is planned apart. */ - public function node(Node $node, FlowContext $context, PlannedNodes $planned): PlannedNode - { + public function node( + Node $node, + FlowContext $context, + PlannedNodes $planned, + ?SourceRows $sources = null, + ): PlannedNode { if ($planned->has($node)) { return $planned->of($node); } @@ -57,7 +64,7 @@ public function node(Node $node, FlowContext $context, PlannedNodes $planned): P $inputs = []; foreach ($node instanceof Node\JoinsFrame ? [$node->children()[0]] : $node->children() as $child) { - $inputs[] = $this->node($child, $context, $planned); + $inputs[] = $this->node($child, $context, $planned, $sources); } $frames = []; @@ -65,8 +72,8 @@ public function node(Node $node, FlowContext $context, PlannedNodes $planned): P if ($node instanceof Node\JoinsFrame) { // the joined frame runs as a pipeline of its own, so a node it shares with this plan needs its own steps $right = new PlannedNodes(); - $this->node($node->right(), $context, $right); - $frames[] = (new PipelineSplit())->of(new LogicalPlan($node->right()), $right, $context); + $this->node($node->right(), $context, $right, $sources); + $frames[] = (new PipelineSplit())->of(new LogicalPlan($node->right()), $right, $context, $sources); } $steps = NodeTranslator::toSteps($node, $context, $frames); diff --git a/src/core/etl/src/Flow/ETL/Planner/NodeTranslator.php b/src/core/etl/src/Flow/ETL/Planner/NodeTranslator.php index be3d32668e..354deaf9cb 100644 --- a/src/core/etl/src/Flow/ETL/Planner/NodeTranslator.php +++ b/src/core/etl/src/Flow/ETL/Planner/NodeTranslator.php @@ -19,6 +19,7 @@ use Flow\ETL\Processor\CachingProcessor; use Flow\ETL\Processor\CollectingProcessor; use Flow\ETL\Processor\ConstrainedProcessor; +use Flow\ETL\Processor\CountingProcessor; use Flow\ETL\Processor\OffsetProcessor; use Flow\ETL\Processor\TopNProcessor; use Flow\ETL\Processor\VoidProcessor; @@ -71,6 +72,7 @@ public static function toSteps(Node $node, FlowContext $context, array $frames): $node instanceof Node\Collect => [new CollectingProcessor()], $node instanceof Node\CollectRefs => [new CollectReferencesTransformer($node->references)], $node instanceof Node\Constrain => [new ConstrainedProcessor($node->constraints)], + $node instanceof Node\Count => [new CountingProcessor()], $node instanceof Node\CrossJoin => [new CrossJoinRowsTransformer( $frames[0], $context->config->executor(), diff --git a/src/core/etl/src/Flow/ETL/Planner/PipelineSplit.php b/src/core/etl/src/Flow/ETL/Planner/PipelineSplit.php index 55f761b675..fb5645b886 100644 --- a/src/core/etl/src/Flow/ETL/Planner/PipelineSplit.php +++ b/src/core/etl/src/Flow/ETL/Planner/PipelineSplit.php @@ -10,6 +10,7 @@ use Flow\ETL\Executor\Pipeline; use Flow\ETL\Executor\Raw; use Flow\ETL\Executor\Segments; +use Flow\ETL\Executor\SourceRows; use Flow\ETL\FlowContext; use Flow\ETL\Plan\LogicalPlan; use Flow\ETL\Plan\Materialization; @@ -25,11 +26,16 @@ * @param LogicalPlan $logical a whole frame's plan, or a join's right side * @param PlannedNodes $planned every node of the plan already planned * @param FlowContext $context the frame this plan belongs to + * @param null|SourceRows $sources handed to the pipeline that reads the source, when the run counts its rows * * @throws InvalidLogicException when the spine's leaf is not a Read, or a sink shares no node with the spine */ - public function of(LogicalPlan $logical, PlannedNodes $planned, FlowContext $context): PhysicalPlan - { + public function of( + LogicalPlan $logical, + PlannedNodes $planned, + FlowContext $context, + ?SourceRows $sources = null, + ): PhysicalPlan { $root = $logical->root; $top = $logical->spine(); $spine = []; @@ -70,14 +76,15 @@ public function of(LogicalPlan $logical, PlannedNodes $planned, FlowContext $con } if ($node->materialization() === Materialization::blocking && $node !== $top) { - $input = new Pipeline($id++, $segments, $context, $input, $limit, $pathFilter); + $input = new Pipeline($id++, $segments, $context, $input, $limit, $pathFilter, $sources); $segments = new Segments(); $limit = null; $pathFilter = new OnlyFiles(); + $sources = null; } } - $pipeline = new Pipeline($id, $segments, $context, $input, $limit, $pathFilter); + $pipeline = new Pipeline($id, $segments, $context, $input, $limit, $pathFilter, $sources); $refusal = $planned->refusal(); return $refusal === null diff --git a/src/core/etl/src/Flow/ETL/Processor/CachingProcessor.php b/src/core/etl/src/Flow/ETL/Processor/CachingProcessor.php index 2b80da2cbe..af160151f0 100644 --- a/src/core/etl/src/Flow/ETL/Processor/CachingProcessor.php +++ b/src/core/etl/src/Flow/ETL/Processor/CachingProcessor.php @@ -54,7 +54,7 @@ public function process(Generator $rows, FlowContext $context): Generator foreach ($rows as $batch) { $cacheKey = bin2hex(random_bytes(16)); $cache->set($cacheKey, $batch); - $index->add($cacheKey); + $index->add($cacheKey, $batch->count()); if (!$stopped) { $signal = yield $batch; diff --git a/src/core/etl/src/Flow/ETL/Processor/CountingProcessor.php b/src/core/etl/src/Flow/ETL/Processor/CountingProcessor.php new file mode 100644 index 0000000000..485304d5c5 --- /dev/null +++ b/src/core/etl/src/Flow/ETL/Processor/CountingProcessor.php @@ -0,0 +1,49 @@ +schema()); + } + + public function process(Generator $rows, FlowContext $context): Generator + { + $count = 0; + + foreach ($rows as $batch) { + $count += $batch->count(); + } + + yield $this->rows($count); + } + + /** + * The one row a count hands out, whether it was counted or known upfront. + */ + public function rows(int $count): Rows + { + return rows($this->schema(), row(['count' => $count])); + } + + public function schema(): Schema + { + return schema(int_schema('count')); + } +} diff --git a/src/core/etl/src/Flow/Floe/FloeExtractor.php b/src/core/etl/src/Flow/Floe/FloeExtractor.php index dbe37f00e5..2de58e36ef 100644 --- a/src/core/etl/src/Flow/Floe/FloeExtractor.php +++ b/src/core/etl/src/Flow/Floe/FloeExtractor.php @@ -14,6 +14,7 @@ use Flow\ETL\Extractor\MetadataColumnsExtractor; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Row; use Flow\ETL\Row\Hydrator; @@ -28,6 +29,7 @@ use Flow\Floe\Codec\NoopCodec; use Generator; +use function iterator_count; use function sprintf; final class FloeExtractor implements @@ -44,6 +46,8 @@ final class FloeExtractor implements private ?int $offset = null; + private ?Statistics $statistics = null; + private bool $unionByName = false; private readonly Filesystem $filesystem; @@ -166,6 +170,11 @@ public function schema(): Schema ); } + public function statistics(): Statistics + { + return $this->statistics ??= $this->declare(); + } + /** * Reconcile every listed file's footer instead of trusting the first one. */ @@ -173,6 +182,7 @@ public function unionByName(bool $union = true): self { $this->unionByName = $union; $this->derivedSchema = null; + $this->statistics = null; return $this; } @@ -194,10 +204,24 @@ public function withOffset(int $offset): self } $this->offset = $offset; + $this->statistics = null; return $this; } + /** + * The footers schema() reads - the first file's, or every file's under unionByName - read once whichever of the + * two asks first, and scaled to the listing when only the first was read. + */ + private function declare(): Statistics + { + $footers = $this->schema === null + ? $this->derivedFooters($this->files(), $this->unionByName) + : $this->footers($this->files(), $this->unionByName); + + return $footers->of(iterator_count($this->sourceFiles($this->filesystem, $this->path)), $this->offset ?? 0); + } + /** * @return Generator */ diff --git a/src/core/etl/src/Flow/Floe/FloeMerger.php b/src/core/etl/src/Flow/Floe/FloeMerger.php index 79e3177326..29b46851b5 100644 --- a/src/core/etl/src/Flow/Floe/FloeMerger.php +++ b/src/core/etl/src/Flow/Floe/FloeMerger.php @@ -215,7 +215,7 @@ private function mergeSplice(array $sources, array $layouts, ?Schema $merged, Pa $sections[] = new Section($section->offset - $copyStart + $outputStart, $section->rowCount); } - $totalRows += $footer->totalRows; + $totalRows += $footer->statistics->rows; $mergedMetadata = $mergedMetadata->merge($footer->metadata); } @@ -227,7 +227,7 @@ private function mergeSplice(array $sources, array $layouts, ?Schema $merged, Pa FloeStreamWriter::writerVersion(), $schema, $sections, - $totalRows, + new Statistics($totalRows, $frameWriter->position() - Format::HEADER_LENGTH), $mergedMetadata->merge($metadata), ))->toJson(); diff --git a/src/core/etl/src/Flow/Floe/FloeSerializer.php b/src/core/etl/src/Flow/Floe/FloeSerializer.php index bb8d79aa9d..3b7c8d0e90 100644 --- a/src/core/etl/src/Flow/Floe/FloeSerializer.php +++ b/src/core/etl/src/Flow/Floe/FloeSerializer.php @@ -63,11 +63,11 @@ public function unserialize(SourceStream $source): Rows } } - if (count($rows) !== $footer->totalRows) { + if (count($rows) !== $footer->statistics->rows) { throw new FloeException(sprintf( 'Floe payload is corrupted, decoded %d of %d rows', count($rows), - $footer->totalRows, + $footer->statistics->rows, )); } diff --git a/src/core/etl/src/Flow/Floe/FloeSourceFile.php b/src/core/etl/src/Flow/Floe/FloeSourceFile.php index 556332d696..ec5cf384e5 100644 --- a/src/core/etl/src/Flow/Floe/FloeSourceFile.php +++ b/src/core/etl/src/Flow/Floe/FloeSourceFile.php @@ -4,8 +4,10 @@ namespace Flow\Floe; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor\SelfDescribingFile; use Flow\ETL\Extractor\SourceFile; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\Schema; final readonly class FloeSourceFile implements SelfDescribingFile @@ -29,4 +31,12 @@ public function source(): SourceFile { return $this->source; } + + public function statistics(): Statistics + { + return new Statistics( + rows: Cardinality::exact($this->reader->totalRows()), + size: Cardinality::exact($this->reader->byteSize()), + ); + } } diff --git a/src/core/etl/src/Flow/Floe/FloeStreamReader.php b/src/core/etl/src/Flow/Floe/FloeStreamReader.php index 6ed8396b1f..ab3b818e96 100644 --- a/src/core/etl/src/Flow/Floe/FloeStreamReader.php +++ b/src/core/etl/src/Flow/Floe/FloeStreamReader.php @@ -373,12 +373,22 @@ public function schema(): Schema return $this->schema ??= $this->footer()->schema(); } + /** + * Uncompressed bytes of the data frames, as the footer records them. + * + * @throws FloeException + */ + public function byteSize(): int + { + return $this->footer()->statistics->byteSize; + } + /** * @throws FloeException */ public function totalRows(): int { - return $this->footer()->totalRows; + return $this->footer()->statistics->rows; } /** diff --git a/src/core/etl/src/Flow/Floe/FloeStreamWriter.php b/src/core/etl/src/Flow/Floe/FloeStreamWriter.php index 8e29e520f0..c28a22cd10 100644 --- a/src/core/etl/src/Flow/Floe/FloeStreamWriter.php +++ b/src/core/etl/src/Flow/Floe/FloeStreamWriter.php @@ -115,7 +115,7 @@ public function resume(DestinationStream $stream, Footer $footer, int $fileLengt } } - $this->totalRows = $footer->totalRows; + $this->totalRows = $footer->statistics->rows; $this->open = true; } @@ -136,7 +136,7 @@ public function close(): void self::writerVersion(), $schema, $this->sections, - $this->totalRows, + new Statistics($this->totalRows, $this->frameWriter()->position() - Format::HEADER_LENGTH), $this->metadata, ))->toJson(); diff --git a/src/core/etl/src/Flow/Floe/Footer.php b/src/core/etl/src/Flow/Floe/Footer.php index 602b1ec640..3e41cb89d5 100644 --- a/src/core/etl/src/Flow/Floe/Footer.php +++ b/src/core/etl/src/Flow/Floe/Footer.php @@ -33,7 +33,7 @@ public function __construct( public string $writer, public array $schema, public array $sections, - public int $totalRows, + public Statistics $statistics, public Metadata $metadata, ) {} @@ -69,9 +69,9 @@ public static function fromArray(array $data): self 'writer' => type_string(), 'schema' => type_array(), 'sections' => type_list(type_array()), - 'totalRows' => type_integer(), + 'statistics' => type_array(), 'metadata' => type_array(), - ])->assert($data); + ], allow_extra: true)->assert($data); } catch (InvalidTypeException $e) { throw new FloeException('Floe footer is malformed: ' . $e->getMessage(), 0, $e); } @@ -95,7 +95,14 @@ public static function fromArray(array $data): self throw new FloeException('Floe footer metadata is malformed: ' . $e->getMessage(), 0, $e); } - return new self($data['version'], $data['writer'], $schema, $sections, $data['totalRows'], $metadata); + return new self( + $data['version'], + $data['writer'], + $schema, + $sections, + Statistics::fromArray($data['statistics']), + $metadata, + ); } public function schema(): Schema @@ -114,7 +121,7 @@ public function schemaBody(): string * writer: string, * schema: array>, * sections: array, - * totalRows: int, + * statistics: array{rows: int, byteSize: int}, * metadata: array, * } */ @@ -131,7 +138,7 @@ public function normalize(): array 'writer' => $this->writer, 'schema' => $this->schema, 'sections' => $sections, - 'totalRows' => $this->totalRows, + 'statistics' => $this->statistics->normalize(), 'metadata' => $this->metadata->normalize(), ]; } diff --git a/src/core/etl/src/Flow/Floe/Section.php b/src/core/etl/src/Flow/Floe/Section.php index 808edaefc4..8e180bdb5c 100644 --- a/src/core/etl/src/Flow/Floe/Section.php +++ b/src/core/etl/src/Flow/Floe/Section.php @@ -28,7 +28,7 @@ public static function fromArray(array $data): self $data = type_structure([ 'offset' => type_integer(), 'rowCount' => type_integer(), - ])->assert($data); + ], allow_extra: true)->assert($data); } catch (InvalidTypeException $e) { throw new FloeException('Floe footer section is malformed: ' . $e->getMessage(), 0, $e); } diff --git a/src/core/etl/src/Flow/Floe/Statistics.php b/src/core/etl/src/Flow/Floe/Statistics.php new file mode 100644 index 0000000000..2ff2ae3f2d --- /dev/null +++ b/src/core/etl/src/Flow/Floe/Statistics.php @@ -0,0 +1,63 @@ + $data + * + * @throws FloeException + */ + public static function fromArray(array $data): self + { + try { + $data = type_structure([ + 'rows' => type_integer(), + 'byteSize' => type_integer(), + ], allow_extra: true)->assert($data); + } catch (InvalidTypeException $e) { + throw new FloeException('Floe footer statistics are malformed: ' . $e->getMessage(), 0, $e); + } + + return new self($data['rows'], $data['byteSize']); + } + + /** + * @return array{rows: int, byteSize: int} + */ + public function normalize(): array + { + return [ + 'rows' => $this->rows, + 'byteSize' => $this->byteSize, + ]; + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Context/MemoryFiles.php b/src/core/etl/tests/Flow/ETL/Tests/Context/MemoryFiles.php new file mode 100644 index 0000000000..357e621e12 --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Context/MemoryFiles.php @@ -0,0 +1,29 @@ + $contents file content keyed by memory:// URI + */ + public static function with(array $contents): MemoryFilesystem + { + $filesystem = memory_filesystem(); + + foreach ($contents as $uri => $content) { + $stream = $filesystem->writeTo(path($uri)); + $stream->append($content); + $stream->close(); + } + + return $filesystem; + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/CountingCache.php b/src/core/etl/tests/Flow/ETL/Tests/Double/CountingCache.php new file mode 100644 index 0000000000..a76cc15c8e --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/CountingCache.php @@ -0,0 +1,53 @@ +inner->clear(); + } + + public function delete(string $key): void + { + $this->inner->delete($key); + } + + public function get(string $key): Rows + { + $this->getCalls++; + + return $this->inner->get($key); + } + + public function has(string $key): bool + { + return $this->inner->has($key); + } + + public function schema(string $key): Schema + { + return $this->inner->schema($key); + } + + public function set(string $key, Rows $value): void + { + $this->inner->set($key, $value); + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/CountingExtractor.php b/src/core/etl/tests/Flow/ETL/Tests/Double/CountingExtractor.php index 070822d587..08010bdf56 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/CountingExtractor.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/CountingExtractor.php @@ -9,6 +9,7 @@ use Flow\ETL\Extractor\Batches; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -94,4 +95,9 @@ public function withSchema(Schema $schema): static { return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/DeclaringExtractor.php b/src/core/etl/tests/Flow/ETL/Tests/Double/DeclaringExtractor.php new file mode 100644 index 0000000000..7a70182d65 --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/DeclaringExtractor.php @@ -0,0 +1,39 @@ +schema()); + } + + public function schema(): Schema + { + return new Schema(); + } + + public function statistics(): Statistics + { + return $this->statistics; + } + + public function withSchema(Schema $schema): static + { + return $this; + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/EmptyExtractor.php b/src/core/etl/tests/Flow/ETL/Tests/Double/EmptyExtractor.php index 6292cb2b73..945f35ac3b 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/EmptyExtractor.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/EmptyExtractor.php @@ -5,6 +5,7 @@ namespace Flow\ETL\Tests\Double; use Flow\ETL\Extractor; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Schema; use Generator; @@ -28,4 +29,9 @@ public function withSchema(Schema $schema): static { return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/FakeExtractor.php b/src/core/etl/tests/Flow/ETL/Tests/Double/FakeExtractor.php index 1b1629fca2..d6e616ac4e 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/FakeExtractor.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/FakeExtractor.php @@ -7,6 +7,7 @@ use DateTimeImmutable; use Flow\ETL\Extractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -144,4 +145,9 @@ public function withSchema(Schema $schema): static { return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/FakeRandomOrdersExtractor.php b/src/core/etl/tests/Flow/ETL/Tests/Double/FakeRandomOrdersExtractor.php index 2dbc8ecf30..6d8db4a73a 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/FakeRandomOrdersExtractor.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/FakeRandomOrdersExtractor.php @@ -8,6 +8,7 @@ use Faker\Factory; use Flow\ETL\Extractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Schema; use Generator; @@ -158,4 +159,9 @@ public function withSchema(Schema $schema): static { return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/FakeSelfDescribingFile.php b/src/core/etl/tests/Flow/ETL/Tests/Double/FakeSelfDescribingFile.php index 670f07b550..bcdd55f5eb 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/FakeSelfDescribingFile.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/FakeSelfDescribingFile.php @@ -7,6 +7,7 @@ use Flow\ETL\Exception\RuntimeException; use Flow\ETL\Extractor\SelfDescribingFile; use Flow\ETL\Extractor\SourceFile; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\Schema; final class FakeSelfDescribingFile implements SelfDescribingFile @@ -17,6 +18,7 @@ public function __construct( private readonly Schema $schema, private readonly SourceFile $source, private readonly bool $describingThrows = false, + private readonly Statistics $statistics = new Statistics(), ) {} public function close(): void @@ -37,4 +39,9 @@ public function source(): SourceFile { return $this->source; } + + public function statistics(): Statistics + { + return $this->statistics; + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/FakeStaticOrdersExtractor.php b/src/core/etl/tests/Flow/ETL/Tests/Double/FakeStaticOrdersExtractor.php index 25c329f82d..5446922670 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/FakeStaticOrdersExtractor.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/FakeStaticOrdersExtractor.php @@ -7,6 +7,7 @@ use DateTimeImmutable; use Flow\ETL\Extractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Row\AdaptiveRowHydrator; use Flow\ETL\Row\Hydrator; @@ -153,4 +154,9 @@ public function withSchema(Schema $schema): static { return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/KeepPaths.php b/src/core/etl/tests/Flow/ETL/Tests/Double/KeepPaths.php new file mode 100644 index 0000000000..c55a3c59e5 --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/KeepPaths.php @@ -0,0 +1,36 @@ + + */ + private array $uris; + + /** + * @param list $paths + */ + public function __construct(array $paths) + { + $this->uris = array_map(static fn(Path $path): string => $path->uri(), $paths); + } + + public function accept(FileStatus $status): bool + { + return $status->isFile() && in_array($status->path->uri(), $this->uris, true); + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/MixedSequenceGenerator.php b/src/core/etl/tests/Flow/ETL/Tests/Double/MixedSequenceGenerator.php index 983662234d..e970fba631 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/MixedSequenceGenerator.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/MixedSequenceGenerator.php @@ -4,6 +4,7 @@ namespace Flow\ETL\Tests\Double; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor\SequenceGenerator\SequenceGenerator; use Generator; @@ -14,4 +15,9 @@ public function generate(): Generator yield 1000; yield 'AB-01'; } + + public function rows(): Cardinality + { + return Cardinality::exact(2); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/OneRowBatchesExtractor.php b/src/core/etl/tests/Flow/ETL/Tests/Double/OneRowBatchesExtractor.php index 70f6cc5142..de0daac4e3 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/OneRowBatchesExtractor.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/OneRowBatchesExtractor.php @@ -9,6 +9,7 @@ use Flow\ETL\Extractor\Batches; use Flow\ETL\Extractor\RewindableExtractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -51,4 +52,9 @@ public function withSchema(Schema $schema): static { return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/RecordingExtractor.php b/src/core/etl/tests/Flow/ETL/Tests/Double/RecordingExtractor.php index dc83c4a006..082249c8a2 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/RecordingExtractor.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/RecordingExtractor.php @@ -5,6 +5,7 @@ namespace Flow\ETL\Tests\Double; use Flow\ETL\Extractor; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -52,4 +53,9 @@ public function withSchema(Schema $schema): static { return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/RecordingFileExtractor.php b/src/core/etl/tests/Flow/ETL/Tests/Double/RecordingFileExtractor.php index a73941f15f..52cffddc9c 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/RecordingFileExtractor.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/RecordingFileExtractor.php @@ -5,6 +5,7 @@ namespace Flow\ETL\Tests\Double; use Flow\ETL\Extractor\FileExtractor; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -84,4 +85,9 @@ public function withSchema(Schema $schema): static { return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/RecordingSequenceGenerator.php b/src/core/etl/tests/Flow/ETL/Tests/Double/RecordingSequenceGenerator.php index 0b99103cef..1cfcb08c66 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/RecordingSequenceGenerator.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/RecordingSequenceGenerator.php @@ -4,16 +4,19 @@ namespace Flow\ETL\Tests\Double; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor\SequenceGenerator\SequenceGenerator; use Generator; /** - * Counts generate() calls so a test can assert how many passes a run makes over the sequence. + * Counts generate() and rows() calls so a test can assert how many passes a run makes over the sequence. */ final class RecordingSequenceGenerator implements SequenceGenerator { public int $generateCalls = 0; + public int $rowsCalls = 0; + public function __construct( private readonly SequenceGenerator $inner, ) {} @@ -24,4 +27,11 @@ public function generate(): Generator yield from $this->inner->generate(); } + + public function rows(): Cardinality + { + $this->rowsCalls++; + + return $this->inner->rows(); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/RepeatableExtractor.php b/src/core/etl/tests/Flow/ETL/Tests/Double/RepeatableExtractor.php index b01be67adc..fc488b488b 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/RepeatableExtractor.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/RepeatableExtractor.php @@ -7,6 +7,7 @@ use Flow\ETL\Extractor; use Flow\ETL\Extractor\OverridingExtractor; use Flow\ETL\Extractor\RewindableExtractor; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -59,4 +60,9 @@ public function withSchema(Schema $schema): static { return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/RowLessExtractor.php b/src/core/etl/tests/Flow/ETL/Tests/Double/RowLessExtractor.php index 3d2fed8e0e..847b90134e 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/RowLessExtractor.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/RowLessExtractor.php @@ -5,6 +5,7 @@ namespace Flow\ETL\Tests\Double; use Flow\ETL\Extractor; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Schema; use Generator; @@ -41,4 +42,9 @@ public function withSchema(Schema $schema): static return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/StopIgnoringExtractor.php b/src/core/etl/tests/Flow/ETL/Tests/Double/StopIgnoringExtractor.php index 31b9ee48e4..93bfa17393 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/StopIgnoringExtractor.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/StopIgnoringExtractor.php @@ -8,6 +8,7 @@ use Flow\ETL\Extractor\BatchableExtractor; use Flow\ETL\Extractor\Batches; use Flow\ETL\Extractor\RewindableExtractor; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -48,4 +49,9 @@ public function withSchema(Schema $schema): static { return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/ThrowingAfterFirstBatchExtractor.php b/src/core/etl/tests/Flow/ETL/Tests/Double/ThrowingAfterFirstBatchExtractor.php index ba618b08c6..b600a182b0 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/ThrowingAfterFirstBatchExtractor.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/ThrowingAfterFirstBatchExtractor.php @@ -5,6 +5,7 @@ namespace Flow\ETL\Tests\Double; use Flow\ETL\Extractor; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Schema; use Generator; @@ -33,4 +34,9 @@ public function withSchema(Schema $schema): static { return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/UndescribableRowLessExtractor.php b/src/core/etl/tests/Flow/ETL/Tests/Double/UndescribableRowLessExtractor.php index ffcdd0c264..1a74cfc87b 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/UndescribableRowLessExtractor.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/UndescribableRowLessExtractor.php @@ -6,6 +6,7 @@ use Flow\ETL\Exception\SchemaNotDerivableException; use Flow\ETL\Extractor; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Schema; use Generator; @@ -32,4 +33,9 @@ public function withSchema(Schema $schema): static { return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/UnsizedFilesystem.php b/src/core/etl/tests/Flow/ETL/Tests/Double/UnsizedFilesystem.php new file mode 100644 index 0000000000..2b9b501c95 --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/UnsizedFilesystem.php @@ -0,0 +1,85 @@ + $unsizedUris + */ + public function __construct( + private Filesystem $inner, + private array $unsizedUris, + ) {} + + public function appendTo(Path $path): DestinationStream + { + return $this->inner->appendTo($path); + } + + public function getSystemTmpDir(): Path + { + return $this->inner->getSystemTmpDir(); + } + + public function list(Path $path, Filter $pathFilter = new KeepAll()): Generator + { + foreach ($this->inner->list($path, $pathFilter) as $status) { + yield in_array($status->path->uri(), $this->unsizedUris, true) + ? new FileStatus($status->path, $status->isFile(), null, $status->lastModifiedAt) + : $status; + } + } + + public function mount(): Mount + { + return $this->inner->mount(); + } + + public function mv(Path $from, Path $to): bool + { + return $this->inner->mv($from, $to); + } + + public function readFrom(Path $path): SourceStream + { + return $this->inner->readFrom($path); + } + + public function rm(Path $path): bool + { + return $this->inner->rm($path); + } + + public function status(Path $path): ?FileStatus + { + return $this->inner->status($path); + } + + public function supports(Path $path): bool + { + return $this->inner->supports($path); + } + + public function writeTo(Path $path): DestinationStream + { + return $this->inner->writeTo($path); + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Double/VaryingBatchesExtractor.php b/src/core/etl/tests/Flow/ETL/Tests/Double/VaryingBatchesExtractor.php index 90bc7b0f4a..2b73216114 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Double/VaryingBatchesExtractor.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Double/VaryingBatchesExtractor.php @@ -5,6 +5,7 @@ namespace Flow\ETL\Tests\Double; use Flow\ETL\Extractor; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -42,4 +43,9 @@ public function withSchema(Schema $schema): static { return $this; } + + public function statistics(): Statistics + { + return new Statistics(); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/AnalyzeTest.php b/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/AnalyzeTest.php index 742c6c1c55..9b149c6ff1 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/AnalyzeTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/AnalyzeTest.php @@ -6,10 +6,14 @@ use DateTimeImmutable; use Flow\Clock\FakeClock; +use Flow\ETL\Cardinality; use Flow\ETL\Dataset\Report; +use Flow\ETL\Dataset\SourceStatistics; use Flow\ETL\Dataset\Statistics\Columns; use Flow\ETL\Dataset\Statistics\HighResolutionTime; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; +use Flow\ETL\Join\Join; use Flow\ETL\Rows; use Flow\ETL\Tests\Double\InlineLoader; use Flow\ETL\Tests\FlowIntegrationTestCase; @@ -22,6 +26,9 @@ use function Flow\ETL\DSL\from_array; use function Flow\ETL\DSL\infer_schema; use function Flow\ETL\DSL\int_schema; +use function Flow\ETL\DSL\join_on; +use function Flow\ETL\DSL\lit; +use function Flow\ETL\DSL\ref; use function Flow\ETL\DSL\schema; use function Flow\ETL\DSL\str_schema; @@ -169,6 +176,46 @@ public function test_analyzing_partitioned_datasets(): void ); } + public function test_analyzing_source_statistics_counts_every_source_apart_from_the_output(): void + { + $report = df() + ->read(from_array([['id' => 1], ['id' => 2], ['id' => 3], ['id' => 4]])) + ->filter(ref('id')->greaterThan(lit(2))) + ->join( + df()->read(from_array([['id' => 3], ['id' => 4], ['id' => 5]])), + join_on(['id' => 'id']), + Join::inner, + ) + ->run(analyze: analyze()->withSourceStatistics()); + + static::assertNotNull($report); + static::assertSame(2, $report->statistics()->totalRows()); + static::assertEquals( + [ + new SourceStatistics('ArrayExtractor', new Statistics(Cardinality::exact(3)), 3, true), + new SourceStatistics('ArrayExtractor', new Statistics(Cardinality::exact(4)), 4, true), + ], + $report->sources(), + ); + } + + public function test_a_source_read_under_a_pushed_limit_reports_no_rows_error(): void + { + $report = df() + ->read(from_array([['id' => 1], ['id' => 2], ['id' => 3], ['id' => 4]])) + ->limit(2) + ->run(analyze: analyze()->withSourceStatistics()); + + static::assertNotNull($report); + $sources = $report->sources(); + static::assertNotNull($sources); + static::assertEquals( + [new SourceStatistics('ArrayExtractor', new Statistics(Cardinality::exact(4)), 4, false)], + $sources, + ); + static::assertNull($sources[0]->rowsError()); + } + public function test_run_explicit_analyze_overrides_config(): void { $config = config_builder()->analyze(analyze())->build(); diff --git a/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/CacheTest.php b/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/CacheTest.php index d146e90064..96db55cdaa 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/CacheTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/CacheTest.php @@ -8,6 +8,7 @@ use Flow\ETL\Cache\Implementation\FilesystemCache; use Flow\ETL\Cache\Implementation\InMemoryCache; use Flow\ETL\Extractor; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -67,6 +68,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator return $this->extractor->extract($context); } + + public function statistics(): Statistics + { + return new Statistics(); + } }; $cache = new InMemoryCache(); diff --git a/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/DisplayTest.php b/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/DisplayTest.php index 336ee7db3c..e362c2e8d8 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/DisplayTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/DisplayTest.php @@ -7,6 +7,7 @@ use DateTimeImmutable; use Flow\ETL\Extractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -111,6 +112,11 @@ enum_schema('enum', BackedStringEnum::class), ); } } + + public function statistics(): Statistics + { + return new Statistics(); + } })->collect(); self::assertCommandOutputIdentical(<<<'ASCIITABLE' @@ -191,6 +197,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator ); } } + + public function statistics(): Statistics + { + return new Statistics(); + } }) ->collect() ->repartition(ref('group')); diff --git a/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/LimitTest.php b/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/LimitTest.php index 1336941293..4d3a548d51 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/LimitTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/LimitTest.php @@ -7,6 +7,7 @@ use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Extractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Optimizer; use Flow\ETL\Planner; @@ -105,6 +106,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(integer_schema('id')), row(['id' => $i])); } } + + public function statistics(): Statistics + { + return new Statistics(); + } })->fetch(); static::assertCount(20, $rows); @@ -135,6 +141,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(integer_schema('id')), row(['id' => $i + 1]), row(['id' => $i + 2])); } } + + public function statistics(): Statistics + { + return new Statistics(); + } }) ->limit(10) ->fetch(); @@ -206,6 +217,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator ); } } + + public function statistics(): Statistics + { + return new Statistics(); + } }) ->withEntries([ 'expanded' => ref('ids')->expand(), @@ -248,6 +264,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(integer_schema('id')), row(['id' => $i + 1]), row(['id' => $i + 2])); } } + + public function statistics(): Statistics + { + return new Statistics(); + } }) ->batchSize(50) ->limit(10) @@ -281,6 +302,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(integer_schema('id')), row(['id' => $i + 1]), row(['id' => $i + 2])); } } + + public function statistics(): Statistics + { + return new Statistics(); + } }) ->limit(10) ->collect() @@ -314,6 +340,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(integer_schema('id')), row(['id' => $i])); } } + + public function statistics(): Statistics + { + return new Statistics(); + } }) ->limit(10) ->fetch(); diff --git a/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/OffsetTest.php b/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/OffsetTest.php index a33bf10584..e9fb7349ff 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/OffsetTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Integration/DataFrame/OffsetTest.php @@ -7,6 +7,7 @@ use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Extractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Rows; use Flow\ETL\Schema; @@ -141,6 +142,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(integer_schema('id')), row(['id' => $i + 1])); } } + + public function statistics(): Statistics + { + return new Statistics(); + } }) ->batchSize(3) ->offset(4) @@ -184,6 +190,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(integer_schema('id')), row(['id' => $i + 1])); } } + + public function statistics(): Statistics + { + return new Statistics(); + } }) ->offset(2) ->collect() @@ -243,6 +254,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator ); } } + + public function statistics(): Statistics + { + return new Statistics(); + } }) ->withEntries([ 'expanded' => ref('ids')->expand(), @@ -306,6 +322,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(integer_schema('id')), row(['id' => $i + 1])); } } + + public function statistics(): Statistics + { + return new Statistics(); + } }) ->offset(4) ->fetch(); diff --git a/src/core/etl/tests/Flow/ETL/Tests/Integration/Extractor/CacheExtractorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Integration/Extractor/CacheExtractorTest.php index dc5362b4f8..d4b0ce9658 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Integration/Extractor/CacheExtractorTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Integration/Extractor/CacheExtractorTest.php @@ -7,9 +7,13 @@ use Flow\ETL\Cache\CacheIndex; use Flow\ETL\Cache\Implementation\FilesystemCache; use Flow\ETL\Cache\Implementation\InMemoryCache; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor\Signal; use Flow\ETL\Rows; +use Flow\ETL\Tests\Double\CountingCache; +use Flow\ETL\Tests\Double\CountingExtractor; use Flow\ETL\Tests\FlowIntegrationTestCase; +use Flow\ETL\Tests\Mother\RowsMother; use function array_map; use function array_merge; @@ -28,6 +32,76 @@ final class CacheExtractorTest extends FlowIntegrationTestCase { + public function test_a_cached_index_declares_its_rows_exactly(): void + { + $cache = new InMemoryCache(); + df(config_builder()->cache($cache)) + ->read(from_array([['id' => 1], ['id' => 2], ['id' => 3]])) + ->cache('cached') + ->run(); + + $extractor = from_cache('cached', cache: $cache); + + static::assertEquals(Cardinality::exact(3), $extractor->statistics()->rows); + static::assertSame($extractor->statistics(), $extractor->statistics()); + } + + public function test_schema_and_statistics_load_the_index_once(): void + { + $cache = new CountingCache(new InMemoryCache()); + df(config_builder()->cache($cache)) + ->read(from_array([['id' => 1], ['id' => 2]])) + ->cache('cached') + ->run(); + $loads = $cache->getCalls; + + $extractor = from_cache('cached', cache: $cache); + $extractor->schema(); + $extractor->statistics(); + + static::assertSame($loads + 1, $cache->getCalls); + } + + public function test_a_cache_from_the_context_declares_nothing(): void + { + static::assertEquals( + Cardinality::unknown(), + from_cache('cached', from_array([['id' => 1]]))->statistics()->rows, + ); + } + + public function test_a_missing_entry_without_a_fallback_declares_zero_rows(): void + { + static::assertEquals( + Cardinality::exact(0), + from_cache('missing', cache: new InMemoryCache())->statistics()->rows, + ); + } + + public function test_a_missing_entry_declares_what_the_fallback_declares(): void + { + static::assertEquals( + Cardinality::exact(2), + from_cache( + 'missing', + from_array([['id' => 1], ['id' => 2]]), + cache: new InMemoryCache(), + )->statistics()->rows, + ); + } + + public function test_a_missing_entry_with_an_undeclaring_fallback_declares_nothing(): void + { + static::assertEquals( + Cardinality::unknown(), + from_cache( + 'missing', + new CountingExtractor(schema(int_schema('id')), RowsMother::sequentialIds(2)), + cache: new InMemoryCache(), + )->statistics()->rows, + ); + } + public function test_extracting_rows_from_cache(): void { $cache = new InMemoryCache(); diff --git a/src/core/etl/tests/Flow/ETL/Tests/Integration/Extractor/PathPartitionsExtractorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Integration/Extractor/PathPartitionsExtractorTest.php index a0d7edd552..138eac2586 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Integration/Extractor/PathPartitionsExtractorTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Integration/Extractor/PathPartitionsExtractorTest.php @@ -4,9 +4,12 @@ namespace Flow\ETL\Tests\Integration\Extractor; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor\PathPartitionsExtractor; use Flow\ETL\Rows; use Flow\ETL\Tests\Context\ExtractedRows; +use Flow\ETL\Tests\Context\MemoryFiles; +use Flow\ETL\Tests\Double\CountingFilesystem; use Flow\ETL\Tests\FlowIntegrationTestCase; use Flow\Filesystem\Tests\OperatingSystem; @@ -27,6 +30,35 @@ final class PathPartitionsExtractorTest extends FlowIntegrationTestCase { use OperatingSystem; + public function test_it_declares_one_row_per_listed_file(): void + { + $statistics = from_path_partitions('memory://dir/**/*', MemoryFiles::with([ + 'memory://dir/year=2024/a.txt' => 'a', + 'memory://dir/year=2024/b.txt' => 'b', + 'memory://dir/year=2025/c.txt' => 'c', + ]))->statistics(); + + static::assertEquals(Cardinality::exact(3), $statistics->rows); + static::assertEquals(Cardinality::unknown(), $statistics->size); + } + + public function test_an_empty_listing_declares_zero_rows(): void + { + static::assertEquals( + Cardinality::exact(0), + from_path_partitions('memory://dir/**/*', MemoryFiles::with([]))->statistics()->rows, + ); + } + + public function test_the_listing_is_read_once(): void + { + $filesystem = new CountingFilesystem(MemoryFiles::with(['memory://dir/year=2024/a.txt' => 'a'])); + $extractor = from_path_partitions('memory://dir/**/*', $filesystem); + + static::assertSame($extractor->statistics(), $extractor->statistics()); + static::assertSame(1, $filesystem->listCalls); + } + public function test_partition_directories_are_declared_as_string_columns_next_to_the_map(): void { $extractor = from_path_partitions(path_real(__DIR__ . '/Fixtures/multi_partitioned/**/*')); diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Cache/CacheIndexTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Cache/CacheIndexTest.php index af9781ceaa..7f36e3845c 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Cache/CacheIndexTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Cache/CacheIndexTest.php @@ -5,6 +5,7 @@ namespace Flow\ETL\Tests\Unit\Cache; use Flow\ETL\Cache\CacheIndex; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Tests\FlowTestCase; @@ -26,6 +27,28 @@ public function test_empty_index_round_trip(): void static::assertEquals($index, CacheIndex::fromRows('dataset-id', $indexRows)); } + public function test_an_empty_index_declares_zero_rows_exactly(): void + { + static::assertEquals(Cardinality::exact(0), (new CacheIndex('dataset-id'))->rows()); + } + + public function test_an_index_without_a_rows_column_declares_unknown_rows(): void + { + $index = CacheIndex::fromRows('dataset-id', rows(schema(str_schema('key')), row(['key' => 'chunk-1']))); + + static::assertSame(['chunk-1'], $index->values()); + static::assertEquals(Cardinality::unknown(), $index->rows()); + } + + public function test_a_chunk_without_a_count_makes_the_rows_unknown(): void + { + $index = new CacheIndex('dataset-id'); + $index->add('chunk-1', 3); + $index->add('chunk-2'); + + static::assertEquals(Cardinality::unknown(), $index->rows()); + } + public function test_from_rows_key_comes_from_the_argument(): void { $index = new CacheIndex('original-key'); @@ -52,6 +75,27 @@ public function test_from_rows_with_non_string_key_entry_throws(): void CacheIndex::fromRows('dataset-id', rows(schema(str_schema('key', nullable: true)), row(['key' => null]))); } + public function test_from_rows_with_non_integer_rows_entry_throws(): void + { + $this->expectException(InvalidArgumentException::class); + $this->expectExceptionMessage('CacheIndex expects rows with an integer "rows" entry, got: string'); + + CacheIndex::fromRows('dataset-id', rows( + schema(str_schema('key'), str_schema('rows')), + row(['key' => 'chunk-1', 'rows' => 'three']), + )); + } + + public function test_rows_sum_the_chunk_counts(): void + { + $index = new CacheIndex('dataset-id'); + $index->add('chunk-1', 3); + $index->add('chunk-2', 0); + $index->add('chunk-3', 4); + + static::assertEquals(Cardinality::exact(7), CacheIndex::fromRows('dataset-id', $index->toRows())->rows()); + } + public function test_to_rows_from_rows_round_trip_preserves_order(): void { $index = new CacheIndex('dataset-id'); @@ -65,14 +109,18 @@ public function test_to_rows_from_rows_round_trip_preserves_order(): void static::assertSame(['chunk-b', 'chunk-a', 'chunk-c'], $reconstructed->values()); } - public function test_to_rows_stores_one_key_entry_per_row(): void + public function test_to_rows_stores_one_key_and_rows_entry_per_chunk(): void { $index = new CacheIndex('dataset-id'); - $index->add('chunk-1'); + $index->add('chunk-1', 2); $index->add('chunk-2'); static::assertEquals( - rows(schema(str_schema('key')), row(['key' => 'chunk-1']), row(['key' => 'chunk-2'])), + rows( + schema(str_schema('key'), int_schema('rows', nullable: true)), + row(['key' => 'chunk-1', 'rows' => 2]), + row(['key' => 'chunk-2', 'rows' => null]), + ), $index->toRows(), ); } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/CardinalityTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/CardinalityTest.php new file mode 100644 index 0000000000..79b74667ce --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/CardinalityTest.php @@ -0,0 +1,168 @@ +merge(Cardinality::exact(20))->atMost); + static::assertNull(Cardinality::atMost(10)->merge(Cardinality::approximately(20))->atMost); + } + + public function test_a_negative_bound_is_rejected(): void + { + $this->expectException(InvalidArgumentException::class); + $this->expectExceptionMessage('Cardinality upper bound must not be negative, given: -1'); + + new Cardinality(atMost: -1); + } + + public function test_a_negative_estimate_is_rejected(): void + { + $this->expectException(InvalidArgumentException::class); + $this->expectExceptionMessage('Cardinality estimate must not be negative, given: -1'); + + new Cardinality(estimate: -1); + } + + public function test_a_negative_relative_error_is_rejected(): void + { + $this->expectException(InvalidArgumentException::class); + $this->expectExceptionMessage('Relative error must not be negative, given: -0.1'); + + new Cardinality(relativeError: -0.1); + } + + public function test_an_estimate_survives_only_if_both_sides_have_one(): void + { + static::assertSame(30, Cardinality::approximately(10)->merge(Cardinality::exact(20))->estimate); + static::assertNull(Cardinality::approximately(10)->merge(Cardinality::atMost(20))->estimate); + } + + public function test_approximately_is_an_estimate_without_a_bound(): void + { + $cardinality = Cardinality::approximately(16820); + + static::assertNull($cardinality->atMost); + static::assertSame(16820, $cardinality->estimate); + static::assertSame(Cardinality::DEFAULT_RELATIVE_ERROR, $cardinality->relativeError); + } + + public function test_approximately_is_confident_within_its_declared_error(): void + { + static::assertSame(16820, Cardinality::approximately(16820, 0.5)->confident(0.5)); + } + + public function test_approximately_is_not_confident_below_its_declared_error(): void + { + static::assertNull(Cardinality::approximately(16820, 0.5)->confident(0.1)); + } + + public function test_at_most_is_a_bound_without_an_estimate(): void + { + $cardinality = Cardinality::atMost(500); + + static::assertSame(500, $cardinality->atMost); + static::assertNull($cardinality->estimate); + } + + public function test_at_most_is_never_confident(): void + { + static::assertNull(Cardinality::atMost(500)->confident(1.0)); + } + + public function test_exact_is_both_a_bound_and_an_estimate(): void + { + $cardinality = Cardinality::exact(1000); + + static::assertSame(1000, $cardinality->atMost); + static::assertSame(1000, $cardinality->estimate); + static::assertSame(0.0, $cardinality->relativeError); + } + + public function test_exact_is_confident_at_zero_error(): void + { + static::assertSame(1000, Cardinality::exact(1000)->confident(0.0)); + } + + public function test_the_sum_carries_the_larger_relative_error(): void + { + static::assertSame( + 0.3, + Cardinality::approximately(10, 0.1)->merge(Cardinality::approximately(20, 0.3))->relativeError, + ); + static::assertSame( + 0.3, + Cardinality::approximately(10, 0.3)->merge(Cardinality::approximately(20, 0.1))->relativeError, + ); + } + + public function test_two_exact_counts_add_up_to_an_exact_count(): void + { + static::assertEquals(Cardinality::exact(30), Cardinality::exact(10)->merge(Cardinality::exact(20))); + } + + public function test_an_exact_count_reads_back_exactly(): void + { + static::assertSame(10, Cardinality::exact(10)->exactly()); + static::assertSame(0, Cardinality::exact(0)->exactly()); + } + + public static function inexact_counts(): Generator + { + yield 'approximate' => [Cardinality::approximately(10)]; + yield 'approximate with no error, no bound' => [Cardinality::approximately(10, 0.0)]; + yield 'estimate equal to its bound, with an error' => [new Cardinality(10, 10)]; + yield 'estimate under its bound, no error' => [new Cardinality(10, 8, 0.0)]; + yield 'upper bound alone' => [Cardinality::atMost(10)]; + yield 'unknown' => [Cardinality::unknown()]; + } + + #[DataProvider('inexact_counts')] + public function test_a_count_that_is_not_exact_reads_back_as_null(Cardinality $cardinality): void + { + static::assertNull($cardinality->exactly()); + } + + public function test_only_a_count_with_neither_bound_nor_estimate_is_unknown(): void + { + static::assertTrue(Cardinality::unknown()->isUnknown()); + static::assertFalse(Cardinality::atMost(10)->isUnknown()); + static::assertFalse(Cardinality::approximately(10)->isUnknown()); + static::assertFalse(Cardinality::exact(0)->isUnknown()); + } + + public function test_unknown_knows_nothing(): void + { + $cardinality = Cardinality::unknown(); + + static::assertNull($cardinality->atMost); + static::assertNull($cardinality->estimate); + static::assertNull($cardinality->confident(1.0)); + } + + public function test_unknown_merged_with_anything_is_unknown(): void + { + static::assertEquals(Cardinality::unknown(), Cardinality::unknown()->merge(Cardinality::exact(10))); + static::assertEquals(Cardinality::unknown(), Cardinality::exact(10)->merge(Cardinality::unknown())); + } + + public function test_a_validated_count_reads_back_as_a_non_negative_int(): void + { + // str_repeat() takes int<0, max>: the analyzer accepts these properties only because the guard narrowed them + static::assertSame(5, strlen(str_repeat('x', Cardinality::atMost(5)->atMost ?? 0))); + static::assertSame(7, strlen(str_repeat('x', Cardinality::approximately(7)->estimate ?? 0))); + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Config/Telemetry/TelemetryContextTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Config/Telemetry/TelemetryContextTest.php index cd5e9c28b8..64da37b0d7 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Config/Telemetry/TelemetryContextTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Config/Telemetry/TelemetryContextTest.php @@ -10,6 +10,7 @@ use Flow\ETL\Loader\StreamLoader; use Flow\ETL\Optimizer\Rule\CombineLimits; use Flow\ETL\Optimizer\Rule\CombineSortAndLimit; +use Flow\ETL\Optimizer\Rule\CountFromStatistics; use Flow\ETL\Optimizer\Rule\PushFilterIntoSource; use Flow\ETL\Optimizer\Rule\PushLimitIntoSource; use Flow\ETL\Tests\Context\MemoryTelemetryContext; @@ -970,7 +971,13 @@ public function test_optimizer_rules_are_logged(): void static::assertCount(1, $debugLogs); static::assertSame( - [CombineLimits::class, CombineSortAndLimit::class, PushLimitIntoSource::class, PushFilterIntoSource::class], + [ + CombineLimits::class, + CombineSortAndLimit::class, + PushLimitIntoSource::class, + PushFilterIntoSource::class, + CountFromStatistics::class, + ], $debugLogs[0]->record->attributes->get('optimizer_rules'), ); } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/DataFrameTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/DataFrameTest.php index d9457f7952..dc403af5a5 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/DataFrameTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/DataFrameTest.php @@ -13,6 +13,7 @@ use Flow\ETL\Exception\InvalidLogicException; use Flow\ETL\Extractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Loader; use Flow\ETL\Memory\ArrayMemory; @@ -147,6 +148,41 @@ public function test_count(): void static::assertSame(5, $count); } + public function test_count_of_a_frame_that_yields_no_rows_is_zero(): void + { + static::assertSame( + 0, + df() + ->read(from_array([['id' => 1], ['id' => 2]])) + ->filter(lit(false)) + ->count(), + ); + } + + public function test_count_counts_the_rows_left_after_a_filter(): void + { + static::assertSame( + 2, + df() + ->read(from_array([['id' => 1], ['id' => 2], ['id' => 3]])) + ->filter(ref('id')->greaterThan(lit(1))) + ->count(), + ); + } + + public function test_count_still_writes_the_frames_sinks(): void + { + $memory = new ArrayMemory(); + + $count = df() + ->read(from_array([['id' => 1], ['id' => 2]])) + ->write(to_memory($memory)) + ->count(); + + static::assertSame(2, $count); + static::assertSame([['id' => 1], ['id' => 2]], $memory->dump()); + } + public function test_drop(): void { $rows = df() @@ -263,6 +299,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(integer_schema('id')), row(['id' => $i])); } } + + public function statistics(): Statistics + { + return new Statistics(); + } }) ->filter(ref('id')->mod(lit(2))->same(lit(0))) ->fetch(); @@ -413,6 +454,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(integer_schema('id')), row(['id' => $i])); } } + + public function statistics(): Statistics + { + return new Statistics(); + } }) ->withEntry('odd', ref('id')->mod(lit(2))->equals(lit(0))) ->fetch(); @@ -490,6 +536,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator ]), ); } + + public function statistics(): Statistics + { + return new Statistics(); + } }; $addStampStringEntry = new class implements Transformer { @@ -772,6 +823,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator row(['id' => 10]), ); } + + public function statistics(): Statistics + { + return new Statistics(); + } }) ->with(new class implements Transformer { public function bind(Schema $input): BoundStep @@ -825,6 +881,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(integer_schema('id')), row(['id' => 2])); yield rows(schema(integer_schema('id')), row(['id' => 3])); } + + public function statistics(): Statistics + { + return new Statistics(); + } }) ->with(new class implements Transformer { public function bind(Schema $input): BoundStep diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Dataset/SourceStatisticsTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Dataset/SourceStatisticsTest.php new file mode 100644 index 0000000000..74fbfb53c0 --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Dataset/SourceStatisticsTest.php @@ -0,0 +1,79 @@ +rowsError(), + 0.000_001, + ); + } + + public function test_an_exact_declaration_that_holds_has_no_error(): void + { + static::assertSame( + 0.0, + (new SourceStatistics('Source', new Statistics(Cardinality::exact(5)), 5, true))->rowsError(), + ); + } + + public function test_an_incomplete_read_has_no_error(): void + { + static::assertNull( + (new SourceStatistics('Source', new Statistics(Cardinality::exact(100)), 10, false))->rowsError(), + ); + } + + public function test_there_is_no_error_without_declared_rows(): void + { + static::assertNull((new SourceStatistics('Source', new Statistics(), 5, true))->rowsError()); + } + + public function test_an_upper_bound_alone_has_no_error(): void + { + static::assertNull( + (new SourceStatistics('Source', new Statistics(Cardinality::atMost(10)), 5, true))->rowsError(), + ); + } + + public function test_an_estimate_of_zero_against_zero_rows_has_no_error(): void + { + static::assertSame( + 0.0, + (new SourceStatistics('Source', new Statistics(Cardinality::approximately(0)), 0, true))->rowsError(), + ); + } + + public function test_a_positive_estimate_against_zero_rows_is_infinitely_off(): void + { + static::assertSame( + INF, + (new SourceStatistics('Source', new Statistics(Cardinality::approximately(500)), 0, true))->rowsError(), + ); + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/ETLErrorHandlingTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/ETLErrorHandlingTest.php index e53d89341a..f5ae242a1b 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/ETLErrorHandlingTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/ETLErrorHandlingTest.php @@ -8,6 +8,7 @@ use Flow\ETL\BoundStep; use Flow\ETL\Extractor; use Flow\ETL\Extractor\Signal; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Loader; use Flow\ETL\Rows; @@ -88,6 +89,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator 'phase' => null, ])); } + + public function statistics(): Statistics + { + return new Statistics(); + } }; $brokenTransformer = new class implements Transformer { @@ -164,6 +170,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator 'phase' => null, ])); } + + public function statistics(): Statistics + { + return new Statistics(); + } }; $brokenTransformer = new class implements Transformer { @@ -262,6 +273,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator 'phase' => null, ])); } + + public function statistics(): Statistics + { + return new Statistics(); + } }; $brokenTransformer = new class implements Transformer { diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/OffsetPipelineTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/OffsetPipelineTest.php index 55911d589f..6e3e8cafac 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/OffsetPipelineTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/OffsetPipelineTest.php @@ -7,6 +7,7 @@ use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Executor\Segments; use Flow\ETL\Extractor; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; use Flow\ETL\Processor\OffsetProcessor; use Flow\ETL\Schema; @@ -100,6 +101,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(int_schema('id')), row(['id' => 3]), row(['id' => 4])); yield rows(schema(int_schema('id')), row(['id' => 5]), row(['id' => 6])); } + + public function statistics(): Statistics + { + return new Statistics(); + } }); $segments->add(new OffsetProcessor(4)); $result = iterator_to_array(ExecutedSegments::of($segments, flow_context(config()))); @@ -126,6 +132,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(int_schema('id')), row(['id' => 3]), row(['id' => 4]), row(['id' => 5])); yield rows(schema(int_schema('id')), row(['id' => 6])); } + + public function statistics(): Statistics + { + return new Statistics(); + } }); $segments->add(new OffsetProcessor(3)); $result = iterator_to_array(ExecutedSegments::of($segments, flow_context(config()))); @@ -152,6 +163,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(int_schema('id')), row(['id' => 1]), row(['id' => 2]), row(['id' => 3])); yield rows(schema(int_schema('id')), row(['id' => 4]), row(['id' => 5])); } + + public function statistics(): Statistics + { + return new Statistics(); + } }); $segments->add(new OffsetProcessor(1)); $result = iterator_to_array(ExecutedSegments::of($segments, flow_context(config()))); @@ -200,6 +216,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema()); yield rows(schema(int_schema('id')), row(['id' => 3])); } + + public function statistics(): Statistics + { + return new Statistics(); + } }); $segments->add(new OffsetProcessor(2)); $result = iterator_to_array(ExecutedSegments::of($segments, flow_context(config()))); diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/PipelineTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/PipelineTest.php index 2a913c2d1a..c7da1b6421 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/PipelineTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/PipelineTest.php @@ -6,6 +6,7 @@ use Flow\ETL\Executor\Pipeline; use Flow\ETL\Executor\Segments; +use Flow\ETL\Executor\SourceRows; use Flow\ETL\Tests\FlowTestCase; use Flow\ETL\Tests\Mother\NodeMother; use Flow\Filesystem\Path\Filter\OnlyFiles; @@ -22,7 +23,9 @@ public function test_accessors_return_the_values_it_was_built_with(): void $input = new Pipeline(0, new Segments(from_array([['id' => 1]])), $context); $filter = new RejectingFilter(); - $pipeline = new Pipeline(1, $segments, $context, $input, 3, $filter); + $sources = new SourceRows(); + + $pipeline = new Pipeline(1, $segments, $context, $input, 3, $filter, $sources); static::assertSame(1, $pipeline->id); static::assertSame($segments, $pipeline->segments()); @@ -30,6 +33,7 @@ public function test_accessors_return_the_values_it_was_built_with(): void static::assertSame($input, $pipeline->input()); static::assertSame(3, $pipeline->limit()); static::assertSame($filter, $pipeline->pathFilter()); + static::assertSame($sources, $pipeline->sources()); } public function test_a_leaf_pipeline_defaults_to_no_input_no_limit_and_only_files(): void @@ -39,5 +43,6 @@ public function test_a_leaf_pipeline_defaults_to_no_input_no_limit_and_only_file static::assertNull($pipeline->input()); static::assertNull($pipeline->limit()); static::assertInstanceOf(OnlyFiles::class, $pipeline->pathFilter()); + static::assertNull($pipeline->sources()); } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/SourceReadTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/SourceReadTest.php new file mode 100644 index 0000000000..a21771c626 --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/SourceReadTest.php @@ -0,0 +1,63 @@ +rows()); + static::assertTrue($read->isComplete()); + } + + public function test_an_open_read_is_not_complete_until_it_closes(): void + { + $read = new SourceRead(); + $read->opened(false); + $read->counted(3); + + static::assertFalse($read->isComplete()); + + $read->closed(); + + static::assertSame(3, $read->rows()); + static::assertTrue($read->isComplete()); + } + + public function test_every_read_must_close(): void + { + $read = new SourceRead(); + $read->opened(false); + $read->opened(false); + $read->closed(); + + static::assertFalse($read->isComplete()); + } + + public function test_a_narrowed_read_never_makes_the_source_complete(): void + { + $read = new SourceRead(); + $read->opened(true); + $read->closed(); + $read->opened(false); + $read->closed(); + + static::assertFalse($read->isComplete()); + } + + public function test_counts_add_up_across_reads(): void + { + $read = new SourceRead(); + $read->counted(2); + $read->counted(5); + + static::assertSame(7, $read->rows()); + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/SourceRowsTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/SourceRowsTest.php new file mode 100644 index 0000000000..59bc66e0d8 --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/SourceRowsTest.php @@ -0,0 +1,141 @@ + 1], ['id' => 2], ['id' => 3]])->withBatchSize(2); + $sources = new SourceRows(); + + $batches = iterator_to_array($sources->count($extractor, $extractor->extract(flow_context()), false), false); + + static::assertEquals(iterator_to_array($extractor->extract(flow_context()), false), $batches); + static::assertEquals( + [new SourceStatistics('ArrayExtractor', new Statistics(Cardinality::exact(3)), 3, true)], + $sources->statistics(), + ); + } + + public function test_a_stop_reaches_the_source_and_ends_the_count(): void + { + $extractor = new CountingExtractor( + schema(int_schema('id')), + rows(schema(int_schema('id')), row(['id' => 1]), row(['id' => 2]), row(['id' => 3])), + ); + $extractor->withBatchSize(1); + $sources = new SourceRows(); + $counted = $sources->count($extractor, $extractor->extract(flow_context()), false); + + static::assertCount(1, $counted->current()); + $counted->send(Signal::STOP); + + static::assertFalse($counted->valid()); + static::assertSame(1, $extractor->batchesYielded); + static::assertSame(1, $sources->statistics()[0]->rows); + static::assertFalse($sources->statistics()[0]->complete); + } + + public function test_a_narrowed_read_is_not_complete(): void + { + $extractor = from_array([['id' => 1], ['id' => 2]]); + $sources = new SourceRows(); + + iterator_to_array($sources->count($extractor, $extractor->extract(flow_context()), true)); + + static::assertSame(2, $sources->statistics()[0]->rows); + static::assertFalse($sources->statistics()[0]->complete); + } + + public function test_one_narrowed_read_makes_the_source_incomplete(): void + { + $extractor = from_array([['id' => 1]]); + $sources = new SourceRows(); + + iterator_to_array($sources->count($extractor, $extractor->extract(flow_context()), false)); + iterator_to_array($sources->count($extractor, $extractor->extract(flow_context()), true)); + + static::assertFalse($sources->statistics()[0]->complete); + } + + public function test_interleaved_reads_of_one_source_add_up(): void + { + $extractor = from_array([['id' => 1], ['id' => 2]])->withBatchSize(1); + $sources = new SourceRows(); + $first = $sources->count($extractor, $extractor->extract(flow_context()), false); + $second = $sources->count($extractor, $extractor->extract(flow_context()), false); + + $first->current(); + iterator_to_array($second); + static::assertFalse($sources->statistics()[0]->complete); + + $first->next(); + $first->next(); + + static::assertSame(4, $sources->statistics()[0]->rows); + static::assertTrue($sources->statistics()[0]->complete); + } + + public function test_a_source_read_twice_is_listed_once_with_both_counts(): void + { + $extractor = from_array([['id' => 1], ['id' => 2]]); + $sources = new SourceRows(); + + iterator_to_array($sources->count($extractor, $extractor->extract(flow_context()), false)); + iterator_to_array($sources->count($extractor, $extractor->extract(flow_context()), false)); + + static::assertCount(1, $sources->statistics()); + static::assertSame(4, $sources->statistics()[0]->rows); + } + + public function test_sources_are_listed_in_the_order_they_were_first_read(): void + { + $first = from_array([['id' => 1]]); + $second = from_array([['id' => 1], ['id' => 2]]); + $sources = new SourceRows(); + + iterator_to_array($sources->count($second, $second->extract(flow_context()), false)); + iterator_to_array($sources->count($first, $first->extract(flow_context()), false)); + + static::assertSame([2, 1], [$sources->statistics()[0]->rows, $sources->statistics()[1]->rows]); + } + + public function test_a_source_that_yields_nothing_is_listed_with_zero_rows(): void + { + $extractor = from_array([]); + $sources = new SourceRows(); + + iterator_to_array($sources->count($extractor, $extractor->extract(flow_context()), false)); + + static::assertSame(0, $sources->statistics()[0]->rows); + } + + public function test_a_source_never_read_is_not_listed(): void + { + $extractor = from_array([['id' => 1]]); + $sources = new SourceRows(); + + $sources->count($extractor, $extractor->extract(flow_context()), false); + + static::assertSame([], $sources->statistics()); + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/StatisticsCollectorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/StatisticsCollectorTest.php index 684fdc62a3..9c466dac8a 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/StatisticsCollectorTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Executor/StatisticsCollectorTest.php @@ -6,20 +6,25 @@ use DateTimeImmutable; use Flow\Clock\FakeClock; +use Flow\ETL\Cardinality; use Flow\ETL\Config\ConfigBuilder; use Flow\ETL\Dataset\Report; +use Flow\ETL\Dataset\SourceStatistics; use Flow\ETL\Dataset\Statistics\Columns; use Flow\ETL\Dataset\Statistics\HighResolutionTime; use Flow\ETL\Executor\StatisticsCollector; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\Tests\FlowTestCase; use function Flow\ETL\DSL\analyze; use function Flow\ETL\DSL\flow_context; +use function Flow\ETL\DSL\from_array; use function Flow\ETL\DSL\int_schema; use function Flow\ETL\DSL\row; use function Flow\ETL\DSL\rows; use function Flow\ETL\DSL\schema; use function Flow\ETL\DSL\str_schema; +use function iterator_to_array; final class StatisticsCollectorTest extends FlowTestCase { @@ -171,6 +176,34 @@ public function test_report_returns_report_with_memory_consumption(): void static::assertGreaterThanOrEqual(0, $report->statistics()->memory->initial()->inBytes()); } + public function test_sources_are_counted_and_reported_when_enabled(): void + { + $collector = new StatisticsCollector(analyze()->withSourceStatistics(), flow_context()); + $extractor = from_array([['id' => 1], ['id' => 2]]); + $sources = $collector->sources(); + static::assertNotNull($sources); + + iterator_to_array($sources->count($extractor, $extractor->extract(flow_context()), false)); + + static::assertEquals( + [new SourceStatistics('ArrayExtractor', new Statistics(Cardinality::exact(2)), 2, true)], + $collector->report()?->sources(), + ); + } + + public function test_sources_are_not_counted_when_not_enabled(): void + { + $collector = new StatisticsCollector(analyze(), flow_context()); + + static::assertNull($collector->sources()); + static::assertNull($collector->report()?->sources()); + } + + public function test_sources_are_not_counted_when_analyze_is_false(): void + { + static::assertNull((new StatisticsCollector(false, flow_context()))->sources()); + } + public function test_schema_is_null_when_not_enabled(): void { $collector = new StatisticsCollector(analyze(), flow_context()); diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/ArrayExtractorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/ArrayExtractorTest.php index 045a17c6e9..15ce8a1999 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/ArrayExtractorTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/ArrayExtractorTest.php @@ -6,9 +6,11 @@ use ArrayIterator; use ArrayObject; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InferredSchemaException; use Flow\ETL\Exception\InvalidLogicException; use Flow\ETL\Extractor\ArrayExtractor; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\Row\PhpRowHydrator; use Flow\ETL\Schema\Definition; use Flow\ETL\Schema\Definition\UnionDefinition; @@ -400,4 +402,29 @@ public function test_array_extractor_honours_the_batch_contract(): void RowsMother::sequentialIds(5), ); } + + public function test_an_array_dataset_declares_an_exact_row_count(): void + { + static::assertEquals( + new Statistics(rows: Cardinality::exact(3)), + from_array([['id' => 1], ['id' => 2], ['id' => 3]])->statistics(), + ); + } + + public function test_a_generator_dataset_declares_nothing(): void + { + /** @var callable(): Generator> $generator */ + $generator = static function (): Generator { + yield ['id' => 1]; + }; + + static::assertEquals(new Statistics(), from_array($generator())->statistics()); + } + + public function test_statistics_are_computed_at_most_once(): void + { + $extractor = from_array([['id' => 1]]); + + static::assertSame($extractor->statistics(), $extractor->statistics()); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/BatchByExtractorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/BatchByExtractorTest.php index 37ce2393dc..e927239705 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/BatchByExtractorTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/BatchByExtractorTest.php @@ -4,14 +4,19 @@ namespace Flow\ETL\Tests\Unit\Extractor; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Exception\SchemaMismatchException; +use Flow\ETL\Extractor\Statistics; +use Flow\ETL\Memory\ArrayMemory; +use Flow\ETL\Tests\Double\DeclaringExtractor; use Flow\ETL\Tests\Double\VaryingBatchesExtractor; use PHPUnit\Framework\TestCase; use function Flow\ETL\DSL\batched_by; use function Flow\ETL\DSL\config; use function Flow\ETL\DSL\flow_context; +use function Flow\ETL\DSL\from_memory; use function Flow\ETL\DSL\from_rows; use function Flow\ETL\DSL\int_schema; use function Flow\ETL\DSL\ref; @@ -186,4 +191,17 @@ public function test_is_repeatable(): void batched_by(from_rows(rows(schema(int_schema('id')), row(['id' => 1]))), ref('id'))->isRepeatable(), ); } + + public function test_it_passes_through_the_child_statistics(): void + { + $child = new DeclaringExtractor(new Statistics(Cardinality::exact(3), Cardinality::exact(300))); + $extractor = batched_by($child, 'id'); + + static::assertEquals(new Statistics(Cardinality::exact(3), Cardinality::exact(300)), $extractor->statistics()); + + $child->statistics = new Statistics(Cardinality::exact(1)); + + static::assertEquals(new Statistics(Cardinality::exact(1)), $extractor->statistics()); + static::assertEquals(new Statistics(), batched_by(from_memory(new ArrayMemory()), 'id')->statistics()); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/BatchExtractorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/BatchExtractorTest.php index e7b5bcd069..5b34d76fd4 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/BatchExtractorTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/BatchExtractorTest.php @@ -4,9 +4,13 @@ namespace Flow\ETL\Tests\Unit\Extractor; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InvalidArgumentException; use Flow\ETL\Exception\SchemaMismatchException; use Flow\ETL\Extractor\BatchExtractor; +use Flow\ETL\Extractor\Statistics; +use Flow\ETL\Memory\ArrayMemory; +use Flow\ETL\Tests\Double\DeclaringExtractor; use Flow\ETL\Tests\Double\FakeExtractor; use Flow\ETL\Tests\Double\VaryingBatchesExtractor; use Flow\ETL\Tests\FlowTestCase; @@ -14,6 +18,7 @@ use function Flow\ETL\DSL\batches; use function Flow\ETL\DSL\flow_context; +use function Flow\ETL\DSL\from_memory; use function Flow\ETL\DSL\from_rows; use function Flow\ETL\DSL\int_schema; use function Flow\ETL\DSL\row; @@ -113,4 +118,17 @@ public function test_is_repeatable(): void { static::assertTrue(batches(new FakeExtractor(10), 5)->isRepeatable()); } + + public function test_it_passes_through_the_child_statistics(): void + { + $child = new DeclaringExtractor(new Statistics(Cardinality::exact(3), Cardinality::exact(300))); + $extractor = batches($child, 10); + + static::assertEquals(new Statistics(Cardinality::exact(3), Cardinality::exact(300)), $extractor->statistics()); + + $child->statistics = new Statistics(Cardinality::exact(1)); + + static::assertEquals(new Statistics(Cardinality::exact(1)), $extractor->statistics()); + static::assertEquals(new Statistics(), batches(from_memory(new ArrayMemory()), 10)->statistics()); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/ChainExtractorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/ChainExtractorTest.php index 57fd54dff3..92768cfdb4 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/ChainExtractorTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/ChainExtractorTest.php @@ -4,17 +4,23 @@ namespace Flow\ETL\Tests\Unit\Extractor; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor; use Flow\ETL\Extractor\BatchableExtractor; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\FlowContext; +use Flow\ETL\Memory\ArrayMemory; use Flow\ETL\Schema; +use Flow\ETL\Tests\Double\DeclaringExtractor; use Flow\ETL\Tests\FlowTestCase; use Generator; use function Flow\ETL\DSL\df; use function Flow\ETL\DSL\flow_context; use function Flow\ETL\DSL\from_all; +use function Flow\ETL\DSL\from_array; use function Flow\ETL\DSL\from_data_frame; +use function Flow\ETL\DSL\from_memory; use function Flow\ETL\DSL\from_rows; use function Flow\ETL\DSL\int_schema; use function Flow\ETL\DSL\row; @@ -60,6 +66,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(int_schema('id')), row(['id' => 1])); yield rows(schema(int_schema('id')), row(['id' => 2])); } + + public function statistics(): Statistics + { + return new Statistics(); + } }, new class implements Extractor { public function withSchema(Schema $schema): static { @@ -76,6 +87,11 @@ public function extract(FlowContext $context, ?int $limit = null): Generator yield rows(schema(int_schema('id')), row(['id' => 3])); yield rows(schema(int_schema('id')), row(['id' => 4])); } + + public function statistics(): Statistics + { + return new Statistics(); + } }); self::assertExtractedRowsEquals( @@ -106,4 +122,40 @@ public function test_is_repeatable(): void { static::assertTrue(from_all(from_rows(rows(schema(int_schema('id')), row(['id' => 1]))))->isRepeatable()); } + + public function test_it_sums_the_statistics_of_its_children(): void + { + static::assertEquals( + new Statistics(rows: Cardinality::exact(5), size: Cardinality::exact(300)), + from_all( + new DeclaringExtractor(new Statistics(Cardinality::exact(2), Cardinality::exact(100))), + new DeclaringExtractor(new Statistics(Cardinality::exact(3), Cardinality::exact(200))), + )->statistics(), + ); + } + + public function test_one_unknown_child_makes_the_chain_unknown(): void + { + static::assertEquals( + new Statistics(), + from_all(from_array([['id' => 1], ['id' => 2]]), from_memory(new ArrayMemory([['id' => 3]])))->statistics(), + ); + } + + public function test_one_estimated_child_makes_the_chain_estimated(): void + { + $estimated = new DeclaringExtractor(new Statistics(rows: Cardinality::approximately(10, 0.2))); + $chain = from_all(from_array([['id' => 1], ['id' => 2]]), $estimated); + + static::assertEquals(Cardinality::approximately(12, 0.2), $chain->statistics()->rows); + + $estimated->statistics = new Statistics(rows: Cardinality::exact(10)); + + static::assertEquals(Cardinality::exact(12), $chain->statistics()->rows); + } + + public function test_an_empty_chain_declares_zero_rows_and_bytes_exactly(): void + { + static::assertEquals(new Statistics(Cardinality::exact(0), Cardinality::exact(0)), from_all()->statistics()); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/CollectingExtractorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/CollectingExtractorTest.php index 95077e06e3..8c4877f1fb 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/CollectingExtractorTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/CollectingExtractorTest.php @@ -4,10 +4,15 @@ namespace Flow\ETL\Tests\Unit\Extractor; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor\CollectingExtractor; +use Flow\ETL\Extractor\Statistics; +use Flow\ETL\Memory\ArrayMemory; +use Flow\ETL\Tests\Double\DeclaringExtractor; use Flow\ETL\Tests\FlowTestCase; use function Flow\ETL\DSL\flow_context; +use function Flow\ETL\DSL\from_memory; use function Flow\ETL\DSL\from_rows; use function Flow\ETL\DSL\int_schema; use function Flow\ETL\DSL\row; @@ -55,4 +60,17 @@ public function test_is_repeatable(): void (new CollectingExtractor(from_rows(rows(schema(int_schema('id')), row(['id' => 1])))))->isRepeatable(), ); } + + public function test_it_passes_through_the_child_statistics(): void + { + $child = new DeclaringExtractor(new Statistics(Cardinality::exact(3), Cardinality::exact(300))); + $extractor = new CollectingExtractor($child); + + static::assertEquals(new Statistics(Cardinality::exact(3), Cardinality::exact(300)), $extractor->statistics()); + + $child->statistics = new Statistics(Cardinality::exact(1)); + + static::assertEquals(new Statistics(Cardinality::exact(1)), $extractor->statistics()); + static::assertEquals(new Statistics(), (new CollectingExtractor(from_memory(new ArrayMemory())))->statistics()); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/FilesExtractorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/FilesExtractorTest.php index 6ab02bd371..85c6c045b6 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/FilesExtractorTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/FilesExtractorTest.php @@ -4,10 +4,14 @@ namespace Flow\ETL\Tests\Unit\Extractor; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor\FilesExtractor; use Flow\ETL\Extractor\Signal; use Flow\ETL\Schema; use Flow\ETL\Tests\Context\ExtractedRows; +use Flow\ETL\Tests\Context\MemoryFiles; +use Flow\ETL\Tests\Double\CountingFilesystem; +use Flow\ETL\Tests\Double\UnsizedFilesystem; use Flow\ETL\Tests\FlowTestCase; use function Flow\ETL\DSL\files; @@ -18,6 +22,48 @@ final class FilesExtractorTest extends FlowTestCase { + public function test_it_declares_one_row_per_listed_file_and_their_bytes_exactly(): void + { + $statistics = files('memory://dir/*', MemoryFiles::with([ + 'memory://dir/a.txt' => 'abcdefghij', + 'memory://dir/b.txt' => 'abcde', + ]))->statistics(); + + static::assertEquals(Cardinality::exact(2), $statistics->rows); + static::assertEquals(Cardinality::exact(15), $statistics->size); + } + + public function test_an_empty_listing_declares_zero_rows_and_zero_bytes(): void + { + $statistics = files('memory://dir/*', MemoryFiles::with([]))->statistics(); + + static::assertEquals(Cardinality::exact(0), $statistics->rows); + static::assertEquals(Cardinality::exact(0), $statistics->size); + } + + public function test_a_member_without_a_size_makes_the_size_unknown(): void + { + $statistics = files( + 'memory://dir/*', + new UnsizedFilesystem(MemoryFiles::with([ + 'memory://dir/a.txt' => 'abcdefghij', + 'memory://dir/b.txt' => 'abcde', + ]), ['memory://dir/b.txt']), + )->statistics(); + + static::assertEquals(Cardinality::exact(2), $statistics->rows); + static::assertEquals(Cardinality::unknown(), $statistics->size); + } + + public function test_the_listing_is_read_once(): void + { + $filesystem = new CountingFilesystem(MemoryFiles::with(['memory://dir/a.txt' => 'abc'])); + $extractor = files('memory://dir/*', $filesystem); + + static::assertSame($extractor->statistics(), $extractor->statistics()); + static::assertSame(1, $filesystem->listCalls); + } + public function test_a_zero_string_extension_is_not_null(): void { $batches = iterator_to_array(files(__DIR__ . '/Fixtures/ZeroExtension/*')->extract(flow_context())); diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/FooterStatisticsTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/FooterStatisticsTest.php new file mode 100644 index 0000000000..c8bbc86615 --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/FooterStatisticsTest.php @@ -0,0 +1,55 @@ +of(3, 0), + ); + } + + public function test_the_first_footer_is_scaled_to_the_listing(): void + { + static::assertEquals( + new Statistics(rows: Cardinality::approximately(40), size: Cardinality::approximately(400)), + (new FooterStatistics(new Statistics(Cardinality::exact(10), Cardinality::exact(100)), 1))->of(4, 0), + ); + } + + public function test_the_offset_is_taken_off_the_rows_never_below_zero(): void + { + $footers = new FooterStatistics(new Statistics(Cardinality::exact(10), Cardinality::exact(100)), 1); + + static::assertEquals(Cardinality::exact(7), $footers->of(1, 3)->rows); + static::assertEquals(Cardinality::exact(0), $footers->of(1, 30)->rows); + static::assertEquals(Cardinality::approximately(37), $footers->of(4, 3)->rows); + } + + public function test_no_footer_read_is_zero_exactly(): void + { + static::assertEquals( + new Statistics(rows: Cardinality::exact(0), size: Cardinality::exact(0)), + (new FooterStatistics(new Statistics(Cardinality::exact(0), Cardinality::exact(0)), 0))->of(0, 0), + ); + } + + public function test_a_negative_footer_count_is_refused(): void + { + $this->expectException(InvalidArgumentException::class); + $this->expectExceptionMessage('Read footer count must not be negative, given: -1'); + + new FooterStatistics(new Statistics(), -1); + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/ListedFilesTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/ListedFilesTest.php new file mode 100644 index 0000000000..74e434f185 --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/ListedFilesTest.php @@ -0,0 +1,86 @@ +status(path(__DIR__ . '/../../Fixtures/empty.txt')); + + static::assertNotNull($empty); + static::assertEquals( + Cardinality::unknown(), + ListedFiles::of([ + new SourceFile(path('memory://orders/a.csv'), 100), + new SourceFile($empty->path, $empty->size), + ])->bytes, + ); + } + + public function test_an_empty_listing_is_zero_bytes_exactly(): void + { + static::assertEquals(Cardinality::exact(0), ListedFiles::of([])->bytes); + } + + public function test_it_sums_the_listed_sizes(): void + { + static::assertEquals( + Cardinality::exact(300), + ListedFiles::of([ + new SourceFile(path('memory://orders/a.csv'), 100), + new SourceFile(path('memory://orders/b.csv'), 200), + ])->bytes, + ); + } + + public function test_one_member_without_a_size_makes_the_total_unknown(): void + { + static::assertEquals( + Cardinality::unknown(), + ListedFiles::of([ + new SourceFile(path('memory://orders/a.csv')), + new SourceFile(path('memory://orders/b.csv'), 200), + ])->bytes, + ); + } + + public function test_it_counts_every_listed_file_sized_or_not(): void + { + static::assertSame( + 2, + ListedFiles::of([ + new SourceFile(path('memory://orders/a.csv')), + new SourceFile(path('memory://orders/b.csv'), 200), + ])->count, + ); + } + + public function test_it_reads_file_statuses_as_well(): void + { + $listed = ListedFiles::of([new FileStatus(path('memory://orders/a.csv'), true, 100)]); + + static::assertSame(1, $listed->count); + static::assertEquals(Cardinality::exact(100), $listed->bytes); + } + + public function test_a_negative_count_is_refused(): void + { + $this->expectException(InvalidArgumentException::class); + $this->expectExceptionMessage('Listed file count must not be negative, given: -1'); + + new ListedFiles(-1, Cardinality::unknown()); + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/MemoryExtractorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/MemoryExtractorTest.php index cc61b7d74f..c8300a73a2 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/MemoryExtractorTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/MemoryExtractorTest.php @@ -6,6 +6,7 @@ use Flow\ETL\Exception\InferredSchemaException; use Flow\ETL\Extractor\MemoryExtractor; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\Memory\ArrayMemory; use Flow\ETL\Tests\Double\CountingMemory; use Flow\ETL\Tests\FlowTestCase; @@ -184,4 +185,9 @@ public function test_memory_extractor_honours_the_batch_contract(): void RowsMother::sequentialIds(5), ); } + + public function test_it_declares_nothing_because_memory_can_grow_mid_run(): void + { + static::assertEquals(new Statistics(), from_memory(new ArrayMemory())->statistics()); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/RowsExtractorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/RowsExtractorTest.php index 6b83eb311b..dad9e64077 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/RowsExtractorTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/RowsExtractorTest.php @@ -4,6 +4,8 @@ namespace Flow\ETL\Tests\Unit\Extractor; +use Flow\ETL\Cardinality; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\Tests\FlowTestCase; use function Flow\ETL\DSL\flow_context; @@ -57,4 +59,15 @@ public function test_is_repeatable(): void { static::assertTrue(from_rows(rows(schema(int_schema('number')), row(['number' => 1])))->isRepeatable()); } + + public function test_it_declares_an_exact_row_count(): void + { + $extractor = from_rows( + rows(schema(int_schema('number')), row(['number' => 1]), row(['number' => 2])), + rows(schema(int_schema('number')), row(['number' => 3])), + ); + + static::assertEquals(new Statistics(rows: Cardinality::exact(3)), $extractor->statistics()); + static::assertSame($extractor->statistics(), $extractor->statistics()); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/SequenceExtractorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/SequenceExtractorTest.php index 844a5ae33d..9ae9fb4bc6 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/SequenceExtractorTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/SequenceExtractorTest.php @@ -7,6 +7,7 @@ use DateInterval; use DatePeriod; use DateTimeImmutable; +use Flow\ETL\Cardinality; use Flow\ETL\Exception\InferredSchemaException; use Flow\ETL\Extractor\SequenceExtractor; use Flow\ETL\Tests\Double\MixedSequenceGenerator; @@ -156,6 +157,24 @@ public function test_infer_schema_resets_the_memo(): void static::assertSame('integer', $extractor->schema()->get('code')->type()->toString()); } + public function test_statistics_declare_the_generator_rows_and_no_size(): void + { + $statistics = from_sequence_number('num', 1, 10, 2)->statistics(); + + static::assertEquals(Cardinality::exact(5), $statistics->rows); + static::assertEquals(Cardinality::unknown(), $statistics->size); + } + + public function test_statistics_ask_the_generator_once(): void + { + $generator = new RecordingSequenceGenerator(new MixedSequenceGenerator()); + $extractor = new SequenceExtractor($generator, 'code'); + + static::assertSame($extractor->statistics(), $extractor->statistics()); + static::assertSame(1, $generator->rowsCalls); + static::assertSame(0, $generator->generateCalls); + } + public function test_schema_is_memoised(): void { $extractor = new SequenceExtractor(new MixedSequenceGenerator(), 'code'); diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/SequenceGenerator/DatePeriodSequenceGeneratorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/SequenceGenerator/DatePeriodSequenceGeneratorTest.php new file mode 100644 index 0000000000..7bf6e6935a --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/SequenceGenerator/DatePeriodSequenceGeneratorTest.php @@ -0,0 +1,83 @@ +rows()); + } + + public function test_excluding_the_start_date_is_counted(): void + { + $generator = new DatePeriodSequenceGenerator( + new DatePeriod( + new DateTimeImmutable('2023-01-01'), + new DateInterval('P1D'), + new DateTimeImmutable('2023-01-11'), + DatePeriod::EXCLUDE_START_DATE, + ), + ); + + static::assertEquals(Cardinality::exact(9), $generator->rows()); + static::assertSame(9, iterator_count($generator->generate())); + } + + public function test_including_the_end_date_is_counted(): void + { + $generator = new DatePeriodSequenceGenerator( + new DatePeriod( + new DateTimeImmutable('2023-01-01'), + new DateInterval('P1D'), + new DateTimeImmutable('2023-01-11'), + DatePeriod::INCLUDE_END_DATE, + ), + ); + + static::assertEquals(Cardinality::exact(11), $generator->rows()); + static::assertSame(11, iterator_count($generator->generate())); + } + + public function test_recurrences_are_counted(): void + { + $generator = new DatePeriodSequenceGenerator( + new DatePeriod(new DateTimeImmutable('2023-01-01'), new DateInterval('P1D'), 5), + ); + + static::assertEquals(Cardinality::exact(6), $generator->rows()); + static::assertSame(6, iterator_count($generator->generate())); + } + + public function test_variable_length_months_are_counted(): void + { + $generator = new DatePeriodSequenceGenerator( + new DatePeriod( + new DateTimeImmutable('2023-01-31'), + new DateInterval('P1M'), + new DateTimeImmutable('2023-12-31'), + ), + ); + + static::assertEquals(Cardinality::exact(iterator_count($generator->generate())), $generator->rows()); + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/SequenceGenerator/NumberSequenceGeneratorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/SequenceGenerator/NumberSequenceGeneratorTest.php new file mode 100644 index 0000000000..8be11690e0 --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/SequenceGenerator/NumberSequenceGeneratorTest.php @@ -0,0 +1,52 @@ +rows()); + static::assertSame($expected, iterator_count($generator->generate())); + } + + #[TestWith([1, 10, 0])] + #[TestWith([1, 3, 5])] + #[TestWith([1, 10, -2])] + public function test_a_step_the_range_rejects_is_rejected_the_same_way(int $start, int $end, int $step): void + { + $this->expectException(ValueError::class); + + (new NumberSequenceGenerator($start, $end, $step))->rows(); + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/SourceFileTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/SourceFileTest.php index d09c2e9dba..a87045a380 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/SourceFileTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/SourceFileTest.php @@ -31,4 +31,14 @@ public function test_uri_is_the_paths_uri(): void (new SourceFile(path('memory://orders/year=2024/data.csv')))->uri(), ); } + + public function test_it_carries_the_listed_size(): void + { + static::assertSame(1024, (new SourceFile(path('memory://orders/data.csv'), 1024))->size); + } + + public function test_a_size_may_be_absent(): void + { + static::assertNull((new SourceFile(path('memory://orders/data.csv')))->size); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/StatisticsTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/StatisticsTest.php new file mode 100644 index 0000000000..1f82a4b5a1 --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Extractor/StatisticsTest.php @@ -0,0 +1,46 @@ +rows); + static::assertEquals(Cardinality::unknown(), $statistics->size); + } + + public function test_merge_adds_rows_and_size_independently(): void + { + static::assertEquals( + new Statistics(Cardinality::exact(30), Cardinality::approximately(300)), + (new Statistics(Cardinality::exact(10), Cardinality::approximately(100)))->merge( + new Statistics(Cardinality::exact(20), Cardinality::exact(200)), + ), + ); + } + + public function test_merging_with_an_unknown_partial_yields_unknown(): void + { + static::assertEquals( + new Statistics(), + (new Statistics(Cardinality::exact(10), Cardinality::exact(100)))->merge(new Statistics()), + ); + } + + public function test_rows_and_size_are_declared_independently(): void + { + $statistics = new Statistics(size: Cardinality::exact(1024)); + + static::assertEquals(Cardinality::unknown(), $statistics->rows); + static::assertEquals(Cardinality::exact(1024), $statistics->size); + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Optimizer/Rule/CountFromStatisticsTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Optimizer/Rule/CountFromStatisticsTest.php new file mode 100644 index 0000000000..de5434b921 --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Optimizer/Rule/CountFromStatisticsTest.php @@ -0,0 +1,171 @@ +apply( + df()->read(from_array([['id' => 1], ['id' => 2], ['id' => 3]]))->explain(Trigger::count)->logical, + NodeMother::context(), + ); + + static::assertEquals( + from_rows(rows(schema(int_schema('count')), row(['count' => 3]))), + $plan->source()->extractor(), + ); + } + + public function test_a_count_over_a_plain_sequence_read_reads_the_number_instead(): void + { + $plan = (new CountFromStatistics())->apply( + df()->read(from_sequence_number('id', 1, 10))->explain(Trigger::count)->logical, + NodeMother::context(), + ); + + static::assertEquals( + from_rows(rows(schema(int_schema('count')), row(['count' => 10]))), + $plan->source()->extractor(), + ); + } + + public function test_a_count_over_a_plain_parquet_read_reads_the_number_instead(): void + { + $memory = memory_filesystem(); + df() + ->read(from_sequence_number('id', 1, 25)) + ->write(to_parquet(path('memory://count/file.parquet'), filesystem: $memory)) + ->run(); + + $plan = (new CountFromStatistics())->apply( + df() + ->read(from_parquet(path('memory://count/file.parquet'), filesystem: $memory)) + ->explain(Trigger::count) + ->logical, + NodeMother::context(), + ); + + static::assertEquals( + from_rows(rows(schema(int_schema('count')), row(['count' => 25]))), + $plan->source()->extractor(), + ); + } + + public function test_a_count_over_a_plain_floe_read_reads_the_number_instead(): void + { + $memory = memory_filesystem(); + df() + ->read(from_sequence_number('id', 1, 25)) + ->write(to_floe(path('memory://count/file.floe'), filesystem: $memory)) + ->run(); + + $plan = (new CountFromStatistics())->apply( + df() + ->read(from_floe(path('memory://count/file.floe'), filesystem: $memory)) + ->explain(Trigger::count) + ->logical, + NodeMother::context(), + ); + + static::assertEquals( + from_rows(rows(schema(int_schema('count')), row(['count' => 25]))), + $plan->source()->extractor(), + ); + } + + public static function inexact_rows(): Generator + { + yield 'approximate' => [Cardinality::approximately(10)]; + yield 'estimate equal to its bound, with an error' => [new Cardinality(10, 10)]; + yield 'upper bound alone' => [Cardinality::atMost(10)]; + yield 'unknown' => [Cardinality::unknown()]; + } + + #[DataProvider('inexact_rows')] + public function test_a_source_that_does_not_know_its_rows_exactly_is_counted_by_running(Cardinality $rows): void + { + $plan = Trigger::count->plan(NodeMother::read(new DeclaringExtractor(new Statistics($rows)))); + + static::assertSame($plan, (new CountFromStatistics())->apply($plan, NodeMother::context())); + } + + public function test_a_read_with_a_pushed_limit_is_counted_by_running(): void + { + $plan = Trigger::count->plan(NodeMother::read()->withLimit(1)); + + static::assertSame($plan, (new CountFromStatistics())->apply($plan, NodeMother::context())); + } + + public function test_a_read_with_a_pushed_partition_filter_is_counted_by_running(): void + { + $plan = Trigger::count->plan(NodeMother::read()->withPathFilter(new RejectingFilter())); + + static::assertSame($plan, (new CountFromStatistics())->apply($plan, NodeMother::context())); + } + + public static function frames_counted_by_running(): Generator + { + yield 'filter' => [df()->read(from_array([['id' => 1], ['id' => 2]]))->filter(ref('id')->equals(lit(1)))]; + yield 'limit' => [df()->read(from_array([['id' => 1], ['id' => 2]]))->limit(1)]; + yield 'join' => [df() + ->read(from_array([['id' => 1]])) + ->join(df()->read(from_array([['id' => 1]])), join_on(['id' => 'id']))]; + yield 'a transform that changes the rows' => [df() + ->read(from_array([['id' => 1], ['id' => 1]])) + ->dropDuplicates('id')]; + yield 'a transform that keeps the rows' => [df() + ->read(from_array([['id' => 1]])) + ->withEntry('copy', ref('id'))]; + yield 'a sink' => [df()->read(from_array([['id' => 1]]))->write(to_memory(new ArrayMemory()))]; + } + + #[DataProvider('frames_counted_by_running')] + public function test_a_frame_that_does_more_than_read_is_counted_by_running(DataFrame $frame): void + { + $plan = $frame->explain(Trigger::count)->logical; + + static::assertSame($plan, (new CountFromStatistics())->apply($plan, NodeMother::context())); + } + + public function test_a_plan_that_does_not_count_is_left_alone(): void + { + $plan = df()->read(from_array([['id' => 1]]))->explain(Trigger::rows)->logical; + + static::assertSame($plan, (new CountFromStatistics())->apply($plan, NodeMother::context())); + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/OptimizerTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/OptimizerTest.php index a85b192b93..ab27e626d8 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/OptimizerTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/OptimizerTest.php @@ -9,6 +9,7 @@ use Flow\ETL\Optimizer; use Flow\ETL\Optimizer\Rule\CombineLimits; use Flow\ETL\Optimizer\Rule\CombineSortAndLimit; +use Flow\ETL\Optimizer\Rule\CountFromStatistics; use Flow\ETL\Optimizer\Rule\PushFilterIntoSource; use Flow\ETL\Optimizer\Rule\PushLimitIntoSource; use Flow\ETL\Tests\Double\RecordingRule; @@ -55,10 +56,16 @@ public function test_an_optimizer_without_rules_returns_the_plan_it_was_given(): static::assertSame($plan, (new Optimizer())->optimize($plan, NodeMother::context())); } - public function test_default_registers_combine_limits_combine_sort_and_limit_push_limit_then_push_filter_into_source(): void + public function test_default_registers_combine_limits_combine_sort_and_limit_push_limit_push_filter_then_count_from_statistics(): void { static::assertSame( - [CombineLimits::class, CombineSortAndLimit::class, PushLimitIntoSource::class, PushFilterIntoSource::class], + [ + CombineLimits::class, + CombineSortAndLimit::class, + PushLimitIntoSource::class, + PushFilterIntoSource::class, + CountFromStatistics::class, + ], array_map(static fn($rule) => $rule::class, Optimizer::default()->rules()), ); } @@ -66,7 +73,7 @@ public function test_default_registers_combine_limits_combine_sort_and_limit_pus public function test_without_drops_the_named_rule_and_keeps_the_rest(): void { static::assertSame( - [CombineLimits::class, CombineSortAndLimit::class, PushFilterIntoSource::class], + [CombineLimits::class, CombineSortAndLimit::class, PushFilterIntoSource::class, CountFromStatistics::class], array_map( static fn($rule) => $rule::class, Optimizer::default()->without(PushLimitIntoSource::class)->rules(), diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/Explain/PhysicalOutlineTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/Explain/PhysicalOutlineTest.php index bab0cb46aa..bce4212b75 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/Explain/PhysicalOutlineTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/Explain/PhysicalOutlineTest.php @@ -25,7 +25,10 @@ public function test_a_plan_without_a_blocking_step_is_one_pipeline(): void static::assertSame(['Columns: id'], $root->lines); static::assertCount(1, $root->children); static::assertSame('Pipeline #0', $root->children[0]->name); - static::assertSame(['Extractor: ArrayExtractor'], $root->children[0]->lines); + static::assertSame( + ['Extractor: ArrayExtractor', ' Statistics: rows exact 1 · size unknown'], + $root->children[0]->lines, + ); static::assertSame([], $root->children[0]->children); } @@ -37,6 +40,7 @@ public function test_every_step_is_listed_in_the_order_the_rows_reach_it(): void │ Columns: id └─ Pipeline #0 Extractor: ArrayExtractor + Statistics: rows exact 1 · size unknown Processor: CollectingProcessor Schema: declared Loader: StreamLoader @@ -62,6 +66,7 @@ public function test_a_joined_frame_is_a_plan_of_its_own_under_the_step_that_rea │ Loader: StreamLoader └─ Pipeline #0 │ Extractor: ArrayExtractor + │ Statistics: rows exact 1 · size unknown │ Processor: HashJoinProcessor │ Join: left │ On: id = id @@ -71,6 +76,7 @@ public function test_a_joined_frame_is_a_plan_of_its_own_under_the_step_that_rea │ Batch: 1000 └─ Right side: Pipeline #0 Extractor: ArrayExtractor + Statistics: rows exact 1 · size unknown PLAN, (new TreeLayout())->render((new PhysicalOutline())->of(PhysicalPlanMother::of( data_frame() @@ -92,7 +98,12 @@ public function test_a_pushed_limit_is_listed_under_the_source_that_was_handed_i )); static::assertSame( - ['Extractor: ArrayExtractor', ' Limit: 1', 'Transformer: LimitTransformer'], + [ + 'Extractor: ArrayExtractor', + ' Statistics: rows exact 2 · size unknown', + ' Limit: 1', + 'Transformer: LimitTransformer', + ], $root->children[0]->lines, ); } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/Explain/StatisticsLineTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/Explain/StatisticsLineTest.php new file mode 100644 index 0000000000..70d8da6a20 --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/Explain/StatisticsLineTest.php @@ -0,0 +1,121 @@ +cardinality(Cardinality::exact(1_000))); + } + + public function test_an_upper_bound_alone_is_written_as_at_most(): void + { + static::assertSame('≤ 25 000', (new StatisticsLine())->cardinality(Cardinality::atMost(25_000))); + } + + public function test_an_estimate_alone_carries_its_relative_error_in_percent(): void + { + static::assertSame('~20 000 ±50%', (new StatisticsLine())->cardinality(Cardinality::approximately(20_000))); + static::assertSame( + '~20 000 ±12.5%', + (new StatisticsLine())->cardinality(Cardinality::approximately(20_000, 0.125)), + ); + } + + public function test_an_estimate_with_an_error_of_zero_and_no_bound_is_not_exact(): void + { + static::assertSame('~7 ±0%', (new StatisticsLine())->cardinality(Cardinality::approximately(7, 0.0))); + } + + public function test_an_estimate_under_a_bound_lists_both(): void + { + static::assertSame( + '~800 ±25% (≤ 1 000)', + (new StatisticsLine())->cardinality(new Cardinality(1_000, 800, 0.25)), + ); + } + + public function test_an_estimate_equal_to_its_bound_is_not_exact_while_it_carries_an_error(): void + { + static::assertSame('~1 000 ±50% (≤ 1 000)', (new StatisticsLine())->cardinality(new Cardinality(1_000, 1_000))); + } + + public function test_an_unknown_count_is_written_as_unknown(): void + { + static::assertSame('unknown', (new StatisticsLine())->cardinality(Cardinality::unknown())); + } + + public function test_numbers_are_grouped_in_thousands(): void + { + static::assertSame('exact 0', (new StatisticsLine())->cardinality(Cardinality::exact(0))); + static::assertSame('exact 999', (new StatisticsLine())->cardinality(Cardinality::exact(999))); + static::assertSame( + 'exact 1 234 567 890', + (new StatisticsLine())->cardinality(Cardinality::exact(1_234_567_890)), + ); + } + + public function test_the_unit_follows_every_number(): void + { + static::assertSame('~800 B ±25% (≤ 1 000 B)', (new StatisticsLine())->cardinality( + new Cardinality(1_000, 800, 0.25), + ' B', + )); + static::assertSame('≤ 1 000 B', (new StatisticsLine())->cardinality(Cardinality::atMost(1_000), ' B')); + static::assertSame('exact 309 188 B', (new StatisticsLine())->cardinality(Cardinality::exact(309_188), ' B')); + static::assertSame('unknown', (new StatisticsLine())->cardinality(Cardinality::unknown(), ' B')); + } + + public function test_a_relative_error_is_written_in_percent(): void + { + static::assertSame('0%', (new StatisticsLine())->percent(0.0)); + static::assertSame('45%', (new StatisticsLine())->percent(0.45)); + static::assertSame('33.33%', (new StatisticsLine())->percent(1 / 3)); + static::assertSame('250%', (new StatisticsLine())->percent(2.5)); + } + + public function test_an_infinite_relative_error_is_written_as_infinity(): void + { + static::assertSame('∞', (new StatisticsLine())->percent(INF)); + } + + public function test_a_missing_relative_error_is_written_as_unknown(): void + { + static::assertSame('unknown', (new StatisticsLine())->percent(null)); + } + + public function test_the_line_lists_rows_then_size_in_bytes(): void + { + static::assertSame( + 'rows exact 1 000 · size exact 309 188 B', + (new StatisticsLine())->of(new Statistics(Cardinality::exact(1_000), Cardinality::exact(309_188))), + ); + } + + public function test_the_line_lists_an_unknown_fact_next_to_a_known_one(): void + { + static::assertSame( + 'rows unknown · size exact 10 B', + (new StatisticsLine())->of(new Statistics(size: Cardinality::exact(10))), + ); + static::assertSame( + 'rows ≤ 5 · size unknown', + (new StatisticsLine())->of(new Statistics(rows: Cardinality::atMost(5))), + ); + } + + public function test_there_is_no_line_when_both_facts_are_unknown(): void + { + static::assertNull((new StatisticsLine())->of(new Statistics())); + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/Explain/StepDetailsTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/Explain/StepDetailsTest.php index cfdc09ae52..1515448c22 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/Explain/StepDetailsTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/Explain/StepDetailsTest.php @@ -8,8 +8,10 @@ use Flow\ETL\Bucketing\HashBucketing; use Flow\ETL\Bucketing\NativeHasher; use Flow\ETL\Bucketing\Storage\MemoryBuckets; +use Flow\ETL\Cardinality; use Flow\ETL\Constraint\UniqueConstraint; use Flow\ETL\Executor; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\GroupBy; use Flow\ETL\GroupBy\DeclaredPivotValues; use Flow\ETL\Join\Comparison\Equal; @@ -32,6 +34,7 @@ use Flow\ETL\Processor\TopNProcessor; use Flow\ETL\Processor\VoidProcessor; use Flow\ETL\Processor\WindowProcessor; +use Flow\ETL\Tests\Double\DeclaringExtractor; use Flow\ETL\Tests\FlowTestCase; use Flow\ETL\Tests\Mother\HashJoinProcessorMother; use Flow\ETL\Tests\Mother\PhysicalPlanMother; @@ -52,7 +55,10 @@ public function test_a_step_is_labelled_by_what_it_is(): void { $details = new StepDetails(); - static::assertSame(['Extractor: ArrayExtractor'], $details->lines(from_array([['id' => 1]]))); + static::assertSame( + ['Extractor: ArrayExtractor', ' Statistics: rows exact 1 · size unknown'], + $details->lines(from_array([['id' => 1]])), + ); static::assertSame(['Processor: VoidProcessor'], $details->lines(new VoidProcessor())); static::assertSame( ['Processor: BatchingProcessor', ' Batch: 100'], @@ -212,7 +218,22 @@ public function test_a_window_lists_the_column_it_writes_and_its_function(): voi public function test_a_step_the_explain_does_not_know_lists_nothing(): void { - static::assertSame([], (new StepDetails())->settings(from_array([['id' => 1]]))); + static::assertSame([], (new StepDetails())->settings(new VoidProcessor())); + } + + public function test_a_source_lists_the_statistics_it_declares(): void + { + static::assertSame( + ['Statistics: rows exact 1 000 · size ~309 188 B ±50%'], + (new StepDetails())->settings(new DeclaringExtractor( + new Statistics(Cardinality::exact(1_000), Cardinality::approximately(309_188)), + )), + ); + } + + public function test_a_source_that_declares_nothing_lists_no_statistics(): void + { + static::assertSame([], (new StepDetails())->settings(new DeclaringExtractor(new Statistics()))); } public function test_bucketing_lists_its_strategy_and_storage(): void diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/Node/CountTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/Node/CountTest.php new file mode 100644 index 0000000000..d5f51b1e21 --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/Node/CountTest.php @@ -0,0 +1,48 @@ +withChildren([$input])); + } + + public function test_with_children_returns_a_new_instance_when_a_child_changes(): void + { + $input = NodeMother::read(); + $other = NodeMother::read(); + $node = new Count($input); + + $rebuilt = $node->withChildren([$other]); + + static::assertNotSame($node, $rebuilt); + static::assertInstanceOf(Count::class, $rebuilt); + static::assertSame([$other], $rebuilt->children()); + } + + public function test_declarations(): void + { + $input = NodeMother::read(); + $node = new Count($input); + + static::assertSame(RowCount::reducing, $node->rowCount()); + static::assertSame(Transparency::opaque, $node->transparency()); + static::assertSame(Materialization::blocking, $node->materialization()); + static::assertEquals(Redefined::unknown(), $node->redefines()); + } +} diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/TriggerTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/TriggerTest.php index 85a6b60716..41c1551fc8 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/TriggerTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Plan/TriggerTest.php @@ -5,6 +5,7 @@ namespace Flow\ETL\Tests\Unit\Plan; use Flow\ETL\Memory\ArrayMemory; +use Flow\ETL\Plan\Node\Count; use Flow\ETL\Plan\Node\Outputs; use Flow\ETL\Plan\Node\Result; use Flow\ETL\Plan\Node\Transaction; @@ -42,6 +43,26 @@ public function test_rows_with_sinks_is_the_result_then_every_sink_in_order(): v static::assertSame([$first, $second], [$root->children()[1], $root->children()[2]]); } + public function test_count_without_sinks_is_a_result_over_a_count_of_the_root(): void + { + $read = NodeMother::read(); + + $root = Trigger::count->plan($read)->root; + + static::assertEquals(new Result(new Count($read)), $root); + } + + public function test_count_with_sinks_keeps_every_sink_after_the_count(): void + { + $read = NodeMother::read(); + $write = new Write($read, to_memory(new ArrayMemory())); + + static::assertEquals( + new Outputs(new Result(new Count($read)), $write), + Trigger::count->plan($read, new Sinks($write))->root, + ); + } + public function test_run_with_one_write_over_the_root_makes_that_write_the_root(): void { $read = NodeMother::read(); diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Planner/NodeTranslatorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Planner/NodeTranslatorTest.php index 37365d1768..be3b1624e9 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Planner/NodeTranslatorTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Planner/NodeTranslatorTest.php @@ -17,6 +17,7 @@ use Flow\ETL\Plan\Node\Collect; use Flow\ETL\Plan\Node\CollectRefs; use Flow\ETL\Plan\Node\Constrain; +use Flow\ETL\Plan\Node\Count; use Flow\ETL\Plan\Node\CrossJoin; use Flow\ETL\Plan\Node\Discard; use Flow\ETL\Plan\Node\Distinct; @@ -47,6 +48,7 @@ use Flow\ETL\Processor\CachingProcessor; use Flow\ETL\Processor\CollectingProcessor; use Flow\ETL\Processor\ConstrainedProcessor; +use Flow\ETL\Processor\CountingProcessor; use Flow\ETL\Processor\GroupByAggregationProcessor; use Flow\ETL\Processor\HashJoinProcessor; use Flow\ETL\Processor\MemorySortProcessor; @@ -211,6 +213,17 @@ public function test_cross_join_steps_are_the_exact_list_in_order(): void ); } + public function test_count_steps_are_the_exact_list_in_order(): void + { + static::assertSame( + [CountingProcessor::class], + array_map( + static fn($step) => $step::class, + NodeTranslator::toSteps(new Count(NodeMother::read()), NodeMother::context(), []), + ), + ); + } + public function test_discard_steps_are_the_exact_list_in_order(): void { static::assertSame( diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Planner/PipelineSplitTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Planner/PipelineSplitTest.php index d7f0e9f2fc..3ec9d45a48 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Planner/PipelineSplitTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Planner/PipelineSplitTest.php @@ -8,6 +8,7 @@ use Flow\ETL\Executor; use Flow\ETL\Executor\Raw; use Flow\ETL\Executor\SinkFeed; +use Flow\ETL\Executor\SourceRows; use Flow\ETL\Executor\TransactionalSinks; use Flow\ETL\Join\Join as JoinType; use Flow\ETL\Loader\ArrayLoader; @@ -667,4 +668,34 @@ public function test_the_leaf_pipeline_carries_the_reads_limit_and_path_filter() static::assertSame(5, $plan->root()->input()?->limit()); static::assertNull($plan->root()->limit()); } + + public function test_only_the_pipeline_that_reads_the_source_counts_its_rows(): void + { + $logical = new LogicalPlan(new Result(NodeMother::limit( + NodeMother::sort(NodeMother::read(from_array([['id' => 1]], schema(int_schema('id'))))), + 5, + ))); + $planned = new PlannedNodes(); + $sources = new SourceRows(); + (new Planner())->node($logical->root, NodeMother::context(), $planned); + + $plan = (new PipelineSplit())->of($logical, $planned, NodeMother::context(), $sources); + + static::assertSame($sources, $plan->root()->input()?->sources()); + static::assertNull($plan->root()->sources()); + } + + public function test_a_pipeline_counts_no_rows_unless_asked(): void + { + $logical = new LogicalPlan(new Result(NodeMother::read(from_array([['id' => 1]], schema(int_schema('id')))))); + $planned = new PlannedNodes(); + (new Planner())->node($logical->root, NodeMother::context(), $planned); + + static::assertNull( + (new PipelineSplit()) + ->of($logical, $planned, NodeMother::context()) + ->root() + ->sources(), + ); + } } diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/PlannerTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/PlannerTest.php index 5948767b8b..95c21713d2 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/PlannerTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/PlannerTest.php @@ -6,11 +6,13 @@ use ArrayObject; use Flow\ETL\Adapter\CSV\CSVLoader; +use Flow\ETL\Dataset\SourceStatistics; use Flow\ETL\Exception\DataDependentSchemaException; use Flow\ETL\Exception\SchemaDefinitionNotFoundException; use Flow\ETL\Executor\Described; use Flow\ETL\Executor\Raw; use Flow\ETL\Executor\SinkFeed; +use Flow\ETL\Executor\SourceRows; use Flow\ETL\Join\Join as JoinType; use Flow\ETL\Loader\MemoryLoader; use Flow\ETL\Memory\ArrayMemory; @@ -279,6 +281,29 @@ public function test_a_joins_right_side_runs_with_the_outer_context(): void static::assertSame([$context], $recording->contexts); } + public function test_a_joins_right_side_counts_its_source_rows_too(): void + { + $node = new CrossJoin( + NodeMother::read(from_array([['id' => 1], ['id' => 2]], schema(int_schema('id')))), + df() + ->read(from_array([['id' => 1], ['id' => 2], ['id' => 3]], schema(int_schema('id')))) + ->explain() + ->logical + ->root, + 'r_', + ); + $context = NodeMother::context(config()); + $sources = new SourceRows(); + + $plan = (new Planner(Optimizer::default()))->plan(new LogicalPlan(new Result($node)), $context, $sources); + iterator_to_array($context->config->executor()->execute($plan)); + + static::assertSame( + [2, 3], + array_map(static fn(SourceStatistics $source): int => $source->rows, $sources->statistics()), + ); + } + public function test_a_read_frame_is_a_source_of_this_pipeline(): void { $extractor = from_data_frame(df()->read(from_csv(__DIR__ . '/../Fixtures/orders.csv'))->select('id')); diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Processor/CachingProcessorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Processor/CachingProcessorTest.php index f0b61087f3..9208c0412d 100644 --- a/src/core/etl/tests/Flow/ETL/Tests/Unit/Processor/CachingProcessorTest.php +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Processor/CachingProcessorTest.php @@ -6,6 +6,7 @@ use Flow\ETL\Cache\CacheIndex; use Flow\ETL\Cache\Implementation\InMemoryCache; +use Flow\ETL\Cardinality; use Flow\ETL\Extractor\Signal; use Flow\ETL\Processor\CachingProcessor; use Flow\ETL\Rows; @@ -43,6 +44,7 @@ public function test_a_stop_still_completes_the_cache(): void static::assertInstanceOf(Rows::class, $indexRows); static::assertCount(5, CacheIndex::fromRows('stop-cache', $indexRows)->values()); + static::assertEquals(Cardinality::exact(5), CacheIndex::fromRows('stop-cache', $indexRows)->rows()); } public function test_bind_returns_the_input_schema(): void diff --git a/src/core/etl/tests/Flow/ETL/Tests/Unit/Processor/CountingProcessorTest.php b/src/core/etl/tests/Flow/ETL/Tests/Unit/Processor/CountingProcessorTest.php new file mode 100644 index 0000000000..0c33f90c24 --- /dev/null +++ b/src/core/etl/tests/Flow/ETL/Tests/Unit/Processor/CountingProcessorTest.php @@ -0,0 +1,53 @@ + 1]), row(['id' => 2])); + yield rows(schema(int_schema('id'))); + yield rows(schema(int_schema('id')), row(['id' => 3])); + })(); + + static::assertEquals( + [rows(schema(int_schema('count')), row(['count' => 3]))], + iterator_to_array((new CountingProcessor())->process($batches, flow_context())), + ); + } + + public function test_no_batch_counts_to_zero(): void + { + $batches = (static function () { + yield from []; + })(); + + static::assertEquals( + [rows(schema(int_schema('count')), row(['count' => 0]))], + iterator_to_array((new CountingProcessor())->process($batches, flow_context())), + ); + } + + public function test_bind_outputs_the_count_column_whatever_the_input(): void + { + $bound = (new CountingProcessor())->bind(schema(int_schema('id'), str_schema('name'))); + + static::assertEquals(schema(int_schema('count')), $bound->output); + static::assertEquals(new CountingProcessor(), $bound->step); + } +} diff --git a/src/core/etl/tests/Flow/Floe/Tests/Context/FloeStreamReaderContext.php b/src/core/etl/tests/Flow/Floe/Tests/Context/FloeStreamReaderContext.php index bb2c65fc07..77e61f9e34 100644 --- a/src/core/etl/tests/Flow/Floe/Tests/Context/FloeStreamReaderContext.php +++ b/src/core/etl/tests/Flow/Floe/Tests/Context/FloeStreamReaderContext.php @@ -8,9 +8,11 @@ use Flow\ETL\Schema\Metadata; use Flow\Filesystem\Filesystem; use Flow\Filesystem\Path; +use Flow\Floe\Codec\NoopCodec; use Flow\Floe\FloeReader; use Flow\Floe\FloeWriter; use Flow\Floe\Footer; +use Flow\Floe\FooterReader; use Flow\Floe\Format; use Flow\Floe\FrameReader; @@ -38,6 +40,17 @@ public static function footer(Filesystem $filesystem, Path $path): Footer return Footer::fromJson($source->read($footerLength, $size - Format::TRAILER_LENGTH - $footerLength)); } + /** + * The bytes between the header and the FOOTER frame - what the footer's byteSize has to report. + */ + public static function dataBytes(Filesystem $filesystem, Path $path): int + { + return ( + (new FooterReader())->read($filesystem->readFrom($path), new NoopCodec())->footerFrameStart + - Format::HEADER_LENGTH + ); + } + /** * @return array frame type and body, in file order */ diff --git a/src/core/etl/tests/Flow/Floe/Tests/Fixtures/pre-statistics/heterogeneous.floe b/src/core/etl/tests/Flow/Floe/Tests/Fixtures/pre-statistics/heterogeneous.floe new file mode 100644 index 0000000000..fb2acd05e9 Binary files /dev/null and b/src/core/etl/tests/Flow/Floe/Tests/Fixtures/pre-statistics/heterogeneous.floe differ diff --git a/src/core/etl/tests/Flow/Floe/Tests/Fixtures/v2/all-entry-types.floe b/src/core/etl/tests/Flow/Floe/Tests/Fixtures/v2/all-entry-types.floe index 4c3c94fe9c..d4fac8c506 100644 Binary files a/src/core/etl/tests/Flow/Floe/Tests/Fixtures/v2/all-entry-types.floe and b/src/core/etl/tests/Flow/Floe/Tests/Fixtures/v2/all-entry-types.floe differ diff --git a/src/core/etl/tests/Flow/Floe/Tests/Fixtures/v2/empty.floe b/src/core/etl/tests/Flow/Floe/Tests/Fixtures/v2/empty.floe index b49a2b92a2..433f9706c9 100644 Binary files a/src/core/etl/tests/Flow/Floe/Tests/Fixtures/v2/empty.floe and b/src/core/etl/tests/Flow/Floe/Tests/Fixtures/v2/empty.floe differ diff --git a/src/core/etl/tests/Flow/Floe/Tests/Fixtures/v2/heterogeneous.floe b/src/core/etl/tests/Flow/Floe/Tests/Fixtures/v2/heterogeneous.floe index fb2acd05e9..1d707715ad 100644 Binary files a/src/core/etl/tests/Flow/Floe/Tests/Fixtures/v2/heterogeneous.floe and b/src/core/etl/tests/Flow/Floe/Tests/Fixtures/v2/heterogeneous.floe differ diff --git a/src/core/etl/tests/Flow/Floe/Tests/Integration/FloeExtractorTest.php b/src/core/etl/tests/Flow/Floe/Tests/Integration/FloeExtractorTest.php new file mode 100644 index 0000000000..f297fa9c85 --- /dev/null +++ b/src/core/etl/tests/Flow/Floe/Tests/Integration/FloeExtractorTest.php @@ -0,0 +1,152 @@ +fs(), + $this->cacheDir->suffix('glob/a.floe'), + rows(schema(int_schema('id')), row(['id' => 1]), row(['id' => 2])), + ); + FloeStreamReaderContext::write( + $this->fs(), + $this->cacheDir->suffix('glob/b.floe'), + rows(schema(int_schema('id')), row(['id' => 3])), + ); + $firstBytes = FloeStreamReaderContext::dataBytes($this->fs(), $this->cacheDir->suffix('glob/a.floe')); + + $statistics = from_floe($this->cacheDir->suffix('glob/*.floe'), filesystem: $this->fs())->statistics(); + + static::assertEquals(Cardinality::approximately(4, Cardinality::DEFAULT_RELATIVE_ERROR), $statistics->rows); + static::assertEquals( + Cardinality::approximately(2 * $firstBytes, Cardinality::DEFAULT_RELATIVE_ERROR), + $statistics->size, + ); + } + + public function test_a_single_file_declares_exact_rows_and_byte_size(): void + { + $path = $this->cacheDir->suffix('single.floe'); + FloeStreamReaderContext::write( + $this->fs(), + $path, + rows(schema(int_schema('id')), row(['id' => 1]), row(['id' => 2])), + ); + + $statistics = from_floe($path, filesystem: $this->fs())->statistics(); + + static::assertEquals(Cardinality::exact(2), $statistics->rows); + static::assertEquals( + Cardinality::exact(FloeStreamReaderContext::dataBytes($this->fs(), $path)), + $statistics->size, + ); + } + + public function test_an_offset_is_subtracted_from_the_row_count(): void + { + $path = $this->cacheDir->suffix('offset.floe'); + FloeStreamReaderContext::write( + $this->fs(), + $path, + rows(schema(int_schema('id')), row(['id' => 1]), row(['id' => 2])), + ); + + $extractor = from_floe($path, filesystem: $this->fs()); + + static::assertEquals(Cardinality::exact(2), $extractor->statistics()->rows); + static::assertEquals(Cardinality::exact(1), $extractor->withOffset(1)->statistics()->rows); + static::assertEquals(Cardinality::exact(0), $extractor->withOffset(10)->statistics()->rows); + } + + public function test_statistics_are_computed_at_most_once(): void + { + $path = $this->cacheDir->suffix('memo.floe'); + FloeStreamReaderContext::write($this->fs(), $path, rows(schema(int_schema('id')), row(['id' => 1]))); + $filesystem = new CountingFilesystem($this->fs()); + $extractor = from_floe($path, filesystem: $filesystem); + + static::assertSame($extractor->statistics(), $extractor->statistics()); + static::assertSame(1, $filesystem->readFromCalls); + static::assertSame(1, $filesystem->closedStreams()); + } + + public function test_schema_then_statistics_reads_the_footer_once(): void + { + $path = $this->cacheDir->suffix('schema-first.floe'); + FloeStreamReaderContext::write($this->fs(), $path, rows(schema(int_schema('id')), row(['id' => 1]))); + $filesystem = new CountingFilesystem($this->fs()); + $extractor = from_floe($path, filesystem: $filesystem); + + $extractor->schema(); + + static::assertEquals(Cardinality::exact(1), $extractor->statistics()->rows); + static::assertSame(1, $filesystem->readFromCalls); + } + + public function test_a_declared_schema_still_reads_the_footer_for_statistics(): void + { + $path = $this->cacheDir->suffix('declared.floe'); + FloeStreamReaderContext::write($this->fs(), $path, rows(schema(int_schema('id')), row(['id' => 1]))); + + static::assertEquals( + Cardinality::exact(1), + from_floe($path, filesystem: $this->fs())->withSchema(schema(int_schema('id')))->statistics()->rows, + ); + } + + public function test_union_by_name_sums_every_footer_exactly(): void + { + FloeStreamReaderContext::write( + $this->fs(), + $this->cacheDir->suffix('union/a.floe'), + rows(schema(int_schema('id')), row(['id' => 1]), row(['id' => 2])), + ); + FloeStreamReaderContext::write( + $this->fs(), + $this->cacheDir->suffix('union/b.floe'), + rows(schema(int_schema('id')), row(['id' => 3])), + ); + + $statistics = from_floe($this->cacheDir->suffix('union/*.floe'), filesystem: $this->fs()) + ->unionByName() + ->statistics(); + + static::assertEquals(Cardinality::exact(3), $statistics->rows); + static::assertEquals( + Cardinality::exact( + FloeStreamReaderContext::dataBytes($this->fs(), $this->cacheDir->suffix('union/a.floe')) + + FloeStreamReaderContext::dataBytes($this->fs(), $this->cacheDir->suffix('union/b.floe')), + ), + $statistics->size, + ); + } + + public function test_union_by_name_invalidates_the_memo(): void + { + $path = $this->cacheDir->suffix('union-memo.floe'); + FloeStreamReaderContext::write($this->fs(), $path, rows(schema(int_schema('id')), row(['id' => 1]))); + $filesystem = new CountingFilesystem($this->fs()); + $extractor = from_floe($path, filesystem: $filesystem); + + $extractor->statistics(); + $extractor->unionByName()->statistics(); + + static::assertSame(2, $filesystem->readFromCalls); + } +} diff --git a/src/core/etl/tests/Flow/Floe/Tests/Integration/FloeHardBreakTest.php b/src/core/etl/tests/Flow/Floe/Tests/Integration/FloeHardBreakTest.php new file mode 100644 index 0000000000..3bdef57169 --- /dev/null +++ b/src/core/etl/tests/Flow/Floe/Tests/Integration/FloeHardBreakTest.php @@ -0,0 +1,28 @@ +expectException(FloeException::class); + $this->expectExceptionMessageMatches('/Floe footer is malformed: .*statistics/'); + + FloeEngineContext::phpReader($this->fs()) + ->read(path(__DIR__ . '/../Fixtures/pre-statistics/heterogeneous.floe')) + ->schema(); + } +} diff --git a/src/core/etl/tests/Flow/Floe/Tests/Integration/FloeStatisticsTest.php b/src/core/etl/tests/Flow/Floe/Tests/Integration/FloeStatisticsTest.php new file mode 100644 index 0000000000..a7c0cbeedb --- /dev/null +++ b/src/core/etl/tests/Flow/Floe/Tests/Integration/FloeStatisticsTest.php @@ -0,0 +1,130 @@ +cacheDir->suffix('compact-a.floe'); + $b = $this->cacheDir->suffix('compact-b.floe'); + $spliced = $this->cacheDir->suffix('compact-spliced.floe'); + $compacted = $this->cacheDir->suffix('compact-compacted.floe'); + FloeStreamReaderContext::write( + $this->fs(), + $a, + rows(schema(int_schema('id')), row(['id' => 1]), row(['id' => 2])), + ); + FloeStreamReaderContext::write($this->fs(), $b, rows(schema(int_schema('id')), row(['id' => 3]))); + + merge_floe([$a, $b], $spliced); + merge_floe([$a, $b], $compacted, compact: true); + + static::assertSame(3, FloeStreamReaderContext::footer($this->fs(), $spliced)->statistics->rows); + static::assertSame(3, FloeStreamReaderContext::footer($this->fs(), $compacted)->statistics->rows); + static::assertSame( + FloeStreamReaderContext::dataBytes($this->fs(), $compacted), + FloeStreamReaderContext::footer($this->fs(), $compacted)->statistics->byteSize, + ); + } + + public function test_a_merged_file_reports_the_output_byte_size_not_the_inputs(): void + { + $a = $this->cacheDir->suffix('splice-a.floe'); + $b = $this->cacheDir->suffix('splice-b.floe'); + $out = $this->cacheDir->suffix('splice-out.floe'); + FloeStreamReaderContext::write($this->fs(), $a, rows(schema(str_schema('name')), row(['name' => 'first']))); + FloeStreamReaderContext::write($this->fs(), $b, rows(schema(str_schema('name')), row(['name' => 'second']))); + + merge_floe([$a, $b], $out); + + $statistics = FloeStreamReaderContext::footer($this->fs(), $out)->statistics; + + static::assertSame(FloeStreamReaderContext::dataBytes($this->fs(), $out), $statistics->byteSize); + static::assertSame( + FloeStreamReaderContext::footer($this->fs(), $a)->statistics->byteSize + + FloeStreamReaderContext::footer($this->fs(), $b)->statistics->byteSize, + $statistics->byteSize, + ); + static::assertLessThan( + (int) $this->fs()->readFrom($a)->size() + (int) $this->fs()->readFrom($b)->size(), + $statistics->byteSize, + ); + } + + public function test_a_multi_section_file_sums_its_sections(): void + { + $path = $this->cacheDir->suffix('sections.floe'); + FloeStreamReaderContext::write($this->fs(), $path, rows(schema(int_schema('id')), row(['id' => 1]))); + + foreach ([[2, 3], [4, 5, 6]] as $ids) { + $writer = new FloeWriter($this->fs(), schema(int_schema('id'))); + $writer->append($path); + + foreach ($ids as $id) { + $writer->write(rows(schema(int_schema('id')), row(['id' => $id]))); + } + + $writer->close(); + } + + $footer = FloeStreamReaderContext::footer($this->fs(), $path); + $sectionRows = 0; + + foreach ($footer->sections as $section) { + $sectionRows += $section->rowCount; + } + + static::assertCount(3, $footer->sections); + static::assertSame(6, $sectionRows); + static::assertSame(6, $footer->statistics->rows); + static::assertSame(FloeStreamReaderContext::dataBytes($this->fs(), $path), $footer->statistics->byteSize); + } + + public function test_a_written_file_reports_its_rows_and_byte_size(): void + { + $path = $this->cacheDir->suffix('written.floe'); + FloeStreamReaderContext::write( + $this->fs(), + $path, + rows(schema(int_schema('id')), row(['id' => 1]), row(['id' => 2])), + ); + + $statistics = FloeStreamReaderContext::footer($this->fs(), $path)->statistics; + + static::assertSame(2, $statistics->rows); + static::assertSame(FloeStreamReaderContext::dataBytes($this->fs(), $path), $statistics->byteSize); + static::assertGreaterThan(0, $statistics->byteSize); + } + + public function test_an_appended_file_reports_the_combined_totals(): void + { + $path = $this->cacheDir->suffix('appended.floe'); + FloeStreamReaderContext::write($this->fs(), $path, rows(schema(int_schema('id')), row(['id' => 1]))); + $before = FloeStreamReaderContext::footer($this->fs(), $path)->statistics->byteSize; + + $writer = new FloeWriter($this->fs(), schema(int_schema('id'))); + $writer->append($path); + $writer->write(rows(schema(int_schema('id')), row(['id' => 2]), row(['id' => 3]))); + $writer->close(); + + $statistics = FloeStreamReaderContext::footer($this->fs(), $path)->statistics; + + static::assertSame(3, $statistics->rows); + static::assertSame(FloeStreamReaderContext::dataBytes($this->fs(), $path), $statistics->byteSize); + static::assertGreaterThan($before, $statistics->byteSize); + } +} diff --git a/src/core/etl/tests/Flow/Floe/Tests/Mother/FooterMother.php b/src/core/etl/tests/Flow/Floe/Tests/Mother/FooterMother.php index 00b89b6359..a693414734 100644 --- a/src/core/etl/tests/Flow/Floe/Tests/Mother/FooterMother.php +++ b/src/core/etl/tests/Flow/Floe/Tests/Mother/FooterMother.php @@ -6,6 +6,7 @@ use Flow\ETL\Schema\Metadata; use Flow\Floe\Footer; +use Flow\Floe\Statistics; final class FooterMother { @@ -17,10 +18,18 @@ final class FooterMother public static function footer( array $schema = [], array $sections = [], - int $totalRows = 0, + int $rows = 0, + int $byteSize = 0, array $metadata = [], ): Footer { /** @var array> $schema */ - return new Footer(1, 'test-writer', $schema, $sections, $totalRows, Metadata::fromArray($metadata)); + return new Footer( + 1, + 'test-writer', + $schema, + $sections, + new Statistics($rows, $byteSize), + Metadata::fromArray($metadata), + ); } } diff --git a/src/core/etl/tests/Flow/Floe/Tests/Unit/FloeSerializerTest.php b/src/core/etl/tests/Flow/Floe/Tests/Unit/FloeSerializerTest.php index 76fe9e00f9..04da98eedc 100644 --- a/src/core/etl/tests/Flow/Floe/Tests/Unit/FloeSerializerTest.php +++ b/src/core/etl/tests/Flow/Floe/Tests/Unit/FloeSerializerTest.php @@ -177,10 +177,10 @@ public function test_unserialize_rejects_empty_payload(): void public function test_unserialize_rejects_row_count_mismatch(): void { $bytes = serialize_to_string(new FloeSerializer(), rows(schema(int_schema('id')), row(['id' => 1]))); - // inflate the footer's totalRows while the body still holds a single row; the JSON + // inflate the footer's row count while the body still holds a single row; the JSON // byte-length is unchanged (1 -> 2) so the trailer stays valid and the read reaches // the whole-value row-count guard - $corrupted = str_replace('"totalRows":1', '"totalRows":2', $bytes); + $corrupted = str_replace('"statistics":{"rows":1', '"statistics":{"rows":2', $bytes); $this->expectException(SerializationException::class); $this->expectExceptionMessage('decoded 1 of 2 rows'); diff --git a/src/core/etl/tests/Flow/Floe/Tests/Unit/FloeStreamWriterTest.php b/src/core/etl/tests/Flow/Floe/Tests/Unit/FloeStreamWriterTest.php index b3c9c8082b..5b5f9eee36 100644 --- a/src/core/etl/tests/Flow/Floe/Tests/Unit/FloeStreamWriterTest.php +++ b/src/core/etl/tests/Flow/Floe/Tests/Unit/FloeStreamWriterTest.php @@ -91,7 +91,7 @@ public function test_create_on_stream_round_trips(): void $footer = FloeStreamReaderContext::footer($filesystem, $path); - static::assertSame(3, $footer->totalRows); + static::assertSame(3, $footer->statistics->rows); static::assertSame(['source' => 'stream'], $footer->metadata->normalize()); static::assertSame( [Format::FRAME_ROW, Format::FRAME_ROW, Format::FRAME_ROW, Format::FRAME_FOOTER], @@ -173,7 +173,7 @@ public function test_batch_omitting_a_nullable_session_column_is_accepted_and_ro $writer->close(); $footer = FloeStreamReaderContext::footer($filesystem, $path); - static::assertSame(2, $footer->totalRows); + static::assertSame(2, $footer->statistics->rows); static::assertNotSame([], $footer->schema); static::assertCount(2, FloeStreamReaderContext::readAll($filesystem, $path)->all()); } @@ -262,7 +262,7 @@ public function test_empty_batch_writes_no_schema_or_section(): void static::assertSame([Format::FRAME_FOOTER], FloeStreamReaderContext::frameTypes($filesystem, $path)); $footer = FloeStreamReaderContext::footer($filesystem, $path); - static::assertSame(0, $footer->totalRows); + static::assertSame(0, $footer->statistics->rows); static::assertSame([], $footer->schema); static::assertCount(0, $footer->sections); } diff --git a/src/core/etl/tests/Flow/Floe/Tests/Unit/FloeWriterTest.php b/src/core/etl/tests/Flow/Floe/Tests/Unit/FloeWriterTest.php index 148fd10cff..c5d83e954a 100644 --- a/src/core/etl/tests/Flow/Floe/Tests/Unit/FloeWriterTest.php +++ b/src/core/etl/tests/Flow/Floe/Tests/Unit/FloeWriterTest.php @@ -88,7 +88,7 @@ public function test_append_on_missing_file_behaves_like_create(): void $writer->write(rows(schema(int_schema('id')), row(['id' => 1]))); $writer->close(); - static::assertSame(1, FloeStreamReaderContext::footer($filesystem, $path)->totalRows); + static::assertSame(1, FloeStreamReaderContext::footer($filesystem, $path)->statistics->rows); } public function test_append_on_empty_file_behaves_like_create(): void @@ -102,7 +102,7 @@ public function test_append_on_empty_file_behaves_like_create(): void $writer->write(rows(schema(int_schema('id')), row(['id' => 1]))); $writer->close(); - static::assertSame(1, FloeStreamReaderContext::footer($filesystem, $path)->totalRows); + static::assertSame(1, FloeStreamReaderContext::footer($filesystem, $path)->statistics->rows); } public function test_append_to_a_closed_zero_row_file_derives_the_schema_from_the_first_batch(): void @@ -151,7 +151,7 @@ public function test_create_with_an_explicit_schema_and_no_writes_records_the_sc $footer = FloeStreamReaderContext::footer($filesystem, $path); - static::assertSame(0, $footer->totalRows); + static::assertSame(0, $footer->statistics->rows); static::assertTrue($footer->schema()->isSame(schema(int_schema('id')))); } @@ -267,7 +267,7 @@ public function test_append_with_incompatible_schema_keeps_previous_rows_readabl $footer = FloeStreamReaderContext::footer($filesystem, $path); - static::assertSame(1, $footer->totalRows); + static::assertSame(1, $footer->statistics->rows); static::assertCount(1, $footer->sections); } @@ -333,7 +333,7 @@ public function test_append_with_same_schema_adds_a_section_reusing_the_footer_s $footer = FloeStreamReaderContext::footer($filesystem, $path); - static::assertSame(2, $footer->totalRows); + static::assertSame(2, $footer->statistics->rows); static::assertNotSame([], $footer->schema); static::assertCount(2, $footer->sections); } @@ -374,7 +374,7 @@ public function test_closing_empty_file_writes_header_and_footer_only(): void $footer = FloeStreamReaderContext::footer($filesystem, $path); - static::assertSame(0, $footer->totalRows); + static::assertSame(0, $footer->statistics->rows); static::assertSame([], $footer->sections); static::assertSame([], $footer->schema); } @@ -402,7 +402,7 @@ public function test_two_writes_with_the_same_schema_in_one_session_share_one_se $footer = FloeStreamReaderContext::footer($filesystem, $path); - static::assertSame(2, $footer->totalRows); + static::assertSame(2, $footer->statistics->rows); static::assertCount(1, $footer->sections); static::assertSame( [Format::FRAME_ROW, Format::FRAME_ROW, Format::FRAME_FOOTER], @@ -447,7 +447,7 @@ public function test_within_batch_heterogeneous_rows_encode_under_one_union_sche static::assertNotSame([], $footer->schema); static::assertCount(1, $footer->sections); - static::assertSame(3, $footer->totalRows); + static::assertSame(3, $footer->statistics->rows); static::assertSame(3, $footer->sections[0]->rowCount); $fileSchema = $footer->schema(); diff --git a/src/core/etl/tests/Flow/Floe/Tests/Unit/FooterReaderTest.php b/src/core/etl/tests/Flow/Floe/Tests/Unit/FooterReaderTest.php index e4ad28854b..28e5f7e642 100644 --- a/src/core/etl/tests/Flow/Floe/Tests/Unit/FooterReaderTest.php +++ b/src/core/etl/tests/Flow/Floe/Tests/Unit/FooterReaderTest.php @@ -38,7 +38,7 @@ public function test_reads_footer_location_of_a_written_file(): void $location = (new FooterReader())->read($fs->readFrom($path), new NoopCodec()); - static::assertSame(2, $location->footer->totalRows); + static::assertSame(2, $location->footer->statistics->rows); static::assertGreaterThan(Format::HEADER_LENGTH, $location->footerFrameStart); static::assertSame(Format::FRAME_FOOTER, ord($fs->readFrom($path)->read(1, $location->footerFrameStart))); } diff --git a/src/core/etl/tests/Flow/Floe/Tests/Unit/FooterTest.php b/src/core/etl/tests/Flow/Floe/Tests/Unit/FooterTest.php index ae6e9397c5..ecd34de58e 100644 --- a/src/core/etl/tests/Flow/Floe/Tests/Unit/FooterTest.php +++ b/src/core/etl/tests/Flow/Floe/Tests/Unit/FooterTest.php @@ -66,11 +66,30 @@ public function test_from_json_with_non_integer_version_throws(): void Footer::fromJson('{"version":"1"}'); } - public function test_from_json_with_non_integer_total_rows_throws(): void + public function test_a_footer_without_a_statistics_block_is_rejected(): void { $this->expectException(FloeException::class); + $this->expectExceptionMessage('Floe footer is malformed'); + + Footer::fromJson('{"version":2,"writer":"x","schema":[],"sections":[],"totalRows":0,"metadata":{}}'); + } + + public function test_malformed_statistics_are_rejected(): void + { + $this->expectException(FloeException::class); + $this->expectExceptionMessage('Floe footer statistics are malformed'); + + Footer::fromJson( + '{"version":2,"writer":"x","schema":[],"sections":[],"statistics":{"rows":"0","byteSize":0},"metadata":{}}', + ); + } - Footer::fromJson('{"version":1,"totalRows":"0"}'); + public function test_the_footer_carries_a_statistics_block(): void + { + static::assertSame( + ['rows' => 5, 'byteSize' => 240], + Footer::fromJson(FooterMother::footer(rows: 5, byteSize: 240)->toJson())->normalize()['statistics'], + ); } public function test_from_json_with_non_object_footer_throws(): void @@ -95,7 +114,7 @@ public function test_from_json_with_non_string_writer_throws(): void { $this->expectException(FloeException::class); - Footer::fromJson('{"version":1,"totalRows":0,"writer":42}'); + Footer::fromJson('{"version":1,"statistics":{"rows":0,"byteSize":0},"writer":42}'); } public function test_from_json_round_trips_typed_metadata(): void @@ -120,7 +139,9 @@ public function test_from_json_with_non_scalar_metadata_value_throws(): void $this->expectException(FloeException::class); $this->expectExceptionMessage('metadata is malformed'); - Footer::fromJson('{"version":1,"writer":"x","schema":[],"sections":[],"totalRows":0,"metadata":{"bad":null}}'); + Footer::fromJson( + '{"version":1,"writer":"x","schema":[],"sections":[],"statistics":{"rows":0,"byteSize":0},"metadata":{"bad":null}}', + ); } public function test_schema_body_re_encodes_stored_schema(): void @@ -139,7 +160,8 @@ public function test_to_json_round_trips_every_field(): void $footer = FooterMother::footer( schema: schema(int_schema('id'))->normalize(), sections: [new Section(6, 2), new Section(120, 3)], - totalRows: 5, + rows: 5, + byteSize: 240, metadata: ['source' => 'test'], ); @@ -151,20 +173,20 @@ public function test_normalize_round_trips_through_from_array(): void $footer = FooterMother::footer( schema: schema(int_schema('id'))->normalize(), sections: [new Section(6, 2), new Section(120, 3)], - totalRows: 5, + rows: 5, + byteSize: 240, metadata: ['source' => 'test'], ); static::assertEquals($footer, Footer::fromArray($footer->normalize())); } - public function test_a_footer_carrying_a_partitions_key_is_refused(): void + public function test_an_unknown_footer_key_is_ignored(): void { - // partitions come from the path, so a footer has no place to put them - $this->expectException(FloeException::class); - $this->expectExceptionMessage('Floe footer is malformed'); - - Footer::fromArray([...FooterMother::footer()->normalize(), 'partitions' => []]); + static::assertEquals( + FooterMother::footer(rows: 5), + Footer::fromArray([...FooterMother::footer(rows: 5)->normalize(), 'partitions' => []]), + ); } public function test_reconstruct_rows_round_trips_unpartitioned_rows(): void diff --git a/src/core/etl/tests/Flow/Floe/Tests/Unit/SectionTest.php b/src/core/etl/tests/Flow/Floe/Tests/Unit/SectionTest.php index 3dcf79d861..d8ce1455ab 100644 --- a/src/core/etl/tests/Flow/Floe/Tests/Unit/SectionTest.php +++ b/src/core/etl/tests/Flow/Floe/Tests/Unit/SectionTest.php @@ -24,11 +24,12 @@ public function test_from_array_with_non_integer_field_throws(): void Section::fromArray(['offset' => '6', 'partitionsId' => 0, 'rowCount' => 1]); } - public function test_from_array_with_non_integer_partitions_id_throws(): void + public function test_an_unknown_section_key_is_ignored(): void { - $this->expectException(FloeException::class); - - Section::fromArray(['offset' => 6, 'partitionsId' => '0', 'rowCount' => 1]); + static::assertEquals( + new Section(6, 1), + Section::fromArray(['offset' => 6, 'partitionsId' => '0', 'rowCount' => 1]), + ); } public function test_normalize_round_trips(): void diff --git a/src/core/etl/tests/Flow/Floe/Tests/Unit/StatisticsTest.php b/src/core/etl/tests/Flow/Floe/Tests/Unit/StatisticsTest.php new file mode 100644 index 0000000000..28c83e9440 --- /dev/null +++ b/src/core/etl/tests/Flow/Floe/Tests/Unit/StatisticsTest.php @@ -0,0 +1,51 @@ +expectException(FloeException::class); + $this->expectExceptionMessage('Floe footer statistics are malformed'); + + Statistics::fromArray(['rows' => 1]); + } + + public function test_an_unknown_key_is_ignored(): void + { + static::assertEquals( + new Statistics(3, 120), + Statistics::fromArray(['rows' => 3, 'byteSize' => 120, 'min' => 1]), + ); + } + + public function test_it_normalizes_to_rows_and_byte_size(): void + { + static::assertSame(['rows' => 3, 'byteSize' => 120], (new Statistics(3, 120))->normalize()); + } + + public function test_it_round_trips_through_from_array(): void + { + $statistics = new Statistics(3, 120); + + static::assertEquals($statistics, Statistics::fromArray($statistics->normalize())); + } + + #[TestWith([-1, 0, 'Floe statistics rows must not be negative, given: -1'])] + #[TestWith([0, -1, 'Floe statistics byte size must not be negative, given: -1'])] + public function test_negative_values_are_rejected(int $rows, int $byteSize, string $message): void + { + $this->expectException(FloeException::class); + $this->expectExceptionMessage($message); + + new Statistics($rows, $byteSize); + } +} diff --git a/src/extension/flow-php-ext/php/Flow/ETL/Adapter/CSV/RustCSVReaderNative.php b/src/extension/flow-php-ext/php/Flow/ETL/Adapter/CSV/RustCSVReaderNative.php index aac509ac5c..1a35add1bd 100644 --- a/src/extension/flow-php-ext/php/Flow/ETL/Adapter/CSV/RustCSVReaderNative.php +++ b/src/extension/flow-php-ext/php/Flow/ETL/Adapter/CSV/RustCSVReaderNative.php @@ -26,6 +26,17 @@ public function __construct( throw new RuntimeException('flow_php extension is not loaded'); } + /** + * The bytes, as read, of every row next() and fold() produced so far - line endings included, the header + * record excluded. + * + * @return int<0, max> + */ + public function consumedBytes(): int + { + throw new RuntimeException('flow_php extension is not loaded'); + } + public function feed(string $chunk): void { throw new RuntimeException('flow_php extension is not loaded'); diff --git a/src/extension/flow-php-ext/src/csv/mod.rs b/src/extension/flow-php-ext/src/csv/mod.rs index 979be674b8..6b7fc4e6ae 100644 --- a/src/extension/flow-php-ext/src/csv/mod.rs +++ b/src/extension/flow-php-ext/src/csv/mod.rs @@ -57,6 +57,7 @@ pub struct CsvReader { cells: Vec, record: Record, first_row_pending: bool, + consumed_bytes: u64, } impl CsvReader { @@ -95,6 +96,7 @@ impl CsvReader { cells: Vec::new(), record: Record::default(), first_row_pending: false, + consumed_bytes: 0, }) } @@ -106,6 +108,12 @@ impl CsvReader { self.tokenizer.finish(); } + /// The bytes, as read, of every row `next` and `fold` produced so far: line endings included, the header + /// record excluded. + pub fn consumed_bytes(&self) -> u64 { + self.consumed_bytes + } + /// The resolved header, `[]` until the first record is available. pub fn headers(&mut self) -> Vec<&[u8]> { self.resolve_headers(); @@ -127,7 +135,7 @@ impl CsvReader { }; while batch.len() < batch_size { - if !next_row(&mut self.first_row_pending, &mut self.tokenizer, &mut self.record) { + if !next_row(&mut self.first_row_pending, &mut self.tokenizer, &mut self.record, &mut self.consumed_bytes) { break; } @@ -152,7 +160,7 @@ impl CsvReader { let mut positions = vec![None; self.cells.len()]; while limit.is_none_or(|limit| folded < limit) { - if !next_row(&mut self.first_row_pending, &mut self.tokenizer, &mut self.record) { + if !next_row(&mut self.first_row_pending, &mut self.tokenizer, &mut self.record, &mut self.consumed_bytes) { break; } @@ -215,9 +223,21 @@ impl CsvReader { } } -/// The first row is pending when auto headers were sized by it; otherwise the next complete record. -fn next_row(first_row_pending: &mut bool, tokenizer: &mut Tokenizer, record: &mut Record) -> bool { - std::mem::take(first_row_pending) || tokenizer.next(record) +/// The first row is pending when auto headers were sized by it; otherwise the next complete record. Either way it is +/// the tokenizer's last record, so its bytes are added to `consumed_bytes`. +fn next_row( + first_row_pending: &mut bool, + tokenizer: &mut Tokenizer, + record: &mut Record, + consumed_bytes: &mut u64, +) -> bool { + if !(std::mem::take(first_row_pending) || tokenizer.next(record)) { + return false; + } + + *consumed_bytes += tokenizer.last_record_bytes() as u64; + + true } /// `CSVRowNormalizer::normalize()` for one cell: padding past the record's end, `emptyToNull`; `None` is null. diff --git a/src/extension/flow-php-ext/src/csv/tokenizer.rs b/src/extension/flow-php-ext/src/csv/tokenizer.rs index a3af8a56f0..1596d0ef35 100644 --- a/src/extension/flow-php-ext/src/csv/tokenizer.rs +++ b/src/extension/flow-php-ext/src/csv/tokenizer.rs @@ -68,6 +68,7 @@ pub struct Tokenizer { scan: Scan, finished: bool, first_record: bool, + last_record_bytes: usize, } impl Tokenizer { @@ -81,6 +82,7 @@ impl Tokenizer { scan: Scan::FieldStart, finished: false, first_record: true, + last_record_bytes: 0, } } @@ -98,6 +100,11 @@ impl Tokenizer { self.finished = true; } + /// The bytes the last record `next` returned took in the stream: line ending and BOM included. + pub fn last_record_bytes(&self) -> usize { + self.last_record_bytes + } + pub fn next(&mut self, record: &mut Record) -> bool { let Some(range) = self.next_record() else { return false; @@ -130,6 +137,8 @@ impl Tokenizer { None => return None, }; + self.last_record_bytes = self.record_start - record.start; + while record.end > record.start && matches!(self.buffer[record.end - 1], b'\r' | b'\n') { record.end -= 1; } diff --git a/src/extension/flow-php-ext/src/lib.rs b/src/extension/flow-php-ext/src/lib.rs index a22baa5a84..8072751f50 100644 --- a/src/extension/flow-php-ext/src/lib.rs +++ b/src/extension/flow-php-ext/src/lib.rs @@ -309,6 +309,11 @@ impl RustCSVReaderNative { Ok(self.reader.fold(&mut fold.fold, limit)? as i64) } + #[php(name = "consumedBytes")] + pub fn consumed_bytes(&self) -> i64 { + self.reader.consumed_bytes() as i64 + } + pub fn next(&mut self, batch_size: i64) -> PhpResult { let batch_size = usize::try_from(batch_size) .ok() diff --git a/src/extension/flow-php-ext/tests/phpt/054_csv_consumed_bytes.phpt b/src/extension/flow-php-ext/tests/phpt/054_csv_consumed_bytes.phpt new file mode 100644 index 0000000000..5f2ac050a5 --- /dev/null +++ b/src/extension/flow-php-ext/tests/phpt/054_csv_consumed_bytes.phpt @@ -0,0 +1,88 @@ +--TEST-- +native CSV consumedBytes() counts the bytes of the rows it produced exactly like the PHP path, over any chunk size +--SKIPIF-- + +--FILE-- +records() as $values) { + if (++$taken >= $rows) { + break; + } + } + + return $source->producedBytes(); +}; + +$native = static function (string $raw, bool $withHeader, int $rows, int $chunk): int { + $reader = new RustCSVReaderNative(',', '"', '\\', $withHeader, true, true); + $fold = new RustColumnFoldNative([], []); + $folded = 0; + + foreach (str_split($raw, $chunk) as $piece) { + $reader->feed($piece); + $folded += $reader->fold($fold, $rows - $folded); + } + + $reader->finish(); + $reader->fold($fold, $rows - $folded); + + return $reader->consumedBytes(); +}; + +$cases = [ + 'LF' => ["a,b\n1,2\n33,44\n", true, 2], + 'CRLF' => ["a,b\r\n1,2\r\n33,44\r\n", true, 2], + 'quoted multi-line LF' => ["a,b\n\"x\ny\",1\n2,3\n", true, 2], + 'quoted multi-line CRLF' => ["a,b\r\n\"x\r\ny\",1\r\n2,3\r\n", true, 2], + 'blank lines' => ["a,b\n\n1,2\n\n", true, 3], + 'BOM without header' => ["\xEF\xBB\xBF1,2\n3,4\n", false, 2], + 'first two of three rows' => ["a,b\n1,2\n33,44\n555,666\n", true, 2], +]; + +foreach ($cases as $label => [$raw, $withHeader, $rows]) { + $expected = $php($raw, $withHeader, $rows); + $counts = array_unique(array_map( + static fn(int $chunk): int => $native($raw, $withHeader, $rows, $chunk), + [1, 3, 7, 4096], + )); + + echo $label, ': ', $counts === [$expected] ? 'identical ' . $expected : 'FAIL php=' . $expected . ' native=' . implode(',', $counts), "\n"; +} + +$reader = new RustCSVReaderNative(',', '"', '\\', true, true, true); +$reader->feed("a,b\n1,2\n33,44\n"); +$reader->finish(); +$reader->next(1); +echo 'next() counts too: ', $reader->consumedBytes(), "\n"; +$reader->next(10); +echo 'header excluded: ', $reader->consumedBytes(), "\n"; +?> +--EXPECT-- +LF: identical 10 +CRLF: identical 12 +quoted multi-line LF: identical 12 +quoted multi-line CRLF: identical 15 +blank lines: identical 6 +BOM without header: identical 11 +first two of three rows: identical 10 +next() counts too: 4 +header excluded: 10 diff --git a/web/landing/assets/wasm/data/orders.floe b/web/landing/assets/wasm/data/orders.floe index 6dafe73507..3136f68f7a 100644 Binary files a/web/landing/assets/wasm/data/orders.floe and b/web/landing/assets/wasm/data/orders.floe differ diff --git a/web/landing/assets/wasm/tools/flow.phar b/web/landing/assets/wasm/tools/flow.phar index 1db86033b8..d0c4b09847 100755 Binary files a/web/landing/assets/wasm/tools/flow.phar and b/web/landing/assets/wasm/tools/flow.phar differ diff --git a/web/landing/content/examples/topics/joins/join_each/code.php b/web/landing/content/examples/topics/joins/join_each/code.php index 9cd26e543e..7ca7ccce64 100644 --- a/web/landing/content/examples/topics/joins/join_each/code.php +++ b/web/landing/content/examples/topics/joins/join_each/code.php @@ -4,6 +4,7 @@ use function Flow\ETL\DSL\{data_frame, df, equal, from_array, int_schema, join_on, row, rows, schema, str_schema, to_output}; use Flow\ETL\{DataFrame, DataFrameFactory, Extractor, FlowContext, Rows}; +use Flow\ETL\Extractor\Statistics; use Flow\ETL\Join\Join; use Flow\ETL\Schema; @@ -28,6 +29,11 @@ public function schema(): Schema return $this->schema; } + public function statistics(): Statistics + { + return new Statistics(); + } + public function withSchema(Schema $schema): static { $clone = clone $this; diff --git a/web/landing/resources/api.json b/web/landing/resources/api.json index 6ac7d65e0c..41752dda5e 100644 --- a/web/landing/resources/api.json +++ b/web/landing/resources/api.json @@ -1 +1 @@ -[{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":31,"slug":"and","name":"and","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"function","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"All","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":36,"slug":"andnot","name":"andNot","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"function","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"All","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":41,"slug":"append","name":"append","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"suffix","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Append","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":54,"slug":"arrayfilter","name":"arrayFilter","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ArrayFilter","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBGaWx0ZXJzIGFuIGFycmF5IGJ5IHJlbW92aW5nIGFsbCBlbGVtZW50cyB0aGF0IG1hdGNoZXMgcGFzc2VkIHZhbHVlLgogICAgICogQXBwbGljYWJsZSB0byBhbGwgZGF0YSBzdHJ1Y3R1cmVzIHRoYXQgY2FuIGJlIGNvbnZlcnRlZCB0byBhbiBhcnJheToKICAgICAqICAgIC0ganNvbgogICAgICogICAgLSBsaXN0CiAgICAgKiAgICAtIG1hcAogICAgICogICAgLSBzdHJ1Y3R1cmUuCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":59,"slug":"arrayget","name":"arrayGet","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"path","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ArrayGet","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":67,"slug":"arraygetcollection","name":"arrayGetCollection","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"keys","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ArrayGetCollection","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gU2NhbGFyRnVuY3Rpb258YXJyYXk8YXJyYXkta2V5LCBtaXhlZD4gJGtleXMKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":72,"slug":"arraygetcollectionfirst","name":"arrayGetCollectionFirst","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"keys","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"ArrayGetCollection","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":85,"slug":"arraykeep","name":"arrayKeep","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ArrayKeep","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBGaWx0ZXJzIGFuIGFycmF5IGJ5IGtlZXBpbmcgb25seSBlbGVtZW50cyB0aGF0IG1hdGNoZXMgcGFzc2VkIHZhbHVlLgogICAgICogQXBwbGljYWJsZSB0byBhbGwgZGF0YSBzdHJ1Y3R1cmVzIHRoYXQgY2FuIGJlIGNvbnZlcnRlZCB0byBhbiBhcnJheToKICAgICAqICAgLSBqc29uCiAgICAgKiAgIC0gbGlzdAogICAgICogICAtIG1hcAogICAgICogICAtIHN0cnVjdHVyZS4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":98,"slug":"arraykeys","name":"arrayKeys","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"ArrayKeys","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIGFsbCBrZXlzIGZyb20gYW4gYXJyYXksIGlnbm9yaW5nIHRoZSB2YWx1ZXMuCiAgICAgKiBBcHBsaWNhYmxlIHRvIGFsbCBkYXRhIHN0cnVjdHVyZXMgdGhhdCBjYW4gYmUgY29udmVydGVkIHRvIGFuIGFycmF5OgogICAgICogICAtIGpzb24KICAgICAqICAgLSBsaXN0CiAgICAgKiAgIC0gbWFwCiAgICAgKiAgIC0gc3RydWN0dXJlLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":106,"slug":"arraymerge","name":"arrayMerge","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ArrayMerge","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gU2NhbGFyRnVuY3Rpb258YXJyYXk8YXJyYXkta2V5LCBtaXhlZD4gJHJlZgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":111,"slug":"arraymergecollection","name":"arrayMergeCollection","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"ArrayMergeCollection","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":116,"slug":"arraypathexists","name":"arrayPathExists","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"path","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ArrayPathExists","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":121,"slug":"arrayreverse","name":"arrayReverse","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"preserveKeys","type":[{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"ArrayReverse","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":126,"slug":"arraysort","name":"arraySort","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"sortFunction","type":[{"name":"Sort","namespace":"Flow\\ETL\\Function\\ArraySort","is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"},{"name":"flags","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"},{"name":"recursive","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"true"}],"return_type":[{"name":"ArraySort","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":142,"slug":"arrayvalues","name":"arrayValues","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"ArrayValues","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIGFsbCB2YWx1ZXMgZnJvbSBhbiBhcnJheSwgaWdub3JpbmcgdGhlIGtleXMuCiAgICAgKiBBcHBsaWNhYmxlIHRvIGFsbCBkYXRhIHN0cnVjdHVyZXMgdGhhdCBjYW4gYmUgY29udmVydGVkIHRvIGFuIGFycmF5OgogICAgICogICAtIGpzb24KICAgICAqICAgLSBsaXN0CiAgICAgKiAgIC0gbWFwCiAgICAgKiAgIC0gc3RydWN0dXJlLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":147,"slug":"ascii","name":"ascii","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Ascii","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":157,"slug":"between","name":"between","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"lowerBoundRef","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null},{"name":"upperBoundRef","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null},{"name":"boundary","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"Boundary","namespace":"Flow\\ETL\\Function\\Between","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Function\\Between\\Boundary::..."}],"return_type":[{"name":"Between","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gbWl4ZWR8U2NhbGFyRnVuY3Rpb24gJGxvd2VyQm91bmRSZWYKICAgICAqIEBwYXJhbSBtaXhlZHxTY2FsYXJGdW5jdGlvbiAkdXBwZXJCb3VuZFJlZgogICAgICogQHBhcmFtIEJvdW5kYXJ5fFNjYWxhckZ1bmN0aW9uICRib3VuZGFyeQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":165,"slug":"binarylength","name":"binaryLength","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"BinaryLength","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":174,"slug":"call","name":"call","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"callable","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"returnType","type":[{"name":"Type","namespace":"Flow\\Types","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"arguments","type":[{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"[]"},{"name":"refAlias","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"}],"return_type":[{"name":"CallUserFunc","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gYXJyYXk8YXJyYXkta2V5LCBtaXhlZD4gJGFyZ3VtZW50cwogICAgICogQHBhcmFtIFR5cGU8bWl4ZWQ+ICRyZXR1cm5UeXBlCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":183,"slug":"capitalize","name":"capitalize","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Capitalize","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":191,"slug":"cast","name":"cast","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"type","type":[{"name":"Type","namespace":"Flow\\Types","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Cast","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gc3RyaW5nfFR5cGU8bWl4ZWQ+ICR0eXBlCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":196,"slug":"chunk","name":"chunk","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"size","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Chunk","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":201,"slug":"coalesce","name":"coalesce","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"params","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"Coalesce","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":206,"slug":"codepointlength","name":"codePointLength","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"CodePointLength","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":211,"slug":"collapsewhitespace","name":"collapseWhitespace","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"CollapseWhitespace","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":216,"slug":"concat","name":"concat","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"params","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"Concat","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":221,"slug":"concatwithseparator","name":"concatWithSeparator","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"separator","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"params","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"ConcatWithSeparator","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":228,"slug":"contains","name":"contains","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Contains","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":233,"slug":"dateformat","name":"dateFormat","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"format","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'Y-m-d'"}],"return_type":[{"name":"DateTimeFormat","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":238,"slug":"datetimeformat","name":"dateTimeFormat","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"format","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'Y-m-d H:i:s'"}],"return_type":[{"name":"DateTimeFormat","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":243,"slug":"divide","name":"divide","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"float","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"scale","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"},{"name":"rounding","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"Rounding","namespace":"Flow\\Calculator","is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"Divide","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":251,"slug":"domelementattributescount","name":"domElementAttributesCount","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"DOMElementAttributesCount","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":256,"slug":"domelementattributevalue","name":"domElementAttributeValue","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"attribute","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"DOMElementAttributeValue","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":261,"slug":"domelementnamespace","name":"domElementNamespace","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"attribute","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"DOMElementNamespaceValue","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":266,"slug":"domelementnextsibling","name":"domElementNextSibling","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"DOMElementNextSibling","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":271,"slug":"domelementparent","name":"domElementParent","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"DOMElementParent","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":276,"slug":"domelementprevioussibling","name":"domElementPreviousSibling","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"DOMElementPreviousSibling","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":281,"slug":"domelementvalue","name":"domElementValue","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"DOMElementValue","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":286,"slug":"endswith","name":"endsWith","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"EndsWith","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":291,"slug":"ensureend","name":"ensureEnd","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"suffix","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"EnsureEnd","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":296,"slug":"ensurestart","name":"ensureStart","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"prefix","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"EnsureStart","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":301,"slug":"enumname","name":"enumName","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"EnumName","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":306,"slug":"enumvalue","name":"enumValue","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"EnumValue","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":311,"slug":"equals","name":"equals","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Equals","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":316,"slug":"exists","name":"exists","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Exists","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":345,"slug":"expand","name":"expand","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"expand","type":[{"name":"ArrayExpand","namespace":"Flow\\ETL\\Function\\ArrayExpand","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Function\\ArrayExpand\\ArrayExpand::..."}],"return_type":[{"name":"ArrayExpand","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBFeHBhbmRzIGVhY2ggdmFsdWUgaW50byBlbnRyeSwgaWYgdGhlcmUgYXJlIG1vcmUgdGhhbiBvbmUgdmFsdWUsIG11bHRpcGxlIHJvd3Mgd2lsbCBiZSBjcmVhdGVkLgogICAgICogQXJyYXkga2V5cyBhcmUgaWdub3JlZCwgb25seSB2YWx1ZXMgYXJlIHVzZWQgdG8gY3JlYXRlIG5ldyByb3dzLgogICAgICogTmVzdGVkIGluIGFub3RoZXIgZnVuY3Rpb24gKHN0cnVjdHVyZSgpLCBjb25jYXQoKSwgLi4uKSBpdCBzdGlsbCBnaXZlcyBvbmUgcm93IHBlciBlbGVtZW50LiBTZXZlcmFsCiAgICAgKiBleHBhbmRzIGluIG9uZSBleHByZXNzaW9uIGFyZSB6aXBwZWQgdG8gdGhlIGxvbmdlc3QgbGlzdDsgYSBzaG9ydGVyIG9uZSBnaXZlcyBudWxsLCBzbyBpdHMgZWxlbWVudAogICAgICogdHlwZSBiZWNvbWVzIG51bGxhYmxlLiBJdCBpcyByZWZ1c2VkIGluc2lkZSBhbm90aGVyIGFycmF5X2V4cGFuZCgpIGFuZCBpbiBmaWx0ZXIoKSwgdW50aWwoKSwKICAgICAqIGR1cGxpY2F0ZVJvdygpLCBhZ2dyZWdhdGUoKSwgb3ZlcigpIGFuZCBvbkVhY2goKS4KICAgICAqCiAgICAgKiBCZWZvcmU6CiAgICAgKiAgICstLSstLS0tLS0tLS0tLS0tLS0tLS0tKwogICAgICogICB8aWR8ICAgICAgICAgICAgICBhcnJheXwKICAgICAqICAgKy0tKy0tLS0tLS0tLS0tLS0tLS0tLS0rCiAgICAgKiAgIHwgMXx7ImEiOjEsImIiOjIsImMiOjN9fAogICAgICogICArLS0rLS0tLS0tLS0tLS0tLS0tLS0tLSsKICAgICAqCiAgICAgKiBBZnRlcjoKICAgICAqICAgKy0tKy0tLS0tLS0tKwogICAgICogICB8aWR8ZXhwYW5kZWR8CiAgICAgKiAgICstLSstLS0tLS0tLSsKICAgICAqICAgfCAxfCAgICAgICAxfAogICAgICogICB8IDF8ICAgICAgIDJ8CiAgICAgKiAgIHwgMXwgICAgICAgM3wKICAgICAqICAgKy0tKy0tLS0tLS0tKwogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":350,"slug":"greaterthan","name":"greaterThan","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"GreaterThan","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":355,"slug":"greaterthanequal","name":"greaterThanEqual","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"GreaterThanEqual","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":360,"slug":"hash","name":"hash","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"algorithm","type":[{"name":"Algorithm","namespace":"Flow\\ETL\\Hash","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Hash\\NativePHPHash::..."}],"return_type":[{"name":"Hash","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":365,"slug":"htmlqueryselector","name":"htmlQuerySelector","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"path","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"HTMLQuerySelector","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":370,"slug":"htmlqueryselectorall","name":"htmlQuerySelectorAll","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"path","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"HTMLQuerySelectorAll","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":378,"slug":"indexof","name":"indexOf","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"ignoreCase","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"},{"name":"offset","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"}],"return_type":[{"name":"IndexOf","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIHRoZSBpbmRleCBvZiBnaXZlbiAkbmVlZGxlIGluIHN0cmluZy4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":389,"slug":"indexoflast","name":"indexOfLast","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"ignoreCase","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"},{"name":"offset","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"}],"return_type":[{"name":"IndexOfLast","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIHRoZSBsYXN0IGluZGV4IG9mIGdpdmVuICRuZWVkbGUgaW4gc3RyaW5nLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":397,"slug":"isempty","name":"isEmpty","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"IsEmpty","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":402,"slug":"iseven","name":"isEven","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Equals","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":407,"slug":"isfalse","name":"isFalse","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Same","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":415,"slug":"isin","name":"isIn","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"haystack","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"IsIn","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gU2NhbGFyRnVuY3Rpb258YXJyYXk8YXJyYXkta2V5LCBtaXhlZD4gJGhheXN0YWNrCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":420,"slug":"isnotnull","name":"isNotNull","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"IsNotNull","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":425,"slug":"isnotnumeric","name":"isNotNumeric","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"IsNotNumeric","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":430,"slug":"isnull","name":"isNull","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"IsNull","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":435,"slug":"isnumeric","name":"isNumeric","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"IsNumeric","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":440,"slug":"isodd","name":"isOdd","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"NotEquals","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":445,"slug":"istrue","name":"isTrue","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Same","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":453,"slug":"istype","name":"isType","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"types","type":[{"name":"Type","namespace":"Flow\\Types","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"IsType","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gc3RyaW5nfFR5cGU8bWl4ZWQ+ICR0eXBlcwogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":465,"slug":"isutf8","name":"isUtf8","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"IsUtf8","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBDaGVjayBzdHJpbmcgaXMgdXRmOCBhbmQgcmV0dXJucyB0cnVlIG9yIGZhbHNlLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":470,"slug":"jsondecode","name":"jsonDecode","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"flags","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"4194304"}],"return_type":[{"name":"JsonDecode","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":475,"slug":"jsonencode","name":"jsonEncode","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"flags","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"4194304"}],"return_type":[{"name":"JsonEncode","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":480,"slug":"lessthan","name":"lessThan","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"LessThan","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":485,"slug":"lessthanequal","name":"lessThanEqual","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"LessThanEqual","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":490,"slug":"literal","name":"literal","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Literal","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":495,"slug":"lower","name":"lower","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"ToLower","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":500,"slug":"minus","name":"minus","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"float","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Minus","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":505,"slug":"mod","name":"mod","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Mod","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":510,"slug":"modifydatetime","name":"modifyDateTime","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"modifier","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ModifyDateTime","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":515,"slug":"multiply","name":"multiply","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"float","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Multiply","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":520,"slug":"notequals","name":"notEquals","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"NotEquals","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":525,"slug":"notsame","name":"notSame","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"NotSame","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":530,"slug":"numberformat","name":"numberFormat","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"decimals","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"2"},{"name":"decimalSeparator","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'.'"},{"name":"thousandsSeparator","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"','"}],"return_type":[{"name":"NumberFormat","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":544,"slug":"oneach","name":"onEach","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"function","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"preserveKeys","type":[{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"true"}],"return_type":[{"name":"OnEach","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBFeGVjdXRlIGEgc2NhbGFyIGZ1bmN0aW9uIG9uIGVhY2ggZWxlbWVudCBvZiBhbiBhcnJheS9saXN0L21hcC9zdHJ1Y3R1cmUgZW50cnkuCiAgICAgKiBJbiBvcmRlciB0byB1c2UgdGhpcyBmdW5jdGlvbiwgeW91IG5lZWQgdG8gcHJvdmlkZSBhIHJlZmVyZW5jZSB0byB0aGUgImVsZW1lbnQiIHRoYXQgd2lsbCBiZSB1c2VkIGluIHRoZSBmdW5jdGlvbi4KICAgICAqCiAgICAgKiBFeGFtcGxlOiAkZGYtPndpdGhFbnRyeSgnYXJyYXknLCByZWYoJ2FycmF5JyktPm9uRWFjaChyZWYoJ2VsZW1lbnQnKS0+Y2FzdCh0eXBlX3N0cmluZygpKSkpCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":549,"slug":"or","name":"or","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"function","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Any","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":554,"slug":"ornot","name":"orNot","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"function","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Any","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":559,"slug":"plus","name":"plus","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"float","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Plus","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":564,"slug":"power","name":"power","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Power","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":569,"slug":"prepend","name":"prepend","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"prefix","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Prepend","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":574,"slug":"regex","name":"regex","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"pattern","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"flags","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"},{"name":"offset","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"}],"return_type":[{"name":"Regex","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":579,"slug":"regexall","name":"regexAll","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"pattern","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"flags","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"},{"name":"offset","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"}],"return_type":[{"name":"RegexAll","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":584,"slug":"regexmatch","name":"regexMatch","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"pattern","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"flags","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"},{"name":"offset","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"}],"return_type":[{"name":"RegexMatch","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":592,"slug":"regexmatchall","name":"regexMatchAll","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"pattern","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"flags","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"},{"name":"offset","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"}],"return_type":[{"name":"RegexMatchAll","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":600,"slug":"regexreplace","name":"regexReplace","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"pattern","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"replacement","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"limit","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"RegexReplace","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":608,"slug":"repeat","name":"repeat","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"times","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Repeat","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":613,"slug":"reverse","name":"reverse","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Reverse","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":618,"slug":"round","name":"round","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"precision","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"2"},{"name":"mode","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"1"}],"return_type":[{"name":"Round","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":623,"slug":"same","name":"same","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Same","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":628,"slug":"sanitize","name":"sanitize","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"placeholder","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'*'"},{"name":"skipCharacters","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"Sanitize","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":635,"slug":"size","name":"size","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Size","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":643,"slug":"slug","name":"slug","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"separator","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'-'"},{"name":"locale","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"},{"name":"symbolsMap","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"Slug","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gU2NhbGFyRnVuY3Rpb258bnVsbHxhcnJheTxhcnJheS1rZXksIG1peGVkPiAkc3ltYm9sc01hcAogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":651,"slug":"split","name":"split","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"separator","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"limit","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"9223372036854775807"}],"return_type":[{"name":"Split","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":656,"slug":"sprintf","name":"sprintf","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"params","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"float","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":true,"default_value":null}],"return_type":[{"name":"Sprintf","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":661,"slug":"startswith","name":"startsWith","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"StartsWith","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":669,"slug":"stringafter","name":"stringAfter","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"includeNeedle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"StringAfter","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIHRoZSBjb250ZW50cyBmb3VuZCBhZnRlciB0aGUgZmlyc3Qgb2NjdXJyZW5jZSBvZiB0aGUgZ2l2ZW4gc3RyaW5nLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":677,"slug":"stringafterlast","name":"stringAfterLast","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"includeNeedle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"StringAfterLast","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIHRoZSBjb250ZW50cyBmb3VuZCBhZnRlciB0aGUgbGFzdCBvY2N1cnJlbmNlIG9mIHRoZSBnaXZlbiBzdHJpbmcuCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":687,"slug":"stringbefore","name":"stringBefore","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"includeNeedle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"StringBefore","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIHRoZSBjb250ZW50cyBmb3VuZCBiZWZvcmUgdGhlIGZpcnN0IG9jY3VycmVuY2Ugb2YgdGhlIGdpdmVuIHN0cmluZy4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":697,"slug":"stringbeforelast","name":"stringBeforeLast","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"includeNeedle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"StringBeforeLast","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIHRoZSBjb250ZW50cyBmb3VuZCBiZWZvcmUgdGhlIGxhc3Qgb2NjdXJyZW5jZSBvZiB0aGUgZ2l2ZW4gc3RyaW5nLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":707,"slug":"stringcontainsany","name":"stringContainsAny","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needles","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"StringContainsAny","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gYXJyYXk8c3RyaW5nPnxTY2FsYXJGdW5jdGlvbiAkbmVlZGxlcwogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":712,"slug":"stringequalsto","name":"stringEqualsTo","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"string","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"StringEqualsTo","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":720,"slug":"stringfold","name":"stringFold","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"StringFold","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIGEgc3RyaW5nIHRoYXQgeW91IGNhbiB1c2UgaW4gY2FzZS1pbnNlbnNpdGl2ZSBjb21wYXJpc29ucy4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":725,"slug":"stringmatch","name":"stringMatch","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"pattern","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"StringMatch","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":730,"slug":"stringmatchall","name":"stringMatchAll","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"pattern","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"StringMatchAll","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":735,"slug":"stringnormalize","name":"stringNormalize","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"form","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"16"}],"return_type":[{"name":"StringNormalize","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":744,"slug":"stringstyle","name":"stringStyle","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"style","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"StringStyles","namespace":"Flow\\ETL\\String","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"StringStyle","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBDb3ZlcnQgc3RyaW5nIHRvIGEgc3R5bGUgZnJvbSBlbnVtIGxpc3QsIHBhc3NlZCBpbiBwYXJhbWV0ZXIuCiAgICAgKiBDYW4gYmUgc3RyaW5nICJ1cHBlciIgb3IgU3RyaW5nU3R5bGVzOjpVUFBFUiBmb3IgVXBwZXIgKGV4YW1wbGUpLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":752,"slug":"stringtitle","name":"stringTitle","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"allWords","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"StringTitle","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBDaGFuZ2VzIGFsbCBncmFwaGVtZXMvY29kZSBwb2ludHMgdG8gInRpdGxlIGNhc2UiLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":757,"slug":"stringwidth","name":"stringWidth","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"StringWidth","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":762,"slug":"strpad","name":"strPad","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"length","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"pad_string","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"' '"},{"name":"type","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"1"}],"return_type":[{"name":"StrPad","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":767,"slug":"strpadboth","name":"strPadBoth","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"length","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"pad_string","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"' '"}],"return_type":[{"name":"StrPad","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":772,"slug":"strpadleft","name":"strPadLeft","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"length","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"pad_string","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"' '"}],"return_type":[{"name":"StrPad","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":777,"slug":"strpadright","name":"strPadRight","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"length","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"pad_string","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"' '"}],"return_type":[{"name":"StrPad","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":786,"slug":"strreplace","name":"strReplace","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"search","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"replace","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"StrReplace","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gYXJyYXk8c3RyaW5nPnxTY2FsYXJGdW5jdGlvbnxzdHJpbmcgJHNlYXJjaAogICAgICogQHBhcmFtIGFycmF5PHN0cmluZz58U2NhbGFyRnVuY3Rpb258c3RyaW5nICRyZXBsYWNlCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":794,"slug":"todate","name":"toDate","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"format","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'Y-m-d\\\\TH:i:sP'"},{"name":"timeZone","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"DateTimeZone","namespace":"","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"DateTimeZone::..."}],"return_type":[{"name":"ToDate","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gU2NhbGFyRnVuY3Rpb258c3RyaW5nICRmb3JtYXQgLSBjdXJyZW50IGZvcm1hdCBvZiB0aGUgZGF0ZSB0aGF0IHdpbGwgYmUgdXNlZCB0byBjcmVhdGUgRGF0ZVRpbWVJbW11dGFibGUgaW5zdGFuY2UKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":805,"slug":"todatetime","name":"toDateTime","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"format","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'Y-m-d H:i:s'"},{"name":"timeZone","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"DateTimeZone","namespace":"","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"DateTimeZone::..."}],"return_type":[{"name":"ToDateTime","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gU2NhbGFyRnVuY3Rpb258c3RyaW5nICRmb3JtYXQgLSBjdXJyZW50IGZvcm1hdCBvZiB0aGUgZGF0ZSB0aGF0IHdpbGwgYmUgdXNlZCB0byBjcmVhdGUgRGF0ZVRpbWVJbW11dGFibGUgaW5zdGFuY2UKICAgICAqIEBwYXJhbSBcRGF0ZVRpbWVab25lfFNjYWxhckZ1bmN0aW9uICR0aW1lWm9uZQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":812,"slug":"trim","name":"trim","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"type","type":[{"name":"Type","namespace":"Flow\\ETL\\Function\\Trim","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Function\\Trim\\Type::..."},{"name":"characters","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"' \\t\\n\\r\\0\u000b'"}],"return_type":[{"name":"Trim","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":817,"slug":"truncate","name":"truncate","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"length","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"ellipsis","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'...'"}],"return_type":[{"name":"Truncate","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":822,"slug":"unicodelength","name":"unicodeLength","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"UnicodeLength","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":846,"slug":"unpack","name":"unpack","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"schema","type":[{"name":"Schema","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ArrayUnpack","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBVbnBhY2tzIGVhY2ggZWxlbWVudCBvZiBhbiBhcnJheSBpbnRvIGEgbmV3IGVudHJ5LCB1c2luZyB0aGUgYXJyYXkga2V5IGFzIHRoZSBlbnRyeSBuYW1lLgogICAgICoKICAgICAqIEJlZm9yZToKICAgICAqICAgKy0tKy0tLS0tLS0tLS0tLS0tLS0tLS0rCiAgICAgKiAgIHxpZHwgICAgICAgICAgICAgIGFycmF5fAogICAgICogICArLS0rLS0tLS0tLS0tLS0tLS0tLS0tLSsKICAgICAqICAgfCAxfHsiYSI6MSwiYiI6MiwiYyI6M318CiAgICAgKiAgIHwgMnx7ImQiOjQsImUiOjUsImYiOjZ9fAogICAgICogICArLS0rLS0tLS0tLS0tLS0tLS0tLS0tLSsKICAgICAqCiAgICAgKiBBZnRlcjoKICAgICAqICAgKy0tKy0tLS0tKy0tLS0tKy0tLS0tKy0tLS0tKy0tLS0tKwogICAgICogICB8aWR8YXJyLmJ8YXJyLmN8YXJyLmR8YXJyLmV8YXJyLmZ8CiAgICAgKiAgICstLSstLS0tLSstLS0tLSstLS0tLSstLS0tLSstLS0tLSsKICAgICAqICAgfCAxfCAgICAyfCAgICAzfCAgICAgfCAgICAgfCAgICAgfAogICAgICogICB8IDJ8ICAgICB8ICAgICB8ICAgIDR8ICAgIDV8ICAgIDZ8CiAgICAgKiAgICstLSstLS0tLSstLS0tLSstLS0tLSstLS0tLSstLS0tLSsKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":851,"slug":"upper","name":"upper","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"ToUpper","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":856,"slug":"wordwrap","name":"wordwrap","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"width","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"break","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'\\n'"},{"name":"cut","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"Wordwrap","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":864,"slug":"xpath","name":"xpath","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"string","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"XPath","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ResolvesFromChildren.php","start_line_in_file":12,"slug":"resolved","name":"resolved","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ResolvesFromChildren.php","start_line_in_file":23,"slug":"deterministic","name":"deterministic","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Flow.php","start_line_in_file":23,"slug":"setup","name":"setUp","class":"Flow\\ETL\\Flow","class_slug":"flow","parameters":[{"name":"config","type":[{"name":"ConfigBuilder","namespace":"Flow\\ETL\\Config","is_nullable":false,"is_variadic":false},{"name":"Config","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Flow.php","start_line_in_file":28,"slug":"extract","name":"extract","class":"Flow\\ETL\\Flow","class_slug":"flow","parameters":[{"name":"extractor","type":[{"name":"Extractor","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"DataFrame","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Flow.php","start_line_in_file":33,"slug":"from","name":"from","class":"Flow\\ETL\\Flow","class_slug":"flow","parameters":[{"name":"extractor","type":[{"name":"Extractor","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"DataFrame","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Flow.php","start_line_in_file":38,"slug":"process","name":"process","class":"Flow\\ETL\\Flow","class_slug":"flow","parameters":[{"name":"rows","type":[{"name":"Rows","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"DataFrame","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Flow.php","start_line_in_file":46,"slug":"read","name":"read","class":"Flow\\ETL\\Flow","class_slug":"flow","parameters":[{"name":"extractor","type":[{"name":"Extractor","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"DataFrame","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBBbGlhcyBmb3IgRmxvdzo6ZXh0cmFjdCBmdW5jdGlvbi4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":73,"slug":"aggregate","name":"aggregate","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"aggregations","type":[{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"algorithm","type":[{"name":"GroupByAlgorithmBuilder","namespace":"Flow\\ETL\\Config\\Grouping","is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEBwYXJhbSBBZ2dyZWdhdGlvbnMgJGFnZ3JlZ2F0aW9ucwogICAgICogQHBhcmFtIG51bGx8R3JvdXBCeUFsZ29yaXRobUJ1aWxkZXIgJGFsZ29yaXRobSBudWxsIGRlZmVycyB0byBjb25maWd1cmF0aW9uOyBhIGJ1aWxkZXIgcGlucyB0aGUKICAgICAqICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhbGdvcml0aG0gZm9yIHRoaXMgb3BlcmF0aW9uIGFuZCBza2lwcyBhbnkgYXV0b21hdGljIGNob2ljZQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":97,"slug":"batchby","name":"batchBy","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"column","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"minSize","type":[{"name":"int","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBNZXJnZS9TcGxpdCBSb3dzIHlpZWxkZWQgYnkgRXh0cmFjdG9yIGludG8gYmF0Y2hlcyBidXQga2VlcCB0aG9zZSB3aXRoIGNvbW1vbiB2YWx1ZSBpbiBnaXZlbiBjb2x1bW4gdG9nZXRoZXIuCiAgICAgKiBUaGlzIHdvcmtzIHByb3Blcmx5IG9ubHkgb24gc29ydGVkIGRhdGFzZXRzLgogICAgICoKICAgICAqIFdoZW4gbWluU2l6ZSBpcyBub3QgcHJvdmlkZWQsIGJhdGNoZXMgd2lsbCBiZSBjcmVhdGVkIG9ubHkgd2hlbiB0aGVyZSBpcyBhIGNoYW5nZSBpbiB2YWx1ZSBvZiB0aGUgY29sdW1uLgogICAgICogV2hlbiBtaW5TaXplIGlzIHByb3ZpZGVkLCBiYXRjaGVzIHdpbGwgYmUgY3JlYXRlZCBvbmx5IHdoZW4gdGhlcmUgaXMgYSBjaGFuZ2UgaW4gdmFsdWUgb2YgdGhlIGNvbHVtbiBvcgogICAgICogd2hlbiB0aGVyZSBhcmUgYXQgbGVhc3QgbWluU2l6ZSByb3dzIGluIHRoZSBiYXRjaC4KICAgICAqCiAgICAgKiBAcGFyYW0gUmVmZXJlbmNlfHN0cmluZyAkY29sdW1uIC0gY29sdW1uIHRvIGdyb3VwIGJ5IChhbGwgcm93cyB3aXRoIHNhbWUgdmFsdWUgc3RheSB0b2dldGhlcikKICAgICAqIEBwYXJhbSBudWxsfGludDwxLCBtYXg+ICRtaW5TaXplIC0gb3B0aW9uYWwgbWluaW11bSByb3dzIHBlciBiYXRjaCBmb3IgZWZmaWNpZW5jeQogICAgICoKICAgICAqIEBsYXp5CiAgICAgKgogICAgICogQHRocm93cyBJbnZhbGlkQXJndW1lbnRFeGNlcHRpb24KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":117,"slug":"batchsize","name":"batchSize","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"size","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBNZXJnZS9TcGxpdCBSb3dzIHlpZWxkZWQgYnkgRXh0cmFjdG9yIGludG8gYmF0Y2hlcyBvZiBnaXZlbiBzaXplLgogICAgICogRm9yIGV4YW1wbGUsIHdoZW4gRXh0cmFjdG9yIGlzIHlpZWxkaW5nIG9uZSByb3cgYXQgdGltZSwgdGhpcyBtZXRob2Qgd2lsbCBtZXJnZSB0aGVtIGludG8gYmF0Y2hlcyBvZiBnaXZlbiBzaXplCiAgICAgKiBiZWZvcmUgcGFzc2luZyB0aGVtIHRvIHRoZSBuZXh0IHBpcGVsaW5lIGVsZW1lbnQuCiAgICAgKiBTaW1pbGFybHkgd2hlbiBFeHRyYWN0b3IgaXMgeWllbGRpbmcgYmF0Y2hlcyBvZiByb3dzLCB0aGlzIG1ldGhvZCB3aWxsIHNwbGl0IHRoZW0gaW50byBzbWFsbGVyIGJhdGNoZXMgb2YgZ2l2ZW4KICAgICAqIHNpemUuCiAgICAgKgogICAgICogSW4gb3JkZXIgdG8gbWVyZ2UgYWxsIFJvd3MgaW50byBhIHNpbmdsZSBiYXRjaCB1c2UgRGF0YUZyYW1lOjpjb2xsZWN0KCkgbWV0aG9kIG9yIHNldCBzaXplIHRvIC0xIG9yIDAuCiAgICAgKgogICAgICogQHBhcmFtIGludDwtMSwgbWF4PiAkc2l6ZQogICAgICoKICAgICAqIEBsYXp5CiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":149,"slug":"cache","name":"cache","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"id","type":[{"name":"string","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"},{"name":"cacheBatchSize","type":[{"name":"int","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"},{"name":"cache","type":[{"name":"Cache","namespace":"Flow\\ETL","is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBTdGFydCBwcm9jZXNzaW5nIHJvd3MgdXAgdG8gdGhpcyBtb21lbnQgYW5kIHB1dCBlYWNoIGluc3RhbmNlIG9mIFJvd3MKICAgICAqIGludG8gcHJldmlvdXNseSBkZWZpbmVkIGNhY2hlLgogICAgICogQ2FjaGUgdHlwZSBjYW4gYmUgc2V0IHRocm91Z2ggQ29uZmlnQnVpbGRlci4KICAgICAqIEJ5IGRlZmF1bHQgZXZlcnl0aGluZyBpcyBjYWNoZWQgaW4gc3lzdGVtIHRtcCBkaXIuCiAgICAgKgogICAgICogSW1wb3J0YW50OiBjYWNoZSBiYXRjaCBzaXplIG1pZ2h0IHNpZ25pZmljYW50bHkgaW1wcm92ZSBwZXJmb3JtYW5jZSB3aGVuIHByb2Nlc3NpbmcgbGFyZ2UgYW1vdW50IG9mIHJvd3MuCiAgICAgKiBMYXJnZXIgYmF0Y2ggc2l6ZSB3aWxsIGluY3JlYXNlIG1lbW9yeSBjb25zdW1wdGlvbiBidXQgd2lsbCByZWR1Y2UgbnVtYmVyIG9mIElPIG9wZXJhdGlvbnMuCiAgICAgKiBXaGVuIG5vdCBzZXQsIHRoZSBiYXRjaCBzaXplIGlzIHRha2VuIGZyb20gdGhlIGxhc3QgRGF0YUZyYW1lOjpiYXRjaFNpemUoKSBjYWxsLgogICAgICoKICAgICAqIEBsYXp5CiAgICAgKgogICAgICogQHBhcmFtIG51bGx8c3RyaW5nICRpZAogICAgICogQHBhcmFtIG51bGx8Q2FjaGUgJGNhY2hlIHJlYWRzIG9mIHRoaXMgY2FjaGUgbXVzdCBwYXNzIHRoZSBzYW1lIGluc3RhbmNlIHRvIGZyb21fY2FjaGUoKQogICAgICoKICAgICAqIEB0aHJvd3MgSW52YWxpZEFyZ3VtZW50RXhjZXB0aW9uCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":166,"slug":"collect","name":"collect","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBCZWZvcmUgdHJhbnNmb3JtaW5nIHJvd3MsIGNvbGxlY3QgdGhlbSBhbmQgbWVyZ2UgaW50byBzaW5nbGUgUm93cyBpbnN0YW5jZS4KICAgICAqIFRoaXMgbWlnaHQgbGVhZCB0byBtZW1vcnkgaXNzdWVzIHdoZW4gcHJvY2Vzc2luZyBsYXJnZSBhbW91bnQgb2Ygcm93cywgdXNlIHdpdGggY2F1dGlvbi4KICAgICAqCiAgICAgKiBAbGF6eQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":185,"slug":"collectrefs","name":"collectRefs","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"references","type":[{"name":"References","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBUaGlzIG1ldGhvZCBhbGxvd3MgdG8gY29sbGVjdCByZWZlcmVuY2VzIHRvIGFsbCBlbnRyaWVzIHVzZWQgaW4gdGhpcyBwaXBlbGluZS4KICAgICAqCiAgICAgKiBgYGBwaHAKICAgICAqIChuZXcgRmxvdygpKQogICAgICogICAtPnJlYWQoRnJvbTo6Y2hhaW4oKSkKICAgICAqICAgLT5jb2xsZWN0UmVmcygkcmVmcyA9IHJlZnMoKSkKICAgICAqICAgLT5ydW4oKTsKICAgICAqIGBgYAogICAgICoKICAgICAqIEBsYXp5CiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":192,"slug":"constrain","name":"constrain","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"constraint","type":[{"name":"Constraint","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"constraints","type":[{"name":"Constraint","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":203,"slug":"count","name":"count","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAdHJpZ2dlcgogICAgICogUmV0dXJuIHRvdGFsIGNvdW50IG9mIHJvd3MgcHJvY2Vzc2VkIGJ5IHRoaXMgcGlwZWxpbmUuCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":222,"slug":"crossjoin","name":"crossJoin","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"dataFrame","type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"prefix","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"''"}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":239,"slug":"display","name":"display","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"limit","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"20"},{"name":"truncate","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"20"},{"name":"formatter","type":[{"name":"Formatter","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Formatter\\AsciiTableFormatter::..."}],"return_type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gaW50ICRsaW1pdCBtYXhpbXVtIG51bWJlcnMgb2Ygcm93cyB0byBkaXNwbGF5CiAgICAgKiBAcGFyYW0gYm9vbHxpbnQgJHRydW5jYXRlIGZhbHNlIG9yIGlmIHNldCB0byAwIGNvbHVtbnMgYXJlIG5vdCB0cnVuY2F0ZWQsIG90aGVyd2lzZSBkZWZhdWx0IHRydW5jYXRlIHRvIDIwCiAgICAgKiAgICAgICAgICAgICAgICAgICAgICAgICAgIGNoYXJhY3RlcnMKICAgICAqIEBwYXJhbSBGb3JtYXR0ZXIgJGZvcm1hdHRlcgogICAgICoKICAgICAqIEB0cmlnZ2VyCiAgICAgKgogICAgICogQHRocm93cyBJbnZhbGlkQXJndW1lbnRFeGNlcHRpb24KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":266,"slug":"drop","name":"drop","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"entries","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBEcm9wIGdpdmVuIGVudHJpZXMuCiAgICAgKgogICAgICogQGxhenkKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":280,"slug":"dropduplicates","name":"dropDuplicates","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"entries","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gUmVmZXJlbmNlfHN0cmluZyAuLi4kZW50cmllcwogICAgICoKICAgICAqIEBsYXp5CiAgICAgKgogICAgICogQHJldHVybiAkdGhpcwogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":287,"slug":"duplicaterow","name":"duplicateRow","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"condition","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null},{"name":"entries","type":[{"name":"WithEntry","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":307,"slug":"fetch","name":"fetch","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"limit","type":[{"name":"int","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"Rows","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBCZSBhd2FyZSB0aGF0IGZldGNoIGlzIG5vdCBtZW1vcnkgc2FmZSBhbmQgd2lsbCBsb2FkIGFsbCByb3dzIGludG8gbWVtb3J5LgogICAgICogSWYgeW91IHdhbnQgdG8gc2FmZWx5IGl0ZXJhdGUgb3ZlciBSb3dzIHVzZSBvZSBvZiB0aGUgZm9sbG93aW5nIG1ldGhvZHM6LgogICAgICoKICAgICAqIERhdGFGcmFtZTo6Z2V0KCkgOiBcR2VuZXJhdG9yCiAgICAgKiBEYXRhRnJhbWU6OmdldEFzQXJyYXkoKSA6IFxHZW5lcmF0b3IKICAgICAqIERhdGFGcmFtZTo6Z2V0RWFjaCgpIDogXEdlbmVyYXRvcgogICAgICogRGF0YUZyYW1lOjpnZXRFYWNoQXNBcnJheSgpIDogXEdlbmVyYXRvcgogICAgICoKICAgICAqIEB0cmlnZ2VyCiAgICAgKgogICAgICogQHRocm93cyBJbnZhbGlkQXJndW1lbnRFeGNlcHRpb24KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":324,"slug":"filter","name":"filter","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"function","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":336,"slug":"filters","name":"filters","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"functions","type":[{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEBwYXJhbSBhcnJheTxTY2FsYXJGdW5jdGlvbj4gJGZ1bmN0aW9ucwogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":350,"slug":"foreach","name":"forEach","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"callback","type":[{"name":"callable","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"void","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAdHJpZ2dlcgogICAgICoKICAgICAqIEBwYXJhbSBudWxsfGNhbGxhYmxlKFJvd3MgJHJvd3MsIEZsb3dDb250ZXh0ICRjb250ZXh0KSA6IHZvaWQgJGNhbGxiYWNrCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":371,"slug":"get","name":"get","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"Generator","namespace":"","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBZaWVsZHMgZWFjaCByb3cgYXMgYW4gaW5zdGFuY2Ugb2YgUm93cy4KICAgICAqCiAgICAgKiBAdHJpZ2dlcgogICAgICoKICAgICAqIEByZXR1cm4gXEdlbmVyYXRvcjxSb3dzPgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":390,"slug":"getasarray","name":"getAsArray","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"Generator","namespace":"","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBZaWVsZHMgZWFjaCByb3cgYXMgYW4gYXJyYXkuCiAgICAgKgogICAgICogQHRyaWdnZXIKICAgICAqCiAgICAgKiBAcmV0dXJuIFxHZW5lcmF0b3I8YXJyYXk8YXJyYXk8bWl4ZWQ+Pj4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":409,"slug":"geteach","name":"getEach","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"Generator","namespace":"","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBZaWVsZCBlYWNoIHJvdyBhcyBhbiBpbnN0YW5jZSBvZiBSb3cuCiAgICAgKgogICAgICogQHRyaWdnZXIKICAgICAqCiAgICAgKiBAcmV0dXJuIFxHZW5lcmF0b3I8Um93PgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":430,"slug":"geteachasarray","name":"getEachAsArray","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"Generator","namespace":"","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBZaWVsZCBlYWNoIHJvdyBhcyBhbiBhcnJheS4KICAgICAqCiAgICAgKiBAdHJpZ2dlcgogICAgICoKICAgICAqIEByZXR1cm4gXEdlbmVyYXRvcjxhcnJheTxtaXhlZD4+CiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":451,"slug":"groupby","name":"groupBy","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"entries","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"algorithm","type":[{"name":"GroupByAlgorithmBuilder","namespace":"Flow\\ETL\\Config\\Grouping","is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"GroupedDataFrame","namespace":"Flow\\ETL\\DataFrame","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEBwYXJhbSBHcm91cEJ5UmVmZXJlbmNlc3xSZWZlcmVuY2V8c3RyaW5nICRlbnRyaWVzIGEgc2luZ2xlIGNvbHVtbiBpcyBncm91cGVkIGJ5IG9uIGl0cyBvd24KICAgICAqIEBwYXJhbSBudWxsfEdyb3VwQnlBbGdvcml0aG1CdWlsZGVyICRhbGdvcml0aG0gbnVsbCBkZWZlcnMgdG8gY29uZmlndXJhdGlvbjsgYSBidWlsZGVyIHBpbnMgdGhlCiAgICAgKiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYWxnb3JpdGhtIGZvciB0aGlzIG9wZXJhdGlvbiBhbmQgc2tpcHMgYW55IGF1dG9tYXRpYyBjaG9pY2UKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":470,"slug":"join","name":"join","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"dataFrame","type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"on","type":[{"name":"Expression","namespace":"Flow\\ETL\\Join","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"type","type":[{"name":"Join","namespace":"Flow\\ETL\\Join","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Join\\Join::..."},{"name":"algorithm","type":[{"name":"JoinAlgorithmBuilder","namespace":"Flow\\ETL\\Config\\Join","is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEBwYXJhbSBudWxsfEpvaW5BbGdvcml0aG1CdWlsZGVyICRhbGdvcml0aG0gbnVsbCBkZWZlcnMgdG8gY29uZmlndXJhdGlvbjsgYSBidWlsZGVyIHBpbnMgdGhlIGFsZ29yaXRobQogICAgICogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZvciB0aGlzIG9wZXJhdGlvbiBhbmQgc2tpcHMgYW55IGF1dG9tYXRpYyBjaG9pY2UKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":498,"slug":"joineach","name":"joinEach","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"factory","type":[{"name":"DataFrameFactory","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"on","type":[{"name":"Expression","namespace":"Flow\\ETL\\Join","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"type","type":[{"name":"Join","namespace":"Flow\\ETL\\Join","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Join\\Join::..."}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBKb2lucyBpbiBtZW1vcnkgcGVyIGJhdGNoOyBpdCBpcyBub3QgZ292ZXJuZWQgYnkgdGhlIGpvaW4gYWxnb3JpdGhtIGFuZCB0YWtlcyBubyBhbGdvcml0aG0gb3ZlcnJpZGUuCiAgICAgKgogICAgICogQGxhenkKICAgICAqCiAgICAgKiBAcGFyYW0gc3RyaW5nfEpvaW4gJHR5cGUKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":514,"slug":"limit","name":"limit","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"limit","type":[{"name":"int","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEB0aHJvd3MgSW52YWxpZEFyZ3VtZW50RXhjZXB0aW9uCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":528,"slug":"load","name":"load","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"sink","type":[{"name":"Loader","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"Sink","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":574,"slug":"match","name":"match","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"schema","type":[{"name":"Schema","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"validator","type":[{"name":"SchemaValidator","namespace":"Flow\\ETL","is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEBwYXJhbSBudWxsfFNjaGVtYVZhbGlkYXRvciAkdmFsaWRhdG9yIC0gd2hlbiBudWxsLCBTdHJpY3RWYWxpZGF0b3IgZ2V0cyBpbml0aWFsaXplZAogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":595,"slug":"offset","name":"offset","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"offset","type":[{"name":"int","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBTa2lwIGdpdmVuIG51bWJlciBvZiByb3dzIGZyb20gdGhlIGJlZ2lubmluZyBvZiB0aGUgZGF0YXNldC4KICAgICAqIFdoZW4gJG9mZnNldCBpcyBudWxsLCBub3RoaW5nIGhhcHBlbnMgKG5vIHJvd3MgYXJlIHNraXBwZWQpLgogICAgICoKICAgICAqIFBlcmZvcm1hbmNlIE5vdGU6IERhdGFGcmFtZSBtdXN0IGl0ZXJhdGUgdGhyb3VnaCBhbmQgcHJvY2VzcyBhbGwgc2tpcHBlZCByb3dzCiAgICAgKiB0byByZWFjaCB0aGUgb2Zmc2V0IHBvc2l0aW9uLiBGb3IgbGFyZ2Ugb2Zmc2V0cywgdGhpcyBjYW4gaW1wYWN0IHBlcmZvcm1hbmNlCiAgICAgKiBhcyB0aGUgZGF0YSBzb3VyY2Ugc3RpbGwgbmVlZHMgdG8gYmUgcmVhZCBhbmQgcHJvY2Vzc2VkIHVwIHRvIHRoZSBvZmZzZXQgcG9pbnQuCiAgICAgKgogICAgICogQHBhcmFtID9pbnQ8MCwgbWF4PiAkb2Zmc2V0CiAgICAgKgogICAgICogQGxhenkKICAgICAqCiAgICAgKiBAdGhyb3dzIEludmFsaWRBcmd1bWVudEV4Y2VwdGlvbgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":609,"slug":"onerror","name":"onError","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"handler","type":[{"name":"ErrorHandler","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":621,"slug":"repartition","name":"repartition","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"entry","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"entries","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICogU2h1ZmZsZXMgdGhlIHN0cmVhbSBzbyBldmVyeSByb3cgc2hhcmluZyB0aGUgZ2l2ZW4gY29sdW1ucyBhcnJpdmVzIGluIG9uZSBiYXRjaC4gSXQgZG9lcyBub3QKICAgICAqIHdyaXRlIGRpcmVjdG9yaWVzIC0gdGhhdCBpcyBkZWNsYXJlZCBvbiB0aGUgbG9hZGVyLCBgdG9fY3N2KC4uLiktPnBhcnRpdGlvbkJ5KCdyZWdpb24nKWAuCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":633,"slug":"printrows","name":"printRows","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"limit","type":[{"name":"int","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"20"},{"name":"truncate","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"20"},{"name":"formatter","type":[{"name":"Formatter","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Formatter\\AsciiTableFormatter::..."}],"return_type":[{"name":"void","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAdHJpZ2dlcgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":653,"slug":"explain","name":"explain","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"trigger","type":[{"name":"Trigger","namespace":"Flow\\ETL\\Plan","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Plan\\Trigger::..."}],"return_type":[{"name":"Plan","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBUaGUgcGxhbiAkdHJpZ2dlciB3b3VsZCBydW4gb3ZlciB0aGlzIGZyYW1lLCBmcm96ZW46IGxhdGVyIHZlcmJzIG9uIHRoaXMgZnJhbWUgZG8gbm90IHJlYWNoIGl0LiBUaGUgZGVmYXVsdAogICAgICogYWRkcyBubyBjb25zdW1lciBvZiBpdHMgb3duIC0gaXQgZHJhd3MgdGhlIGZyYW1lIGFzIGJ1aWx0LCBpdHMgY2hhaW4gYW5kIGl0cyBzaW5rcy4gdG9TdHJpbmcoKSBwcmludHMgaXQgYXMgYQogICAgICogdHJlZS4gQW5zd2VycyBmcm9tIHRoZSBwbGFuIHdpdGhvdXQgcmVhZGluZyBhIHJvdy4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":663,"slug":"printschema","name":"printSchema","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"formatter","type":[{"name":"SchemaFormatter","namespace":"Flow\\ETL\\Schema","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Row\\Formatter\\ASCIISchemaFormatter::..."}],"return_type":[{"name":"void","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEB0aHJvd3MgU2NoZW1hTm90RGVyaXZhYmxlRXhjZXB0aW9uCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":671,"slug":"rename","name":"rename","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"from","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"to","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":678,"slug":"renameeach","name":"renameEach","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"strategies","type":[{"name":"RenameEntryStrategy","namespace":"Flow\\ETL\\Transformer\\Rename","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":689,"slug":"rows","name":"rows","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"transformer","type":[{"name":"Transformer","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"Transformation","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICogQWxpYXMgZm9yIEVUTDo6dHJhbnNmb3JtIG1ldGhvZC4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":706,"slug":"run","name":"run","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"analyze","type":[{"name":"Analyze","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"Report","namespace":"Flow\\ETL\\Dataset","is_nullable":true,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAdHJpZ2dlcgogICAgICoKICAgICAqIFdoZW4gYW5hbHl6aW5nIHBpcGVsaW5lIGV4ZWN1dGlvbiB3ZSBjYW4gY2hvc2UgdG8gY29sbGVjdCB2YXJpb3VzIG1ldHJpY3MgdGhyb3VnaCBhbmFseXplKCktPndpdGgqKCkgbWV0aG9kCiAgICAgKgogICAgICogLSBjb2x1bW4gc3RhdGlzdGljcyAtIGFuYWx5emUoKS0+d2l0aENvbHVtblN0YXRpc3RpY3MoKQogICAgICogLSBzY2hlbWEgLSBhbmFseXplKCktPndpdGhTY2hlbWEoKQogICAgICoKICAgICAqIEBwYXJhbSBBbmFseXplfGJvb2wgJGFuYWx5emUgLSB3aGVuIHNldCBydW4gd2lsbCByZXR1cm4gUmVwb3J0CiAgICAgKgogICAgICogQHJldHVybiAoJGFuYWx5emUgaXMgQW5hbHl6ZXx0cnVlID8gUmVwb3J0IDogbnVsbCkKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":731,"slug":"schema","name":"schema","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"Schema","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEB0aHJvd3MgU2NoZW1hTm90RGVyaXZhYmxlRXhjZXB0aW9uCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":744,"slug":"select","name":"select","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"entries","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICogS2VlcCBvbmx5IGdpdmVuIGVudHJpZXMuCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":758,"slug":"sortby","name":"sortBy","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"entries","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"algorithm","type":[{"name":"SortAlgorithmBuilder","namespace":"Flow\\ETL\\Config\\Sort","is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEBwYXJhbSBSZWZlcmVuY2V8U29ydFJlZmVyZW5jZXN8c3RyaW5nICRlbnRyaWVzIGEgc2luZ2xlIGNvbHVtbiBpcyBzb3J0ZWQgYnkgb24gaXRzIG93bgogICAgICogQHBhcmFtIG51bGx8U29ydEFsZ29yaXRobUJ1aWxkZXIgJGFsZ29yaXRobSBudWxsIGRlZmVycyB0byBjb25maWd1cmF0aW9uOyBhIGJ1aWxkZXIgcGlucyB0aGUgYWxnb3JpdGhtCiAgICAgKiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZm9yIHRoaXMgb3BlcmF0aW9uIGFuZCBza2lwcyBhbnkgYXV0b21hdGljIGNob2ljZQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":772,"slug":"transform","name":"transform","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"transformer","type":[{"name":"Transformer","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"Transformation","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"Transformations","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"WithEntry","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBBbGlhcyBmb3IgRGF0YUZyYW1lOjp3aXRoKCkuCiAgICAgKgogICAgICogQGxhenkKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":783,"slug":"until","name":"until","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"function","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBUaGUgZGlmZmVyZW5jZSBiZXR3ZWVuIGZpbHRlciBhbmQgdW50aWwgaXMgdGhhdCBmaWx0ZXIgd2lsbCBrZWVwIGZpbHRlcmluZyByb3dzIHVudGlsIGV4dHJhY3RvcnMgZmluaXNoIHlpZWxkaW5nCiAgICAgKiByb3dzLiBVbnRpbCB3aWxsIHNlbmQgYSBTVE9QIHNpZ25hbCB0byB0aGUgRXh0cmFjdG9yIHdoZW4gdGhlIGNvbmRpdGlvbiBpcyBub3QgbWV0LgogICAgICoKICAgICAqIEBsYXp5CiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":797,"slug":"void","name":"void","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICogVGhpcyBtZXRob2QgaXMgdXNlZnVsIG1vc3RseSBpbiBkZXZlbG9wbWVudCB3aGVuCiAgICAgKiB5b3Ugd2FudCB0byBwYXVzZSBwcm9jZXNzaW5nIGF0IGNlcnRhaW4gbW9tZW50IHdpdGhvdXQKICAgICAqIHJlbW92aW5nIGNvZGUuIEFsbCBvcGVyYXRpb25zIHdpbGwgZ2V0IHByb2Nlc3NlZCB1cCB0byB0aGlzIHBvaW50LAogICAgICogZnJvbSBoZXJlIG5vIHJvd3MgYXJlIHBhc3NlZCBmb3J3YXJkLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":807,"slug":"with","name":"with","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"transformer","type":[{"name":"Transformer","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"Transformation","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"Transformations","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"WithEntry","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":835,"slug":"withentries","name":"withEntries","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"references","type":[{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEBwYXJhbSBhcnJheTxpbnQsIFdpdGhFbnRyeT58YXJyYXk8c3RyaW5nLCBTY2FsYXJGdW5jdGlvbnxXaW5kb3dGdW5jdGlvbnxXaXRoRW50cnk+ICRyZWZlcmVuY2VzCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":853,"slug":"withentry","name":"withEntry","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"entry","type":[{"name":"Definition","namespace":"Flow\\ETL\\Schema","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"reference","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"WindowFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gRGVmaW5pdGlvbjxtaXhlZD58c3RyaW5nICRlbnRyeQogICAgICoKICAgICAqIEBsYXp5CiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":866,"slug":"write","name":"write","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"sink","type":[{"name":"Loader","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"Sink","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICogQWxpYXMgZm9yIEVUTDo6bG9hZCBmdW5jdGlvbi4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame\/GroupedDataFrame.php","start_line_in_file":21,"slug":"aggregate","name":"aggregate","class":"Flow\\ETL\\DataFrame\\GroupedDataFrame","class_slug":"groupeddataframe","parameters":[{"name":"aggregations","type":[{"name":"AggregatingFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"DataFrame","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame\/GroupedDataFrame.php","start_line_in_file":28,"slug":"pivot","name":"pivot","class":"Flow\\ETL\\DataFrame\\GroupedDataFrame","class_slug":"groupeddataframe","parameters":[{"name":"ref","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"values","type":[{"name":"PivotValues","namespace":"Flow\\ETL\\GroupBy","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null}] \ No newline at end of file +[{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":31,"slug":"and","name":"and","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"function","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"All","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":36,"slug":"andnot","name":"andNot","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"function","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"All","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":41,"slug":"append","name":"append","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"suffix","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Append","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":54,"slug":"arrayfilter","name":"arrayFilter","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ArrayFilter","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBGaWx0ZXJzIGFuIGFycmF5IGJ5IHJlbW92aW5nIGFsbCBlbGVtZW50cyB0aGF0IG1hdGNoZXMgcGFzc2VkIHZhbHVlLgogICAgICogQXBwbGljYWJsZSB0byBhbGwgZGF0YSBzdHJ1Y3R1cmVzIHRoYXQgY2FuIGJlIGNvbnZlcnRlZCB0byBhbiBhcnJheToKICAgICAqICAgIC0ganNvbgogICAgICogICAgLSBsaXN0CiAgICAgKiAgICAtIG1hcAogICAgICogICAgLSBzdHJ1Y3R1cmUuCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":59,"slug":"arrayget","name":"arrayGet","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"path","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ArrayGet","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":67,"slug":"arraygetcollection","name":"arrayGetCollection","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"keys","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ArrayGetCollection","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gU2NhbGFyRnVuY3Rpb258YXJyYXk8YXJyYXkta2V5LCBtaXhlZD4gJGtleXMKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":72,"slug":"arraygetcollectionfirst","name":"arrayGetCollectionFirst","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"keys","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"ArrayGetCollection","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":85,"slug":"arraykeep","name":"arrayKeep","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ArrayKeep","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBGaWx0ZXJzIGFuIGFycmF5IGJ5IGtlZXBpbmcgb25seSBlbGVtZW50cyB0aGF0IG1hdGNoZXMgcGFzc2VkIHZhbHVlLgogICAgICogQXBwbGljYWJsZSB0byBhbGwgZGF0YSBzdHJ1Y3R1cmVzIHRoYXQgY2FuIGJlIGNvbnZlcnRlZCB0byBhbiBhcnJheToKICAgICAqICAgLSBqc29uCiAgICAgKiAgIC0gbGlzdAogICAgICogICAtIG1hcAogICAgICogICAtIHN0cnVjdHVyZS4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":98,"slug":"arraykeys","name":"arrayKeys","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"ArrayKeys","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIGFsbCBrZXlzIGZyb20gYW4gYXJyYXksIGlnbm9yaW5nIHRoZSB2YWx1ZXMuCiAgICAgKiBBcHBsaWNhYmxlIHRvIGFsbCBkYXRhIHN0cnVjdHVyZXMgdGhhdCBjYW4gYmUgY29udmVydGVkIHRvIGFuIGFycmF5OgogICAgICogICAtIGpzb24KICAgICAqICAgLSBsaXN0CiAgICAgKiAgIC0gbWFwCiAgICAgKiAgIC0gc3RydWN0dXJlLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":106,"slug":"arraymerge","name":"arrayMerge","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ArrayMerge","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gU2NhbGFyRnVuY3Rpb258YXJyYXk8YXJyYXkta2V5LCBtaXhlZD4gJHJlZgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":111,"slug":"arraymergecollection","name":"arrayMergeCollection","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"ArrayMergeCollection","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":116,"slug":"arraypathexists","name":"arrayPathExists","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"path","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ArrayPathExists","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":121,"slug":"arrayreverse","name":"arrayReverse","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"preserveKeys","type":[{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"ArrayReverse","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":126,"slug":"arraysort","name":"arraySort","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"sortFunction","type":[{"name":"Sort","namespace":"Flow\\ETL\\Function\\ArraySort","is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"},{"name":"flags","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"},{"name":"recursive","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"true"}],"return_type":[{"name":"ArraySort","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":142,"slug":"arrayvalues","name":"arrayValues","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"ArrayValues","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIGFsbCB2YWx1ZXMgZnJvbSBhbiBhcnJheSwgaWdub3JpbmcgdGhlIGtleXMuCiAgICAgKiBBcHBsaWNhYmxlIHRvIGFsbCBkYXRhIHN0cnVjdHVyZXMgdGhhdCBjYW4gYmUgY29udmVydGVkIHRvIGFuIGFycmF5OgogICAgICogICAtIGpzb24KICAgICAqICAgLSBsaXN0CiAgICAgKiAgIC0gbWFwCiAgICAgKiAgIC0gc3RydWN0dXJlLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":147,"slug":"ascii","name":"ascii","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Ascii","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":157,"slug":"between","name":"between","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"lowerBoundRef","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null},{"name":"upperBoundRef","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null},{"name":"boundary","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"Boundary","namespace":"Flow\\ETL\\Function\\Between","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Function\\Between\\Boundary::..."}],"return_type":[{"name":"Between","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gbWl4ZWR8U2NhbGFyRnVuY3Rpb24gJGxvd2VyQm91bmRSZWYKICAgICAqIEBwYXJhbSBtaXhlZHxTY2FsYXJGdW5jdGlvbiAkdXBwZXJCb3VuZFJlZgogICAgICogQHBhcmFtIEJvdW5kYXJ5fFNjYWxhckZ1bmN0aW9uICRib3VuZGFyeQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":165,"slug":"binarylength","name":"binaryLength","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"BinaryLength","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":174,"slug":"call","name":"call","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"callable","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"returnType","type":[{"name":"Type","namespace":"Flow\\Types","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"arguments","type":[{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"[]"},{"name":"refAlias","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"}],"return_type":[{"name":"CallUserFunc","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gYXJyYXk8YXJyYXkta2V5LCBtaXhlZD4gJGFyZ3VtZW50cwogICAgICogQHBhcmFtIFR5cGU8bWl4ZWQ+ICRyZXR1cm5UeXBlCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":183,"slug":"capitalize","name":"capitalize","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Capitalize","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":191,"slug":"cast","name":"cast","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"type","type":[{"name":"Type","namespace":"Flow\\Types","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Cast","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gc3RyaW5nfFR5cGU8bWl4ZWQ+ICR0eXBlCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":196,"slug":"chunk","name":"chunk","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"size","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Chunk","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":201,"slug":"coalesce","name":"coalesce","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"params","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"Coalesce","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":206,"slug":"codepointlength","name":"codePointLength","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"CodePointLength","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":211,"slug":"collapsewhitespace","name":"collapseWhitespace","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"CollapseWhitespace","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":216,"slug":"concat","name":"concat","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"params","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"Concat","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":221,"slug":"concatwithseparator","name":"concatWithSeparator","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"separator","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"params","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"ConcatWithSeparator","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":228,"slug":"contains","name":"contains","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Contains","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":233,"slug":"dateformat","name":"dateFormat","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"format","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'Y-m-d'"}],"return_type":[{"name":"DateTimeFormat","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":238,"slug":"datetimeformat","name":"dateTimeFormat","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"format","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'Y-m-d H:i:s'"}],"return_type":[{"name":"DateTimeFormat","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":243,"slug":"divide","name":"divide","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"float","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"scale","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"},{"name":"rounding","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"Rounding","namespace":"Flow\\Calculator","is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"Divide","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":251,"slug":"domelementattributescount","name":"domElementAttributesCount","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"DOMElementAttributesCount","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":256,"slug":"domelementattributevalue","name":"domElementAttributeValue","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"attribute","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"DOMElementAttributeValue","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":261,"slug":"domelementnamespace","name":"domElementNamespace","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"attribute","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"DOMElementNamespaceValue","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":266,"slug":"domelementnextsibling","name":"domElementNextSibling","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"DOMElementNextSibling","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":271,"slug":"domelementparent","name":"domElementParent","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"DOMElementParent","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":276,"slug":"domelementprevioussibling","name":"domElementPreviousSibling","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"DOMElementPreviousSibling","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":281,"slug":"domelementvalue","name":"domElementValue","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"DOMElementValue","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":286,"slug":"endswith","name":"endsWith","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"EndsWith","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":291,"slug":"ensureend","name":"ensureEnd","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"suffix","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"EnsureEnd","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":296,"slug":"ensurestart","name":"ensureStart","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"prefix","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"EnsureStart","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":301,"slug":"enumname","name":"enumName","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"EnumName","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":306,"slug":"enumvalue","name":"enumValue","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"EnumValue","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":311,"slug":"equals","name":"equals","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Equals","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":316,"slug":"exists","name":"exists","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Exists","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":345,"slug":"expand","name":"expand","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"expand","type":[{"name":"ArrayExpand","namespace":"Flow\\ETL\\Function\\ArrayExpand","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Function\\ArrayExpand\\ArrayExpand::..."}],"return_type":[{"name":"ArrayExpand","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBFeHBhbmRzIGVhY2ggdmFsdWUgaW50byBlbnRyeSwgaWYgdGhlcmUgYXJlIG1vcmUgdGhhbiBvbmUgdmFsdWUsIG11bHRpcGxlIHJvd3Mgd2lsbCBiZSBjcmVhdGVkLgogICAgICogQXJyYXkga2V5cyBhcmUgaWdub3JlZCwgb25seSB2YWx1ZXMgYXJlIHVzZWQgdG8gY3JlYXRlIG5ldyByb3dzLgogICAgICogTmVzdGVkIGluIGFub3RoZXIgZnVuY3Rpb24gKHN0cnVjdHVyZSgpLCBjb25jYXQoKSwgLi4uKSBpdCBzdGlsbCBnaXZlcyBvbmUgcm93IHBlciBlbGVtZW50LiBTZXZlcmFsCiAgICAgKiBleHBhbmRzIGluIG9uZSBleHByZXNzaW9uIGFyZSB6aXBwZWQgdG8gdGhlIGxvbmdlc3QgbGlzdDsgYSBzaG9ydGVyIG9uZSBnaXZlcyBudWxsLCBzbyBpdHMgZWxlbWVudAogICAgICogdHlwZSBiZWNvbWVzIG51bGxhYmxlLiBJdCBpcyByZWZ1c2VkIGluc2lkZSBhbm90aGVyIGFycmF5X2V4cGFuZCgpIGFuZCBpbiBmaWx0ZXIoKSwgdW50aWwoKSwKICAgICAqIGR1cGxpY2F0ZVJvdygpLCBhZ2dyZWdhdGUoKSwgb3ZlcigpIGFuZCBvbkVhY2goKS4KICAgICAqCiAgICAgKiBCZWZvcmU6CiAgICAgKiAgICstLSstLS0tLS0tLS0tLS0tLS0tLS0tKwogICAgICogICB8aWR8ICAgICAgICAgICAgICBhcnJheXwKICAgICAqICAgKy0tKy0tLS0tLS0tLS0tLS0tLS0tLS0rCiAgICAgKiAgIHwgMXx7ImEiOjEsImIiOjIsImMiOjN9fAogICAgICogICArLS0rLS0tLS0tLS0tLS0tLS0tLS0tLSsKICAgICAqCiAgICAgKiBBZnRlcjoKICAgICAqICAgKy0tKy0tLS0tLS0tKwogICAgICogICB8aWR8ZXhwYW5kZWR8CiAgICAgKiAgICstLSstLS0tLS0tLSsKICAgICAqICAgfCAxfCAgICAgICAxfAogICAgICogICB8IDF8ICAgICAgIDJ8CiAgICAgKiAgIHwgMXwgICAgICAgM3wKICAgICAqICAgKy0tKy0tLS0tLS0tKwogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":350,"slug":"greaterthan","name":"greaterThan","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"GreaterThan","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":355,"slug":"greaterthanequal","name":"greaterThanEqual","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"GreaterThanEqual","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":360,"slug":"hash","name":"hash","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"algorithm","type":[{"name":"Algorithm","namespace":"Flow\\ETL\\Hash","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Hash\\NativePHPHash::..."}],"return_type":[{"name":"Hash","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":365,"slug":"htmlqueryselector","name":"htmlQuerySelector","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"path","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"HTMLQuerySelector","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":370,"slug":"htmlqueryselectorall","name":"htmlQuerySelectorAll","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"path","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"HTMLQuerySelectorAll","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":378,"slug":"indexof","name":"indexOf","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"ignoreCase","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"},{"name":"offset","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"}],"return_type":[{"name":"IndexOf","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIHRoZSBpbmRleCBvZiBnaXZlbiAkbmVlZGxlIGluIHN0cmluZy4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":389,"slug":"indexoflast","name":"indexOfLast","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"ignoreCase","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"},{"name":"offset","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"}],"return_type":[{"name":"IndexOfLast","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIHRoZSBsYXN0IGluZGV4IG9mIGdpdmVuICRuZWVkbGUgaW4gc3RyaW5nLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":397,"slug":"isempty","name":"isEmpty","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"IsEmpty","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":402,"slug":"iseven","name":"isEven","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Equals","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":407,"slug":"isfalse","name":"isFalse","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Same","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":415,"slug":"isin","name":"isIn","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"haystack","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"IsIn","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gU2NhbGFyRnVuY3Rpb258YXJyYXk8YXJyYXkta2V5LCBtaXhlZD4gJGhheXN0YWNrCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":420,"slug":"isnotnull","name":"isNotNull","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"IsNotNull","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":425,"slug":"isnotnumeric","name":"isNotNumeric","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"IsNotNumeric","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":430,"slug":"isnull","name":"isNull","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"IsNull","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":435,"slug":"isnumeric","name":"isNumeric","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"IsNumeric","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":440,"slug":"isodd","name":"isOdd","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"NotEquals","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":445,"slug":"istrue","name":"isTrue","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Same","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":453,"slug":"istype","name":"isType","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"types","type":[{"name":"Type","namespace":"Flow\\Types","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"IsType","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gc3RyaW5nfFR5cGU8bWl4ZWQ+ICR0eXBlcwogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":465,"slug":"isutf8","name":"isUtf8","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"IsUtf8","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBDaGVjayBzdHJpbmcgaXMgdXRmOCBhbmQgcmV0dXJucyB0cnVlIG9yIGZhbHNlLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":470,"slug":"jsondecode","name":"jsonDecode","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"flags","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"4194304"}],"return_type":[{"name":"JsonDecode","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":475,"slug":"jsonencode","name":"jsonEncode","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"flags","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"4194304"}],"return_type":[{"name":"JsonEncode","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":480,"slug":"lessthan","name":"lessThan","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"LessThan","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":485,"slug":"lessthanequal","name":"lessThanEqual","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"LessThanEqual","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":490,"slug":"literal","name":"literal","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Literal","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":495,"slug":"lower","name":"lower","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"ToLower","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":500,"slug":"minus","name":"minus","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"float","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Minus","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":505,"slug":"mod","name":"mod","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Mod","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":510,"slug":"modifydatetime","name":"modifyDateTime","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"modifier","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ModifyDateTime","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":515,"slug":"multiply","name":"multiply","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"float","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Multiply","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":520,"slug":"notequals","name":"notEquals","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"NotEquals","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":525,"slug":"notsame","name":"notSame","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"NotSame","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":530,"slug":"numberformat","name":"numberFormat","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"decimals","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"2"},{"name":"decimalSeparator","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'.'"},{"name":"thousandsSeparator","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"','"}],"return_type":[{"name":"NumberFormat","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":544,"slug":"oneach","name":"onEach","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"function","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"preserveKeys","type":[{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"true"}],"return_type":[{"name":"OnEach","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBFeGVjdXRlIGEgc2NhbGFyIGZ1bmN0aW9uIG9uIGVhY2ggZWxlbWVudCBvZiBhbiBhcnJheS9saXN0L21hcC9zdHJ1Y3R1cmUgZW50cnkuCiAgICAgKiBJbiBvcmRlciB0byB1c2UgdGhpcyBmdW5jdGlvbiwgeW91IG5lZWQgdG8gcHJvdmlkZSBhIHJlZmVyZW5jZSB0byB0aGUgImVsZW1lbnQiIHRoYXQgd2lsbCBiZSB1c2VkIGluIHRoZSBmdW5jdGlvbi4KICAgICAqCiAgICAgKiBFeGFtcGxlOiAkZGYtPndpdGhFbnRyeSgnYXJyYXknLCByZWYoJ2FycmF5JyktPm9uRWFjaChyZWYoJ2VsZW1lbnQnKS0+Y2FzdCh0eXBlX3N0cmluZygpKSkpCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":549,"slug":"or","name":"or","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"function","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Any","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":554,"slug":"ornot","name":"orNot","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"function","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Any","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":559,"slug":"plus","name":"plus","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"ref","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"float","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Plus","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":564,"slug":"power","name":"power","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Power","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":569,"slug":"prepend","name":"prepend","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"prefix","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Prepend","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":574,"slug":"regex","name":"regex","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"pattern","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"flags","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"},{"name":"offset","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"}],"return_type":[{"name":"Regex","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":579,"slug":"regexall","name":"regexAll","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"pattern","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"flags","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"},{"name":"offset","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"}],"return_type":[{"name":"RegexAll","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":584,"slug":"regexmatch","name":"regexMatch","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"pattern","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"flags","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"},{"name":"offset","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"}],"return_type":[{"name":"RegexMatch","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":592,"slug":"regexmatchall","name":"regexMatchAll","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"pattern","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"flags","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"},{"name":"offset","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"0"}],"return_type":[{"name":"RegexMatchAll","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":600,"slug":"regexreplace","name":"regexReplace","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"pattern","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"replacement","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"limit","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"RegexReplace","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":608,"slug":"repeat","name":"repeat","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"times","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Repeat","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":613,"slug":"reverse","name":"reverse","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Reverse","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":618,"slug":"round","name":"round","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"precision","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"2"},{"name":"mode","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"1"}],"return_type":[{"name":"Round","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":623,"slug":"same","name":"same","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"value","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"Same","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":628,"slug":"sanitize","name":"sanitize","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"placeholder","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'*'"},{"name":"skipCharacters","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"Sanitize","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":635,"slug":"size","name":"size","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"Size","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":643,"slug":"slug","name":"slug","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"separator","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'-'"},{"name":"locale","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"},{"name":"symbolsMap","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"Slug","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gU2NhbGFyRnVuY3Rpb258bnVsbHxhcnJheTxhcnJheS1rZXksIG1peGVkPiAkc3ltYm9sc01hcAogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":651,"slug":"split","name":"split","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"separator","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"limit","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"9223372036854775807"}],"return_type":[{"name":"Split","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":656,"slug":"sprintf","name":"sprintf","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"params","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"float","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"null","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":true,"default_value":null}],"return_type":[{"name":"Sprintf","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":661,"slug":"startswith","name":"startsWith","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"StartsWith","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":669,"slug":"stringafter","name":"stringAfter","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"includeNeedle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"StringAfter","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIHRoZSBjb250ZW50cyBmb3VuZCBhZnRlciB0aGUgZmlyc3Qgb2NjdXJyZW5jZSBvZiB0aGUgZ2l2ZW4gc3RyaW5nLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":677,"slug":"stringafterlast","name":"stringAfterLast","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"includeNeedle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"StringAfterLast","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIHRoZSBjb250ZW50cyBmb3VuZCBhZnRlciB0aGUgbGFzdCBvY2N1cnJlbmNlIG9mIHRoZSBnaXZlbiBzdHJpbmcuCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":687,"slug":"stringbefore","name":"stringBefore","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"includeNeedle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"StringBefore","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIHRoZSBjb250ZW50cyBmb3VuZCBiZWZvcmUgdGhlIGZpcnN0IG9jY3VycmVuY2Ugb2YgdGhlIGdpdmVuIHN0cmluZy4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":697,"slug":"stringbeforelast","name":"stringBeforeLast","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"includeNeedle","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"StringBeforeLast","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIHRoZSBjb250ZW50cyBmb3VuZCBiZWZvcmUgdGhlIGxhc3Qgb2NjdXJyZW5jZSBvZiB0aGUgZ2l2ZW4gc3RyaW5nLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":707,"slug":"stringcontainsany","name":"stringContainsAny","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"needles","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"StringContainsAny","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gYXJyYXk8c3RyaW5nPnxTY2FsYXJGdW5jdGlvbiAkbmVlZGxlcwogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":712,"slug":"stringequalsto","name":"stringEqualsTo","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"string","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"StringEqualsTo","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":720,"slug":"stringfold","name":"stringFold","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"StringFold","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBSZXR1cm5zIGEgc3RyaW5nIHRoYXQgeW91IGNhbiB1c2UgaW4gY2FzZS1pbnNlbnNpdGl2ZSBjb21wYXJpc29ucy4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":725,"slug":"stringmatch","name":"stringMatch","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"pattern","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"StringMatch","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":730,"slug":"stringmatchall","name":"stringMatchAll","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"pattern","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"StringMatchAll","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":735,"slug":"stringnormalize","name":"stringNormalize","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"form","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"16"}],"return_type":[{"name":"StringNormalize","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":744,"slug":"stringstyle","name":"stringStyle","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"style","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"StringStyles","namespace":"Flow\\ETL\\String","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"StringStyle","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBDb3ZlcnQgc3RyaW5nIHRvIGEgc3R5bGUgZnJvbSBlbnVtIGxpc3QsIHBhc3NlZCBpbiBwYXJhbWV0ZXIuCiAgICAgKiBDYW4gYmUgc3RyaW5nICJ1cHBlciIgb3IgU3RyaW5nU3R5bGVzOjpVUFBFUiBmb3IgVXBwZXIgKGV4YW1wbGUpLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":752,"slug":"stringtitle","name":"stringTitle","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"allWords","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"StringTitle","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBDaGFuZ2VzIGFsbCBncmFwaGVtZXMvY29kZSBwb2ludHMgdG8gInRpdGxlIGNhc2UiLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":757,"slug":"stringwidth","name":"stringWidth","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"StringWidth","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":762,"slug":"strpad","name":"strPad","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"length","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"pad_string","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"' '"},{"name":"type","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"1"}],"return_type":[{"name":"StrPad","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":767,"slug":"strpadboth","name":"strPadBoth","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"length","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"pad_string","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"' '"}],"return_type":[{"name":"StrPad","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":772,"slug":"strpadleft","name":"strPadLeft","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"length","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"pad_string","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"' '"}],"return_type":[{"name":"StrPad","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":777,"slug":"strpadright","name":"strPadRight","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"length","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"pad_string","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"' '"}],"return_type":[{"name":"StrPad","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":786,"slug":"strreplace","name":"strReplace","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"search","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"replace","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"StrReplace","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gYXJyYXk8c3RyaW5nPnxTY2FsYXJGdW5jdGlvbnxzdHJpbmcgJHNlYXJjaAogICAgICogQHBhcmFtIGFycmF5PHN0cmluZz58U2NhbGFyRnVuY3Rpb258c3RyaW5nICRyZXBsYWNlCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":794,"slug":"todate","name":"toDate","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"format","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'Y-m-d\\\\TH:i:sP'"},{"name":"timeZone","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"DateTimeZone","namespace":"","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"DateTimeZone::..."}],"return_type":[{"name":"ToDate","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gU2NhbGFyRnVuY3Rpb258c3RyaW5nICRmb3JtYXQgLSBjdXJyZW50IGZvcm1hdCBvZiB0aGUgZGF0ZSB0aGF0IHdpbGwgYmUgdXNlZCB0byBjcmVhdGUgRGF0ZVRpbWVJbW11dGFibGUgaW5zdGFuY2UKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":805,"slug":"todatetime","name":"toDateTime","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"format","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'Y-m-d H:i:s'"},{"name":"timeZone","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"DateTimeZone","namespace":"","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"DateTimeZone::..."}],"return_type":[{"name":"ToDateTime","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gU2NhbGFyRnVuY3Rpb258c3RyaW5nICRmb3JtYXQgLSBjdXJyZW50IGZvcm1hdCBvZiB0aGUgZGF0ZSB0aGF0IHdpbGwgYmUgdXNlZCB0byBjcmVhdGUgRGF0ZVRpbWVJbW11dGFibGUgaW5zdGFuY2UKICAgICAqIEBwYXJhbSBcRGF0ZVRpbWVab25lfFNjYWxhckZ1bmN0aW9uICR0aW1lWm9uZQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":812,"slug":"trim","name":"trim","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"type","type":[{"name":"Type","namespace":"Flow\\ETL\\Function\\Trim","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Function\\Trim\\Type::..."},{"name":"characters","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"' \\t\\n\\r\\0\u000b'"}],"return_type":[{"name":"Trim","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":817,"slug":"truncate","name":"truncate","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"length","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"ellipsis","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'...'"}],"return_type":[{"name":"Truncate","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":822,"slug":"unicodelength","name":"unicodeLength","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"UnicodeLength","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":846,"slug":"unpack","name":"unpack","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"schema","type":[{"name":"Schema","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"ArrayUnpack","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":"LyoqCiAgICAgKiBVbnBhY2tzIGVhY2ggZWxlbWVudCBvZiBhbiBhcnJheSBpbnRvIGEgbmV3IGVudHJ5LCB1c2luZyB0aGUgYXJyYXkga2V5IGFzIHRoZSBlbnRyeSBuYW1lLgogICAgICoKICAgICAqIEJlZm9yZToKICAgICAqICAgKy0tKy0tLS0tLS0tLS0tLS0tLS0tLS0rCiAgICAgKiAgIHxpZHwgICAgICAgICAgICAgIGFycmF5fAogICAgICogICArLS0rLS0tLS0tLS0tLS0tLS0tLS0tLSsKICAgICAqICAgfCAxfHsiYSI6MSwiYiI6MiwiYyI6M318CiAgICAgKiAgIHwgMnx7ImQiOjQsImUiOjUsImYiOjZ9fAogICAgICogICArLS0rLS0tLS0tLS0tLS0tLS0tLS0tLSsKICAgICAqCiAgICAgKiBBZnRlcjoKICAgICAqICAgKy0tKy0tLS0tKy0tLS0tKy0tLS0tKy0tLS0tKy0tLS0tKwogICAgICogICB8aWR8YXJyLmJ8YXJyLmN8YXJyLmR8YXJyLmV8YXJyLmZ8CiAgICAgKiAgICstLSstLS0tLSstLS0tLSstLS0tLSstLS0tLSstLS0tLSsKICAgICAqICAgfCAxfCAgICAyfCAgICAzfCAgICAgfCAgICAgfCAgICAgfAogICAgICogICB8IDJ8ICAgICB8ICAgICB8ICAgIDR8ICAgIDV8ICAgIDZ8CiAgICAgKiAgICstLSstLS0tLSstLS0tLSstLS0tLSstLS0tLSstLS0tLSsKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":851,"slug":"upper","name":"upper","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"ToUpper","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":856,"slug":"wordwrap","name":"wordwrap","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"width","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"break","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"'\\n'"},{"name":"cut","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"Wordwrap","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ScalarFunctionChain.php","start_line_in_file":864,"slug":"xpath","name":"xpath","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[{"name":"string","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"XPath","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":true,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ResolvesFromChildren.php","start_line_in_file":12,"slug":"resolved","name":"resolved","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Function\/ResolvesFromChildren.php","start_line_in_file":23,"slug":"deterministic","name":"deterministic","class":"Flow\\ETL\\Function\\ScalarFunctionChain","class_slug":"scalarfunctionchain","parameters":[],"return_type":[{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Flow.php","start_line_in_file":23,"slug":"setup","name":"setUp","class":"Flow\\ETL\\Flow","class_slug":"flow","parameters":[{"name":"config","type":[{"name":"ConfigBuilder","namespace":"Flow\\ETL\\Config","is_nullable":false,"is_variadic":false},{"name":"Config","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Flow.php","start_line_in_file":28,"slug":"extract","name":"extract","class":"Flow\\ETL\\Flow","class_slug":"flow","parameters":[{"name":"extractor","type":[{"name":"Extractor","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"DataFrame","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Flow.php","start_line_in_file":33,"slug":"from","name":"from","class":"Flow\\ETL\\Flow","class_slug":"flow","parameters":[{"name":"extractor","type":[{"name":"Extractor","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"DataFrame","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Flow.php","start_line_in_file":38,"slug":"process","name":"process","class":"Flow\\ETL\\Flow","class_slug":"flow","parameters":[{"name":"rows","type":[{"name":"Rows","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"DataFrame","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/Flow.php","start_line_in_file":46,"slug":"read","name":"read","class":"Flow\\ETL\\Flow","class_slug":"flow","parameters":[{"name":"extractor","type":[{"name":"Extractor","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"DataFrame","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBBbGlhcyBmb3IgRmxvdzo6ZXh0cmFjdCBmdW5jdGlvbi4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":74,"slug":"aggregate","name":"aggregate","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"aggregations","type":[{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"algorithm","type":[{"name":"GroupByAlgorithmBuilder","namespace":"Flow\\ETL\\Config\\Grouping","is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEBwYXJhbSBBZ2dyZWdhdGlvbnMgJGFnZ3JlZ2F0aW9ucwogICAgICogQHBhcmFtIG51bGx8R3JvdXBCeUFsZ29yaXRobUJ1aWxkZXIgJGFsZ29yaXRobSBudWxsIGRlZmVycyB0byBjb25maWd1cmF0aW9uOyBhIGJ1aWxkZXIgcGlucyB0aGUKICAgICAqICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhbGdvcml0aG0gZm9yIHRoaXMgb3BlcmF0aW9uIGFuZCBza2lwcyBhbnkgYXV0b21hdGljIGNob2ljZQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":98,"slug":"batchby","name":"batchBy","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"column","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"minSize","type":[{"name":"int","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBNZXJnZS9TcGxpdCBSb3dzIHlpZWxkZWQgYnkgRXh0cmFjdG9yIGludG8gYmF0Y2hlcyBidXQga2VlcCB0aG9zZSB3aXRoIGNvbW1vbiB2YWx1ZSBpbiBnaXZlbiBjb2x1bW4gdG9nZXRoZXIuCiAgICAgKiBUaGlzIHdvcmtzIHByb3Blcmx5IG9ubHkgb24gc29ydGVkIGRhdGFzZXRzLgogICAgICoKICAgICAqIFdoZW4gbWluU2l6ZSBpcyBub3QgcHJvdmlkZWQsIGJhdGNoZXMgd2lsbCBiZSBjcmVhdGVkIG9ubHkgd2hlbiB0aGVyZSBpcyBhIGNoYW5nZSBpbiB2YWx1ZSBvZiB0aGUgY29sdW1uLgogICAgICogV2hlbiBtaW5TaXplIGlzIHByb3ZpZGVkLCBiYXRjaGVzIHdpbGwgYmUgY3JlYXRlZCBvbmx5IHdoZW4gdGhlcmUgaXMgYSBjaGFuZ2UgaW4gdmFsdWUgb2YgdGhlIGNvbHVtbiBvcgogICAgICogd2hlbiB0aGVyZSBhcmUgYXQgbGVhc3QgbWluU2l6ZSByb3dzIGluIHRoZSBiYXRjaC4KICAgICAqCiAgICAgKiBAcGFyYW0gUmVmZXJlbmNlfHN0cmluZyAkY29sdW1uIC0gY29sdW1uIHRvIGdyb3VwIGJ5IChhbGwgcm93cyB3aXRoIHNhbWUgdmFsdWUgc3RheSB0b2dldGhlcikKICAgICAqIEBwYXJhbSBudWxsfGludDwxLCBtYXg+ICRtaW5TaXplIC0gb3B0aW9uYWwgbWluaW11bSByb3dzIHBlciBiYXRjaCBmb3IgZWZmaWNpZW5jeQogICAgICoKICAgICAqIEBsYXp5CiAgICAgKgogICAgICogQHRocm93cyBJbnZhbGlkQXJndW1lbnRFeGNlcHRpb24KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":118,"slug":"batchsize","name":"batchSize","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"size","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBNZXJnZS9TcGxpdCBSb3dzIHlpZWxkZWQgYnkgRXh0cmFjdG9yIGludG8gYmF0Y2hlcyBvZiBnaXZlbiBzaXplLgogICAgICogRm9yIGV4YW1wbGUsIHdoZW4gRXh0cmFjdG9yIGlzIHlpZWxkaW5nIG9uZSByb3cgYXQgdGltZSwgdGhpcyBtZXRob2Qgd2lsbCBtZXJnZSB0aGVtIGludG8gYmF0Y2hlcyBvZiBnaXZlbiBzaXplCiAgICAgKiBiZWZvcmUgcGFzc2luZyB0aGVtIHRvIHRoZSBuZXh0IHBpcGVsaW5lIGVsZW1lbnQuCiAgICAgKiBTaW1pbGFybHkgd2hlbiBFeHRyYWN0b3IgaXMgeWllbGRpbmcgYmF0Y2hlcyBvZiByb3dzLCB0aGlzIG1ldGhvZCB3aWxsIHNwbGl0IHRoZW0gaW50byBzbWFsbGVyIGJhdGNoZXMgb2YgZ2l2ZW4KICAgICAqIHNpemUuCiAgICAgKgogICAgICogSW4gb3JkZXIgdG8gbWVyZ2UgYWxsIFJvd3MgaW50byBhIHNpbmdsZSBiYXRjaCB1c2UgRGF0YUZyYW1lOjpjb2xsZWN0KCkgbWV0aG9kIG9yIHNldCBzaXplIHRvIC0xIG9yIDAuCiAgICAgKgogICAgICogQHBhcmFtIGludDwtMSwgbWF4PiAkc2l6ZQogICAgICoKICAgICAqIEBsYXp5CiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":150,"slug":"cache","name":"cache","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"id","type":[{"name":"string","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"},{"name":"cacheBatchSize","type":[{"name":"int","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"},{"name":"cache","type":[{"name":"Cache","namespace":"Flow\\ETL","is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBTdGFydCBwcm9jZXNzaW5nIHJvd3MgdXAgdG8gdGhpcyBtb21lbnQgYW5kIHB1dCBlYWNoIGluc3RhbmNlIG9mIFJvd3MKICAgICAqIGludG8gcHJldmlvdXNseSBkZWZpbmVkIGNhY2hlLgogICAgICogQ2FjaGUgdHlwZSBjYW4gYmUgc2V0IHRocm91Z2ggQ29uZmlnQnVpbGRlci4KICAgICAqIEJ5IGRlZmF1bHQgZXZlcnl0aGluZyBpcyBjYWNoZWQgaW4gc3lzdGVtIHRtcCBkaXIuCiAgICAgKgogICAgICogSW1wb3J0YW50OiBjYWNoZSBiYXRjaCBzaXplIG1pZ2h0IHNpZ25pZmljYW50bHkgaW1wcm92ZSBwZXJmb3JtYW5jZSB3aGVuIHByb2Nlc3NpbmcgbGFyZ2UgYW1vdW50IG9mIHJvd3MuCiAgICAgKiBMYXJnZXIgYmF0Y2ggc2l6ZSB3aWxsIGluY3JlYXNlIG1lbW9yeSBjb25zdW1wdGlvbiBidXQgd2lsbCByZWR1Y2UgbnVtYmVyIG9mIElPIG9wZXJhdGlvbnMuCiAgICAgKiBXaGVuIG5vdCBzZXQsIHRoZSBiYXRjaCBzaXplIGlzIHRha2VuIGZyb20gdGhlIGxhc3QgRGF0YUZyYW1lOjpiYXRjaFNpemUoKSBjYWxsLgogICAgICoKICAgICAqIEBsYXp5CiAgICAgKgogICAgICogQHBhcmFtIG51bGx8c3RyaW5nICRpZAogICAgICogQHBhcmFtIG51bGx8Q2FjaGUgJGNhY2hlIHJlYWRzIG9mIHRoaXMgY2FjaGUgbXVzdCBwYXNzIHRoZSBzYW1lIGluc3RhbmNlIHRvIGZyb21fY2FjaGUoKQogICAgICoKICAgICAqIEB0aHJvd3MgSW52YWxpZEFyZ3VtZW50RXhjZXB0aW9uCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":167,"slug":"collect","name":"collect","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBCZWZvcmUgdHJhbnNmb3JtaW5nIHJvd3MsIGNvbGxlY3QgdGhlbSBhbmQgbWVyZ2UgaW50byBzaW5nbGUgUm93cyBpbnN0YW5jZS4KICAgICAqIFRoaXMgbWlnaHQgbGVhZCB0byBtZW1vcnkgaXNzdWVzIHdoZW4gcHJvY2Vzc2luZyBsYXJnZSBhbW91bnQgb2Ygcm93cywgdXNlIHdpdGggY2F1dGlvbi4KICAgICAqCiAgICAgKiBAbGF6eQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":186,"slug":"collectrefs","name":"collectRefs","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"references","type":[{"name":"References","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBUaGlzIG1ldGhvZCBhbGxvd3MgdG8gY29sbGVjdCByZWZlcmVuY2VzIHRvIGFsbCBlbnRyaWVzIHVzZWQgaW4gdGhpcyBwaXBlbGluZS4KICAgICAqCiAgICAgKiBgYGBwaHAKICAgICAqIChuZXcgRmxvdygpKQogICAgICogICAtPnJlYWQoRnJvbTo6Y2hhaW4oKSkKICAgICAqICAgLT5jb2xsZWN0UmVmcygkcmVmcyA9IHJlZnMoKSkKICAgICAqICAgLT5ydW4oKTsKICAgICAqIGBgYAogICAgICoKICAgICAqIEBsYXp5CiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":193,"slug":"constrain","name":"constrain","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"constraint","type":[{"name":"Constraint","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"constraints","type":[{"name":"Constraint","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":204,"slug":"count","name":"count","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAdHJpZ2dlcgogICAgICogUmV0dXJuIHRvdGFsIGNvdW50IG9mIHJvd3MgcHJvY2Vzc2VkIGJ5IHRoaXMgcGlwZWxpbmUuCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":224,"slug":"crossjoin","name":"crossJoin","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"dataFrame","type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"prefix","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"''"}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":241,"slug":"display","name":"display","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"limit","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"20"},{"name":"truncate","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"20"},{"name":"formatter","type":[{"name":"Formatter","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Formatter\\AsciiTableFormatter::..."}],"return_type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gaW50ICRsaW1pdCBtYXhpbXVtIG51bWJlcnMgb2Ygcm93cyB0byBkaXNwbGF5CiAgICAgKiBAcGFyYW0gYm9vbHxpbnQgJHRydW5jYXRlIGZhbHNlIG9yIGlmIHNldCB0byAwIGNvbHVtbnMgYXJlIG5vdCB0cnVuY2F0ZWQsIG90aGVyd2lzZSBkZWZhdWx0IHRydW5jYXRlIHRvIDIwCiAgICAgKiAgICAgICAgICAgICAgICAgICAgICAgICAgIGNoYXJhY3RlcnMKICAgICAqIEBwYXJhbSBGb3JtYXR0ZXIgJGZvcm1hdHRlcgogICAgICoKICAgICAqIEB0cmlnZ2VyCiAgICAgKgogICAgICogQHRocm93cyBJbnZhbGlkQXJndW1lbnRFeGNlcHRpb24KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":268,"slug":"drop","name":"drop","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"entries","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBEcm9wIGdpdmVuIGVudHJpZXMuCiAgICAgKgogICAgICogQGxhenkKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":282,"slug":"dropduplicates","name":"dropDuplicates","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"entries","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gUmVmZXJlbmNlfHN0cmluZyAuLi4kZW50cmllcwogICAgICoKICAgICAqIEBsYXp5CiAgICAgKgogICAgICogQHJldHVybiAkdGhpcwogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":289,"slug":"duplicaterow","name":"duplicateRow","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"condition","type":[{"name":"mixed","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null},{"name":"entries","type":[{"name":"WithEntry","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":309,"slug":"fetch","name":"fetch","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"limit","type":[{"name":"int","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"Rows","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBCZSBhd2FyZSB0aGF0IGZldGNoIGlzIG5vdCBtZW1vcnkgc2FmZSBhbmQgd2lsbCBsb2FkIGFsbCByb3dzIGludG8gbWVtb3J5LgogICAgICogSWYgeW91IHdhbnQgdG8gc2FmZWx5IGl0ZXJhdGUgb3ZlciBSb3dzIHVzZSBvZSBvZiB0aGUgZm9sbG93aW5nIG1ldGhvZHM6LgogICAgICoKICAgICAqIERhdGFGcmFtZTo6Z2V0KCkgOiBcR2VuZXJhdG9yCiAgICAgKiBEYXRhRnJhbWU6OmdldEFzQXJyYXkoKSA6IFxHZW5lcmF0b3IKICAgICAqIERhdGFGcmFtZTo6Z2V0RWFjaCgpIDogXEdlbmVyYXRvcgogICAgICogRGF0YUZyYW1lOjpnZXRFYWNoQXNBcnJheSgpIDogXEdlbmVyYXRvcgogICAgICoKICAgICAqIEB0cmlnZ2VyCiAgICAgKgogICAgICogQHRocm93cyBJbnZhbGlkQXJndW1lbnRFeGNlcHRpb24KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":326,"slug":"filter","name":"filter","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"function","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":338,"slug":"filters","name":"filters","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"functions","type":[{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEBwYXJhbSBhcnJheTxTY2FsYXJGdW5jdGlvbj4gJGZ1bmN0aW9ucwogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":352,"slug":"foreach","name":"forEach","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"callback","type":[{"name":"callable","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"void","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAdHJpZ2dlcgogICAgICoKICAgICAqIEBwYXJhbSBudWxsfGNhbGxhYmxlKFJvd3MgJHJvd3MsIEZsb3dDb250ZXh0ICRjb250ZXh0KSA6IHZvaWQgJGNhbGxiYWNrCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":373,"slug":"get","name":"get","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"Generator","namespace":"","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBZaWVsZHMgZWFjaCByb3cgYXMgYW4gaW5zdGFuY2Ugb2YgUm93cy4KICAgICAqCiAgICAgKiBAdHJpZ2dlcgogICAgICoKICAgICAqIEByZXR1cm4gXEdlbmVyYXRvcjxSb3dzPgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":392,"slug":"getasarray","name":"getAsArray","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"Generator","namespace":"","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBZaWVsZHMgZWFjaCByb3cgYXMgYW4gYXJyYXkuCiAgICAgKgogICAgICogQHRyaWdnZXIKICAgICAqCiAgICAgKiBAcmV0dXJuIFxHZW5lcmF0b3I8YXJyYXk8YXJyYXk8bWl4ZWQ+Pj4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":411,"slug":"geteach","name":"getEach","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"Generator","namespace":"","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBZaWVsZCBlYWNoIHJvdyBhcyBhbiBpbnN0YW5jZSBvZiBSb3cuCiAgICAgKgogICAgICogQHRyaWdnZXIKICAgICAqCiAgICAgKiBAcmV0dXJuIFxHZW5lcmF0b3I8Um93PgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":432,"slug":"geteachasarray","name":"getEachAsArray","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"Generator","namespace":"","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBZaWVsZCBlYWNoIHJvdyBhcyBhbiBhcnJheS4KICAgICAqCiAgICAgKiBAdHJpZ2dlcgogICAgICoKICAgICAqIEByZXR1cm4gXEdlbmVyYXRvcjxhcnJheTxtaXhlZD4+CiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":453,"slug":"groupby","name":"groupBy","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"entries","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"algorithm","type":[{"name":"GroupByAlgorithmBuilder","namespace":"Flow\\ETL\\Config\\Grouping","is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"GroupedDataFrame","namespace":"Flow\\ETL\\DataFrame","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEBwYXJhbSBHcm91cEJ5UmVmZXJlbmNlc3xSZWZlcmVuY2V8c3RyaW5nICRlbnRyaWVzIGEgc2luZ2xlIGNvbHVtbiBpcyBncm91cGVkIGJ5IG9uIGl0cyBvd24KICAgICAqIEBwYXJhbSBudWxsfEdyb3VwQnlBbGdvcml0aG1CdWlsZGVyICRhbGdvcml0aG0gbnVsbCBkZWZlcnMgdG8gY29uZmlndXJhdGlvbjsgYSBidWlsZGVyIHBpbnMgdGhlCiAgICAgKiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYWxnb3JpdGhtIGZvciB0aGlzIG9wZXJhdGlvbiBhbmQgc2tpcHMgYW55IGF1dG9tYXRpYyBjaG9pY2UKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":472,"slug":"join","name":"join","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"dataFrame","type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"on","type":[{"name":"Expression","namespace":"Flow\\ETL\\Join","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"type","type":[{"name":"Join","namespace":"Flow\\ETL\\Join","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Join\\Join::..."},{"name":"algorithm","type":[{"name":"JoinAlgorithmBuilder","namespace":"Flow\\ETL\\Config\\Join","is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEBwYXJhbSBudWxsfEpvaW5BbGdvcml0aG1CdWlsZGVyICRhbGdvcml0aG0gbnVsbCBkZWZlcnMgdG8gY29uZmlndXJhdGlvbjsgYSBidWlsZGVyIHBpbnMgdGhlIGFsZ29yaXRobQogICAgICogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZvciB0aGlzIG9wZXJhdGlvbiBhbmQgc2tpcHMgYW55IGF1dG9tYXRpYyBjaG9pY2UKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":500,"slug":"joineach","name":"joinEach","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"factory","type":[{"name":"DataFrameFactory","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"on","type":[{"name":"Expression","namespace":"Flow\\ETL\\Join","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"type","type":[{"name":"Join","namespace":"Flow\\ETL\\Join","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Join\\Join::..."}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBKb2lucyBpbiBtZW1vcnkgcGVyIGJhdGNoOyBpdCBpcyBub3QgZ292ZXJuZWQgYnkgdGhlIGpvaW4gYWxnb3JpdGhtIGFuZCB0YWtlcyBubyBhbGdvcml0aG0gb3ZlcnJpZGUuCiAgICAgKgogICAgICogQGxhenkKICAgICAqCiAgICAgKiBAcGFyYW0gc3RyaW5nfEpvaW4gJHR5cGUKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":516,"slug":"limit","name":"limit","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"limit","type":[{"name":"int","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEB0aHJvd3MgSW52YWxpZEFyZ3VtZW50RXhjZXB0aW9uCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":530,"slug":"load","name":"load","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"sink","type":[{"name":"Loader","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"Sink","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":576,"slug":"match","name":"match","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"schema","type":[{"name":"Schema","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"validator","type":[{"name":"SchemaValidator","namespace":"Flow\\ETL","is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEBwYXJhbSBudWxsfFNjaGVtYVZhbGlkYXRvciAkdmFsaWRhdG9yIC0gd2hlbiBudWxsLCBTdHJpY3RWYWxpZGF0b3IgZ2V0cyBpbml0aWFsaXplZAogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":597,"slug":"offset","name":"offset","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"offset","type":[{"name":"int","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":false,"is_nullable":true,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBTa2lwIGdpdmVuIG51bWJlciBvZiByb3dzIGZyb20gdGhlIGJlZ2lubmluZyBvZiB0aGUgZGF0YXNldC4KICAgICAqIFdoZW4gJG9mZnNldCBpcyBudWxsLCBub3RoaW5nIGhhcHBlbnMgKG5vIHJvd3MgYXJlIHNraXBwZWQpLgogICAgICoKICAgICAqIFBlcmZvcm1hbmNlIE5vdGU6IERhdGFGcmFtZSBtdXN0IGl0ZXJhdGUgdGhyb3VnaCBhbmQgcHJvY2VzcyBhbGwgc2tpcHBlZCByb3dzCiAgICAgKiB0byByZWFjaCB0aGUgb2Zmc2V0IHBvc2l0aW9uLiBGb3IgbGFyZ2Ugb2Zmc2V0cywgdGhpcyBjYW4gaW1wYWN0IHBlcmZvcm1hbmNlCiAgICAgKiBhcyB0aGUgZGF0YSBzb3VyY2Ugc3RpbGwgbmVlZHMgdG8gYmUgcmVhZCBhbmQgcHJvY2Vzc2VkIHVwIHRvIHRoZSBvZmZzZXQgcG9pbnQuCiAgICAgKgogICAgICogQHBhcmFtID9pbnQ8MCwgbWF4PiAkb2Zmc2V0CiAgICAgKgogICAgICogQGxhenkKICAgICAqCiAgICAgKiBAdGhyb3dzIEludmFsaWRBcmd1bWVudEV4Y2VwdGlvbgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":611,"slug":"onerror","name":"onError","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"handler","type":[{"name":"ErrorHandler","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":623,"slug":"repartition","name":"repartition","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"entry","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"entries","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICogU2h1ZmZsZXMgdGhlIHN0cmVhbSBzbyBldmVyeSByb3cgc2hhcmluZyB0aGUgZ2l2ZW4gY29sdW1ucyBhcnJpdmVzIGluIG9uZSBiYXRjaC4gSXQgZG9lcyBub3QKICAgICAqIHdyaXRlIGRpcmVjdG9yaWVzIC0gdGhhdCBpcyBkZWNsYXJlZCBvbiB0aGUgbG9hZGVyLCBgdG9fY3N2KC4uLiktPnBhcnRpdGlvbkJ5KCdyZWdpb24nKWAuCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":635,"slug":"printrows","name":"printRows","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"limit","type":[{"name":"int","namespace":null,"is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"20"},{"name":"truncate","type":[{"name":"int","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"20"},{"name":"formatter","type":[{"name":"Formatter","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Formatter\\AsciiTableFormatter::..."}],"return_type":[{"name":"void","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAdHJpZ2dlcgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":655,"slug":"explain","name":"explain","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"trigger","type":[{"name":"Trigger","namespace":"Flow\\ETL\\Plan","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Plan\\Trigger::..."}],"return_type":[{"name":"Plan","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBUaGUgcGxhbiAkdHJpZ2dlciB3b3VsZCBydW4gb3ZlciB0aGlzIGZyYW1lLCBmcm96ZW46IGxhdGVyIHZlcmJzIG9uIHRoaXMgZnJhbWUgZG8gbm90IHJlYWNoIGl0LiBUaGUgZGVmYXVsdAogICAgICogYWRkcyBubyBjb25zdW1lciBvZiBpdHMgb3duIC0gaXQgZHJhd3MgdGhlIGZyYW1lIGFzIGJ1aWx0LCBpdHMgY2hhaW4gYW5kIGl0cyBzaW5rcy4gdG9TdHJpbmcoKSBwcmludHMgaXQgYXMgYQogICAgICogdHJlZS4gQW5zd2VycyBmcm9tIHRoZSBwbGFuIHdpdGhvdXQgcmVhZGluZyBhIHJvdy4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":665,"slug":"printschema","name":"printSchema","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"formatter","type":[{"name":"SchemaFormatter","namespace":"Flow\\ETL\\Schema","is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"Flow\\ETL\\Row\\Formatter\\ASCIISchemaFormatter::..."}],"return_type":[{"name":"void","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEB0aHJvd3MgU2NoZW1hTm90RGVyaXZhYmxlRXhjZXB0aW9uCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":673,"slug":"rename","name":"rename","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"from","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"to","type":[{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":680,"slug":"renameeach","name":"renameEach","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"strategies","type":[{"name":"RenameEntryStrategy","namespace":"Flow\\ETL\\Transformer\\Rename","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":691,"slug":"rows","name":"rows","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"transformer","type":[{"name":"Transformer","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"Transformation","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICogQWxpYXMgZm9yIEVUTDo6dHJhbnNmb3JtIG1ldGhvZC4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":708,"slug":"run","name":"run","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"analyze","type":[{"name":"Analyze","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"bool","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":true,"is_nullable":false,"is_variadic":false,"default_value":"false"}],"return_type":[{"name":"Report","namespace":"Flow\\ETL\\Dataset","is_nullable":true,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAdHJpZ2dlcgogICAgICoKICAgICAqIFdoZW4gYW5hbHl6aW5nIHBpcGVsaW5lIGV4ZWN1dGlvbiB3ZSBjYW4gY2hvc2UgdG8gY29sbGVjdCB2YXJpb3VzIG1ldHJpY3MgdGhyb3VnaCBhbmFseXplKCktPndpdGgqKCkgbWV0aG9kCiAgICAgKgogICAgICogLSBjb2x1bW4gc3RhdGlzdGljcyAtIGFuYWx5emUoKS0+d2l0aENvbHVtblN0YXRpc3RpY3MoKQogICAgICogLSBzY2hlbWEgLSBhbmFseXplKCktPndpdGhTY2hlbWEoKQogICAgICoKICAgICAqIEBwYXJhbSBBbmFseXplfGJvb2wgJGFuYWx5emUgLSB3aGVuIHNldCBydW4gd2lsbCByZXR1cm4gUmVwb3J0CiAgICAgKgogICAgICogQHJldHVybiAoJGFuYWx5emUgaXMgQW5hbHl6ZXx0cnVlID8gUmVwb3J0IDogbnVsbCkKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":736,"slug":"schema","name":"schema","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"Schema","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEB0aHJvd3MgU2NoZW1hTm90RGVyaXZhYmxlRXhjZXB0aW9uCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":749,"slug":"select","name":"select","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"entries","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICogS2VlcCBvbmx5IGdpdmVuIGVudHJpZXMuCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":763,"slug":"sortby","name":"sortBy","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"entries","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false},{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"algorithm","type":[{"name":"SortAlgorithmBuilder","namespace":"Flow\\ETL\\Config\\Sort","is_nullable":true,"is_variadic":false}],"has_default_value":true,"is_nullable":true,"is_variadic":false,"default_value":"null"}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEBwYXJhbSBSZWZlcmVuY2V8U29ydFJlZmVyZW5jZXN8c3RyaW5nICRlbnRyaWVzIGEgc2luZ2xlIGNvbHVtbiBpcyBzb3J0ZWQgYnkgb24gaXRzIG93bgogICAgICogQHBhcmFtIG51bGx8U29ydEFsZ29yaXRobUJ1aWxkZXIgJGFsZ29yaXRobSBudWxsIGRlZmVycyB0byBjb25maWd1cmF0aW9uOyBhIGJ1aWxkZXIgcGlucyB0aGUgYWxnb3JpdGhtCiAgICAgKiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZm9yIHRoaXMgb3BlcmF0aW9uIGFuZCBza2lwcyBhbnkgYXV0b21hdGljIGNob2ljZQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":777,"slug":"transform","name":"transform","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"transformer","type":[{"name":"Transformer","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"Transformation","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"Transformations","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"WithEntry","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBBbGlhcyBmb3IgRGF0YUZyYW1lOjp3aXRoKCkuCiAgICAgKgogICAgICogQGxhenkKICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":788,"slug":"until","name":"until","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"function","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBUaGUgZGlmZmVyZW5jZSBiZXR3ZWVuIGZpbHRlciBhbmQgdW50aWwgaXMgdGhhdCBmaWx0ZXIgd2lsbCBrZWVwIGZpbHRlcmluZyByb3dzIHVudGlsIGV4dHJhY3RvcnMgZmluaXNoIHlpZWxkaW5nCiAgICAgKiByb3dzLiBVbnRpbCB3aWxsIHNlbmQgYSBTVE9QIHNpZ25hbCB0byB0aGUgRXh0cmFjdG9yIHdoZW4gdGhlIGNvbmRpdGlvbiBpcyBub3QgbWV0LgogICAgICoKICAgICAqIEBsYXp5CiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":802,"slug":"void","name":"void","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICogVGhpcyBtZXRob2QgaXMgdXNlZnVsIG1vc3RseSBpbiBkZXZlbG9wbWVudCB3aGVuCiAgICAgKiB5b3Ugd2FudCB0byBwYXVzZSBwcm9jZXNzaW5nIGF0IGNlcnRhaW4gbW9tZW50IHdpdGhvdXQKICAgICAqIHJlbW92aW5nIGNvZGUuIEFsbCBvcGVyYXRpb25zIHdpbGwgZ2V0IHByb2Nlc3NlZCB1cCB0byB0aGlzIHBvaW50LAogICAgICogZnJvbSBoZXJlIG5vIHJvd3MgYXJlIHBhc3NlZCBmb3J3YXJkLgogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":812,"slug":"with","name":"with","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"transformer","type":[{"name":"Transformer","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"Transformation","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"Transformations","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"WithEntry","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICov"},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":840,"slug":"withentries","name":"withEntries","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"references","type":[{"name":"array","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICoKICAgICAqIEBwYXJhbSBhcnJheTxpbnQsIFdpdGhFbnRyeT58YXJyYXk8c3RyaW5nLCBTY2FsYXJGdW5jdGlvbnxXaW5kb3dGdW5jdGlvbnxXaXRoRW50cnk+ICRyZWZlcmVuY2VzCiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":858,"slug":"withentry","name":"withEntry","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"entry","type":[{"name":"Definition","namespace":"Flow\\ETL\\Schema","is_nullable":false,"is_variadic":false},{"name":"string","namespace":null,"is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"reference","type":[{"name":"ScalarFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false},{"name":"WindowFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAcGFyYW0gRGVmaW5pdGlvbjxtaXhlZD58c3RyaW5nICRlbnRyeQogICAgICoKICAgICAqIEBsYXp5CiAgICAgKi8="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame.php","start_line_in_file":871,"slug":"write","name":"write","class":"Flow\\ETL\\DataFrame","class_slug":"dataframe","parameters":[{"name":"sink","type":[{"name":"Loader","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false},{"name":"Sink","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":"LyoqCiAgICAgKiBAbGF6eQogICAgICogQWxpYXMgZm9yIEVUTDo6bG9hZCBmdW5jdGlvbi4KICAgICAqLw=="},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame\/GroupedDataFrame.php","start_line_in_file":21,"slug":"aggregate","name":"aggregate","class":"Flow\\ETL\\DataFrame\\GroupedDataFrame","class_slug":"groupeddataframe","parameters":[{"name":"aggregations","type":[{"name":"AggregatingFunction","namespace":"Flow\\ETL\\Function","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":true,"default_value":null}],"return_type":[{"name":"DataFrame","namespace":"Flow\\ETL","is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null},{"repository_path":"src\/core\/etl\/src\/Flow\/ETL\/DataFrame\/GroupedDataFrame.php","start_line_in_file":28,"slug":"pivot","name":"pivot","class":"Flow\\ETL\\DataFrame\\GroupedDataFrame","class_slug":"groupeddataframe","parameters":[{"name":"ref","type":[{"name":"Reference","namespace":"Flow\\ETL\\Row","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null},{"name":"values","type":[{"name":"PivotValues","namespace":"Flow\\ETL\\GroupBy","is_nullable":false,"is_variadic":false}],"has_default_value":false,"is_nullable":false,"is_variadic":false,"default_value":null}],"return_type":[{"name":"self","namespace":null,"is_nullable":false,"is_variadic":false}],"attributes":[],"scalar_function_chain":false,"doc_comment":null}] \ No newline at end of file diff --git a/web/landing/templates/documentation/navigation_left.html.twig b/web/landing/templates/documentation/navigation_left.html.twig index 9a29195c0f..2287846322 100644 --- a/web/landing/templates/documentation/navigation_left.html.twig +++ b/web/landing/templates/documentation/navigation_left.html.twig @@ -63,6 +63,7 @@
  • {{ nav.nav_link('/documentation/components/core/caching', 'Caching', currentPath) }}
  • {{ nav.nav_link('/documentation/components/core/floe', 'Floe File Format', currentPath) }}
  • {{ nav.nav_link('/documentation/components/core/partitioning', 'Partitioning', currentPath) }}
    • +
  • {{ nav.nav_link('/documentation/components/core/statistics', 'Source Statistics', currentPath) }}
  • {{ nav.nav_link('/documentation/components/core/constraints', 'Constraints', currentPath) }}
  • {{ nav.nav_link('/documentation/components/core/schema', 'Schema', currentPath) }}
  • {{ nav.nav_link('/documentation/components/core/display', 'Display', currentPath) }}