#1390 - configurable storage for aggregating functions#2473
Conversation
GroupBy aggregation can move state out of the PHP heap instead of holding every result in memory. A new AggregationStorage abstraction offers three backends: memory (default, unchanged behavior), filesystem spill+merge, and PSR-16 key-value (APCu/Redis). - MergeableAggregatingFunction + merge() on all built-in aggregators - AggregationStorage: Memory / External (filesystem spill+merge) / Kv (PSR-16) - KV storage fails loudly on eviction / write failure instead of returning a wrong aggregate - config: AggregationStorageStrategy + FLOW_AGGREGATION_MAX_MEMORY wired into ConfigBuilder - APCu enabled in the nix dev shell (for the PSR-16/APCu storage test)
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 1.x #2473 +/- ##
============================================
- Coverage 85.12% 83.48% -1.65%
+ Complexity 21300 658 -20642
============================================
Files 1608 1627 +19
Lines 65717 67042 +1325
============================================
+ Hits 55944 55969 +25
- Misses 9773 11073 +1300 🚀 New features to boost your workflow:
|
|
| */ | ||
| public function __construct( | ||
| public array $values, | ||
| public array $aggregators, |
There was a problem hiding this comment.
I'm not really sure if I understand this data structure, it's keeping values for all aggregators?
| return $this; | ||
| } | ||
|
|
||
| public function aggregationStorage(AggregationStorageStrategy $strategy): self |
| return $this; | ||
| } | ||
|
|
||
| public function aggregationStore(AggregationStorage $storage): self |
There was a problem hiding this comment.
Those two methods, aggregationStorage and aggregationStore are a bit confusing
|
|
||
| public function flush(): void; | ||
|
|
||
| public function save(string $hash, GroupEntry $entry): void; |
There was a problem hiding this comment.
what's the difference between entry and save?
|
|
||
| namespace Flow\ETL\Function; | ||
|
|
||
| interface MergeableAggregatingFunction extends AggregatingFunction |
There was a problem hiding this comment.
could you walk me through this logic? Like when two AggregatingFunctions can be merged?
2 participants
GroupBy aggregation can move state out of the PHP heap instead of holding every result in memory. A new AggregationStorage abstraction offers three backends: memory (default,
unchanged behavior), filesystem spill+merge, and PSR-16 key-value (APCu/Redis).
Resolves: #1390
Change Log
Added
AggregationStorageabstraction with three backends: in-memory (default), filesystem spill+merge, and PSR-16 key-value (APCu/Redis).ConfigBuilderaggregation settings:aggregationStorage(),aggregationMemoryLimit(),aggregationFilesystem(),aggregationStore(), theAggregationStorageStrategyenum and theFLOW_AGGREGATION_MAX_MEMORYenvironment variable.MergeableAggregatingFunctioninterface withmerge()implemented by all built-in aggregating functions, allowing spilled partial aggregates to be merged.Fixed
Changed
Removed
Deprecated
Security