Repository navigation
fix(analytics): collector undercounts past 100k history rows and stores schedule-dependent units #1074
Description
Problem
Two correctness gaps in the analytics savings-snapshot collector (internal/analytics/collector.go) surfaced during CodeRabbit review of PR #1049. Both are "heavy lift" changes deferred from that PR (which made them safe/observable) into this follow-up so they can be designed properly.
1. Fixed 100k history fetch cap silently undercounts
Collect calls GetAllPurchaseHistory(ctx, purchaseHistoryFetchLimit) with a hardcoded purchaseHistoryFetchLimit = 100000. Once purchase_history grows past that size the source set is silently truncated, so snapshots undercount with no error signal. Older-but-still-active 1y/3y commitments are exactly the rows most likely to fall off.
Interim mitigation in PR #1049: the collector now logs a loud WARNING when the result fills the cap, so the undercount is observable. The proper fix is an active-only store query (filter by status/expiry server-side) or a paginated/cursor scan, which requires a new method on the config store interface.
2. Persisted units are schedule-dependent
Collect converts monthly values to hourly rates (EstimatedSavings / HoursPerMonth, UpfrontCost / (Term*HoursPerYear), MonthlyCost / HoursPerMonth), but the query layer later does SUM(total_savings), SUM(total_commitment), SUM(total_usage) as if they were additive monthly totals. With the default daily schedule a $720/month purchase contributes only ~$30 to that month's series, and changing the schedule changes the numbers.
Fix options (design decision needed): store schedule-invariant monthly amounts (amortize upfront by term in months), OR weight by elapsed time during aggregation. Either touches the materialized-view definitions in migration 000067 and the integration suite, so it was not changed blindly under the PR #1049 timeline. No consumer is currently misled: the Trends frontend is deferred (#1048).
Evidence
CodeRabbit comments on PR #1049:
internal/analytics/collector.go:25/92-95- "Remove the fixed 100k history cap or page this read."internal/analytics/collector.go:146,153- "The persisted units don't match the query-layer aggregations."
Files
internal/analytics/collector.gointernal/analytics/postgres_analytics.go(query-layer SUM/AVG semantics)internal/database/postgres/migrations/000067_*(materialized-view math, if the monthly-amounts approach is chosen)
Acceptance
- Active commitments are never silently dropped (active-only or paginated read).
- The monthly trends series is schedule-invariant: changing the collection cadence does not change the reported monthly totals.
- Regression tests cover both: a >cap dataset, and identical totals under two different collection cadences.
Activity
cristim commentedon Jul 27, 2026
Verification sweep against
main(101f099fb) on 2026-07-27 finds this already resolved.The collector now reads active-only rows via SQL with no cap and stores schedule-invariant monthly run-rates.
Evidence: commit 5fc0899 (#1049) plus b0233d3.
Recommending close.
cristim commentedon Sep 2, 2026
Verified resolved at 3c0f8ac: the collector no longer uses a fixed history cap; it reads active commitments through the uncapped SQL active-only query, and it persists schedule-invariant monthly run-rates (upfront amortized over the term in months) that the trends queries and the monthly materialized view aggregate with a per-day SUM followed by AVG, so the collection cadence no longer changes the reported totals. Evidence: internal/analytics/collector.go:95 and :180-190, internal/analytics/postgres_analytics.go:353 (#1221, #1209, migration 000078). Residual axes checked: Term<=0 rows are skipped and counted rather than dividing by zero; nil MonthlyCost leaves usage unknown instead of $0; no other caller of the old capped GetAllPurchaseHistory remains in the collector. Closing as completed; reopen if the behaviour recurs.