Python, DuckDB, and DBT utilities for processing Wintap/Lintap telemetry data.
DBT is now the primary post-processing path. The canonical input is a raw_sensor parquet dataset:
<dataset>/raw_sensor/<event_type>/dayPK=YYYYMMDD/hourPK=HH/*.parquet
<dataset>/raw_sensor/raw_process_conn_incr/dayPK=YYYYMMDD/hourPK=HH/protoPK=tcp|udp/*.parquet
<dataset>/compacted/<event_type>/dayPK=YYYYMMDD/hourPK=HH[/protoPK=...]/<merged>.parquet
DBT reads both raw and compacted locations with name-aligned UNION ALL semantics.
Compaction validates then deletes source prefixes after the UTC upload grace;
run DBT after successful migration, not during overlap or an unreconciled failure.
WINTAP_DBT_COMPACTED_ROOT optionally overrides the direct compacted event root.
See the DBT guide.
The hourly compactor lives in
extras/raw-sensor-compact/ as a standalone UV project, with its own lockfile
and DuckDB 2.0.0 environment. It does not upgrade the main pipeline's
DuckDB 1.5.6 dependency. From this repository root:
uv run --project extras/raw-sensor-compact --locked raw-sensor-compact --helpThe same worker optionally maintains durable process snapshots in S3. Infrequent DBT builds use them for historical/exited parent context without adding lifecycle observations to activity counts. Bootstrap history explicitly and exclude readers/writers during updates and builds. See DBT configuration.
For the full current run process, start with:
review-notes/PipelineRunbook.md
- Python 3.10, 3.11, or 3.12
- uv
- DuckDB CLI for QA scripts
- DuckDB-compatible parquet data
Install uv if needed:
curl -LsSf https://astral.sh/uv/install.sh | shCreate a local run config from the example:
cd Wintap-PyUtil
cp wintap-run.env.example wintap-run.env
# edit wintap-run.env
source wintap-run.envThe local config defines one run root and derived DBT settings:
WINTAP_DATA_ROOT
WINTAP_DBT_DATASET
WINTAP_DBT_DATABASE
WINTAP_DBT_START_DAY
WINTAP_DBT_END_DAY
Useful targets:
make print-dbt-config
make dbt-debug
make dbt-build
make dbt-test
make qa-pid-hash
make dbt-docsDBT Makefile targets use uv run --isolated --dev by default so a missing or broken local .venv does not block processing. Override with DBT_UV_RUN='uv run' only if you intentionally want to use the project virtual environment.
Create the development environment and install the package in editable mode:
The canonical Python/DBT environment pins DuckDB 1.5.6 in pyproject.toml.
The DBT adapter remains on the 1.9 line; DuckDB 2.0 is used only by the isolated compactor.
uv sync --devRun commands through uv:
uv run python -c "import wintappy; print(wintappy.VERSION)"The pre-DBT Python/DuckDB pipeline remains available as legacy/reference tooling while the DBT path is hardened:
uv run rawtorolling --help
uv run rawtostdview --help
uv run ubersummary --helpUse DBT for new processing work.
make fmt # black + isort
make fmt-check # formatting checks
make lint # mypy + sqlfluff
make test # pytest
make ci # fmt-check + lint + test + dbt-build
make cleanLLNL-CODE-837816
