Skip to content

About

No description, website, or topics provided.

Resources

Stars

3 stars

Watchers

2 watching

Forks

Latest commit

 

History

99 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Wintap-PyUtil

Python, DuckDB, and DBT utilities for processing Wintap/Lintap telemetry data.

DBT is now the primary post-processing path. The canonical input is a raw_sensor parquet dataset:

<dataset>/raw_sensor/<event_type>/dayPK=YYYYMMDD/hourPK=HH/*.parquet
<dataset>/raw_sensor/raw_process_conn_incr/dayPK=YYYYMMDD/hourPK=HH/protoPK=tcp|udp/*.parquet
<dataset>/compacted/<event_type>/dayPK=YYYYMMDD/hourPK=HH[/protoPK=...]/<merged>.parquet

DBT reads both raw and compacted locations with name-aligned UNION ALL semantics. Compaction validates then deletes source prefixes after the UTC upload grace; run DBT after successful migration, not during overlap or an unreconciled failure. WINTAP_DBT_COMPACTED_ROOT optionally overrides the direct compacted event root. See the DBT guide.

The hourly compactor lives in extras/raw-sensor-compact/ as a standalone UV project, with its own lockfile and DuckDB 2.0.0 environment. It does not upgrade the main pipeline's DuckDB 1.5.6 dependency. From this repository root:

uv run --project extras/raw-sensor-compact --locked raw-sensor-compact --help

The same worker optionally maintains durable process snapshots in S3. Infrequent DBT builds use them for historical/exited parent context without adding lifecycle observations to activity counts. Bootstrap history explicitly and exclude readers/writers during updates and builds. See DBT configuration.

For the full current run process, start with:

review-notes/PipelineRunbook.md

Minimum system requirements

  • Python 3.10, 3.11, or 3.12
  • uv
  • DuckDB CLI for QA scripts
  • DuckDB-compatible parquet data

Install uv if needed:

curl -LsSf https://astral.sh/uv/install.sh | sh

DBT pipeline quick setup

Create a local run config from the example:

cd Wintap-PyUtil
cp wintap-run.env.example wintap-run.env
# edit wintap-run.env
source wintap-run.env

The local config defines one run root and derived DBT settings:

WINTAP_DATA_ROOT
WINTAP_DBT_DATASET
WINTAP_DBT_DATABASE
WINTAP_DBT_START_DAY
WINTAP_DBT_END_DAY

Useful targets:

make print-dbt-config
make dbt-debug
make dbt-build
make dbt-test
make qa-pid-hash
make dbt-docs

DBT Makefile targets use uv run --isolated --dev by default so a missing or broken local .venv does not block processing. Override with DBT_UV_RUN='uv run' only if you intentionally want to use the project virtual environment.

Development setup

Create the development environment and install the package in editable mode:

The canonical Python/DBT environment pins DuckDB 1.5.6 in pyproject.toml. The DBT adapter remains on the 1.9 line; DuckDB 2.0 is used only by the isolated compactor.

uv sync --dev

Run commands through uv:

uv run python -c "import wintappy; print(wintappy.VERSION)"

Legacy Python ETL commands

The pre-DBT Python/DuckDB pipeline remains available as legacy/reference tooling while the DBT path is hardened:

uv run rawtorolling --help
uv run rawtostdview --help
uv run ubersummary --help

Use DBT for new processing work.

Development tasks

make fmt          # black + isort
make fmt-check    # formatting checks
make lint         # mypy + sqlfluff
make test         # pytest
make ci           # fmt-check + lint + test + dbt-build
make clean

Release

LLNL-CODE-837816

About

No description, website, or topics provided.

Resources

Stars

3 stars

Watchers

2 watching

Forks

Releases

Packages

Used by

Contributors

Languages