Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .agent/skills/beam-concepts/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,38 @@ description: Explains core Apache Beam programming model concepts including PCol
## The Beam Model
Evolved from Google's MapReduce, FlumeJava, and Millwheel projects. Originally called the "Dataflow Model."

### Protos

The `/model` directory defines the official, language-agnostic Protocol Buffer (`.proto`) and gRPC service specifications that establish the **Beam Model** and the **Beam Portability Framework**.

#### Why `/model` Exists (Portability & Decoupling)

Without a standardized model representation, supporting $N$ SDK languages across $M$ execution runners would require $N \times M$ separate translation layers. By defining all core pipeline concepts, data encodings, metrics, and worker RPC protocols as Protobuf messages and gRPC services, `/model` acts as the universal lingua franca:

* **SDKs** compile user pipelines into standardized Runner API protobuf graphs.

* **Runners** inspect, optimize, and distribute these graphs without needing SDK-specific language runtimes.

* **Workers (SDK Harnesses)** execute user code (`DoFn`s) and communicate with runners over standardized Fn API gRPC channels.

#### Core Directories & What They Do

1. **`/model/pipeline` (Runner API & Core Model)**: Defines the SDK- and runner-independent representation of pipelines (`Pipeline`, `Components`, `PTransform`, `PCollection`, `Coder`), timestamps/constants, Beam Schemas (`Row`, `Field`), and execution metrics (`MonitoringInfo`).
2. **`/model/fn-execution` (Fn API & Provisioning)**: Defines bidirectional gRPC services between runners and worker SDK harnesses for bundle execution (`Control`), element streaming (`Data`), state/timer access (`State`), log forwarding (`Logging`), and container initialization (`Provisioning`).
3. **`/model/job-management` (Job, Expansion, & Artifact APIs)**: Defines gRPC interfaces for submitting and monitoring jobs on remote servers (`JobService`), resolving cross-language transforms in remote SDKs (`ExpansionService`), and staging dependency artifacts or container images (`ArtifactService`).
4. **`/model/interactive` (Interactive API)**: Defines metadata and stream headers for recording and replaying data in Interactive Beam notebooks.

#### What Agents Need to Pay Special Attention To

* **Conservative Proto Changes**: Proto changes are generally conservative and accepted only when there is a compelling reason and community consensus. Every addition introduces a new obligation that each SDK and runner must support; adding new Beam model elements (portable types, capabilities) increases the compatibility gap between SDK and runner capabilities.
* **URNs Are the API Contract**: Transforms, coders, windowing strategies, environments, and metrics are bound together by standardized string URNs (e.g., `beam:transform:pardo:v1`, `beam:coder:bytes:v1`). When inspecting or creating transforms across languages, always verify URN mappings and registry handlers in both the SDK and Runner runtimes.
* **Strict Backward & Wire Compatibility**:
* Never renumber, delete, or modify existing field IDs or URN strings in `.proto` files, as they are used across distributed RPC boundaries and persisted checkpoints.
* When extending Java/Python SDK classes to support new proto fields, use **Builder patterns** (e.g., `OutputBuilder`) rather than adding arguments to legacy static factory methods (`WindowedValue.of(...)`), which breaks API compatibility.
* **Build System & Naming Collisions**:
* Modifying files in `/model` requires re-generating language bindings (e.g., `./gradlew :model:pipeline:generateProto`).
* Avoid protobuf field names that conflict with reserved keywords in target languages (e.g., `class` in Java or `output` in Python, as noted in `beam_fn_api.proto` comments).

## Key Abstractions

### Pipeline
Expand Down
3 changes: 3 additions & 0 deletions .agent/skills/contributing/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,9 @@ description: Guides the contribution workflow for Apache Beam, including creatin
- Every source file needs Apache license header
- New dependencies must have Apache-compatible open source licenses
- Add unit tests for your changes
- Document public-facing APIs in a standard way (Javadoc, docstrings/pydoc).
- Documentation should be user-friendly, informative.
- Implementation details belong in inline code comments.
- Use descriptive commit messages

### 5. Create Pull Request
Expand Down
Loading