diff --git a/.agent/skills/beam-concepts/SKILL.md b/.agent/skills/beam-concepts/SKILL.md index da3dd9fbf319..4807ea849cbe 100644 --- a/.agent/skills/beam-concepts/SKILL.md +++ b/.agent/skills/beam-concepts/SKILL.md @@ -25,6 +25,38 @@ description: Explains core Apache Beam programming model concepts including PCol ## The Beam Model Evolved from Google's MapReduce, FlumeJava, and Millwheel projects. Originally called the "Dataflow Model." +### Protos + +The `/model` directory defines the official, language-agnostic Protocol Buffer (`.proto`) and gRPC service specifications that establish the **Beam Model** and the **Beam Portability Framework**. + +#### Why `/model` Exists (Portability & Decoupling) + +Without a standardized model representation, supporting $N$ SDK languages across $M$ execution runners would require $N \times M$ separate translation layers. By defining all core pipeline concepts, data encodings, metrics, and worker RPC protocols as Protobuf messages and gRPC services, `/model` acts as the universal lingua franca: + +* **SDKs** compile user pipelines into standardized Runner API protobuf graphs. + +* **Runners** inspect, optimize, and distribute these graphs without needing SDK-specific language runtimes. + +* **Workers (SDK Harnesses)** execute user code (`DoFn`s) and communicate with runners over standardized Fn API gRPC channels. + +#### Core Directories & What They Do + +1. **`/model/pipeline` (Runner API & Core Model)**: Defines the SDK- and runner-independent representation of pipelines (`Pipeline`, `Components`, `PTransform`, `PCollection`, `Coder`), timestamps/constants, Beam Schemas (`Row`, `Field`), and execution metrics (`MonitoringInfo`). +2. **`/model/fn-execution` (Fn API & Provisioning)**: Defines bidirectional gRPC services between runners and worker SDK harnesses for bundle execution (`Control`), element streaming (`Data`), state/timer access (`State`), log forwarding (`Logging`), and container initialization (`Provisioning`). +3. **`/model/job-management` (Job, Expansion, & Artifact APIs)**: Defines gRPC interfaces for submitting and monitoring jobs on remote servers (`JobService`), resolving cross-language transforms in remote SDKs (`ExpansionService`), and staging dependency artifacts or container images (`ArtifactService`). +4. **`/model/interactive` (Interactive API)**: Defines metadata and stream headers for recording and replaying data in Interactive Beam notebooks. + +#### What Agents Need to Pay Special Attention To + +* **Conservative Proto Changes**: Proto changes are generally conservative and accepted only when there is a compelling reason and community consensus. Every addition introduces a new obligation that each SDK and runner must support; adding new Beam model elements (portable types, capabilities) increases the compatibility gap between SDK and runner capabilities. +* **URNs Are the API Contract**: Transforms, coders, windowing strategies, environments, and metrics are bound together by standardized string URNs (e.g., `beam:transform:pardo:v1`, `beam:coder:bytes:v1`). When inspecting or creating transforms across languages, always verify URN mappings and registry handlers in both the SDK and Runner runtimes. +* **Strict Backward & Wire Compatibility**: + * Never renumber, delete, or modify existing field IDs or URN strings in `.proto` files, as they are used across distributed RPC boundaries and persisted checkpoints. + * When extending Java/Python SDK classes to support new proto fields, use **Builder patterns** (e.g., `OutputBuilder`) rather than adding arguments to legacy static factory methods (`WindowedValue.of(...)`), which breaks API compatibility. +* **Build System & Naming Collisions**: + * Modifying files in `/model` requires re-generating language bindings (e.g., `./gradlew :model:pipeline:generateProto`). + * Avoid protobuf field names that conflict with reserved keywords in target languages (e.g., `class` in Java or `output` in Python, as noted in `beam_fn_api.proto` comments). + ## Key Abstractions ### Pipeline diff --git a/.agent/skills/contributing/SKILL.md b/.agent/skills/contributing/SKILL.md index bac50c5d0cd5..2a60c5a3cd98 100644 --- a/.agent/skills/contributing/SKILL.md +++ b/.agent/skills/contributing/SKILL.md @@ -66,6 +66,9 @@ description: Guides the contribution workflow for Apache Beam, including creatin - Every source file needs Apache license header - New dependencies must have Apache-compatible open source licenses - Add unit tests for your changes +- Document public-facing APIs in a standard way (Javadoc, docstrings/pydoc). + - Documentation should be user-friendly, informative. + - Implementation details belong in inline code comments. - Use descriptive commit messages ### 5. Create Pull Request