Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -591,7 +591,7 @@ GO_GENERATE_PACKAGES := \
./runway/extension/merger/... \
./stovepipe/core/requestlog/... \
./stovepipe/extension/buildrunner/... \
./stovepipe/extension/projectresult/... \
./stovepipe/extension/projectstatus/... \
./stovepipe/extension/queueconfig/... \
./stovepipe/extension/sourcecontrol/... \
./stovepipe/extension/storage/... \
Expand Down
8 changes: 4 additions & 4 deletions doc/rfc/stovepipe/workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Greenness is recorded as a **health degree** where **`0` means green** and **hig

### Project — greenness at a finer grain

A **project** is a caller-defined slice of the repository. Whole-repo greenness answers "is the branch green at this URI"; project greenness answers the question deployments actually need — **"is *this project* green at this URI"**, and its dual, "what is the latest URI at which this project is green". The running pipeline does not derive projects from a target graph. On a succeeded or failed request, `record` gives the request's terminal build id to `projectresult.Resolver` and writes one validation fact per result it returns. The example server wires the noop resolver, which returns no results. How a resolver chooses projects is implementer-specific. The separate analyze stage that would map a target graph to project-scoped builds is not built; see [Designed, not built](#designed-not-built-project-analysis).
A **project** is a caller-defined slice of the repository. Whole-repo greenness answers "is the branch green at this URI"; project greenness answers the question deployments actually need — **"is *this project* green at this URI"**, and its dual, "what is the latest URI at which this project is green". The running pipeline does not derive projects from a target graph. On a succeeded or failed request, `record` gives the request's terminal build id to `projectstatus.Resolver` and writes one validation fact per result it returns. The example server wires the noop resolver, which returns no results. How a resolver chooses projects is implementer-specific. The separate analyze stage that would map a target graph to project-scoped builds is not built; see [Designed, not built](#designed-not-built-project-analysis).

### Promotion ref — the last green commit, by name

Expand All @@ -64,7 +64,7 @@ Hooks are the notification boundary. When validation of a commit begins, when a

## Workflow

What runs is one pass per Request. It establishes whole-repository greenness, and on a succeeded or failed outcome `record` also gives the Request's terminal build id to `projectresult.Resolver` before writing whatever project facts it returns. That call is inline on the record delivery. It does not publish to another stage, and it does not start another build.
What runs is one pass per Request. It establishes whole-repository greenness, and on a succeeded or failed outcome `record` also gives the Request's terminal build id to `projectstatus.Resolver` before writing whatever project facts it returns. That call is inline on the record delivery. It does not publish to another stage, and it does not start another build.

```
external poller ──(Queue name)──► ┌──────────────────────────────┐
Expand Down Expand Up @@ -116,7 +116,7 @@ What runs is one pass per Request. It establishes whole-repository greenness, an
2. **process** — decides build strategy (incremental since last-green vs full monorepo), gates concurrent work per Queue, coalesces backlog to the latest head, publishes a **hook event** announcing that validation of the commit has begun, and publishes to `build`. See [process.md](steps/process.md).
3. **build** — runs the build-runner for the chosen scope. A flag derived from `process` decides whether to build relative to the last-green **baseline URI** (incremental) or from scratch (full). It records a build and publishes the BuildID.
4. **buildsignal** — polls until the build is terminal, records that status, releases the Queue's `in_flight_count` slot, projects the terminal status and build id onto the Request (`succeeded` / `failed` / `cancelled`), and publishes the RequestID to `record`.
5. **record** — for a succeeded or failed Request, writes the whole-repo greenness for the head URI (`0` green / `1` broken to start), derived from the Request's build outcome. On green it advances the Queue's **last-green URI** so the next `process` can build incrementally from here, and asks `SourceControl` to advance the Queue's **promotion ref** to the same commit (see [Promotion ref](#promotion-ref--the-last-green-commit-by-name)). It then gives the Request's terminal build id to `projectresult.Resolver` and writes one validation fact per returned result. The example server uses the noop resolver, so that list is empty unless a deployment supplies another. It publishes `validation.repository.recorded` for that fact. The Queue's `in_flight_count` was already released by `buildsignal` when the build went terminal. A cancelled Request writes no fact and publishes `validation.repository.cancelled`, so a consumer can stop waiting on the commit.
5. **record** — for a succeeded or failed Request, writes the whole-repo greenness for the head URI (`0` green / `1` broken to start), derived from the Request's build outcome. On green it advances the Queue's **last-green URI** so the next `process` can build incrementally from here, and asks `SourceControl` to advance the Queue's **promotion ref** to the same commit (see [Promotion ref](#promotion-ref--the-last-green-commit-by-name)). It then gives the Request's terminal build id to `projectstatus.Resolver` and writes one validation fact per returned result. The example server uses the noop resolver, so that list is empty unless a deployment supplies another. It publishes `validation.repository.recorded` for that fact. The Queue's `in_flight_count` was already released by `buildsignal` when the build went terminal. A cancelled Request writes no fact and publishes `validation.repository.cancelled`, so a consumer can stop waiting on the commit.

### Designed, not built: project analysis

Expand Down Expand Up @@ -154,4 +154,4 @@ Callers gate deployments on greenness, so the dangerous failure is a Request tha

- **Greenness degree semantics.** The endpoints (`0` green, `1` fully broken) are fixed; the meaning of intermediate values once projects exist (fraction of projects broken? weighted severity?) is deferred until project analysis is concrete.
- **Poller vs. webhook ingestion.** Only the external poller is in scope now. The dedup key is designed so a webhook producer can be added later without changing identity, but that producer is out of scope for this RFC.
- **Project mapping contract.** Not built. There is no analyze controller or topic. `record` already persists facts from `projectresult.Resolver` on the same delivery as the repository fact. The unbuilt analyze design — a target-graph mapping, project-scoped builds, and whether that mapping is an extension or an external service — is still open.
- **Project mapping contract.** Not built. There is no analyze controller or topic. `record` already persists facts from `projectstatus.Resolver` on the same delivery as the repository fact. The unbuilt analyze design — a target-graph mapping, project-scoped builds, and whether that mapping is an extension or an external service — is still open.
2 changes: 1 addition & 1 deletion service/stovepipe/server/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ go_library(
"//stovepipe/core/requestlog:go_default_library",
"//stovepipe/extension/buildrunner:go_default_library",
"//stovepipe/extension/buildrunner/fake:go_default_library",
"//stovepipe/extension/projectresult/noop:go_default_library",
"//stovepipe/extension/projectstatus/noop:go_default_library",
"//stovepipe/extension/queueconfig/default:go_default_library",
"//stovepipe/extension/sourcecontrol:go_default_library",
"//stovepipe/extension/sourcecontrol/fake:go_default_library",
Expand Down
4 changes: 2 additions & 2 deletions service/stovepipe/server/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ import (
"github.com/uber/submitqueue/stovepipe/core/requestlog"
"github.com/uber/submitqueue/stovepipe/extension/buildrunner"
buildrunnerfake "github.com/uber/submitqueue/stovepipe/extension/buildrunner/fake"
projectresultnoop "github.com/uber/submitqueue/stovepipe/extension/projectresult/noop"
projectstatusnoop "github.com/uber/submitqueue/stovepipe/extension/projectstatus/noop"
queueconfigdefault "github.com/uber/submitqueue/stovepipe/extension/queueconfig/default"
"github.com/uber/submitqueue/stovepipe/extension/sourcecontrol"
sourcecontrolfake "github.com/uber/submitqueue/stovepipe/extension/sourcecontrol/fake"
Expand Down Expand Up @@ -441,7 +441,7 @@ func registerPrimaryControllers(
}
count++

recordController := record.NewController(logger, scope, store, materializer, projectresultnoop.New(), sourceControl, registry, stovepipemq.TopicKeyRecord, "stovepipe-record")
recordController := record.NewController(logger, scope, store, materializer, projectstatusnoop.New(), sourceControl, registry, stovepipemq.TopicKeyRecord, "stovepipe-record")
if err := c.Register(recordController); err != nil {
return count, fmt.Errorf("failed to register record controller: %w", err)
}
Expand Down
8 changes: 4 additions & 4 deletions stovepipe/controller/record/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ go_library(
"//stovepipe/core/messagequeue:go_default_library",
"//stovepipe/core/requestlog:go_default_library",
"//stovepipe/entity:go_default_library",
"//stovepipe/extension/projectresult:go_default_library",
"//stovepipe/extension/projectstatus:go_default_library",
"//stovepipe/extension/sourcecontrol:go_default_library",
"//stovepipe/extension/storage:go_default_library",
"@com_github_uber_go_tally//:go_default_library",
Expand Down Expand Up @@ -47,9 +47,9 @@ go_test(
"//stovepipe/core/requestlog:go_default_library",
"//stovepipe/core/requestlog/mock:go_default_library",
"//stovepipe/entity:go_default_library",
"//stovepipe/extension/projectresult:go_default_library",
"//stovepipe/extension/projectresult/mock:go_default_library",
"//stovepipe/extension/projectresult/noop:go_default_library",
"//stovepipe/extension/projectstatus:go_default_library",
"//stovepipe/extension/projectstatus/mock:go_default_library",
"//stovepipe/extension/projectstatus/noop:go_default_library",
"//stovepipe/extension/sourcecontrol:go_default_library",
"//stovepipe/extension/sourcecontrol/mock:go_default_library",
"//stovepipe/extension/storage:go_default_library",
Expand Down
56 changes: 28 additions & 28 deletions stovepipe/controller/record/record.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ import (
stovepipemq "github.com/uber/submitqueue/stovepipe/core/messagequeue"
"github.com/uber/submitqueue/stovepipe/core/requestlog"
"github.com/uber/submitqueue/stovepipe/entity"
"github.com/uber/submitqueue/stovepipe/extension/projectresult"
"github.com/uber/submitqueue/stovepipe/extension/projectstatus"
"github.com/uber/submitqueue/stovepipe/extension/sourcecontrol"
"github.com/uber/submitqueue/stovepipe/extension/storage"
"go.uber.org/zap"
Expand All @@ -58,15 +58,15 @@ import (
// when that fact is green advances the queue's last-green bookmark and promotes
// the commit. Implements consumer.Controller.
type Controller struct {
logger *zap.SugaredLogger
metricsScope tally.Scope
stores storage.Factory
materializer requestlog.Materializer
projectResultFactory projectresult.Factory
sourceControl sourcecontrol.Factory
registry consumer.TopicRegistry
topicKey consumer.TopicKey
consumerGroup string
logger *zap.SugaredLogger
metricsScope tally.Scope
stores storage.Factory
materializer requestlog.Materializer
projectStatusResolverFactory projectstatus.ResolverFactory
sourceControl sourcecontrol.Factory
registry consumer.TopicRegistry
topicKey consumer.TopicKey
consumerGroup string
}

// Verify Controller implements consumer.Controller interface at compile time.
Expand All @@ -85,23 +85,23 @@ func NewController(
scope tally.Scope,
stores storage.Factory,
materializer requestlog.Materializer,
projectResultFactory projectresult.Factory,
projectStatusResolverFactory projectstatus.ResolverFactory,
sourceControl sourcecontrol.Factory,
registry consumer.TopicRegistry,
topicKey consumer.TopicKey,
consumerGroup string,
) *Controller {
name := string(topicKey) + "_controller"
return &Controller{
logger: logger.Named(name),
metricsScope: scope.SubScope(name),
stores: stores,
materializer: materializer,
projectResultFactory: projectResultFactory,
sourceControl: sourceControl,
registry: registry,
topicKey: topicKey,
consumerGroup: consumerGroup,
logger: logger.Named(name),
metricsScope: scope.SubScope(name),
stores: stores,
materializer: materializer,
projectStatusResolverFactory: projectStatusResolverFactory,
sourceControl: sourceControl,
registry: registry,
topicKey: topicKey,
consumerGroup: consumerGroup,
}
}

Expand Down Expand Up @@ -184,16 +184,16 @@ func (c *Controller) Process(ctx context.Context, delivery consumer.Delivery) er
}

func (c *Controller) recordProjectFacts(ctx context.Context, store storage.Storage, request entity.Request) error {
resolver, err := c.projectResultFactory.For(projectresult.Config{QueueName: request.Queue})
resolver, err := c.projectStatusResolverFactory.For(projectstatus.ResolverConfig{QueueName: request.Queue})
if err != nil {
return fmt.Errorf("failed to resolve project result resolver for queue %q: %w", request.Queue, err)
return fmt.Errorf("resolve project status resolver for queue %q: %w", request.Queue, err)
}
results, err := resolver.Resolve(ctx, request, request.TerminalBuildID)
if err != nil {
return fmt.Errorf("failed to resolve project results for request %q: %w", request.ID, err)
return fmt.Errorf("resolve project status results for request %q: %w", request.ID, err)
}

if err := validateProjectResults(request, results); err != nil {
if err := validateProjectStatusResults(request, results); err != nil {
return err
}
for _, result := range results {
Expand All @@ -214,18 +214,18 @@ func (c *Controller) recordProjectFacts(ctx context.Context, store storage.Stora
return c.persistProjectFactsRecordedLog(ctx, store, request, len(results))
}

func validateProjectResults(request entity.Request, results []projectresult.Result) error {
func validateProjectStatusResults(request entity.Request, results []projectstatus.Result) error {
seen := make(map[string]struct{}, len(results))
for _, result := range results {
if result.Project == "" {
return fmt.Errorf("project result for request %q has an empty project", request.ID)
return fmt.Errorf("project status result for request %q has an empty project", request.ID)
}
if _, ok := seen[result.Project]; ok {
return fmt.Errorf("project result for request %q contains duplicate project %q", request.ID, result.Project)
return fmt.Errorf("project status result for request %q contains duplicate project %q", request.ID, result.Project)
}
seen[result.Project] = struct{}{}
if math.IsNaN(result.Degree) || result.Degree < entity.DegreeGreen || result.Degree > entity.DegreeBroken {
return fmt.Errorf("project result for request %q and project %q has invalid degree %v", request.ID, result.Project, result.Degree)
return fmt.Errorf("project status result for request %q and project %q has invalid degree %v", request.ID, result.Project, result.Degree)
}
}
return nil
Expand Down
Loading
Loading