Aspire-based load testing harness

.github/workflows/testing-tool-ci.yml

@@ -0,0 +1,49 @@
+name: Testing Tool CI
+
+on:
+  push:
+    branches:
+      - master
+      - release-*
+    paths:
+      - 'tools/testing-tool/**'
+      - '.github/workflows/testing-tool-ci.yml'
+  pull_request:
+    paths:
+      - 'tools/testing-tool/**'
+      - '.github/workflows/testing-tool-ci.yml'
+  workflow_dispatch:
+
+env:
+  DOTNET_NOLOGO: true
+
+defaults:
+  run:
+    shell: pwsh
+
+jobs:
+  build:
+    name: Build solution
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v7.0.1
+        with:
+          fetch-depth: 0
+
+      - name: Setup .NET SDK
+        uses: actions/setup-dotnet@v6.0.0
+        with:
+          global-json-file: global.json
+
+      - name: Build
+        run: dotnet build tools/testing-tool/TestingTool.slnx --configuration Release
+
+      - name: Build container image
+        uses: docker/build-push-action@v7.2.0
+        with:
+          context: .
+          file: tools/testing-tool/Dockerfile
+          push: false
+          load: true
+          tags: particular/testing-tool:ci

README.md

@@ -113,6 +113,7 @@ Local testing guides:
 - [Forward Headers Testing](docs/forward-headers-testing.md)
 - [Authentication Testing](docs/authentication-testing.md)
 - [Persistence Tests](docs/testing-persistence.md)
+- [Load Testing](docs/load-testing.md)
 
 ## How to developer test the PowerShell Module
 

docs/load-testing.md

@@ -0,0 +1,47 @@
+# Load Testing
+
+The [ServiceControl Testing Tool](../tools/testing-tool/README.md) is a stateless .NET service that generates error load and real-world failure scenarios against a test ServiceControl instance, to validate error-ingestion performance. It ships with a web UI for manual control, built-in failure scenarios (third-party outage, timeout spike, poison message, and more), and OpenTelemetry telemetry.
+
+This guide covers getting it up and running quickly. For the full reference — every scenario, API endpoint, configuration setting, container usage, and scaling notes — see the [testing tool README](../tools/testing-tool/README.md).
+
+## Quick start: full stack with Aspire
+
+The Aspire AppHost brings up the whole system in a single command: the ServiceControl platform (Learning transport), the testing tool, and a complete observability stack (OTel Collector, Jaeger, Prometheus, and Grafana with a prebuilt dashboard).
+
+```bash
+aspire run tools/testing-tool/TestingTool.AppHost/TestingTool.AppHost.csproj
+```
+
+Aspire assigns dynamic ports — open the testing tool's web UI via the link in the Aspire dashboard.
+
+Useful flags, passed after `--`:
+
+| Flag | Description |
+|---|---|
+| `--tag <tag>` | Test a specific ServiceControl image tag, e.g. a PR prerelease like `pr-1234` |
+| `--persistence <type>` | Persistence for the error instance: `RavenDb` (default), `SqlServer`, or `PostgreSql` |
+| `--error-ingestion-scale-unit <n>` | Spin up `n` additional error-ingestion-only ServiceControl instances |
+
+## Run against an existing ServiceControl
+
+```bash
+dotnet build tools/testing-tool/TestingTool.slnx --configuration Release
+dotnet run --project tools/testing-tool/TestingTool --configuration Release
+```
+
+Open http://localhost:5290. The tool defaults to a ServiceControl instance at `http://localhost:33333`; point it elsewhere with the `TestingTool__ServiceControlApiUrl` environment variable.
+
+## Generating load
+
+Everything is controllable from the web UI, or via the HTTP API:
+
+- **Scenarios** — the five built-in failure scenarios (`third-party-outage`, `timeout-spike`, `poison-message`, `deserialization-failure`, `background-noise`), each producing naturally-grouped errors. Start with `POST /api/scenarios/{name}/start` (optional `rate` and `durationSeconds`).
+- **Bypass writer** — high-throughput load that writes failed-message envelopes directly to the ServiceControl error queue, skipping the message handler (`POST /api/bypass/start`).
+- **Jobs** — retry, archive, search, retention sweep, and custom-check-failure jobs exercise ServiceControl's recoverability, FTS search, and retention pipelines against the load. Jobs do not auto-start; kick them off from the UI or `/api/jobs`.
+- **Release-test presets** — named presets mapped from the [testing scenarios](testing-scenarios.md), e.g. `POST /api/release-tests/ingestion-load/start`.
+
+## Observing the run
+
+- The Aspire dashboard shows logs and traces for every service.
+- Grafana (auto-provisioned; log in with `admin`/`admin`) ships a prebuilt "Testing Tool" dashboard — errors/sec by scenario, search latency p95, replay/archive rates, and errors raised vs ingested.
+- Jaeger for trace analysis, Prometheus for metrics. The tool also exposes a Prometheus scraping endpoint at `/metrics`.

tools/testing-tool/Directory.Build.props

@@ -0,0 +1,24 @@
+<Project>
+
+  <!--
+    The testing tool is a self-contained, stateless .NET 10 service kept outside the main
+    ServiceControl `src/` tree so it is not swept into the platform build. These properties mirror
+    the conventions used by the rest of the repository (warnings-as-errors, code-style enforcement,
+    nullable reference types, implicit usings) without inheriting the platform packaging targets.
+  -->
+  <PropertyGroup>
+    <TargetFramework>net10.0</TargetFramework>
+    <ImplicitUsings>enable</ImplicitUsings>
+    <Nullable>enable</Nullable>
+    <TreatWarningsAsErrors Condition="'$(Configuration)' != 'Debug'">true</TreatWarningsAsErrors>
+    <EnableNETAnalyzers>true</EnableNETAnalyzers>
+    <EnforceCodeStyleInBuild>true</EnforceCodeStyleInBuild>
+    <NuGetAuditLevel>low</NuGetAuditLevel>
+    <NuGetAuditMode Condition="'$(NuGetAuditMode)' == ''">all</NuGetAuditMode>
+  </PropertyGroup>
+
+  <PropertyGroup Condition="'$(CI)' != ''">
+    <ContinuousIntegrationBuild>true</ContinuousIntegrationBuild>
+  </PropertyGroup>
+
+</Project>

tools/testing-tool/Dockerfile

@@ -0,0 +1,38 @@
+# Multi-stage build for the ServiceControl testing tool.
+# Build context is the repository root so global.json + nuget.config are available.
+ARG RUNTIMEVERSION=10.0
+
+# --- build stage ---
+FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:10.0 AS build
+ARG TARGETARCH
+WORKDIR /src
+ENV CI=true
+
+# global.json pins the SDK; nuget.config adds the Particular feed for later NServiceBus deps.
+COPY global.json nuget.config ./
+COPY tools/testing-tool/ ./tools/testing-tool/
+
+# Publish the runnable host; its ProjectReferences pull in Contracts + Scenarios.
+RUN dotnet publish tools/testing-tool/TestingTool/TestingTool.csproj \
+    --configuration Release \
+    --arch $TARGETARCH \
+    --output /app \
+    /p:UseAppHost=false
+
+# --- runtime stage ---
+# Uses the same chiseled composite base as the rest of the ServiceControl images so that
+# globalization/ICU dependencies are present.
+FROM mcr.microsoft.com/dotnet/aspnet:$RUNTIMEVERSION-noble-chiseled-composite-extra
+WORKDIR /app
+
+COPY --from=build /app ./
+
+ENV ASPNETCORE_URLS=http://+:8080
+EXPOSE 8080
+
+# Health probes are served by the /health/live and /health/ready HTTP endpoints (added in
+# Phase 6). Kubernetes uses HTTP GET probes directly; the chiseled base ships no shell/curl so
+# exec-based probes are not supported.
+
+USER $APP_UID
+ENTRYPOINT ["dotnet", "TestingTool.dll"]