WPB-26489 backend adminless scheduled jobs model and teardown support by battermann · Pull Request #5289 · wireapp/wire-server

battermann · 2026-06-25T07:39:56Z

Checklist

Add a new entry in an appropriate subdirectory of changelog.d
Read and follow the PR guidelines

Copilot

Pull request overview

This PR introduces an Arbiter-backed scheduled-jobs subsystem across services, migrating the meetings-cleanup worker to run via Arbiter and adding initial “adminless” reminder/deletion scheduled jobs, along with an internal /i/jobs UI/API mount in Brig.

Changes:

Add Wire.API.Jobs, Wire.JobSubsystem, and Wire.JobStore to model/schedule jobs and persist minimal app-level metadata in Postgres.
Replace the standalone meetings cleanup cron thread with an Arbiter worker pool (scheduledJobs.pollInterval controls polling cadence).
Expose Arbiter’s internal UI + REST API through Brig’s internal routes under /i/jobs and /i/jobs/api/v1.

Reviewed changes

Copilot reviewed 56 out of 57 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
services/galley/src/Galley/Env.hs	Add Galley env field for passing a connection string into the job subsystem.
services/galley/src/Galley/App.hs	Wire `JobSubsystem` + `JobStore` interpreters and derive a Postgres connection string for Arbiter usage.
services/galley/galley.cabal	Add `arbiter-core` dependency for Galley.
services/galley/default.nix	Add `arbiter-core` Nix dependency for Galley.
services/brig/src/Brig/Run.hs	Adjust internal API wiring to pass `env` into `IAPI.servantSitemap`.
services/brig/src/Brig/App.hs	Initialize embedded Arbiter server/UI application and store it in Brig env.
services/brig/src/Brig/API/Internal.hs	Mount internal `/i/jobs` Raw app into Brig internal routes.
services/brig/default.nix	Add Arbiter packages (`arbiter-core`, `arbiter-servant`, `arbiter-servant-ui`).
services/brig/brig.cabal	Add Arbiter packages to Brig build-depends.
services/background-worker/test/Test/Wire/Util.hs	Extend test env with `arbiterConnStr` placeholder.
services/background-worker/test/Test/Wire/BackendNotificationPusherSpec.hs	Extend test env with `arbiterConnStr` placeholder.
services/background-worker/src/Wire/MeetingsCleanupWorker.hs	Refactor to expose `runCleanupOldMeetings` instead of owning the cron thread; move metrics increment into cleanup routine.
services/background-worker/src/Wire/Effects.hs	Add `JobSubsystem`/`JobStore` effects + interpreters and extend migration handling.
services/background-worker/src/Wire/BackgroundWorker/ScheduledJobs.hs	New scheduled-jobs dispatcher/worker wiring for meetings cleanup + adminless jobs via Arbiter.
services/background-worker/src/Wire/BackgroundWorker/Options.hs	Add `scheduledJobs` config section (poll interval).
services/background-worker/src/Wire/BackgroundWorker/Env.hs	Add `arbiterConnStr` to runtime env, derived from Postgres settings.
services/background-worker/src/Wire/BackgroundWorker.hs	Start the unified scheduled-jobs worker instead of the old meetings-cleanup worker thread.
services/background-worker/src/Wire/AdminlessJobsWorker.hs	New adminless reminder/deletion job handlers + scheduled job cleanup in app-level store.
services/background-worker/default.nix	Add `arbiter-core` dependency for background-worker.
services/background-worker/background-worker.integration.yaml	Add integration config for `scheduledJobs.pollInterval`.
services/background-worker/background-worker.cabal	Expose new modules and add `arbiter-core` dependency.
postgres-schema.sql	Update schema snapshot to include scheduled jobs table and related DB objects.
nix/manual-overrides.nix	Add overrides/jailbreaks for Arbiter package set.
nix/haskell-pins.nix	Add Arbiter source pin and package mapping.
libs/wire-subsystems/wire-subsystems.cabal	Add Arbiter deps and expose new job subsystem/store modules; add `cron` dep.
libs/wire-subsystems/test/unit/Wire/ConversationSubsystem/InterpreterSpec.hs	Add a `JobSubsystem` mock interpreter for conversation subsystem unit tests.
libs/wire-subsystems/src/Wire/Options/Keys.hs	Fix ASN.1 imports to use `crypton-asn1-*` packages explicitly.
libs/wire-subsystems/src/Wire/JobSubsystem/Workers.hs	New Arbiter worker runner helpers (recurring + one-off) with explicit defaults.
libs/wire-subsystems/src/Wire/JobSubsystem/Interpreter.hs	New Polysemy interpreter to schedule jobs into Arbiter and start workers.
libs/wire-subsystems/src/Wire/JobSubsystem.hs	New Polysemy effect and configuration types for scheduling/starting jobs.
libs/wire-subsystems/src/Wire/JobStore/Postgres.hs	New Postgres-backed interpreter for persisting minimal scheduled job metadata.
libs/wire-subsystems/src/Wire/JobStore.hs	New Polysemy effect for scheduled job metadata persistence.
libs/wire-subsystems/src/Wire/ConversationSubsystem/Update.hs	Schedule adminless reminder/deletion jobs when last admin leaves; add reminder event emission.
libs/wire-subsystems/src/Wire/ConversationSubsystem/Interpreter.hs	Add internal actions for adminless deletion/reminder execution paths.
libs/wire-subsystems/src/Wire/ConversationSubsystem/Action.hs	Factor conversation deletion into reusable `removeConversation`.
libs/wire-subsystems/src/Wire/ConversationSubsystem.hs	Add new internal conversation actions for adminless job execution.
libs/wire-subsystems/postgres-migrations/20260624095944-extend-job-model.sql	Add `scheduled_jobs` table + indexes migration.
libs/wire-subsystems/default.nix	Add Arbiter and `cron` dependencies for wire-subsystems.
libs/wire-api/wire-api.cabal	Expose new Jobs API modules in wire-api.
libs/wire-api/test/unit/Test/Wire/API/Conversation.hs	Extend event tests for new `ConvAdminlessReminder` type classification.
libs/wire-api/src/Wire/API/Team/FeatureFlags.hs	Accept both legacy and current JSON keys for search visibility defaults.
libs/wire-api/src/Wire/API/Routes/Internal/Jobs.hs	New internal route type for mounting the embedded jobs app.
libs/wire-api/src/Wire/API/Routes/Internal/Brig.hs	Add Jobs app route into Brig internal API type.
libs/wire-api/src/Wire/API/Jobs.hs	New job payload types, queue names, and registry definition for Arbiter.
libs/wire-api/src/Wire/API/Event/Conversation.hs	Add `ConvAdminlessReminder` event type and corresponding event payload schema.
libs/types-common/src/Data/Id.hs	Introduce `ScheduledJobId` and `ScheduledJob` id tag.
libs/extended/src/Hasql/Pool/Extended.hs	Add shared `postgresqlConnectionString` helper and refactor pool init to use it.
integration/test/Test/AdminlessGroups.hs	Add integration coverage for adminless scheduling and Arbiter execution path.
integration/test/Notifications.hs	Add helper to detect adminless reminder notifications.
flake.nix	Add Arbiter git input.
flake.lock	Lock Arbiter input revision/hash.
docs/src/developer/reference/config-options.md	Document `scheduledJobs.pollInterval`.
charts/wire-server/values.yaml	Add Helm value for `background-worker.config.scheduledJobs.pollInterval`.
charts/wire-server/templates/background-worker/configmap.yaml	Render scheduledJobs config into background-worker configmap.
changelog.d/2-features/WPB-26489	Feature changelog entry for Arbiter-backed scheduled jobs + adminless jobs.
changelog.d/0-release-notes/WPB-26489	Release notes for Arbiter scheduling and `/i/jobs` endpoints.
cabal.project	Add `source-repository-package` entries for Arbiter packages.

 initPostgresPool :: PoolConfig -> Map Text Text -> Maybe FilePathSecrets -> IO HasqlPool.Pool
 initPostgresPool config pgConfig mFpSecrets = do
-  mPw <- for mFpSecrets initCredentials
-  let pgSettings =
-        HasqlConnSettings.connectionString (PostgresqlConnectionString.toUrl $ PostgresqlConnectionString.fromKeyValueParams pgConfig)
-          <> foldMap HasqlConnSettings.password mPw
+  connStr <- postgresqlConnectionString pgConfig mFpSecrets
+  let pgSettings = HasqlConnSettings.connectionString connStr
  metrics <- initHasqlPoolMetrics


 idTagName HistoryClient = "HistoryClient"
+idTagName ScheduledJob = "ScheduledJob "



+testAdminlessJobsExecuteViaArbiterApi = do
+  (alice, tid, _) <- createTeam OwnDomain 1
+
+  -- we diable the feature and create an adminless group


+  bindResponse (GalleyI.getConversation conv) $ \resp -> do
+    resp.status `shouldMatchInt` 200
+
+  -- now we enabled the feature so that the job will get excecuted


    _convCodeURI :: Either HttpsUrl (Map Text HttpsUrl),
-    _passwordHashingRateLimitEnv :: RateLimitEnv
+    _passwordHashingRateLimitEnv :: RateLimitEnv,
+    _jobsApiConnStr :: Text
  }


akshaymankar

Haven't read everything, comments based on just trying to understand the arbiter stuff.

akshaymankar · 2026-06-29T13:01:17Z

+
+source-repository-package
+  type: git
+  location: https://github.com/velveteer/arbiter.git
+  tag: 96097411a0480e182d0cbce819c45023fe8aeec7
+  subdir:
+    arbiter-core
+    arbiter-hasql
+    arbiter-migrations
+    arbiter-servant
+    arbiter-servant-ui
+    arbiter-simple
+    arbiter-worker
+    arbiter-test-common
+


This is already pinned using nix, so it shouldn't be needed.

Suggested change

source-repository-package

type: git

location: https://github.com/velveteer/arbiter.git

tag: 96097411a0480e182d0cbce819c45023fe8aeec7

subdir:

arbiter-core

arbiter-hasql

arbiter-migrations

arbiter-servant

arbiter-servant-ui

arbiter-simple

arbiter-worker

arbiter-test-common

akshaymankar · 2026-06-29T13:13:31Z

+  jobsApiConnStr <- postgresqlConnectionString opts.postgresql opts.postgresqlPassword
+  jobsApiApp <- do
+    config <- ArbServer.initArbiterServer (Proxy @ScheduledJobsRegistry) (encodeUtf8 jobsApiConnStr) ArbiterCore.defaultSchemaName
+    pure $ ArbUI.arbiterAppWithAdmin config
+  Log.info lgr $ Log.msg (Log.val "Internal jobs API initialized")


This would create a new connections outside of our regular connection pool. Also it uses postgresql-simple, making our project use both postgresql libraries.
Perhaps we can fork arbiter-servant and make it work with hasql in a separate PR and not have this admin UI until then?

akshaymankar · 2026-06-29T13:15:38Z

+jobsApp :: App.Env -> ServerT JobsIRoutes.JobsAppAPI (Handler r)
+jobsApp env = Tagged env.jobsApiApp


Why is this stored in the env?

akshaymankar · 2026-06-29T13:17:35Z

+    . case env.postgresMigration.teamFeatures of
+      CassandraStorage ->
+        interpretTeamFeatureStoreToCassandra
+      MigrationToPostgresql ->
+        interpretTeamFeatureStoreToCassandraAndPostgres
+      PostgresqlStorage ->
+        interpretTeamFeatureStoreToPostgres


Can we please extract this into a let binding?

akshaymankar · 2026-06-29T13:18:59Z

+      JobSubsystemConfig
+        { jobSubsystemArbiterConnStr = env.arbiterConnStr,
+          jobSubsystemSchemaName = ArbiterCore.defaultSchemaName
+        }


Also it'd be nice to extract this into a let binding. This series of function application usually generates really long errors, I'd keep everything extractable into a let binding.

akshaymankar · 2026-06-29T13:28:44Z

+type JobsAppAPI =
+  "jobs"
+    :> Raw


Why is this using Raw? The upstream seems to have the servant types.

zebot added the ok-to-test Approved for running tests in CI, overrides not-ok-to-test if both labels exist label Jun 25, 2026

battermann changed the base branch from WPB-26487-backend-scheduled-jobs-http-control-plane-using-arbiter to develop June 29, 2026 10:13

battermann added 28 commits June 29, 2026 11:13

bootstrap arbiter into wire-server

88473da

migrate the meetings clean up job to arbiter

f5d3bf7

brig internal: route plumbing the arbiter UI

ffda689

refactoring and implementing admin ui

c0a7767

resolve warning, formatting

6367153

inline arbiter ui/api

9fa39a8

removed unused code

846cd9b

job store

3fca127

schema

fad2961

job subsystem

a22982f

resolve warning

5a00b4c

revomve obsolete job subsystem operations

6189d46

formatting

3fa80c2

linting

dcada2a

single worker start effect, renaming

ea7eabc

clean up

23c6461

reminder and deletion jobs

cd0de87

migrate the meetings clean up to a real arbiter cron

9721fa3

send reminder events

0ca1706

fixed parser for tean feature

535d257

remove wire-server scheduled_job entry when job is finished

117753c

test that jobs get created correctly

a2d741f

test execution of job

b4b018d

make the arbiter config explicit

9c720c9

make arbiter worker parallelism setting explicit

9a95083

use correct feature store

762f898

logging

8d797ff

poll intervall configurable

cae2224

battermann added 5 commits June 29, 2026 12:13

comment and formatting

9f459b8

formatting

de07b58

update test for job excecution

9ada476

fix bw config map and add config options doc

6964759

changelog

4e704c1

battermann force-pushed the WPB-26489-backend-adminless-scheduled-jobs-model-and-teardown-support branch from 7737422 to 4e704c1 Compare June 29, 2026 12:28

battermann marked this pull request as ready for review June 29, 2026 12:30

battermann requested review from a team as code owners June 29, 2026 12:30

formatting

3d8477c

battermann requested a review from Copilot June 29, 2026 12:52

Copilot started reviewing on behalf of battermann June 29, 2026 12:52 View session

Copilot AI reviewed Jun 29, 2026

View reviewed changes

akshaymankar reviewed Jun 29, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WPB-26489 backend adminless scheduled jobs model and teardown support#5289

WPB-26489 backend adminless scheduled jobs model and teardown support#5289
battermann wants to merge 34 commits into
developfrom
WPB-26489-backend-adminless-scheduled-jobs-model-and-teardown-support

battermann commented Jun 25, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

akshaymankar left a comment

Uh oh!

akshaymankar Jun 29, 2026

Uh oh!

akshaymankar Jun 29, 2026

Uh oh!

akshaymankar Jun 29, 2026

Uh oh!

akshaymankar Jun 29, 2026

Uh oh!

akshaymankar Jun 29, 2026

Uh oh!

akshaymankar Jun 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		idTagName HistoryClient = "HistoryClient"
		idTagName ScheduledJob = "ScheduledJob "

		jobsApp :: App.Env -> ServerT JobsIRoutes.JobsAppAPI (Handler r)
		jobsApp env = Tagged env.jobsApiApp

Uh oh!

Conversation

battermann commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

akshaymankar left a comment

Choose a reason for hiding this comment

Uh oh!

akshaymankar Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

akshaymankar Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

akshaymankar Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

akshaymankar Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

akshaymankar Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

akshaymankar Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

battermann commented Jun 25, 2026 •

edited

Loading