refactor(persistence): audit pagination and add record iteration helpers

[drew]

Description

Audit every gateway API and internal persistence caller that lists records to ensure pagination is explicit, complete, and consistent. Add reusable helpers for iterating every raw record or decoded message so callers that require a full scan do not manually implement offset loops or accidentally process only the first page.

The audit should distinguish between:

• externally paginated list APIs, which must enforce bounded page sizes and expose consistent continuation semantics; and
• internal full-scan operations, which must iterate every page with checked offset advancement and well-defined decode/error handling.

Context

While reviewing the compute-driver lifecycle stack (#2743, #2744, and #2786), we found several direct calls such as list_by_type(..., 1000, 0). Some are intentionally bounded, while others are logically full-table operations and can silently omit records after the first page.

The repository already contains working but duplicated pagination loops, including sandbox scans in grpc/provider.rs, provider refresh-state scans in provider_refresh.rs, and provider-policy validation in grpc/policy.rs. Persistence exposes page-level methods such as list, list_by_type, and list_all_messages, but it does not expose a common abstraction for safely visiting every page.

This issue is intentionally deferred from the compute-driver lifecycle stack so pagination behavior can be reviewed and corrected consistently across the gateway rather than patched in one subsystem.

Definition of Done

• Inventory all gateway RPC list APIs and all internal callers of persistence list methods.
• Classify each caller as a bounded page request or an operation that must visit every matching record.
• Verify externally paginated APIs enforce page-size limits and use consistent offset or continuation semantics.
• Add reusable asynchronous helpers for iterating all matching raw ObjectRecord values and decoded protobuf messages.
• Support the common query scopes needed by existing callers, including all workspaces and a specific workspace; selectors and membership queries should use the same abstraction where applicable.
• Use checked conversions and checked offset advancement, and define consistent behavior for database and protobuf decode failures.
• Migrate internal full-scan callers away from one-page reads and duplicated pagination loops.
• Add regression tests with more records than one page, including coverage that proves every record is visited exactly once.
• Document which low-level persistence methods return one page and which helpers exhaust all pages.