Stack/htcondor 6 contract suite

.github/workflows/integration.yml

@@ -10,6 +10,7 @@ on:
       - ".github/workflows/integration.yml"
       - "tests/conftest.py"
       - "pyproject.toml"
+      - "src/**"
 
 permissions:
   contents: read
@@ -66,10 +67,12 @@ jobs:
             # TODO(#13/#14): ship the suite in the image and exec as an unprivileged user.
             runner=(docker compose -f "$COMPOSE_FILE" exec -T backend pytest)
           else
-            runner=(pixi run -e py313 pytest)
+            # Independent tests, run side by side (stacks.yml `workers`).
+            runner=(pixi run -e py313 pytest -n "${{ matrix.workers }}")
           fi
           set +e
-          "${runner[@]}" tests/integration -m "${{ matrix.markers }}" \
+          # -rs --no-fold-skipped: name every skipped test and why.
+          "${runner[@]}" tests/integration -m "${{ matrix.markers }}" -rs --no-fold-skipped \
             --stack="${{ matrix.stack }}" --config="${{ matrix.config }}"
           rc=$?
           # Exit 5 = no tests selected; tolerated until the contract suite lands.

docs/dev/explanations/integration-testing.md

@@ -21,14 +21,17 @@ know which version caused the problem.
 
 Some backends do not support every feature. The tests check for a feature in the same way that
 users of the library do (`isinstance` against a `runtime_checkable` protocol), and skip the test
-when the backend does not have it. The helper for this will be added in phase 2, together with the
-first tests that use it.
+when the backend does not have it. A test asks for a capability by taking its fixture
+(`output_retriever`, `killable`, `purgeable`).
 
 ## Phases
 
-- **Phase 0 (now).** The interCEde protocols do not exist yet. Each stack is tested with the
-  backend's own command-line tools (`arcsub`, `arcstat`, `arcget`, `condor_submit`, `sbatch`).
-- **Phase 2.** One set of tests, written against the interCEde protocols, runs on every stack
-  (`test_submit.py`, `test_status.py`, `test_output.py`, `test_kill.py`). It replaces the phase 0
-  tests. Behaviour that only one backend has (for example HTCondor's one-time output download)
-  gets its own test and marker (`destructive_fetch`), not an `if` inside a shared test.
+- **Phase 0.** Before the interCEde protocols existed, a stack was tested with the backend's own
+  command-line tools (`arcsub`, `arcstat`, `arcget`, `condor_submit`, `sbatch`).
+- **Phase 2 (now).** One set of tests, written against the interCEde protocols, runs on every stack
+  (`test_submit.py`, `test_status.py`, `test_output.py`, `test_kill.py`, `test_purge.py`, and
+  `test_errors.py` for what a consumer sees when things go wrong). These tests carry the
+  `contract` marker, which selects them whatever a stack's `markers` say. A new backend writes no
+  tests of its own: its stack runs this suite. Behaviour that only one backend has (for example
+  HTCondor's one-time output download) gets its own test and marker (`destructive_fetch`), not an
+  `if` inside a shared test.

docs/dev/how-to/add-an-integration-stack.md

@@ -24,7 +24,7 @@ You do not need to change any CI workflow. CI finds the stack in `stacks.yml`.
 
    ```sh
    docker compose -f tests/integration/stacks/<id>/compose.yml up --wait
-   pixi run -e py313 pytest tests/integration -m "<markers>" --stack=<id> --config=basic
+   pixi run -e py313 pytest tests/integration -n 4 -m "<markers>" --stack=<id> --config=basic
    pixi run -e py313 pytest tests/integration/test_manifest.py
    ```
 
@@ -35,8 +35,9 @@ You do not need to change any CI workflow. CI finds the stack in `stacks.yml`.
 - [ ] The images use Docker, not Podman.
 - [ ] `intercede-client.toml` is filled in: `id`, endpoint, queue, credential type and path.
 - [ ] `pytest tests/integration/test_manifest.py` passes.
-- [ ] At least one test runs for this stack (phase 0: a test that uses the backend's own
-      command-line tools). Without one, the CI job passes but tests nothing.
+- [ ] The shared suite passes on this stack (`pytest tests/integration -m "<markers>" --stack=<id>`).
+      You do not write tests for a new backend: the stack runs the `contract` tests, skipping
+      those for capabilities the backend does not have.
 - [ ] If your pull request had its own workflow for this stack, remove it. `integration.yml` runs
       on every pull request that changes `tests/integration/`. It does not block merging yet
       (#14), so keep your own workflow only if you need a check that blocks merging.

docs/dev/how-to/run-integration-tests-locally.md

@@ -16,9 +16,12 @@ CI runs the same commands as below, so you can reproduce any CI job on your mach
 2. Run the tests. Copy `<markers>` from the stack's entry in `stacks.yml`:
 
    ```sh
-   pixi run -e py313 pytest tests/integration -m "<markers>" --stack=<id> --config=basic
+   pixi run -e py313 pytest tests/integration -n 4 -m "<markers>" --stack=<id> --config=basic
    ```
 
+`-n 4` runs four tests at the same time, as CI does. Use the stack's `workers` value from
+`stacks.yml` if it sets one, or leave `-n` out to run one test at a time while debugging.
+
 For stacks with `exec_in_container: true` (the `local-*` stacks), run the tests inside the
 container instead:
 

docs/dev/reference/integration-stacks.md

@@ -44,6 +44,7 @@ tests/integration/
 | `versions` | no | Backend versions to test, for example `[lts, latest]`. Default: `[latest]`. |
 | `markers` | yes | `pytest -m` expression that selects the tests for this stack. |
 | `exec_in_container` | no | `true`: run pytest inside the backend container. Default: `false`. |
+| `workers` | no | How many tests run at the same time (pytest-xdist). Default: `4`. Set `1` for a backend that cannot take several jobs at once. |
 
 ## When a stack gets CI jobs
 
@@ -112,9 +113,20 @@ Defined in `tests/conftest.py` (options) and `tests/integration/conftest.py` (fi
 | `stack` | fixture | The stack's entry from `stacks.yml`. |
 | `stack_dir` | fixture | Folder with the stack's `compose.yml`. |
 | `client_config` | fixture | The stack's `intercede-client.toml`, already parsed. |
+| `refusals` | fixture | What `[refusals]` in `intercede-client.toml` says the backend cannot honour: `containers`, `url_inputs`, `url_outputs`. The shared suite checks each is refused at submit. |
+| `credentials` | fixture | A credential provider holding what `[credentials]` points to (types: `token`). |
+| `backend` | fixture | The backend `[backend]` describes, built by the registry and open. |
+| `credential` | fixture | The credential `[credentials]` points to, read from the stack's credentials folder. |
+| `rejected_credentials` | fixture | A provider holding a credential of the right kind that the resource rejects. |
+| `unreachable_config` | fixture | `[backend]`, pointed at a closed port, as when the resource is down. |
+| `output_retriever`, `killable`, `purgeable` | fixture | The backend as that capability; skips the test when the backend does not have it. |
 
 If `intercede-client.toml` is missing, the test fails in CI (when `$CI` is set) and is skipped
 locally. In CI, a job where every test is skipped would look green but test nothing.
 
-Markers, defined in `pyproject.toml`: `remote`, `scheduler`, `arc`, `htcondor`, `slurm`, `ssh`,
-`destructive_fetch`.
+Markers, defined in `pyproject.toml`: `contract`, `remote`, `scheduler`, `arc`, `htcondor`,
+`slurm`, `ssh`, `destructive_fetch`.
+
+Tests marked `contract` are the shared suite and run on every stack: with `--stack`, they get the
+markers that the stack's `markers` expression asks for, so `-m "<markers>"` selects them. The
+other markers select tests for behaviour only some backends have.

docs/user/reference/api.md

@@ -19,6 +19,14 @@ Anything else, including every name starting with `_`, is internal.
 
 ::: intercede.registry
 
+## Credentials
+
+::: intercede.auth.models
+
+::: intercede.auth.protocols
+
+::: intercede.auth.providers
+
 ## Exceptions
 
 ::: intercede.exceptions

pyproject.toml

@@ -51,6 +51,7 @@ markers = [
   "backend_only: restrict a test to the named backend fixture(s)",
   "xfail_backend: strict xfail a test on the named backend fixture(s)",
   # integration suite (selected per stack in tests/integration/stacks.yml)
+  "contract: the shared integration suite, run on every stack whatever its markers",
   "remote: drives a remote CE backend (grid gateway: ARC, HTCondor-CE)",
   "scheduler: drives a Transport x Scheduler composition (BatchBackend)",
   "arc: requires an ARC CE",
@@ -144,6 +145,11 @@ py313 = { features = ["testing", "py313"] }
 py314 = { features = ["testing", "py314"] }
 docs = { features = ["docs"], solve-group = "default" }
 
+[tool.pixi.feature.testing.dependencies]
+# The HTCondor tools (condor_submit, condor_q, ...) the htcondor-ce backend runs;
+# the integration suite drives the backend against a real CE with them.
+htcondor-utils = "*"
+
 [tool.pixi.feature.testing.pypi-dependencies]
 intercede = { path = ".", editable = true }
 

src/intercede/_backends/_htcondor/__init__.py

@@ -0,0 +1,11 @@
+"""What interCEde knows about HTCondor, shared by every backend that drives it.
+
+Plain functions and data only, with no dependency on any backend class: the
+HTCondor-CE backend uses them directly, and the HTCondor scheduler of the
+composed backend (issue #21) will use the same ones over its transport. They
+share these functions, never a subtype.
+
+- ``submit``: a submit description built from a ``SubmissionSpec``
+- ``commands``: the argv of each HTCondor tool, and parsers for their output
+- ``status``: the state mapping from job ads to ``JobStatus``
+"""

src/intercede/_backends/_htcondor/commands.py

@@ -0,0 +1,117 @@
+"""The argv of each HTCondor tool, and parsers for what they print.
+
+``remote`` is how a tool reaches its schedd: ``schedd_options(pool, name)``
+for a remote schedd, as an HTCondor-CE is reached, or ``()`` for the schedd
+of the host the tool runs on, as the composed backend reaches it.
+"""
+
+from __future__ import annotations
+
+import re
+from collections.abc import Sequence
+
+from intercede.models import JobID
+
+# Ids per command line, so a status poll of thousands of jobs stays bounded.
+CHUNK = 100
+
+
+def schedd_options(pool: str, name: str) -> list[str]:
+    """Reach the schedd ``name`` through the collector ``pool`` (host:port)."""
+    return ["-pool", pool, "-name", name]
+
+
+def submit(
+    description: str, pool: str | None = None, name: str | None = None
+) -> list[str]:
+    """Submit a description file and spool its inputs to the schedd.
+
+    Spooling copies the inputs at submit time, so nothing on the submitting
+    host is needed afterwards. With ``pool`` and ``name`` the schedd is a
+    remote one (``-remote`` implies spooling); without, it is the local one.
+    """
+    if (pool is None) != (name is None):
+        raise ValueError("a remote schedd needs both pool and name")
+
+    command = ["condor_submit", "-terse"]
+
+    if pool is None or name is None:
+        command.append("-spool")
+    else:
+        command.extend(["-pool", pool, "-remote", name])
+
+    command.append(description)
+    return command
+
+
+def query(
+    ids: Sequence[JobID], attributes: Sequence[str], remote: Sequence[str]
+) -> list[str]:
+    """Read the ads of jobs still in the queue."""
+    return ["condor_q", *remote, *ids, "-attributes", ",".join(attributes), "-json"]
+
+
+def history(
+    ids: Sequence[JobID], attributes: Sequence[str], remote: Sequence[str]
+) -> list[str]:
+    """Read the ads of jobs that have left the queue."""
+    return [
+        "condor_history",
+        *remote,
+        *ids,
+        "-attributes",
+        ",".join(attributes),
+        "-json",
+    ]
+
+
+def remove(ids: Sequence[JobID], remote: Sequence[str]) -> list[str]:
+    """Remove jobs: cancels active ones, releases finished ones."""
+    return ["condor_rm", *remote, *ids]
+
+
+def transfer_data(ids: Sequence[JobID], remote: Sequence[str]) -> list[str]:
+    """Fetch the spooled output of jobs into the directory they were submitted from."""
+    return ["condor_transfer_data", *remote, *ids]
+
+
+def chunks(ids: Sequence[JobID]) -> list[Sequence[JobID]]:
+    """Split ids into groups of at most ``CHUNK``."""
+    return [ids[start : start + CHUNK] for start in range(0, len(ids), CHUNK)]
+
+
+_TERSE = re.compile(
+    r"\s*(?P<cluster>\d+)\.(?P<first>\d+)"
+    r"\s*-\s*"
+    r"(?P<last_cluster>\d+)\.(?P<last>\d+)\s*"
+)
+
+
+def parse_submit(raw: str) -> list[JobID]:
+    """Parse job IDs from condor_submit -terse output, e.g. '12.0 - 12.3'."""
+    lines = [line for line in raw.splitlines() if line.strip()]
+
+    if len(lines) != 1:
+        raise ValueError(f"unexpected condor_submit output: {raw.strip()!r}")
+
+    match = _TERSE.fullmatch(lines[0])
+    if match is None:
+        raise ValueError(f"unexpected condor_submit output: {raw.strip()!r}")
+
+    cluster = int(match["cluster"])
+    first = int(match["first"])
+    last_cluster = int(match["last_cluster"])
+    last = int(match["last"])
+
+    if cluster != last_cluster or last < first:
+        raise ValueError(f"unexpected condor_submit output: {raw.strip()!r}")
+
+    return [JobID(f"{cluster}.{proc}") for proc in range(first, last + 1)]
+
+
+_REMOVED = re.compile(r"^Job (\d+\.\d+) marked for removal", re.MULTILINE)
+
+
+def parse_remove(raw: str) -> set[JobID]:
+    """Return the ids ``condor_rm`` reports as removed; any other id was not."""
+    return {JobID(job) for job in _REMOVED.findall(raw)}