Skip to content

Run the live/keyed test suite in CI on push to main #131

Description

@martinkersner

What

Add a CI lane that runs the live keyed test suite (DATAMAXI_API_KEY=... python -m pytest tests/) on push to main only.

Why

.github/workflows/python-package.yml currently runs on every push but executes only the offline mocked lane (python -m pytest tests/ -m "not integration", keyless). The live lanes — tests/test_integration.py (integration marker) and tests/test_call.py (smoke lane) — hit prod endpoints and are deliberately skipped in CI (smoke skips cleanly without a key; integration is deselected). So real API/SDK regressions against prod are never caught in CI.

Scope / approach

  • New job (or gated step) that sets DATAMAXI_API_KEY from a repo secret and runs python -m pytest tests/ (full suite, incl. integration + smoke).
  • Gate to main only, e.g. if: github.ref == 'refs/heads/main' (or a separate workflow with on: push: branches: [main]) — do not run on every branch push (avoids leaking the key to fork/branch runs and burning prod quota).
  • Requires adding a DATAMAXI_API_KEY GitHub Actions secret (repo settings → Secrets).

Caveat — expect intermittent red

PR #130 removed the flaky-prod xfail markers, so the live lane is no longer masked and can go red intermittently through no code fault:

  • Cold API pods → funding_rate.* and naver.* return 500 "no data found" (NATS-warmed caches empty on cold start).
  • premium() on an empty page raises ValueError('no data found') instead of returning empty (client-side SDK gap; limit=10 reduces but doesn't eliminate the empty-window risk).

Because of this, make the live lane non-blocking for now — a scheduled/nightly run, or continue-on-error: true, rather than a required check — until the empty-result handling is normalized in the SDK. Note the DATAMAXI_TIMEOUT env knob (conftest, default 30s) for slow prod endpoints.

Done when

  • A CI run on push to main executes the full keyed suite against prod using the DATAMAXI_API_KEY secret.
  • The lane is non-blocking (does not fail the required build on transient prod-data flakiness).
  • Branch/fork pushes do not run the keyed lane (secret not exposed).

Related: #130

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions