Dev

.github/pull_request_template.md

@@ -0,0 +1,26 @@
+## Summary
+
+- 
+
+## Type
+
+- [ ] Data contract / API
+- [ ] Provider integration
+- [ ] Streaming / Redis
+- [ ] VN preload / storage
+- [ ] Diagnostics / monitoring
+- [ ] Docs / repo hygiene
+
+## Checks
+
+- [ ] I did not commit directly to `main`.
+- [ ] Docs are updated for public behavior changes.
+- [ ] Tests are added or updated.
+- [ ] Docker tests pass, or a blocker is documented.
+- [ ] No secrets, generated logs, parquet data, or local caches are included.
+
+## Test Evidence
+
+```text
+
+```

.github/workflows/ci.yml

@@ -0,0 +1,23 @@
+name: CI
+
+on:
+  pull_request:
+    branches: ["dev", "main"]
+  push:
+    branches: ["dev"]
+
+jobs:
+  unit-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build data_layer image
+        run: docker compose build data_layer
+
+      - name: Run unit tests
+        run: docker compose run --rm test_runner python -m unittest discover -s tests

.gitignore

@@ -1,4 +1,5 @@
 .env
+.mailmap
 .env.local
 .env.*.local
 __pycache__/

.pre-commit-config.yaml

@@ -0,0 +1,19 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.6.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-added-large-files
+        args: ["--maxkb=1024"]
+      - id: check-case-conflict
+      - id: check-merge-conflict
+      - id: check-yaml
+      - id: mixed-line-ending
+        args: ["--fix=lf"]
+
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.5.7
+    hooks:
+      - id: ruff
+        args: ["--fix", "--exit-non-zero-on-fix"]

CODE_OF_CONDUCT.md

@@ -0,0 +1,22 @@
+# Code of Conduct
+
+`data_layer` is maintained as a professional engineering project. Contributors are expected to keep discussions respectful, technical, and focused on improving reliability for downstream users.
+
+## Expected Behavior
+
+- Be clear, patient, and constructive.
+- Discuss code, design, and operational risk directly without personal attacks.
+- Assume good intent, but verify technical claims with tests, logs, docs, or reproducible examples.
+- Respect security boundaries and never request or expose credentials.
+- Keep issue and pull-request threads on topic.
+
+## Unacceptable Behavior
+
+- Harassment, threats, or discriminatory language.
+- Publishing private information, credentials, logs with secrets, or production account details.
+- Repeated off-topic comments that block maintainers or contributors from resolving work.
+- Intentionally submitting malicious code or misleading test results.
+
+## Enforcement
+
+Maintainers may edit or remove comments, close issues, block users, or reject contributions that violate this code. Security-sensitive incidents should be reported privately using the process in `SECURITY.md`.

CONTRIBUTING.md

@@ -0,0 +1,68 @@
+# Contributing
+
+Thanks for helping improve `data_layer`. This service is used as a market-data gateway by execution and alpha services, so changes should keep API contracts stable and observable.
+
+## Branch Flow
+
+- Do not commit directly to `main`.
+- Work on `dev` or a feature branch from `dev`.
+- Open pull requests into `dev` first.
+- Merge `dev` into `main` only through a release pull request after tests and smoke checks pass.
+- Keep commits focused and use messages that name the subsystem and behavior changed.
+
+Recommended flow:
+
+```bash
+git checkout dev
+git pull origin dev
+git checkout -b feature/binance-derivatives-contract
+```
+
+## Local Checks
+
+This server is Docker-first. Prefer container tests over installing Python packages on the host:
+
+```bash
+docker compose run --rm test_runner python -m unittest discover -s tests
+```
+
+For changed Python files, a quick compile check is useful:
+
+```bash
+python3 -m py_compile app/path/to_file.py
+```
+
+## Pre-Commit
+
+Install hooks in a development environment:
+
+```bash
+pre-commit install
+pre-commit run --all-files
+```
+
+The hook set checks whitespace, YAML, merge conflicts, large files, and Ruff linting.
+
+## API Contract Rules
+
+- Keep existing response shapes stable unless a migration plan is documented.
+- Add new endpoints instead of mutating old endpoint contracts when downstream services already depend on them.
+- Put provider-specific raw fields under explicit payload sections and include metadata such as provider, market, params, cached, and stored.
+- Do not let alpha containers call external providers directly. Add or extend `data_layer` wrappers instead.
+- Do not store ephemeral crypto derivatives metrics unless a design note explicitly approves it.
+
+## Documentation
+
+Update these files when changing public behavior:
+
+- `README.md` for project-level capabilities and quickstart.
+- `DATA_LAYER_SERVICE_ACCESS_GUIDE.md` for downstream service contracts.
+- Tests under `tests/` for endpoint and SDK behavior.
+
+## Pull Request Checklist
+
+- [ ] I did not commit directly to `main`.
+- [ ] I updated docs for public API or operational changes.
+- [ ] I added/updated tests for the changed contract.
+- [ ] Docker unit tests pass or the reason is documented.
+- [ ] No secrets, credentials, generated logs, parquet data, or local caches are included.

DATA_LAYER_SERVICE_ACCESS_GUIDE.md

@@ -327,6 +327,61 @@ Important crypto parser note:
 - This was the root cause of the rsibound warmup issue where `RIFUSDT`, `COMPUSDT`, and similar symbols returned data from data_layer but alpha logged `rows=0`.
 - The batch endpoint is additive. It does not replace or mutate the existing single-symbol endpoint contract.
 
+### Pattern E: Binance Futures Derivatives Metrics For Basis-Arb
+
+Basis-arbitrage and futures microstructure alpha must request derivatives metrics through `data_layer`.
+Do not call Binance directly from alpha containers.
+
+These endpoints are direct, non-storage wrappers around official Binance USDⓈ-M Futures market-data APIs:
+
+- `GET /v1/binance/futures/exchange-info?symbol=BTCUSDT_260925`
+- `GET /v1/binance/futures/klines/BTCUSDT_260925?interval=1d&limit=30`
+- `GET /v1/binance/futures/depth/BTCUSDT?limit=5`
+- `GET /v1/binance/futures/open-interest/BTCUSDT`
+- `GET /v1/binance/futures/open-interest-history/BTCUSDT?period=1d&limit=30`
+- `GET /v1/binance/futures/long-short/global_account/BTCUSDT?period=1d&limit=30`
+- `GET /v1/binance/futures/long-short/top_account/BTCUSDT?period=1d&limit=30`
+- `GET /v1/binance/futures/long-short/top_position/BTCUSDT?period=1d&limit=30`
+- `GET /v1/binance/futures/taker-long-short/BTCUSDT?period=1d&limit=30`
+- `GET /v1/binance/futures/funding-rate/BTCUSDT?limit=100`
+- `GET /v1/binance/futures/basis/BTCUSDT?contract_type=CURRENT_QUARTER&period=1d&limit=30`
+
+Always resolve delivery symbols from `exchange-info` first. Do not guess date suffixes:
+
+```bash
+GET http://data_layer:8100/v1/binance/futures/exchange-info
+```
+
+At the 2026-07-08 smoke test, Binance listed `BTCUSDT_260925` as `CURRENT_QUARTER` and
+`BTCUSDT_261225` as `NEXT_QUARTER`; `BTCUSDT_260926` was rejected by Binance as an invalid symbol.
+
+Preferred alpha warmup/rebalance request for basis-arb:
+
+```bash
+POST http://data_layer:8100/v1/binance/futures/basis-bundle
+Content-Type: application/json
+
+{
+  "perp_symbol": "BTCUSDT",
+  "delivery_symbol": "BTCUSDT_260925",
+  "pair": "BTCUSDT",
+  "interval": "1d",
+  "period": "1d",
+  "limit": 30,
+  "include_depth": true,
+  "depth_limit": 5,
+  "contract_type": "CURRENT_QUARTER"
+}
+```
+
+Bundle response contract:
+
+- `components` contains successful raw provider payloads keyed by component name.
+- `errors` contains per-component failures.
+- `partial=true` means at least one component failed; alpha must decide whether the missing component is fatal.
+- `cached=false` and `stored=false`; data_layer does not persist these derivatives metrics to parquet/Redis history.
+- Futures-data history endpoints are latest-30-days style datasets. Consumers must not assume older history is available from these wrappers.
+
 Fallback semantics:
 
 - `GET http://data_layer:8100/v1/fallback/crypto/status/BTCUSDT?interval=1m`

README.md

@@ -1,13 +1,61 @@
 # data_layer
 
-A centralized, high-performance market data gateway service that aggregates and distributes real-time financial market data from multiple sources — **Binance** (crypto) and **Vietnamese stock market** (via DNSE WebSocket + vnstock fallback) — to downstream trading services via **Redis Pub/Sub** and a **REST API**.
+`data_layer` is the market-data gateway for Bobby's trading stack. It centralizes provider connectivity, historical warmup, live streaming, fallback recovery, and service-to-service data contracts so alpha and execution services do not connect to exchanges directly.
+
+It currently serves:
+
+- Binance crypto spot and USD-M futures market data.
+- Binance derivatives metrics for basis-arbitrage research and execution.
+- VN stock and derivative data through DNSE/vnstock integrations.
+- Redis Pub/Sub streams for live consumers.
+- REST endpoints for warmup, recovery, diagnostics, and health checks.
+
+## Quick Links
+
+- [Integration guide](./DATA_LAYER_SERVICE_ACCESS_GUIDE.md)
+- [Contributing guide](./CONTRIBUTING.md)
+- [Security policy](./SECURITY.md)
+- [Code of conduct](./CODE_OF_CONDUCT.md)
+- [License](./LICENSE)
+
+## Repository Policy
+
+`main` is treated as the protected release branch. Do not commit directly to `main`.
+
+Use this flow:
+
+```bash
+git checkout dev
+git pull origin dev
+git checkout -b feature/my-change
+```
+
+Open pull requests into `dev`; merge `dev` into `main` only through a release pull request after tests and smoke checks pass.
+
+## What This Service Owns
+
+- Provider connections and retry/backoff policy.
+- Normalized service contracts for downstream systems.
+- Redis live-stream publication.
+- VN preload parquet storage and materialized views.
+- Latest-state recovery endpoints.
+- Diagnostics for provider health and data freshness.
+
+## What This Service Does Not Own
+
+- Trading decisions.
+- Portfolio, risk, order routing, or broker account state.
+- Alpha-specific signal logic.
+- Direct broker execution.
+- Long-term storage for ephemeral Binance derivatives metrics unless a design note explicitly approves it.
 
 ## Features
 
 - **Real-time streaming** — WebSocket multiplexer for Binance (spot & futures trade + kline) and DNSE (VN stock live quotes)
 - **Automatic failover** — DNSE as primary VN source, vnstock REST poller as secondary fallback
 - **Redis Pub/Sub distribution** — single upstream connection shared across many downstream consumers
 - **Historical warmup (VN)** — Parquet-backed preload service for 1-minute OHLCV candle warmup
+- **Binance derivatives REST wrappers** — OHLCV, funding, open interest, long/short ratios, taker ratio, depth, and basis bundle endpoints
 - **Preload watchdog** — auto-refreshes VN candle data during market hours, sleeps until next open otherwise
 - ⚡ **High-performance serialization** — `orjson` throughout for minimal latency
 - **Alpha strategy example** — moving-average crossover strategy included as a reference implementation
@@ -50,6 +98,21 @@ Downstream services (alpha strategies, paper trading engines, execution services
 | Runtime | Python 3.10+ |
 | Container | Docker + Docker Compose |
 
+## Development Checks
+
+This repo is Docker-first. Prefer container tests over installing Python packages directly on the server:
+
+```bash
+docker compose run --rm test_runner python -m unittest discover -s tests
+```
+
+Optional pre-commit hooks:
+
+```bash
+pre-commit install
+pre-commit run --all-files
+```
+
 ## Project Structure
 
 ```

SECURITY.md

@@ -0,0 +1,27 @@
+# Security Policy
+
+`data_layer` handles market-data connectivity and service-to-service data distribution. It should never contain trading credentials, broker secrets, personal API keys, or production account identifiers in source control.
+
+## Reporting a Vulnerability
+
+Please report security issues privately to the maintainers. Do not open a public issue with exploit details, credentials, or production logs.
+
+Include:
+
+- affected component or endpoint;
+- reproduction steps;
+- expected impact;
+- relevant logs with secrets removed;
+- suggested mitigation, if known.
+
+## Supported Security Practices
+
+- Keep `.env` local and out of git.
+- Use Docker networks for internal services.
+- Do not expose Redis or provider credentials publicly.
+- Prefer API wrappers in `data_layer` over direct provider calls from alpha containers.
+- Redact API keys, tokens, account IDs, and investor identifiers before sharing logs.
+
+## Dependency Updates
+
+Dependency and Docker image updates should be tested in `dev` before merge to `main`.