Enable Sentry tracing and profiling and fix some serious performance issues! (#403)

* Enable Sentry tracing and logging

* disable Honeycomb beeline and configure sampling for nonessential Sentry traces

* speed bost

* another speed boost to auth endpoint, and cleanup

* update documentation

Author: Irving Popovetsky

.dev/dev-jwt-key

@@ -11,4 +11,23 @@ dist
 .coverage
 html/*
 **/__pycache__
-**/*.pyc
+**/*.pyc
+
+# Development files - should not be in production
+.dev/
+src/.dev/
+src/.dev
+**/.dev/
+**/.dev
+*.sqlite3
+*.db
+db.sqlite3
+src/db.sqlite3
+**/db.sqlite3
+
+# Test artifacts
+.pytest_cache/
+src/.pytest_cache/
+**/.pytest_cache/
+.coverage
+htmlcov/

.dev/dev-jwt-key.pub

@@ -2,6 +2,7 @@
 *.env
 !example.env
 local.py
+.dev/
 env/
 venv/
 ENV/
@@ -81,4 +82,8 @@ pip-delete-this-directory.txt
 
 ## Database backups and tools with credentials
 backups/
-scripts/db-tools.sh
+scripts/db-tools.sh
+
+# Claude
+settings.local.json
+.mcp.json

.dockerignore

@@ -29,14 +29,14 @@ ENV VIRTUAL_ENV=/venv \
     PATH="/venv/bin:$PATH"
 
 # Install production dependencies only
-RUN poetry install --only=main --no-interaction --no-cache --compile
+RUN poetry install --only=main --no-interaction --no-cache
 
 # =============================================================================
 # Test builder: add dev dependencies
 # =============================================================================
 FROM builder AS test-builder
 
-RUN poetry install --no-interaction --no-cache --compile
+RUN poetry install --no-interaction --no-cache
 
 # =============================================================================
 # Runtime base: minimal image shared by test and production
@@ -50,7 +50,6 @@ RUN apk upgrade --no-cache && \
     pip install --no-cache-dir --upgrade pip
 
 ENV PYTHONUNBUFFERED=1 \
-    PYTHONDONTWRITEBYTECODE=1 \
     PATH="/venv/bin:$PATH"
 
 WORKDIR /app
@@ -79,7 +78,6 @@ FROM runtime-base AS production
 
 COPY --from=builder /venv /venv
 COPY src ./src
-COPY .dev ./src/.dev
 
 # Pre-compile Python bytecode for faster cold starts
 RUN python -m compileall -q ./src/

.gitignore

@@ -1,20 +1,42 @@
 # Deploying a new backend version
 
-Once a release is built and deployed by CircleCI, deploy it to an environment using ArgoCD.
+The backend is deployed to AWS ECS (Elastic Container Service) with separate staging and production environments.
 
-1. First, to connect to ArgoCD:
-```
-kubectl -n argocd port-forward service/argocd-server 8443:443 &
-open https://localhost:8443
+## Building and Pushing Docker Images
+
+Use the `docker-build.sh` script to build multi-architecture images and push to AWS ECR:
+
+```bash
+# Build and push staging images
+./docker-build.sh staging
+
+# Build and push production images
+./docker-build.sh prod
 ```
-2. login - credentials are in 1password, or ask someone for help
-3. pick up the new version in staging.
-  - go to https://localhost:8443/applications/pyback-staging,
-  - click the hamburger menu (3 dots, blue button), -> Details -> Parameters
-  - update the images field with the build ID as the tag, like: `operationcode/back-end:staging-846`
-  - as the new pods deploy, tail their logs to check for errors
-  - validate the staging environment (notes below)
-4. repeat those steps for the production environment
+
+This creates:
+- `back-end:staging-amd64` and `back-end:staging-arm64` images
+- A multi-arch manifest at `back-end:staging`
+
+## Deploying to ECS
+
+After images are pushed to ECR, deploy by updating the ECS service:
+
+1. **Update task definition** with new image tag
+2. **Deploy to staging first** - Update ECS service to use new task definition
+3. **Monitor logs** in CloudWatch or Sentry
+4. **Validate staging** (see below)
+5. **Deploy to production** - Repeat for production ECS service
+
+## Important: JWT Secret Key Migration
+
+**Before deploying these performance changes**, you must update the production `JWT_SECRET_KEY` environment variable:
+
+1. Generate a new secret: `openssl rand -base64 64 | tr -d '\n'`
+2. Set `JWT_SECRET_KEY` env var in ECS task definition to the generated string
+3. Remove `JWT_PUBLIC_KEY` env var (no longer needed with HS256)
+
+⚠️ **This will log out all users** (one-time migration from RS256 to HS256)
 
 # Validating the staging environment
 
@@ -25,33 +47,30 @@ When you run the front-end repo in localdev mode, it automatically connects to t
 2. run the dev server:  `docker run -it -v ${PWD}:/src -w /src -p 127.0.0.1:3000:3000/tcp node:lts yarn dev --hostname 0.0.0.0`
 3. Connect to the dev server: `open http://localhost:3000`
 
-# Certificate management with certbot
+# Monitoring
 
-Certbot runs continously as a kube operator and refreshes certs for you.  To ensure it is working,
-check the logs of the `cert-manager` pod, like:
-```
-kubectl -n cert-manager logs -f cert-manager-dcc48bf99-skhn7
-```
+## Sentry Performance Monitoring
 
-Current version running is v0.10.1
+The application is instrumented with Sentry for error tracking and performance monitoring:
+- Error tracking with breadcrumbs and context
+- Transaction tracing for HTTP requests
+- Database query performance tracking
+- Python profiling for CPU-intensive operations
 
-if you need for some reason to upgrade:
-1. read the release notes for all versions between current and desired, watch for breaking changes
-2. ignore the instructions about helm and kubectly apply, one minor version at a time
-```
-kubectl apply \
-  --validate=false \
-  -f https://github.com/jetstack/cert-manager/releases/download/v0.10.1/cert-manager.yaml
-```
+Configure via environment variables (see `example.env`):
+- `SENTRY_DSN` - Sentry project DSN
+- `SENTRY_TRACES_SAMPLE_RATE` - Percentage of requests to trace (0.0-1.0)
+- `SENTRY_PROFILES_SAMPLE_RATE` - Percentage of transactions to profile (0.0-1.0)
 
-certificates installed:
-```
-$ kubectl get Certificates --all-namespaces
-NAMESPACE               NAME                READY   SECRET              AGE
-monitoring              grafana-tls         True    grafana-tls         299d
-operationcode-staging   back-end-tls        True    back-end-tls        264d
-operationcode-staging   resources-api-tls   True    resources-api-tls   299d
-operationcode           back-end-tls        True    back-end-tls        264d
-operationcode           resources-api-tls   True    resources-api-tls   299d
+## CloudWatch Logs
+
+Application logs are sent to CloudWatch Logs. Access via AWS Console or CLI:
+
+```bash
+# View recent logs for staging
+aws logs tail /ecs/back-end-staging --follow
+
+# View recent logs for production
+aws logs tail /ecs/back-end-production --follow
 ```
 

Dockerfile

@@ -25,13 +25,13 @@ For information about the maintainers of the project, check out [MAINTAINERS.md]
 
 ## Quick Start
 Recommended versions of tools used within the repo:
-- `python@3.7` or greater (in some environments, you may need to specify version of python i.e. `python test.py` vs `python3 test.py`))
+- `python@3.12` or greater
 - `git@2.17.1` or greater
-- `poetry@0.12.11` or greater
-    - [Poetry](https://poetry.eustace.io/) is a packaging and dependency manager, similar to pip or pipenv
-    - Poetry provides a custom installer that can be ran via `curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python`
-    - Alternatively, poetry can be installed via pip/pip3 with `pip install --user poetry` or `pip3 install --user poetry`
-    - See https://poetry.eustace.io/docs/
+- `poetry@2.3.0` or greater
+    - [Poetry](https://python-poetry.org/) is a packaging and dependency manager
+    - Install via: `curl -sSL https://install.python-poetry.org | python3 -`
+    - Or via pip: `pip install --user poetry`
+    - See https://python-poetry.org/docs/
 
 
 ```bash
@@ -57,7 +57,7 @@ poetry run pytest
 poetry run black .
 # the next line shouldn't output anything to the terminal if it passes
 poetry run flake8
-poetry run isort -rc .
+poetry run isort .
 ```
 
 ## Running [Bandit](https://github.com/PyCQA/bandit)