|
| 1 | +# 07 - Monitoring & API Integration (Read-First, Safe-by-Default) |
| 2 | +Home: [README](./README.md) |
| 3 | + |
| 4 | +## Who this is for |
| 5 | +- Learners integrating external monitoring or telemetry APIs into Python data workflows. |
| 6 | +- Teams needing reliable API-based reporting and downstream dashboards. |
| 7 | + |
| 8 | +## What you will build |
| 9 | +- Read-only ingestion jobs for external monitoring APIs. |
| 10 | +- Shared database cache tables for dashboard-friendly access. |
| 11 | +- A documented field mapping worksheet and data contract. |
| 12 | + |
| 13 | +## Prerequisites |
| 14 | +- SQL ETL baseline from [06_SQL.md](./06_SQL.md). |
| 15 | +- API credentials (key or token) for your monitoring platform. |
| 16 | +- Database write access to cache schema. |
| 17 | + |
| 18 | +## Step-by-step lab pack |
| 19 | + |
| 20 | +### Step 1 - Read-only policy baseline |
| 21 | +Before any code: |
| 22 | +- confirm API scopes, |
| 23 | +- enforce read-only endpoints, |
| 24 | +- define timeout and retry standards. |
| 25 | + |
| 26 | +No write operations are allowed until validation checklist passes. |
| 27 | + |
| 28 | +### Step 2 - Monitoring API ingestion starter |
| 29 | +Use any monitoring or telemetry API you have access to. Good free options for learning: |
| 30 | +- [OpenWeatherMap API](https://openweathermap.org/api) (weather monitoring) |
| 31 | +- [GitHub API](https://docs.github.com/en/rest) (repository/event monitoring) |
| 32 | +- [UptimeRobot API](https://uptimerobot.com/api/) (uptime monitoring) |
| 33 | + |
| 34 | +Initial pulls: |
| 35 | +- active alerts or events, |
| 36 | +- resource/node health status, |
| 37 | +- key metrics or utilization data. |
| 38 | + |
| 39 | +### Step 3 - Secondary source ingestion |
| 40 | +Add a second data source to practice multi-source integration: |
| 41 | +- a different API endpoint from the same platform, |
| 42 | +- or a complementary monitoring service. |
| 43 | + |
| 44 | +Initial pulls: |
| 45 | +- monitored instances or resources, |
| 46 | +- health and performance metrics, |
| 47 | +- alert/event summary. |
| 48 | + |
| 49 | +### Step 4 - Shared database cache contract |
| 50 | +Create cache tables with source metadata: |
| 51 | +- `cache_monitoring_alerts` |
| 52 | +- `cache_monitoring_nodes` |
| 53 | +- `cache_perf_instances` |
| 54 | +- `cache_perf_metrics` |
| 55 | + |
| 56 | +Required columns: |
| 57 | +- `source_system` |
| 58 | +- `collected_at_utc` |
| 59 | +- `entity_key` |
| 60 | +- `payload_hash` |
| 61 | + |
| 62 | +### Step 5 - Field mapping worksheet |
| 63 | +Maintain this worksheet in your project docs: |
| 64 | + |
| 65 | +| Entity | Source system | Collection cadence | Destination table | Owner | |
| 66 | +|---|---|---|---|---| |
| 67 | +| Active alerts | Monitoring API | Every 15 min | cache_monitoring_alerts | Monitoring team | |
| 68 | +| Node status | Monitoring API | Every 15 min | cache_monitoring_nodes | Monitoring team | |
| 69 | +| Instance health | Performance API | Every 30 min | cache_perf_instances | DBA team | |
| 70 | +| Performance metrics | Performance API | Hourly | cache_perf_metrics | DBA team | |
| 71 | + |
| 72 | +### Step 6 - Daily ops report build |
| 73 | +Outputs: |
| 74 | +- `output/monitoring_daily.xlsx` |
| 75 | +- `output/monitoring_daily.html` |
| 76 | +- `logs/monitoring_run_YYYYMMDD.log` |
| 77 | + |
| 78 | +Include: |
| 79 | +- top critical alerts, |
| 80 | +- down resource summary, |
| 81 | +- instance health snapshot, |
| 82 | +- stale data warnings. |
| 83 | + |
| 84 | +## Expected output |
| 85 | +- Stable read-only ingestion from monitoring APIs. |
| 86 | +- Reliable cache tables for dashboards. |
| 87 | +- Traceable field mappings and data ownership. |
| 88 | + |
| 89 | +## Break/fix drills |
| 90 | +1. Force token/auth failure and verify sanitized error logging. |
| 91 | +2. Simulate API timeout and validate retry/backoff. |
| 92 | +3. Remove required field from payload transform and verify reject behavior. |
| 93 | + |
| 94 | +## Troubleshooting |
| 95 | +- auth errors: |
| 96 | + - validate credential scope, |
| 97 | + - test endpoint manually, |
| 98 | + - confirm clock/time sync for token validity. |
| 99 | +- schema drift: |
| 100 | + - compare payload keys against mapping worksheet, |
| 101 | + - route unknown fields to audit logs. |
| 102 | +- stale dashboards: |
| 103 | + - verify cache job schedule and `collected_at_utc` freshness checks. |
| 104 | + |
| 105 | +## Mastery check |
| 106 | +Advance when you can: |
| 107 | +- describe your API data contracts, |
| 108 | +- prove read-only ingestion reliability, |
| 109 | +- explain fallback when source APIs are unavailable. |
| 110 | + |
| 111 | +## Learning-style options (Play/Build/Dissect/Teach-back) |
| 112 | +- Play: vary polling intervals and compare freshness/cost tradeoffs. |
| 113 | +- Build: implement both ingestion jobs and cache writes. |
| 114 | +- Dissect: annotate one end-to-end payload transform. |
| 115 | +- Teach-back: present source-to-cache architecture to a peer. |
| 116 | + |
| 117 | +## Primary Sources |
| 118 | +- [OpenWeatherMap API](https://openweathermap.org/api) |
| 119 | +- [GitHub REST API](https://docs.github.com/en/rest) |
| 120 | +- [requests library](https://docs.python-requests.org/en/latest/) |
| 121 | +- [SQLAlchemy tutorial](https://docs.sqlalchemy.org/en/20/tutorial/index.html) |
| 122 | + |
| 123 | +## Optional Resources |
| 124 | +- [UptimeRobot API](https://uptimerobot.com/api/) |
| 125 | +- [httpx documentation](https://www.python-httpx.org/) |
| 126 | + |
| 127 | +## Sample database schemas |
| 128 | +- Use this schema pack for cache and downstream marts: |
| 129 | + - [13_SAMPLE_DATABASE_SCHEMAS.md](./13_SAMPLE_DATABASE_SCHEMAS.md) |
| 130 | + |
| 131 | +--- |
| 132 | + |
| 133 | +| [← Prev](projects/level-5/README.md) | [Home](README.md) | [Next →](concepts/http-explained.md) | |
| 134 | +|:---|:---:|---:| |
0 commit comments