Skip to content

Commit a70b2cc

Browse files
wu-shengwu-sheng
authored
feat: support Python 3.10-3.14, drop 3.8/3.9, update plugin compatibility (#386)
## Summary Support Python 3.10 through 3.14, drop end-of-life Python 3.8 and 3.9. Update plugin `support_matrix` and test services for newer library versions on 3.12+. Supersedes #374. ### Core agent fixes - Fix missing `import importlib.util` that caused `NameError` on Python 3.12+ (root cause of #374 failures) - Replace deprecated `find_module`/`load_module` with `find_spec`/`module_from_spec` in plugin loader and doc generator - Remove Python <3.8 `pkg_resources` fallback (dead code) ### Dependency updates - Python range: `>=3.10, <3.15` - Unpin `psutil`, `packaging`; loosen `wrapt>=1.14`, `uvloop>=0.17` - Remove `uwsgi` from dev deps (doesn't build on 3.12+) - Mark plugins/lint groups as optional; use `>=` ranges so `poetry lock` resolves on all Python versions - Upgrade pylint pin from `2.13.9` to `>=2.13.9` - Remove Poetry 1.5.1 pin on Linux (can't read Poetry 2.x lock files) ### Plugin support_matrix updates for 3.12+ All hook points verified against new library versions: | Plugin | >=3.10 | >=3.12 | >=3.13 | >=3.14 | |--------|--------|--------|--------|--------| | bottle | 0.12.23 | 0.12.23 | 0.13 | 0.13 | | django | 3.2 | 3.2 | 5.1 | 5.1 | | flask | 2.0 | 2.0 | 2.0 | 3.0 | | tornado | 6.0, 6.1 | 6.0, 6.1 | 6.0, 6.1 | 6.4 | | pyramid | 1.10, 2.0 | 2.1 | 2.1 | 2.1 | | kafka-python | 2.0 | 2.3 | 2.3 | 2.3 | | pulsar-client | 3.3.0 | 3.9.0 | 3.9.0 | 3.9.0 | | psycopg | 3.0/3.1 | 3.1 | 3.2 | 3.2 | | httpx | 0.22/0.23 | 0.22/0.23 | 0.23 | 0.28/0.23 | | happybase | 1.2.0 | 1.3.0 | 1.3.0 | 1.3.0 | | urllib3 | 1.25/1.26 | skipped | skipped | skipped | - **urllib3**: skipped on 3.12+ — `urllib3.request.RequestMethods` was removed in urllib3 2.x, plugin needs code adaptation - **happybase**: upgraded to 1.3.0 for 3.12+ (thriftpy2 0.6.0 has cp312/cp313 wheels) ### Test service fixes - Django test services: replace removed `django.conf.urls.url` with `django.urls.path` (compatible with Django 2.0+ through 5.1) - Fix `testcontainers` DockerCompose API for v4 compatibility (`filepath` -> `context`) ### CI updates - Test matrix: Python 3.10/3.11/3.12/3.13/3.14 (dropped 3.8/3.9) - Fix test step: `poetry install --without plugins,lint` (avoids installing incompatible old plugin deps on host) - Replace unapproved `getsentry/paths-filter` with approved `dorny/paths-filter` - Exclude `profiling_greenlet` E2E on 3.14 (gevent doesn't have 3.14 wheels yet) ### Docker & docs - Update docker/Makefile for 3.10-3.14 - Update Container.md with version range and known limitations - Add CLAUDE.md and Claude Code skills for plugin development ### Locally verified (with span data validation) - django==5.1 on Python 3.13 — PASSED - psycopg[binary]==3.2.* on Python 3.13 — PASSED - happybase==1.3.0 on Python 3.13 — PASSED - httpx==0.23.* on Python 3.13 — PASSED - requests==2.26/2.25 on Python 3.13 — PASSED - Agent loads all 35 plugins on Python 3.10/3.11/3.12/3.13/3.14
1 parent b91ebc4 commit a70b2cc

32 files changed

+4388
-2572
lines changed

‎.claude/skills/new-plugin/SKILL.md‎

Lines changed: 408 additions & 0 deletions
Large diffs are not rendered by default.

‎.claude/skills/plugin-test/SKILL.md‎

Lines changed: 240 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,240 @@
1+
---
2+
name: plugin-test
3+
description: Build Docker test images and run SkyWalking Python plugin/unit/e2e tests locally, mirroring the CI pipeline
4+
user-invocable: true
5+
---
6+
7+
# Run Plugin Tests Locally
8+
9+
This skill runs the same tests that CI runs, but locally. Ask the user what they want to test, or infer from recent changes.
10+
11+
## Options
12+
13+
The user can specify:
14+
- **Scope**: `all`, `unit`, a plugin name (e.g., `flask`, `redis`), a category (`data`, `http`, `web`), or `e2e`
15+
- **Python version**: e.g., `3.11` (default: `3.11-slim` for Docker image)
16+
- **Rebuild image**: whether to rebuild the Docker image (default: only if not already built)
17+
18+
If the user just says `/plugin-test` with no args, ask what to test.
19+
20+
## Prerequisites Check
21+
22+
Before running tests, verify:
23+
```bash
24+
# Check Docker is running
25+
docker info > /dev/null 2>&1 || echo "ERROR: Docker is not running"
26+
27+
# Check Poetry is available
28+
poetry --version || echo "ERROR: Poetry not found, run 'make poetry'"
29+
30+
# Check if environment is set up
31+
poetry env info > /dev/null 2>&1 || echo "WARNING: Poetry env not set up, will run 'make env'"
32+
```
33+
34+
If prerequisites fail, help the user fix them before proceeding.
35+
36+
## Step 1: Setup Environment (if needed)
37+
38+
```bash
39+
make env
40+
```
41+
42+
This installs Poetry and all dependencies including dev and plugin groups.
43+
44+
## Step 2: Build Plugin Test Docker Image
45+
46+
The plugin test image must be built before running any plugin integration tests (not needed for unit tests).
47+
48+
```bash
49+
# Default Python version
50+
docker build --build-arg BASE_PYTHON_IMAGE=3.11-slim \
51+
-t apache/skywalking-python-agent:latest-plugin --no-cache \
52+
. -f tests/plugin/Dockerfile.plugin
53+
```
54+
55+
To test with a specific Python version:
56+
```bash
57+
docker build --build-arg BASE_PYTHON_IMAGE=3.12-slim \
58+
-t apache/skywalking-python-agent:latest-plugin --no-cache \
59+
. -f tests/plugin/Dockerfile.plugin
60+
```
61+
62+
**Important**: The image tag is always `apache/skywalking-python-agent:latest-plugin` — this is what docker-compose files reference. Rebuilding with a different Python version replaces it.
63+
64+
### Testing Multiple Python Versions
65+
66+
The Python version **inside the Docker container** (not the host) determines what Python the plugin runs under. To test multiple versions, rebuild the image for each:
67+
68+
```bash
69+
# Test with Python 3.11
70+
docker build --build-arg BASE_PYTHON_IMAGE=3.11-slim \
71+
-t apache/skywalking-python-agent:latest-plugin --no-cache . \
72+
-f tests/plugin/Dockerfile.plugin
73+
poetry run pytest -v tests/plugin/web/sw_flask/
74+
75+
# Then test with Python 3.13
76+
docker build --build-arg BASE_PYTHON_IMAGE=3.13-slim \
77+
-t apache/skywalking-python-agent:latest-plugin --no-cache . \
78+
-f tests/plugin/Dockerfile.plugin
79+
poetry run pytest -v tests/plugin/web/sw_flask/
80+
```
81+
82+
CI does this as a matrix (builds images for each Python version in parallel, then runs all tests against each). Locally, you rebuild and re-run sequentially.
83+
84+
**Available Python base images**: `3.9-slim`, `3.10-slim`, `3.11-slim`, `3.12-slim`, `3.13-slim`, `3.14-slim`
85+
86+
Check if image already exists:
87+
```bash
88+
docker images apache/skywalking-python-agent:latest-plugin --format "{{.ID}} {{.CreatedAt}}"
89+
```
90+
91+
If it exists and is recent, ask the user if they want to rebuild or which Python version to use (rebuilding takes ~1-2 minutes).
92+
93+
## Step 3: Run Tests
94+
95+
### Unit Tests (no Docker needed)
96+
```bash
97+
poetry run pytest -v tests/unit/
98+
```
99+
100+
### Single Plugin Test
101+
Map the plugin name to its test path:
102+
- Web frameworks: `tests/plugin/web/sw_<name>/`
103+
- HTTP clients: `tests/plugin/http/sw_<name>/`
104+
- Data stores: `tests/plugin/data/sw_<name>/`
105+
106+
```bash
107+
poetry run pytest -v tests/plugin/web/sw_flask/
108+
poetry run pytest -v tests/plugin/data/sw_redis/
109+
poetry run pytest -v tests/plugin/http/sw_requests/
110+
```
111+
112+
### Plugin Category
113+
```bash
114+
poetry run pytest -v tests/plugin/data/ # All data plugins
115+
poetry run pytest -v tests/plugin/http/ # All HTTP plugins
116+
poetry run pytest -v tests/plugin/web/ # All web plugins
117+
```
118+
119+
### All Tests
120+
```bash
121+
poetry run pytest -v tests/unit/ tests/plugin/data/ tests/plugin/http/ tests/plugin/web/
122+
```
123+
124+
### E2E Tests
125+
E2E tests need a separate Docker image and use SkyWalking infra-e2e:
126+
```bash
127+
# Build E2E image
128+
docker build --build-arg BASE_PYTHON_IMAGE=3.11-slim \
129+
-t apache/skywalking-python-agent:latest-e2e --no-cache \
130+
. -f tests/e2e/base/Dockerfile.e2e
131+
132+
# E2E tests use infra-e2e framework, not pytest directly
133+
# They are defined in tests/e2e/case/*/e2e.yaml
134+
```
135+
136+
Note: E2E tests require the `e2e` CLI tool from SkyWalking infra-e2e. They are typically only run in CI. Inform the user if they ask for E2E.
137+
138+
## Step 4: Interpret Results
139+
140+
### Success
141+
All tests pass with green output. Report the summary.
142+
143+
### Failure - Docker Compose Timeout
144+
```
145+
Wait time exceeded 150 secs
146+
```
147+
This means services didn't start. Check:
148+
1. Is Docker running with enough resources?
149+
2. Is port 9090/9091/12800/19876 already in use? (`lsof -i :9090`)
150+
3. Check Docker logs: `docker compose -f tests/plugin/<category>/sw_<name>/docker-compose.yml logs`
151+
152+
### Failure - Validation Mismatch
153+
The test prints a diff between expected and actual span data. Common causes:
154+
- Wrong `componentId` in expected.data.yml
155+
- Missing or extra spans
156+
- Tag values don't match
157+
- Operation name mismatch
158+
159+
### Failure - Import/Install Error
160+
Plugin library failed to install in Docker. Check:
161+
- Library version compatibility with Python version
162+
- requirements.txt content in the test directory
163+
- Docker compose command for the service
164+
165+
### Failure - Permanent 502 / Empty Segments
166+
**IMPORTANT: Check your HTTP proxy first!** If `http_proxy` or `https_proxy` environment variables are set, ALL test HTTP requests (prepare, validate) go through the proxy instead of reaching Docker containers directly. This causes 502 responses and empty segment data.
167+
168+
```bash
169+
# Check for proxy
170+
echo $http_proxy $https_proxy
171+
172+
# Run tests without proxy
173+
export http_proxy="" https_proxy="" no_proxy="*" NO_PROXY="*"
174+
poetry run pytest -v tests/plugin/web/sw_flask/
175+
```
176+
177+
If still failing after clearing proxy:
178+
- Check container logs: `docker logs sw_<name>-consumer-1`
179+
- Look for whether the service actually started
180+
- Debug in foreground: `docker compose -f <path>/docker-compose.yml up`
181+
- Check ports: `lsof -i :9090 -i :9091 -i :12800`
182+
183+
## Plugin Name to Test Path Mapping
184+
185+
| Plugin | Test Path |
186+
|--------|-----------|
187+
| flask | tests/plugin/web/sw_flask/ |
188+
| django | tests/plugin/web/sw_django/ |
189+
| fastapi | tests/plugin/web/sw_fastapi/ |
190+
| sanic | tests/plugin/web/sw_sanic/ |
191+
| tornado | tests/plugin/web/sw_tornado/ |
192+
| bottle | tests/plugin/web/sw_bottle/ |
193+
| pyramid | tests/plugin/web/sw_pyramid/ |
194+
| falcon | tests/plugin/web/sw_falcon/ |
195+
| grpc | tests/plugin/web/sw_grpc/ |
196+
| requests | tests/plugin/http/sw_requests/ |
197+
| urllib3 | tests/plugin/http/sw_urllib3/ |
198+
| aiohttp | tests/plugin/http/sw_aiohttp/ |
199+
| httpx | tests/plugin/http/sw_httpx/ |
200+
| http | tests/plugin/http/sw_http/ |
201+
| http_wsgi | tests/plugin/http/sw_http_wsgi/ |
202+
| websockets | tests/plugin/http/sw_websockets/ |
203+
| redis | tests/plugin/data/sw_redis/ |
204+
| pymongo | tests/plugin/data/sw_pymongo/ |
205+
| pymysql | tests/plugin/data/sw_pymysql/ |
206+
| mysqlclient | tests/plugin/data/sw_mysqlclient/ |
207+
| psycopg | tests/plugin/data/sw_psycopg/ |
208+
| psycopg2 | tests/plugin/data/sw_psycopg2/ |
209+
| elasticsearch | tests/plugin/data/sw_elasticsearch/ |
210+
| kafka | tests/plugin/data/sw_kafka/ |
211+
| rabbitmq | tests/plugin/data/sw_rabbitmq/ |
212+
| pulsar | tests/plugin/data/sw_pulsar/ |
213+
| neo4j | tests/plugin/data/sw_neo4j/ |
214+
| happybase | tests/plugin/data/sw_happybase/ |
215+
| loguru | tests/plugin/data/sw_loguru/ |
216+
217+
## Useful Debug Commands
218+
219+
```bash
220+
# Check what containers are running during a test
221+
docker ps
222+
223+
# Check container logs for a stuck test
224+
docker compose -f tests/plugin/<category>/sw_<name>/docker-compose.yml logs
225+
226+
# Clean up orphaned containers from failed tests
227+
docker compose -f tests/plugin/<category>/sw_<name>/docker-compose.yml down --remove-orphans
228+
229+
# Check what the mock collector received (while containers are running)
230+
curl http://localhost:12800/receiveData
231+
232+
# Validate expected data manually
233+
curl -X POST http://localhost:12800/dataValidate -d @tests/plugin/<category>/sw_<name>/expected.data.yml
234+
235+
# Check if ports are in use
236+
lsof -i :9090 -i :9091 -i :12800 -i :19876
237+
238+
# Run with verbose output
239+
poetry run pytest -v -s tests/plugin/web/sw_flask/
240+
```

‎.github/workflows/CI.yaml‎

Lines changed: 18 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -45,10 +45,12 @@ jobs:
4545
- name: Set up Python
4646
uses: actions/setup-python@v5
4747
with:
48-
python-version: 3.8
48+
python-version: "3.11"
4949
- name: Lint codes
5050
run: |
51-
make env
51+
make poetry
52+
poetry install --all-extras --with lint
53+
make gen
5254
make lint
5355
5456
# Check if the plugin doc is generated correctly
@@ -65,7 +67,7 @@ jobs:
6567
- name: Set up Python
6668
uses: actions/setup-python@v5
6769
with:
68-
python-version: 3.8
70+
python-version: "3.14"
6971
- name: Check plugin doc
7072
run: |
7173
make env
@@ -86,10 +88,9 @@ jobs:
8688
with:
8789
persist-credentials: false
8890
- name: Check for file changes
89-
uses: getsentry/paths-filter@v2.11.1
91+
uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
9092
id: filter
9193
with:
92-
token: ${{ github.token }}
9394
# The following filters indicate a category along with
9495
# the files that should not be ignored by CI when modified.
9596
filters: |
@@ -138,7 +139,7 @@ jobs:
138139
runs-on: ubuntu-latest
139140
strategy:
140141
matrix: # may support pypy in the future
141-
python-version: [ "3.8-slim", "3.9-slim", "3.10-slim", "3.11-slim" ]
142+
python-version: [ "3.10-slim", "3.11-slim", "3.12-slim", "3.13-slim", "3.14-slim" ]
142143
fail-fast: false
143144
env:
144145
BASE_PYTHON_IMAGE: ${{ matrix.python-version }}
@@ -171,7 +172,7 @@ jobs:
171172
timeout-minutes: 20
172173
strategy:
173174
matrix:
174-
python-version: [ "3.8", "3.9", "3.10", "3.11" ]
175+
python-version: [ "3.10", "3.11", "3.12", "3.13", "3.14" ]
175176
test-path: ${{fromJson(needs.prep-plugin-and-unit-tests.outputs.matrix)}}
176177
fail-fast: false
177178
env:
@@ -205,7 +206,9 @@ jobs:
205206
python-version: ${{ matrix.python-version }}
206207
- name: Run unit tests
207208
run: |
208-
make env
209+
make poetry
210+
poetry install --all-extras --without plugins,lint
211+
make gen
209212
poetry run pytest -v ${{ matrix.test-path }}
210213
211214
docker-e2e:
@@ -219,7 +222,7 @@ jobs:
219222
timeout-minutes: 10
220223
strategy:
221224
matrix:
222-
python-image-variant: [ "3.8-slim", "3.9-slim", "3.10-slim", "3.11-slim" ]
225+
python-image-variant: [ "3.10-slim", "3.11-slim", "3.12-slim", "3.13-slim", "3.14-slim" ]
223226
fail-fast: false
224227
env:
225228
BASE_PYTHON_IMAGE: ${{ matrix.python-image-variant }}
@@ -251,7 +254,7 @@ jobs:
251254
timeout-minutes: 20
252255
strategy:
253256
matrix:
254-
python-image-variant: [ "3.8-slim", "3.9-slim", "3.10-slim", "3.11-slim" ]
257+
python-image-variant: [ "3.10-slim", "3.11-slim", "3.12-slim", "3.13-slim", "3.14-slim" ]
255258
case:
256259
- name: gRPC-single-process
257260
path: tests/e2e/case/grpc/single/e2e.yaml
@@ -275,6 +278,11 @@ jobs:
275278
path: tests/e2e/case/profiling/threading/e2e.yaml
276279
- name: profiling_greenlet
277280
path: tests/e2e/case/profiling/greenlet/e2e.yaml
281+
exclude:
282+
# gevent/greenlet does not support Python 3.14 yet
283+
- python-image-variant: "3.14-slim"
284+
case:
285+
name: profiling_greenlet
278286
fail-fast: false
279287
steps:
280288
- name: Checkout source codes

0 commit comments

Comments
 (0)