From 35f70a28bb6c63dd1f053b9398ce6e74bc433f92 Mon Sep 17 00:00:00 2001 From: Zoran Simic Date: Tue, 23 Jun 2026 07:43:16 -0700 Subject: [PATCH 1/5] Added docs in OKF format --- .claude/settings.json | 10 ++ CLAUDE.md | 8 +- docs/architecture/build-setup.md | 51 +++++++++ docs/architecture/config.md | 55 ++++++++++ docs/architecture/cpython.md | 44 ++++++++ docs/architecture/index.md | 31 ++++++ docs/architecture/module-builder.md | 59 ++++++++++ docs/architecture/ppg.md | 43 ++++++++ docs/architecture/python-builder.md | 38 +++++++ docs/architecture/python-inspector.md | 57 ++++++++++ docs/cli/build-report.md | 38 +++++++ docs/cli/build.md | 48 +++++++++ docs/cli/diagnostics.md | 24 +++++ docs/cli/index.md | 29 +++++ docs/cli/inspect.md | 48 +++++++++ docs/cli/lib-auto-correct.md | 41 +++++++ docs/cli/list.md | 35 ++++++ docs/cli/recompress.md | 35 ++++++ docs/concepts/build-layout.md | 54 ++++++++++ docs/concepts/folder-masking.md | 31 ++++++ docs/concepts/index.md | 9 ++ docs/concepts/portability.md | 39 +++++++ docs/concepts/static-linking.md | 36 +++++++ docs/concepts/telltale-detection.md | 54 ++++++++++ docs/configuration/index.md | 7 ++ docs/configuration/portable-python-yml.md | 89 ++++++++++++++++ docs/guides/add-an-external-module.md | 64 +++++++++++ docs/guides/build-a-portable-python.md | 65 ++++++++++++ docs/guides/index.md | 9 ++ docs/guides/local-development.md | 70 ++++++++++++ docs/index.md | 30 ++++++ docs/log.md | 12 +++ docs/modules/external-modules.md | 59 ++++++++++ docs/modules/index.md | 7 ++ docs/overview.md | 69 ++++++++++++ requirements.txt | 14 +-- scripts/check_okf.py | 124 ++++++++++++++++++++++ 37 files changed, 1526 insertions(+), 10 deletions(-) create mode 100644 .claude/settings.json create mode 100644 docs/architecture/build-setup.md create mode 100644 docs/architecture/config.md create mode 100644 docs/architecture/cpython.md create mode 100644 docs/architecture/index.md create mode 100644 docs/architecture/module-builder.md create mode 100644 docs/architecture/ppg.md create mode 100644 docs/architecture/python-builder.md create mode 100644 docs/architecture/python-inspector.md create mode 100644 docs/cli/build-report.md create mode 100644 docs/cli/build.md create mode 100644 docs/cli/diagnostics.md create mode 100644 docs/cli/index.md create mode 100644 docs/cli/inspect.md create mode 100644 docs/cli/lib-auto-correct.md create mode 100644 docs/cli/list.md create mode 100644 docs/cli/recompress.md create mode 100644 docs/concepts/build-layout.md create mode 100644 docs/concepts/folder-masking.md create mode 100644 docs/concepts/index.md create mode 100644 docs/concepts/portability.md create mode 100644 docs/concepts/static-linking.md create mode 100644 docs/concepts/telltale-detection.md create mode 100644 docs/configuration/index.md create mode 100644 docs/configuration/portable-python-yml.md create mode 100644 docs/guides/add-an-external-module.md create mode 100644 docs/guides/build-a-portable-python.md create mode 100644 docs/guides/index.md create mode 100644 docs/guides/local-development.md create mode 100644 docs/index.md create mode 100644 docs/log.md create mode 100644 docs/modules/external-modules.md create mode 100644 docs/modules/index.md create mode 100644 docs/overview.md create mode 100755 scripts/check_okf.py diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..da322a9 --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,10 @@ +{ + "permissions": { + "allow": [ + "Bash(.venv/bin/portable-python:*)", + "Bash(.venv/bin/pytest:*)", + "Bash(.venv/bin/python scripts/check_okf.py:*)", + "Bash(tox:*)" + ] + } +} diff --git a/CLAUDE.md b/CLAUDE.md index 7e54760..23d363d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,9 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## What This Project Does -portable-python is a CLI tool and Python library for compiling portable CPython binaries from source. Binaries are statically linked so they can be extracted to any folder and used without installation. Supports Linux and macOS (not Windows). +portable-python is a CLI tool and Python library for compiling portable CPython binaries from source. +Binaries are statically linked so they can be extracted to any folder and used without installation. +Supports Linux and macOS (not Windows). ## Common Commands @@ -16,14 +18,14 @@ tox tox -e py313 # Run a single test -pytest tests/test_build.py::test_build_rc -vv +.venv/bin/pytest tests/test_build.py::test_build_rc -vv # Lint check / auto-fix tox -e style tox -e reformat # Run tests directly (from venv) -pytest tests/ +.venv/bin/pytest tests/ # CI uses: uvx --with tox-uv tox -e py ``` diff --git a/docs/architecture/build-setup.md b/docs/architecture/build-setup.md new file mode 100644 index 0000000..ef291ae --- /dev/null +++ b/docs/architecture/build-setup.md @@ -0,0 +1,51 @@ +--- +type: Class +title: BuildSetup +description: Coordinates the overall compilation — resolves the python spec, prepares folders, builds external modules then CPython, validates, and compresses the result. +resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py +tags: [class, build, coordinator] +timestamp: 2026-06-23T00:00:00Z +--- + +# BuildSetup + +`BuildSetup` drives a build from end to end. The CLI's [`build`](/cli/build.md) command (and library users) construct one and call `compile()`. Defined in `__init__.py`. + +```python +from portable_python import BuildSetup + +setup = BuildSetup("cpython:3.13.2") +setup.compile() +``` + +# Schema + +| Member | Kind | Purpose | +|--------|------|---------| +| `__init__(python_spec, modules, prefix)` | constructor | Resolve the [`PythonSpec`](/cli/build.md), set up [`Folders`](/architecture/config.md), pick the extension, and instantiate the family's `python_builder`. | +| `python_spec` | attribute | The resolved family + version (full `X.Y.Z` required). | +| `folders` | attribute (`Folders`) | Resolved build/dist/sources/logs/destdir paths. | +| `prefix` | attribute | Optional `--prefix`; when set the build is **not** [portable](/concepts/portability.md). | +| `tarball_name` | attribute | Composed output name, e.g. `cpython-3.13.2-macos-arm64.tar.gz`. | +| `python_builder` | attribute (`PythonBuilder`) | The concrete builder, e.g. [`Cpython`](/architecture/cpython.md). | +| `validate_module_selection(fatal)` | method | Check every selected/candidate module's `linker_outcome`; abort on a `failed` outcome. | +| `compile()` | method | The full pipeline (see below). | + +## The `compile()` pipeline + +1. **Clean** the build folder (and logs); set up file logging to `logs/00-portable-python.log`. +2. `python_builder.validate_setup()`. +3. Enter a [`BuildContext`](/concepts/folder-masking.md) — applies macOS `/usr/local` masking and other isolation. +4. Log the platform, config files in use, and the build report. +5. `validate_module_selection()` (fatal unless dry-run / debug). +6. Clean `components/` and `deps/`. +7. `build_context.compile()` then `python_builder.compile()` — external modules first, then CPython. +8. If a `dist/` folder is configured, **compress** the install into `dist/{tarball_name}`. + +## Spec validation + +`BuildSetup` insists on a **full** version: a spec like `3.13` is rejected — you must give `3.13.2`. `"latest"` (or empty) resolves to `PPG.cpython.latest`. Invalid specs abort early with a red error. + +# Citations + +[1] [src/portable_python/__init__.py — BuildSetup](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py) diff --git a/docs/architecture/config.md b/docs/architecture/config.md new file mode 100644 index 0000000..5ed4074 --- /dev/null +++ b/docs/architecture/config.md @@ -0,0 +1,55 @@ +--- +type: Class +title: Config & Folders +description: Config loads and merges YAML configuration with platform-specific overrides; Folders resolves the templated build/dist/sources paths. +resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/config.py +tags: [class, configuration, folders, yaml] +timestamp: 2026-06-23T00:00:00Z +--- + +# Config & Folders + +`Config` (in `config.py`) loads, merges, and queries the YAML [configuration](/configuration/portable-python-yml.md). `Folders` (in `versions.py`) turns templated path settings into concrete filesystem paths. Both are reached through [`PPG`](/architecture/ppg.md). + +# Schema + +## Config + +| Member | Purpose | +|--------|---------| +| `__init__(paths, target)` | Layer the built-in `DEFAULT_CONFIG` under any user config files, for the given target platform. | +| `get_value(*key, by_platform=True)` | Fetch a config value, honoring platform-specific overrides (most specific wins). | +| `get_entry(*key, by_platform=True)` | Like `get_value` but returns the raw entry. | +| `resolved_path(*key)` | Fetch a value and resolve it to a path. | +| `config_files_report()` / `represented()` | Human-readable summary of which config files are in effect (used by [`diagnostics`](/cli/diagnostics.md)). | +| `cleanup_globs(...)` / `symlink_duplicates(...)` / `ensure_main_file_symlinks(...)` | Finalize-time file housekeeping in the install. | + +## Folders + +| Member | Purpose | +|--------|---------| +| `build_folder`, `components`, `deps`, `sources`, `logs`, `dist`, `destdir` | Resolved component paths — see [build layout](/concepts/build-layout.md). | +| `ppp-marker` | Template for the in-progress install folder name. | +| `formatted(text)` | Expand `{build}`, `{version}`, `{abi_suffix}`, … placeholders. | +| `resolved_destdir(relative_path)` | Compose a path under the marker'd destdir. | + +## Precedence & overrides + +Configuration merges multiple sources. The built-in `DEFAULT_CONFIG` provides sane defaults; user files (default `portable-python.yml`, plus any `include:`d files) layer on top. Within a file, **platform-specific** sections override generic ones: + +```yaml +ext: gz # generic default +windows: + ext: zip # used only when targeting windows +macos: + env: + MACOSX_DEPLOYMENT_TARGET: 13 +``` + +`get_value(..., by_platform=True)` returns the most specific match for the current `PPG.target`. + +# Citations + +[1] [src/portable_python/config.py — Config, DEFAULT_CONFIG](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/config.py) +[2] [src/portable_python/versions.py — Folders](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/versions.py) +[3] [CONFIGURATION.md](https://github.com/codrsquad/portable-python/blob/main/CONFIGURATION.md) diff --git a/docs/architecture/cpython.md b/docs/architecture/cpython.md new file mode 100644 index 0000000..4e16028 --- /dev/null +++ b/docs/architecture/cpython.md @@ -0,0 +1,44 @@ +--- +type: Class +title: Cpython +description: The concrete PythonBuilder that compiles CPython — configure/make/install, optimization flags, and the finalize step that makes the install relocatable. +resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cpython.py +tags: [class, cpython, build, finalize] +timestamp: 2026-06-23T00:00:00Z +--- + +# Cpython + +`Cpython` is the concrete [`PythonBuilder`](/architecture/python-builder.md) that actually compiles CPython. It is returned by `CPythonFamily.get_builder()` (see [`PPG`](/architecture/ppg.md)) and lives in `cpython.py`. + +# Schema + +| Member | Kind | Purpose | +|--------|------|---------| +| `candidate_modules()` | classmethod | The external modules CPython can statically link (the [external modules](/modules/index.md) set). | +| `url` | property | CPython source URL (`python.org/ftp/...Python-X.Y.Z.tar.xz`), overridable via config. | +| `c_configure_args()` | method | Assemble `./configure` args (from `cpython-configure` config + computed flags). | +| `has_configure_opt(name, *variants)` | method | Test whether a configure option is in effect. | +| `xenv_LDFLAGS_NODIST`, `xenv_LIBZSTD_*`, `xenv_LIBMPDEC_*` | methods | Environment for linking specific static deps (zstd, mpdec). | +| `_do_linux_compile()` | method | configure → make → make install (DESTDIR). | +| `_finalize()` | method | Clean, byte-compile, relativize — see below. | + +## Configure arguments + +CPython is configured with, by default, `--enable-optimizations`, `--with-lto`, and `--with-ensurepip=upgrade` (from `cpython-configure` in [config](/configuration/portable-python-yml.md)), plus computed flags pointing at the static deps. Following the [no-patches principle](/overview.md), **all** behavior comes from configure flags — never source edits. + +## Finalize: making it relocatable + +After `make install` the `_finalize()` step is what turns a normal install into a [portable](/concepts/portability.md) one: + +- **Clean passes** — remove tests/idle/2to3 before byte-compiling (`cpython-clean-1st-pass`, ~94 MB) and prune seldom-used pycaches after (`cpython-clean-2nd-pass`, ~1.8 MB). +- **Byte-compile** the standard library (`cpython-compile-all`). +- `_relativize_shebangs()` — rewrite `bin/` script shebangs to be relative. +- `_relativize_sysconfig()` (`RelSysConf`) — rewrite absolute paths in the `sysconfig` data so the install resolves relative to its own location. +- `_apply_pep668()` — mark the environment per PEP 668. +- `_validate_venv_creation()` / `_validate_venv_module()` — sanity-check that `venv` works in the finished build. + +# Citations + +[1] [src/portable_python/cpython.py](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cpython.py) +[2] [src/portable_python/config.py — cpython-* defaults](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/config.py) diff --git a/docs/architecture/index.md b/docs/architecture/index.md new file mode 100644 index 0000000..cffa058 --- /dev/null +++ b/docs/architecture/index.md @@ -0,0 +1,31 @@ +# Architecture + +The core classes and how they collaborate during a build. See [ARCHITECTURE.md](https://github.com/codrsquad/portable-python/blob/main/ARCHITECTURE.md) for the original hierarchy diagram. + +## Global state & configuration + +* [PPG](/architecture/ppg.md) - Global singleton holding config, target platform, and version families. +* [Config](/architecture/config.md) - Loads and merges YAML configuration; the `Folders` helper resolves build paths. + +## Build coordination + +* [BuildSetup](/architecture/build-setup.md) - Drives the overall compilation: resolve spec, select modules, build, validate, compress. +* [ModuleBuilder](/architecture/module-builder.md) - Abstract base for anything that gets compiled (external C libs and Python itself). +* [PythonBuilder](/architecture/python-builder.md) - `ModuleBuilder` specialization for python implementations. +* [Cpython](/architecture/cpython.md) - Concrete builder: CPython's configure/make/install, optimization, and finalization. + +## Validation + +* [PythonInspector](/architecture/python-inspector.md) - Validates the portability of a built (or any) python by checking shared-library dependencies and paths. + +## Collaboration at a glance + +``` +PPG (config, target, families) + └─ BuildSetup(python_spec) + ├─ Folders (resolved build/dist/... paths) + ├─ BuildContext (macOS folder masking, isolation) + └─ python_builder : Cpython (a PythonBuilder, a ModuleBuilder) + └─ modules : ModuleCollection + └─ candidates/selected : ModuleBuilder (Openssl, Zlib, ...) +``` diff --git a/docs/architecture/module-builder.md b/docs/architecture/module-builder.md new file mode 100644 index 0000000..827a305 --- /dev/null +++ b/docs/architecture/module-builder.md @@ -0,0 +1,59 @@ +--- +type: Class +title: ModuleBuilder +description: Abstract base for everything that gets compiled — external C libraries and CPython itself — providing the common download/configure/make/install flow and environment injection. +resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py +tags: [class, abstract, build, module] +timestamp: 2026-06-23T00:00:00Z +--- + +# ModuleBuilder + +`ModuleBuilder` is the abstract base for anything compiled by the tool. Both [external C modules](/modules/index.md) and [`PythonBuilder`](/architecture/python-builder.md) (hence [`Cpython`](/architecture/cpython.md)) extend it, so every component shares the same [build layout](/concepts/build-layout.md) and flow. + +# Schema + +## Declarative class attributes + +| Attribute | Purpose | +|-----------|---------| +| `m_name` | Module name (derived from the class name). | +| `m_telltale` | Marker file(s) indicating the lib is present on the system. See [telltale detection](/concepts/telltale-detection.md). | +| `m_debian` | Debian dev package, with `!` / `+` / `-` sigils encoding build constraints. | +| `m_include` | Optional subfolder to add to `CPATH` when active (e.g. `openssl`). | +| `m_build_cwd` | Optional subfolder (relative to unpacked source) to run configure/make from. | + +## Source resolution (overridable) + +| Member | Purpose | +|--------|---------| +| `url` | Download URL of the source tarball. | +| `version` | Version to build (default per module, overridable via config). | +| `headers` / `src_suffix` | HTTP headers and archive suffix when the URL lacks an extension. | +| `cfg_version`, `cfg_url`, `cfg_configure`, `cfg_patches`, `cfg_http_headers` | Read per-module overrides from [config](/configuration/portable-python-yml.md) (keys like `openssl-version`, `openssl-url`, …). | + +## Environment injection (`xenv_*`) + +Each `xenv_*` method supplies one environment variable for the compile, pointing tools at the shared `deps/` prefix — the mechanism behind [static linking](/concepts/static-linking.md): + +`xenv_CPATH`, `xenv_LDFLAGS`, `xenv_PATH`, `xenv_LD_LIBRARY_PATH`, `xenv_PKG_CONFIG_PATH`. `_find_all_env_vars()` gathers every `xenv_*` on the instance just before running. + +## Build flow + +| Method | Role | +|--------|------| +| `compile()` | Download → unpack → patch → `_prepare()` → platform compile → `_finalize()`. | +| `run_configure(program, *args, prefix)` | Run `./configure` with the deps prefix. | +| `run_make(*args, cpu_count)` | Run `make` (parallelized by CPU count). | +| `_do_linux_compile()` / `_do_macos_compile()` | Platform-specific compile, dispatched by `PPG.target`. | +| `linker_outcome(is_selected)` | Decide `static` / `shared` / `absent` / `failed` for this module. | +| `captured_logs()` | Context manager routing this module's output to its own numbered log file. | + +## Adding a module + +To add a new external module you subclass `ModuleBuilder`, set the `m_*` attributes, implement `url`/`version`, and implement `_do_linux_compile()` (macOS reuses it unless overridden). See the [guide](/guides/add-an-external-module.md). + +# Citations + +[1] [src/portable_python/__init__.py — ModuleBuilder](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py) +[2] [src/portable_python/external/xcpython.py — concrete modules](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/external/xcpython.py) diff --git a/docs/architecture/ppg.md b/docs/architecture/ppg.md new file mode 100644 index 0000000..47cef10 --- /dev/null +++ b/docs/architecture/ppg.md @@ -0,0 +1,43 @@ +--- +type: Class +title: PPG +description: Global singleton holding shared configuration, the target platform, and the registry of supported python version families. +resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/versions.py +tags: [class, global-state, singleton, versions] +timestamp: 2026-06-23T00:00:00Z +--- + +# PPG + +`PPG` is the global state holder for portable-python. Every module reaches shared state — configuration, the target platform, and version families — through `PPG`'s class attributes and classmethods. It is defined in `versions.py`. + +# Schema + +| Member | Kind | Purpose | +|--------|------|---------| +| `config` | attribute (`Config`) | The active merged [configuration](/architecture/config.md). | +| `target` | attribute (`PlatformId`) | Target OS/arch; drives platform dispatch and telltale expansion. | +| `families` | attribute (`dict`) | Registry mapping family name → `VersionFamily` (default: `{"cpython": ...}`). | +| `cpython` | attribute (`CPythonFamily`) | The built-in CPython family implementation. | +| `grab_config(paths, target)` | classmethod | (Re)load config from the given paths and set `target`. Called by the CLI's `main`. | +| `get_folders(base, family, version, abi_suffix)` | classmethod | Build a [`Folders`](/architecture/config.md) for a given family/version. | +| `family(name, fatal=True)` | classmethod | Look up a `VersionFamily` by name. | +| `find_python(spec)` | classmethod | Resolve a python on `PATH` via a cached `runez` `PythonDepot`. | +| `find_telltale(*telltales)` | classmethod | Expand `{include}` placeholders against `target.sys_include` and return the first existing path. See [telltale detection](/concepts/telltale-detection.md). | + +## Version families + +A `VersionFamily` (e.g. `CPythonFamily`) knows how to **list available versions** and **provide a builder**: + +- `available_versions` / `latest` — lazily fetched and cached (CPython fetches from `python.org/ftp`, or GitHub tags when `cpython-use-github` is set). This lazy fetch is why [`list`](/cli/list.md) hits the network on first use. +- `get_builder()` — returns the concrete builder class ([`Cpython`](/architecture/cpython.md) for the cpython family). +- `min_version` — earliest non-EOL version known to compile well (currently `3.9`). + +## Why a singleton + +The build touches global, process-wide facts: which config file is active, what platform we are targeting, and which python versions exist. Centralizing them in `PPG` avoids threading that state through every constructor, and gives the CLI a single place ([`main`](/cli/index.md)) to initialize it. + +# Citations + +[1] [src/portable_python/versions.py](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/versions.py) +[2] [CLAUDE.md — Key classes](https://github.com/codrsquad/portable-python/blob/main/CLAUDE.md) diff --git a/docs/architecture/python-builder.md b/docs/architecture/python-builder.md new file mode 100644 index 0000000..4ed1796 --- /dev/null +++ b/docs/architecture/python-builder.md @@ -0,0 +1,38 @@ +--- +type: Class +title: PythonBuilder +description: A ModuleBuilder specialization for python implementations — adds module selection, the python install layout, and helpers to run the freshly built interpreter. +resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py +tags: [class, abstract, python, build] +timestamp: 2026-06-23T00:00:00Z +--- + +# PythonBuilder + +`PythonBuilder` extends [`ModuleBuilder`](/architecture/module-builder.md) with behavior specific to building a python interpreter. It is still abstract — the concrete implementation is [`Cpython`](/architecture/cpython.md). Defined in `__init__.py`. + +# Schema + +| Member | Kind | Purpose | +|--------|------|---------| +| `modules` | attribute (`ModuleCollection`) | The external modules selected/available for this build. | +| `selected_modules()` | method | Builds the `ModuleCollection` from candidates + the desired list. | +| `bin_python` | property | Path to the built interpreter (`.../bin/python`). | +| `version` | property | The python version being built. | +| `validate_setup()` | method | Pre-flight checks before compilation begins. | +| `run_python(*args)` | method | Invoke the freshly built interpreter (used during finalize/validation). | +| `xenv_LDFLAGS()` | method | Python-specific linker flags layered on the base `ModuleBuilder` behavior. | +| `_prepare()` | method | Hook run before the platform compile. | + +## ModuleCollection + +A `PythonBuilder` owns a `ModuleCollection`, which models the set of candidate external modules and resolves which are actually built: + +- `candidates` — every possible module for this builder (from `candidate_modules()`). +- `selected` — only the modules chosen for *this* build (config + `--modules` + auto-selection). This is the list that actually gets compiled. +- `auto_selected` — modules force-selected because a build can't succeed without them (each module's `auto_select_reason()`). +- `report()` / `report_rows()` — the human-readable table shown by [`build-report`](/cli/build-report.md). + +# Citations + +[1] [src/portable_python/__init__.py — PythonBuilder, ModuleCollection](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py) diff --git a/docs/architecture/python-inspector.md b/docs/architecture/python-inspector.md new file mode 100644 index 0000000..8d5ccda --- /dev/null +++ b/docs/architecture/python-inspector.md @@ -0,0 +1,57 @@ +--- +type: Class +title: PythonInspector +description: Validates the portability of a python installation by parsing the dynamic-library dependencies of every executable and .so, and reporting any non-portable references. +resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/inspector.py +tags: [class, inspection, validation, portability] +timestamp: 2026-06-23T00:00:00Z +--- + +# PythonInspector + +`PythonInspector` (in `inspector.py`) answers the question the whole project is built around: *is this python installation [portable](/concepts/portability.md), and if not, why not?* It powers the [`inspect`](/cli/inspect.md) command and the post-build validation in [`BuildSetup.compile()`](/architecture/build-setup.md). + +```python +from portable_python.inspector import PythonInspector + +inspector = PythonInspector("/usr/bin/python3") +print(inspector.represented()) +problem = inspector.full_so_report.get_problem(portable=True) +if problem: + print("not portable: %s" % problem) +``` + +# Schema + +| Member | Kind | Purpose | +|--------|------|---------| +| `__init__(spec, modules)` | constructor | Resolve the python to inspect and the module set to report. | +| `full_so_report` | property (`FullSoReport`) | Aggregated scan of every `.so`/executable and its lib references. | +| `module_info` | property | Per-module info (versions, extra detail). | +| `represented(verbose)` | method | Render the human-readable report. | +| `get_problem(portable)` | method (on `FullSoReport`) | Return a description of the first portability problem, or empty if clean. | + +## Supporting types + +| Type | Role | +|------|------| +| `SoInfo` | One `.so`/executable; parses `ldd` (Linux) / `otool` (macOS) output into references. | +| `CLibInfo` | One referenced C library, classified by `LibType`. | +| `LibType` | Enum classifying a reference (system, relative/self, problematic, …). | +| `FullSoReport` | The complete scan and its `get_problem()` verdict. | +| `LibAutoCorrect` | Rewrites absolute lib references in binaries to **relative** form (`@loader_path` / `$ORIGIN`), making an install relocatable. Exposed via [`lib-auto-correct`](/cli/lib-auto-correct.md). | + +## How it detects problems + +For each binary it lists dynamic dependencies, then classifies every reference: + +- **System / standard** library → acceptable. +- **Relative / self** reference → acceptable (this is what `LibAutoCorrect` produces). +- **Absolute, non-system** reference → a portability problem; for a portable build this fails the inspection. + +`LibAutoCorrect` is also used **during** the build's finalize step to convert references to relative form before the portable check runs. + +# Citations + +[1] [src/portable_python/inspector.py](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/inspector.py) +[2] [src/portable_python/tracking.py — Tracker/Trackable](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/tracking.py) diff --git a/docs/cli/build-report.md b/docs/cli/build-report.md new file mode 100644 index 0000000..fd76c32 --- /dev/null +++ b/docs/cli/build-report.md @@ -0,0 +1,38 @@ +--- +type: CLI Command +title: build-report +description: Show the status of buildable modules — which will be auto-compiled, which are present on the system, and which would fail — without building anything. +resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py +tags: [cli, build, report, command] +timestamp: 2026-06-23T00:00:00Z +--- + +# build-report + +Show which modules would be compiled for a given spec, and validate that the selection can actually build on this host — without running a build. It prints the [`ModuleCollection.report()`](/architecture/python-builder.md) table and then calls `validate_module_selection()`. + +``` +portable-python build-report [OPTIONS] [PYTHON_SPEC] +``` + +# Schema + +| Argument / Option | Purpose | +|-------------------|---------| +| `PYTHON_SPEC` (optional) | Version to report on; defaults to the latest known version. | +| `--modules, -m CSV` | Specific modules to check. | + +## What it tells you + +For each candidate module the report shows its [telltale](/concepts/telltale-detection.md) status and its `linker_outcome`: whether it will be compiled `static`, linked `shared`, is `absent`, or is `failed` (a combination that cannot build on this host — e.g. a `-`-sigil Debian dev package being present). This is the fastest way to catch a "broken" module before committing to a full build. + +# Examples + +```shell +portable-python build-report 3.13.2 +portable-python build-report 3.13.2 -m openssl,sqlite +``` + +# Citations + +[1] [src/portable_python/cli.py — build_report](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py) diff --git a/docs/cli/build.md b/docs/cli/build.md new file mode 100644 index 0000000..0f9dfb0 --- /dev/null +++ b/docs/cli/build.md @@ -0,0 +1,48 @@ +--- +type: CLI Command +title: build +description: Build a portable (or --prefix) python binary from source for the given version spec. +resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py +tags: [cli, build, command] +timestamp: 2026-06-23T00:00:00Z +--- + +# build + +Build a portable python binary. Constructs a [`BuildSetup`](/architecture/build-setup.md) and calls `compile()`. + +``` +portable-python build [OPTIONS] PYTHON_SPEC +``` + +# Schema + +| Argument / Option | Purpose | +|-------------------|---------| +| `PYTHON_SPEC` (required) | Full version to build, e.g. `3.13.2` or `cpython:3.13.2`. A bare `X.Y` is rejected — give `X.Y.Z`. `latest` resolves to the newest known version. | +| `--modules, -m CSV` | External modules to include, e.g. `-m openssl,zlib`. Default comes from config + auto-selection. | +| `--prefix, -p PATH` | Build for a fixed install location. **Disables portability** — the result must live at `PATH`. | + +## Output + +The finished tarball lands in `dist/`, named for the family, version, platform, and arch, e.g. `dist/cpython-3.13.2-macos-arm64.tar.gz`. See [build layout](/concepts/build-layout.md). + +# Examples + +```shell +# Portable build, modules auto-selected from config +portable-python build 3.13.2 + +# Restrict to specific modules +portable-python build 3.13.2 -m openssl,zlib,xz + +# Non-portable build pinned to a prefix +portable-python build 3.13.2 --prefix /apps/python3.13 + +# See exactly what would happen, without compiling +portable-python --dryrun build 3.13.2 +``` + +# Citations + +[1] [src/portable_python/cli.py — build](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py) diff --git a/docs/cli/diagnostics.md b/docs/cli/diagnostics.md new file mode 100644 index 0000000..dd0a5b5 --- /dev/null +++ b/docs/cli/diagnostics.md @@ -0,0 +1,24 @@ +--- +type: CLI Command +title: diagnostics +description: Show system diagnostics (invoker python, platform info) alongside the active configuration. +resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py +tags: [cli, diagnostics, command] +timestamp: 2026-06-23T00:00:00Z +--- + +# diagnostics + +Print system diagnostics next to the resolved [configuration](/architecture/config.md), as a two-column table. Useful when a build behaves unexpectedly and you need to confirm which config files and platform are in effect. + +``` +portable-python diagnostics +``` + +# Schema + +Takes no arguments. The left column comes from `runez.SYS_INFO` (invoker python, platform, etc.); the right column is `PPG.config.represented()` — the merged config and the files it was loaded from. + +# Citations + +[1] [src/portable_python/cli.py — diagnostics](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py) diff --git a/docs/cli/index.md b/docs/cli/index.md new file mode 100644 index 0000000..70ef93a --- /dev/null +++ b/docs/cli/index.md @@ -0,0 +1,29 @@ +# CLI + +`portable-python` is a single-entry-point CLI built with `click` (via `runez.click`). All commands are defined in `cli.py`. + +``` +portable-python [GLOBAL OPTIONS] COMMAND [ARGS] +``` + +## Global options + +| Option | Purpose | +|--------|---------| +| `--config, -c PATH` | Config file to use (default: `portable-python.yml`). | +| `--quiet, -q` | Turn off DEBUG logging. | +| `--dryrun, -n` | Show what would be done without doing it. | +| `--target, -t` | Override the detected platform (internal/testing). | +| `--version` / `--color` | Standard `runez` options. | + +The `main` group initializes logging and calls [`PPG.grab_config()`](/architecture/ppg.md) before any subcommand runs. + +## Commands + +* [build](/cli/build.md) - Build a portable python binary. +* [build-report](/cli/build-report.md) - Show which modules will be compiled and whether they can build. +* [inspect](/cli/inspect.md) - Check whether a python installation is portable. +* [list](/cli/list.md) - List available versions for a family. +* [diagnostics](/cli/diagnostics.md) - Show system diagnostics and active config. +* [recompress](/cli/recompress.md) - Re-compress an existing tarball/folder (compare sizes). +* [lib-auto-correct](/cli/lib-auto-correct.md) - Rewrite a build's lib references to relative paths. diff --git a/docs/cli/inspect.md b/docs/cli/inspect.md new file mode 100644 index 0000000..6f66bd8 --- /dev/null +++ b/docs/cli/inspect.md @@ -0,0 +1,48 @@ +--- +type: CLI Command +title: inspect +description: Inspect a python installation for non-portable dynamic library usage, reporting any references that would break portability. +resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py +tags: [cli, inspect, portability, command] +timestamp: 2026-06-23T00:00:00Z +--- + +# inspect + +Inspect any python installation and report whether it is [portable](/concepts/portability.md). Backed by the [`PythonInspector`](/architecture/python-inspector.md); aborts non-zero if a portability problem is found (unless `--prefix` is given). + +``` +portable-python inspect [OPTIONS] PATH +``` + +# Schema + +| Argument / Option | Purpose | +|-------------------|---------| +| `PATH` (required) | Path to a python installation or interpreter. The special value `invoker` inspects the python running the CLI. | +| `--modules, -m MODULES` | Which modules to inspect (`all` to check everything). | +| `--verbose, -v` | Show the full `.so` report. | +| `--prefix, -p` | The build used `--prefix` (not portable) — relaxes the portability check accordingly. | +| `--skip-so, -s` | Don't scan every `.so` file. | + +## Exit behavior + +After printing the report, if scanning `.so` files (and not skipped), it asks `full_so_report.get_problem(portable=not prefix)`. A non-empty problem aborts with a non-zero exit — making `inspect` usable as a CI gate. + +# Examples + +```shell +# Is the system python portable? (it usually is not) +portable-python inspect /usr/bin/python3 + +# Full detail +portable-python inspect -v ~/versions/3.13.2 + +# Inspect the interpreter running this CLI +portable-python inspect invoker +``` + +# Citations + +[1] [src/portable_python/cli.py — inspect](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py) +[2] [src/portable_python/inspector.py — PythonInspector](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/inspector.py) diff --git a/docs/cli/lib-auto-correct.md b/docs/cli/lib-auto-correct.md new file mode 100644 index 0000000..bebd37a --- /dev/null +++ b/docs/cli/lib-auto-correct.md @@ -0,0 +1,41 @@ +--- +type: CLI Command +title: lib-auto-correct +description: Scan a python installation and rewrite executables/libraries to use relative paths — the same relativization the build applies internally, runnable standalone. +resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py +tags: [cli, lib-auto-correct, relativize, command] +timestamp: 2026-06-23T00:00:00Z +--- + +# lib-auto-correct + +Scan a python installation and auto-correct its executables and libraries to reference dependencies by **relative** path. This exercises the [`LibAutoCorrect`](/architecture/python-inspector.md) logic that the build runs internally — handy for iterating on relativization without waiting for a full build. + +``` +portable-python lib-auto-correct [OPTIONS] PATH +``` + +# Schema + +| Argument / Option | Purpose | +|-------------------|---------| +| `PATH` (required) | The python installation to scan. | +| `--commit` | Actually perform the changes. Without it, the command runs in dry-run mode. | +| `--prefix, -p PATH` | The `--prefix` the program was built with. Defaults to the scanned python's own `sysconfig` prefix. | + +By default this is **dry-run**: it shows what it would rewrite. Pass `--commit` to apply the changes. See [static linking](/concepts/static-linking.md) for why relativization is needed. + +# Examples + +```shell +# Preview corrections (dry-run) +portable-python lib-auto-correct ~/versions/3.13.2 + +# Apply them +portable-python lib-auto-correct --commit ~/versions/3.13.2 +``` + +# Citations + +[1] [src/portable_python/cli.py — lib_auto_correct](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py) +[2] [src/portable_python/inspector.py — LibAutoCorrect](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/inspector.py) diff --git a/docs/cli/list.md b/docs/cli/list.md new file mode 100644 index 0000000..300bda4 --- /dev/null +++ b/docs/cli/list.md @@ -0,0 +1,35 @@ +--- +type: CLI Command +title: list +description: List the latest available versions for a python family (default cpython), optionally as JSON. +resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py +tags: [cli, list, versions, command] +timestamp: 2026-06-23T00:00:00Z +--- + +# list + +List the latest available versions for a [version family](/architecture/ppg.md). Versions are fetched lazily (from `python.org/ftp`, or GitHub tags when `cpython-use-github` is configured) and cached. + +``` +portable-python list [OPTIONS] [FAMILY] +``` + +# Schema + +| Argument / Option | Purpose | +|-------------------|---------| +| `FAMILY` (optional) | Family to list; defaults to `cpython`. Unknown families abort with a message. | +| `--json` | Emit the available versions as JSON instead of a table. | + +# Examples + +```shell +portable-python list +portable-python list cpython --json +``` + +# Citations + +[1] [src/portable_python/cli.py — list_cmd](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py) +[2] [src/portable_python/versions.py — CPythonFamily.get_available_versions](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/versions.py) diff --git a/docs/cli/recompress.md b/docs/cli/recompress.md new file mode 100644 index 0000000..7397007 --- /dev/null +++ b/docs/cli/recompress.md @@ -0,0 +1,35 @@ +--- +type: CLI Command +title: recompress +description: Re-compress an existing binary tarball or folder into another format, mainly to compare resulting sizes. +resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py +tags: [cli, recompress, command] +timestamp: 2026-06-23T00:00:00Z +--- + +# recompress + +Re-compress an existing build (folder or tarball) into a different compression format. Mildly useful for comparing the size trade-offs of different compressions. + +``` +portable-python recompress PATH EXT +``` + +# Schema + +| Argument | Purpose | +|----------|---------| +| `PATH` (required) | An existing folder or tarball. Resolved against the configured base/build/dist/destdir folders if not absolute. | +| `EXT` (required) | Target compression extension. Constrained to the platform's `supported_compression` choices. | + +After re-compressing, the command prints the size of both the source and the result so they can be compared. + +# Examples + +```shell +portable-python recompress dist/cpython-3.13.2-linux-x86_64.tar.gz tar.zst +``` + +# Citations + +[1] [src/portable_python/cli.py — recompress](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py) diff --git a/docs/concepts/build-layout.md b/docs/concepts/build-layout.md new file mode 100644 index 0000000..2665031 --- /dev/null +++ b/docs/concepts/build-layout.md @@ -0,0 +1,54 @@ +--- +type: Concept +title: Build Folder Layout +description: Every component follows the same conventional layout under build/, with external libraries sharing a deps prefix and the finished binary landing in dist/. +tags: [build-layout, concept, folders] +timestamp: 2026-06-23T00:00:00Z +--- + +# Build Folder Layout + +Every component — external modules and CPython alike — is built using the same conventional folder layout. This uniformity is what lets a single [`ModuleBuilder`](/architecture/module-builder.md) framework drive every compile. Folder paths are configurable; the defaults are shown below. + +# Schema + +``` +build/ + sources/ # Downloaded source tarballs (downloaded once, reused) + openssl-3.5.6.tar.gz + Python-3.13.2.tar.xz + components/ # Each module is unpacked + compiled here + openssl/ + cpython/ + deps/ # Shared --prefix for ALL external modules + include/ # → injected into CPATH + lib/ lib64/ # → injected into LDFLAGS + bin/ # → injected into PATH + logs/ # Per-component build logs, ordered + 00-portable-python.log + 01-openssl.log + 02-cpython.log + ppp-marker/{version}/ # Full installation after build completes +dist/ + cpython-3.13.2-macos-arm64.tar.gz # Ready-to-go portable tarball +``` + +## Key folders + +| Folder | Role | +|--------|------| +| `sources/` | Cache of downloaded tarballs. Survives if you point it outside `build/`. | +| `components/` | Scratch space where each component's source is unpacked and compiled. | +| `deps/` | The shared install prefix passed to every external module's `./configure --prefix=build/deps`. CPython finds these via injected env vars — see [static linking](/concepts/static-linking.md). | +| `logs/` | One log file per component, numbered so they sort in compile order. | +| `ppp-marker/{version}/` | The completed installation, named via the `ppp-marker` template before compression. | +| `dist/` | Final portable tarball(s). | + +## Configurability + +All of these paths are templated and overridable in [configuration](/configuration/portable-python-yml.md) under the `folders:` key, using placeholders like `{build}`, `{version}`, and `{abi_suffix}`. The [`Folders`](/architecture/config.md) helper resolves them. For example, the dev config keeps builds per-version with `build: "build/{family}-{version}{abi_suffix}"`. + +# Citations + +[1] [README.rst — Build folder structure](https://github.com/codrsquad/portable-python/blob/main/README.rst) +[2] [src/portable_python/config.py — DEFAULT_CONFIG folders](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/config.py) diff --git a/docs/concepts/folder-masking.md b/docs/concepts/folder-masking.md new file mode 100644 index 0000000..ee7e908 --- /dev/null +++ b/docs/concepts/folder-masking.md @@ -0,0 +1,31 @@ +--- +type: Concept +title: Folder Masking (macOS) +description: On macOS, /usr/local is temporarily masked with a RAM disk during the build so that CPython's configure cannot accidentally discover and dynamically link Homebrew libraries. +tags: [macos, folder-mask, concept, isolation] +timestamp: 2026-06-23T00:00:00Z +--- + +# Folder Masking (macOS) + +On macOS, `/usr/local` is where Homebrew and other package managers install libraries and headers. If CPython's `configure` discovers those during a build, it may **dynamically link** against them — silently breaking [portability](/concepts/portability.md), because the resulting binary would depend on libraries that won't exist on another machine. + +## The mask + +To prevent this, the build temporarily **masks** `/usr/local` with an empty RAM disk for the duration of the compile. With `/usr/local` appearing empty, `configure` falls back to the statically-compiled libraries in `build/deps/` (see [static linking](/concepts/static-linking.md)) instead of system ones. + +This is implemented by `FolderMask` and orchestrated by `BuildContext`: + +- `BuildContext.__enter__` mounts the mask (and any other isolation hacks) before compilation begins. +- `BuildContext.__exit__` / `cleanup` unmounts it afterward, restoring `/usr/local`. + +Because it is a RAM disk mounted over the path (not a deletion), the real `/usr/local` is untouched and reappears the moment the mask is removed. + +## Scope + +This mask applies to **macOS only**. On Linux, isolation is achieved through telltale-driven module selection and the injected `CPATH`/`LDFLAGS` pointing at `build/deps/` first. + +# Citations + +[1] [src/portable_python/__init__.py — FolderMask, BuildContext](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py) +[2] [ARCHITECTURE.md — Folder Masking pattern](https://github.com/codrsquad/portable-python/blob/main/ARCHITECTURE.md) diff --git a/docs/concepts/index.md b/docs/concepts/index.md new file mode 100644 index 0000000..9475836 --- /dev/null +++ b/docs/concepts/index.md @@ -0,0 +1,9 @@ +# Concepts + +The domain ideas behind portable-python. Read these to understand *why* the build works the way it does. + +* [Portability](/concepts/portability.md) - What "portable" means here, and how it is validated. +* [Static linking](/concepts/static-linking.md) - Why dependencies are compiled statically and paths relativized. +* [Telltale detection](/concepts/telltale-detection.md) - How the tool decides whether a library must be compiled or already exists. +* [Folder masking](/concepts/folder-masking.md) - The macOS `/usr/local` RAM-disk mask that prevents accidental dynamic linking. +* [Build layout](/concepts/build-layout.md) - The `build/` and `dist/` folder structure shared by every component. diff --git a/docs/concepts/portability.md b/docs/concepts/portability.md new file mode 100644 index 0000000..435c643 --- /dev/null +++ b/docs/concepts/portability.md @@ -0,0 +1,39 @@ +--- +type: Concept +title: Portability +description: A python installation is portable when it can be unpacked into any folder and run without depending on non-standard system shared libraries. +tags: [portability, concept, validation] +timestamp: 2026-06-23T00:00:00Z +--- + +# Portability + +A python installation is **portable** when it can be copied or unpacked into an arbitrary folder and used as-is, without: + +- An installer or post-install step. +- A dependency on shared libraries that may be absent (or a different version) on the target machine. +- Hard-coded absolute paths baked into binaries, scripts, or `sysconfig`. + +Portability is the central guarantee of this project. Two design choices deliver it: + +1. **[Static linking](/concepts/static-linking.md)** of dependencies, so the binary carries what it needs rather than loading it from the system at runtime. +2. **Path relativization**, so shebangs and `sysconfig` values resolve relative to the install folder instead of an absolute build-time prefix. + +## Portable vs `--prefix` builds + +The tool can produce two kinds of output: + +| Build kind | Description | Relocatable? | +|------------|-------------|--------------| +| Portable (default) | Statically linked, paths relativized | Yes — run from anywhere | +| `--prefix PATH` | Built to live at a fixed location (e.g. `/apps/python3.13`) | No — must live at `PATH` | + +See the [`build`](/cli/build.md) command for how to choose. + +## Validation + +Portability is not assumed — it is **verified**. The [`PythonInspector`](/architecture/python-inspector.md) walks every executable and `.so` file, parses their dynamic dependencies (`ldd` on Linux, `otool` on macOS), and classifies each reference as a system library, a relative reference, or a problem. The [`inspect`](/cli/inspect.md) command surfaces this report and fails if a non-portable reference is found. + +# Citations + +[1] [README.rst — Supported operating systems](https://github.com/codrsquad/portable-python/blob/main/README.rst) diff --git a/docs/concepts/static-linking.md b/docs/concepts/static-linking.md new file mode 100644 index 0000000..58a1b9b --- /dev/null +++ b/docs/concepts/static-linking.md @@ -0,0 +1,36 @@ +--- +type: Concept +title: Static Linking +description: Dependencies are compiled statically into a shared deps prefix and linked into CPython, so the binary carries no runtime dependency on system shared libraries. +tags: [static-linking, concept, build] +timestamp: 2026-06-23T00:00:00Z +--- + +# Static Linking + +To make a binary [portable](/concepts/portability.md), its dependencies must travel with it rather than be loaded from the host at runtime. portable-python achieves this by compiling each external C library **statically**. + +## How it works + +1. Each [external module](/modules/index.md) is compiled with flags that disable shared output and enable static, position-independent code — typically `--enable-shared=no --enable-static=yes` (and `-fPIC` where needed). +2. All modules install into a single shared prefix, `build/deps/` (see [build layout](/concepts/build-layout.md)). +3. When CPython is compiled, environment variables injected by the [`ModuleBuilder`](/architecture/module-builder.md) point its `configure`/`make` at that prefix: + - `CPATH` → `build/deps/include` + - `LDFLAGS` → `build/deps/lib` (and `lib64`) + - `PKG_CONFIG_PATH`, `PATH`, `LD_LIBRARY_PATH` as needed. +4. CPython links those static archives into `libpython` and its extension `.so` files. + +The `xenv_*` methods on `ModuleBuilder` are the mechanism: each returns the value for one environment variable, and they are gathered just before `configure`/`make` runs. + +## No source patches + +Static linking is achieved **only** through `configure`/`make` flags — never by editing upstream source. This keeps the tool maintainable as upstream versions change. See the [guiding principles](/overview.md). + +## Relativization + +Static linking removes runtime library dependencies, but absolute *paths* can still leak into the install (shebang lines, `sysconfig` variables). The [`Cpython`](/architecture/cpython.md) finalize step rewrites these to be relative, and [`LibAutoCorrect`](/architecture/python-inspector.md) rewrites dynamic-library references in binaries to relative form (`@loader_path`/`$ORIGIN` style). Together these make the install relocatable. + +# Citations + +[1] [src/portable_python/__init__.py — ModuleBuilder.xenv_* methods](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py) +[2] [src/portable_python/cpython.py — finalize / relativize steps](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cpython.py) diff --git a/docs/concepts/telltale-detection.md b/docs/concepts/telltale-detection.md new file mode 100644 index 0000000..6d9c4c8 --- /dev/null +++ b/docs/concepts/telltale-detection.md @@ -0,0 +1,54 @@ +--- +type: Concept +title: Telltale Detection +description: Modules check for marker header files to determine whether a library is already present on the system, which drives whether it is compiled statically, linked dynamically, or skipped. +tags: [telltale, concept, module-selection] +timestamp: 2026-06-23T00:00:00Z +--- + +# Telltale Detection + +Before compiling, the tool must decide, for each candidate [external module](/modules/index.md), whether the library is already available on the build host. It does this by looking for a **telltale** — a marker file (usually a development header) whose presence indicates the system has that library installed. + +## The `m_telltale` attribute + +Each [`ModuleBuilder`](/architecture/module-builder.md) subclass declares `m_telltale`, a path (or list of paths) using an `{include}` placeholder that is expanded against the target's system include directories: + +```python +class Openssl(ModuleBuilder): + m_telltale = "{include}/openssl/ssl.h" + +class Sqlite(ModuleBuilder): + m_telltale = ["{include}/sqlite3.h"] +``` + +`PPG.find_telltale()` expands the placeholder against each `target.sys_include` directory and returns the first existing path, or `None`. + +## Linker outcome + +The telltale result, combined with whether the module was *selected* and the platform, feeds `ModuleBuilder.linker_outcome()`, which classifies the module as one of: + +| Outcome | Meaning | +|---------|---------| +| `static` | Will be compiled and statically linked. | +| `shared` | Present on system, will be dynamically linked (only acceptable for non-portable / system libs). | +| `absent` | Not present and not selected — skipped. | +| `failed` | A configuration that cannot succeed (see Debian markers below). | + +## Debian markers (`m_debian`) + +On Linux, `m_debian` names the Debian dev package and a leading sigil encodes a build constraint: + +| Prefix | Meaning | Example | +|--------|---------|---------| +| `!` | Required to compile **at all** — build is broken without it. | `!zlib1g-dev`, `!libffi-dev` | +| `+` | Required when the module is **selected** for static compile. | `+libsqlite3-dev`, `+uuid-dev` | +| `-` | Presence of the dev package **breaks** static compilation. | `-libreadline-dev`, `-tk-dev` | +| (none) | Used for detection/reporting only. | `libgdbm-dev` | + +This is why [`build-report`](/cli/build-report.md) can warn that a module is "broken" on the current host before a build is even attempted. + +# Citations + +[1] [src/portable_python/versions.py — PPG.find_telltale](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/versions.py) +[2] [src/portable_python/__init__.py — ModuleBuilder.linker_outcome](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py) diff --git a/docs/configuration/index.md b/docs/configuration/index.md new file mode 100644 index 0000000..43a90d5 --- /dev/null +++ b/docs/configuration/index.md @@ -0,0 +1,7 @@ +# Configuration + +How portable-python is configured, and how settings are merged and overridden. + +* [portable-python.yml](/configuration/portable-python-yml.md) - The config file: sections, per-module keys, path templates, and platform overrides. + +Configuration is loaded and merged by [`Config`](/architecture/config.md). The built-in `DEFAULT_CONFIG` provides defaults; user files layer on top, with platform-specific sections winning over generic ones. See also the root [CONFIGURATION.md](https://github.com/codrsquad/portable-python/blob/main/CONFIGURATION.md). diff --git a/docs/configuration/portable-python-yml.md b/docs/configuration/portable-python-yml.md new file mode 100644 index 0000000..abcfe35 --- /dev/null +++ b/docs/configuration/portable-python-yml.md @@ -0,0 +1,89 @@ +--- +type: Configuration +title: portable-python.yml +description: The YAML configuration file controlling folders, module selection, configure flags, cleanup, and per-module overrides — with platform-specific sections. +resource: https://github.com/codrsquad/portable-python/blob/main/portable-python.yml +tags: [configuration, yaml, config] +timestamp: 2026-06-23T00:00:00Z +--- + +# portable-python.yml + +portable-python is configured via a YAML file — by default `portable-python.yml` in the current directory, overridable with `--config`. The built-in `DEFAULT_CONFIG` (in `config.py`) supplies defaults; user files layer on top and may `include:` other files. Values are read through [`Config.get_value()`](/architecture/config.md). + +# Schema + +## Top-level keys + +| Key | Type | Purpose | +|-----|------|---------| +| `folders` | map | Build path templates (see below). | +| `ext` | string | Output archive extension (`gz` default; `zip` on windows). | +| `cpython-modules` | CSV/list | Which [external modules](/modules/index.md) to auto-select. | +| `cpython-configure` | list | Extra `./configure` args (default: `--enable-optimizations`, `--with-lto`, `--with-ensurepip=upgrade`). | +| `cpython-clean-1st-pass` | list | Files removed before byte-compiling (~94 MB: tests, idle, 2to3). | +| `cpython-clean-2nd-pass` | list | Pycaches pruned after byte-compiling (~1.8 MB). | +| `cpython-compile-all` | bool | Whether to byte-compile the stdlib. | +| `cpython-additional-packages` | list | Extra pip packages to install into the build. | +| `cpython-use-github` | bool | Fetch the available version list from GitHub tags instead of python.org. | +| `include` | path | Layer another config file (a leading `+` puts it in front). | + +## `folders` + +All support path templates: `{build}`, `{family}`, `{version}`, `{abi_suffix}`. + +| Key | Default | Role | +|-----|---------|------| +| `build` | `build` | Root build folder. | +| `destdir` | `{build}` | `make install` DESTDIR. | +| `dist` | `dist` | Final tarball folder. | +| `logs` | `{build}/logs` | Per-component logs. | +| `sources` | `build/sources` | Downloaded tarball cache. | +| `ppp-marker` | `/ppp-marker/{version}{abi_suffix}` | In-progress install folder name. | + +See [build layout](/concepts/build-layout.md) for how these map onto disk. + +## Per-module keys + +Replace `{module}` with the lowercased module name (e.g. `openssl`): + +| Key | Purpose | +|-----|---------| +| `{module}-version` | Version to build. | +| `{module}-url` | Source URL (`$version` is substituted). | +| `{module}-src-suffix` | Archive extension when the URL lacks one. | +| `{module}-configure` | **Replace** the default configure args (`$deps_lib` substituted). | +| `{module}-http-headers` | Headers for the download (env vars expanded). | +| `{module}-patches` | Source patches to apply before compiling. | +| `{module}-debian` | Debian package name for dependency detection. | + +## Platform-specific overrides + +Sections named for a platform (`linux`, `macos`, `windows`) override generic keys; the most specific match for `PPG.target` wins. `MACOSX_DEPLOYMENT_TARGET` defaults to `13` (Ventura). + +# Examples + +```yaml +include: +pp-dev.yml + +folders: + build: "build/{family}-{version}{abi_suffix}" # keep builds per-version + +cpython-modules: libffi zlib xz bzip2 openssl uuid sqlite + +# Override a dependency version and source +libffi-version: 3.3 +openssl-url: https://my-openssl-mirror/openssl-$version.tar.gz +openssl-http-headers: + - Authorization: Bearer ${GITHUB_TOKEN} + +# Per-platform configure args +macos: + openssl-configure: --with-terminfo-dirs=/usr/share/terminfo +``` + +# Citations + +[1] [portable-python.yml — sample dev config](https://github.com/codrsquad/portable-python/blob/main/portable-python.yml) +[2] [src/portable_python/config.py — DEFAULT_CONFIG](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/config.py) +[3] [CONFIGURATION.md](https://github.com/codrsquad/portable-python/blob/main/CONFIGURATION.md) diff --git a/docs/guides/add-an-external-module.md b/docs/guides/add-an-external-module.md new file mode 100644 index 0000000..14f30f6 --- /dev/null +++ b/docs/guides/add-an-external-module.md @@ -0,0 +1,64 @@ +--- +type: Playbook +title: Add an External Module +description: Add a new statically-linked C library to CPython by creating a ModuleBuilder subclass and wiring it into the candidate list. +tags: [playbook, modules, contributing] +timestamp: 2026-06-23T00:00:00Z +--- + +# Add an External Module + +To statically link a new C library into CPython, add a [`ModuleBuilder`](/architecture/module-builder.md) subclass. The existing [external modules](/modules/external-modules.md) are the best templates. + +## Steps + +1. **Create the class** in `src/portable_python/external/xcpython.py`: + + ```python + class Mylib(ModuleBuilder): + """See https://docs.python.org/3/library/mymodule.html""" + + m_telltale = "{include}/mylib.h" # marker if system already has it + m_debian = "+libmylib-dev" # Linux build constraint (see telltale detection) + + @property + def url(self): + return self.cfg_url(self.version) or f"https://example.org/mylib-{self.version}.tar.gz" + + @property + def version(self): + return self.cfg_version("1.2.3") # default, overridable via mylib-version config + + def c_configure_args(self): + if args := self.cfg_configure(self.deps_lib_dir, self.deps_lib64_dir): + yield args + else: + yield "--enable-shared=no" + yield "--enable-static=yes" + + def _do_linux_compile(self): + self.run_configure("./configure", self.c_configure_args()) + self.run_make() + self.run_make("install") + ``` + +2. **Set the declarative attributes**: `m_telltale` (see [telltale detection](/concepts/telltale-detection.md)), `m_debian` with the right sigil, and `m_include`/`m_build_cwd` if needed. + +3. **Implement source resolution**: `url` and `version` (using `cfg_url`/`cfg_version` so config overrides work). + +4. **Implement the compile**: `_do_linux_compile()`. macOS reuses it unless you add `_do_macos_compile()`. + +5. **Register it** in `Cpython.candidate_modules()` (in `cpython.py`) so it becomes a candidate. + +6. **Add tests** in `tests/test_setup.py` (the suite runs in `--dryrun`, so it asserts the planned commands without compiling). + +## Tips + +- Prefer config flags over source patches — see the [no-patches principle](/overview.md). +- Use `--dryrun` to confirm the configure/make commands look right before a real build. +- If the library is only needed when explicitly selected, leave `auto_select_reason()` unset; if a build cannot succeed without it, return a short reason there. + +# Citations + +[1] [DEVELOP.md — Add a New External Module](https://github.com/codrsquad/portable-python/blob/main/DEVELOP.md) +[2] [src/portable_python/external/xcpython.py — existing modules](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/external/xcpython.py) diff --git a/docs/guides/build-a-portable-python.md b/docs/guides/build-a-portable-python.md new file mode 100644 index 0000000..4142002 --- /dev/null +++ b/docs/guides/build-a-portable-python.md @@ -0,0 +1,65 @@ +--- +type: Playbook +title: Build a Portable Python +description: Install portable-python, build a statically-linked python binary, and use it from any folder. +tags: [playbook, build, getting-started] +timestamp: 2026-06-23T00:00:00Z +--- + +# Build a Portable Python + +End-to-end: from installing the tool to running the binary it produces. + +## 1. Install the tool + +`portable-python` is a regular python CLI: + +```shell +pipx install portable-python # or: pickley install portable-python +portable-python --help +``` + +## 2. (Optional) Preview the build + +A dry run shows exactly what would happen — downloads, configure/make commands, and the selected [modules](/modules/index.md) — without compiling: + +```shell +portable-python --dryrun build 3.13.2 +``` + +Use [`build-report`](/cli/build-report.md) to confirm the module selection is buildable on this host: + +```shell +portable-python build-report 3.13.2 +``` + +## 3. Build + +```shell +cd some-temp-folder +portable-python build 3.13.2 +ls -l dist/cpython-3.13.2-*.tar.gz +``` + +This runs the full [`BuildSetup.compile()`](/architecture/build-setup.md) pipeline: external libraries first, then CPython, then finalize + validate, then compress into `dist/`. + +## 4. Use it + +The tarball is [portable](/concepts/portability.md) — unpack anywhere and run, no installer: + +```shell +tar -C ~/versions/ -xf dist/cpython-3.13.2-macos-arm64.tar.gz +~/versions/3.13.2/bin/python --version +``` + +## 5. Verify portability (optional) + +```shell +portable-python inspect ~/versions/3.13.2 +``` + +A clean exit means no non-portable dynamic-library references — see [`inspect`](/cli/inspect.md). + +# Citations + +[1] [README.rst — Building a portable cpython](https://github.com/codrsquad/portable-python/blob/main/README.rst) diff --git a/docs/guides/index.md b/docs/guides/index.md new file mode 100644 index 0000000..cd28233 --- /dev/null +++ b/docs/guides/index.md @@ -0,0 +1,9 @@ +# Guides + +Task-oriented playbooks for using and extending portable-python. + +* [Build a portable python](/guides/build-a-portable-python.md) - Install the tool, build a binary, and use it. +* [Add an external module](/guides/add-an-external-module.md) - Add a new C library to the static-link set. +* [Local development](/guides/local-development.md) - Dev venv, tests, dryrun, and building Linux binaries via Docker. + +See also the root [DEVELOP.md](https://github.com/codrsquad/portable-python/blob/main/DEVELOP.md) for the original notes these guides are based on. diff --git a/docs/guides/local-development.md b/docs/guides/local-development.md new file mode 100644 index 0000000..4df3732 --- /dev/null +++ b/docs/guides/local-development.md @@ -0,0 +1,70 @@ +--- +type: Playbook +title: Local Development +description: Set up a dev venv, run the tests, use dryrun mode, and build Linux binaries via Docker. +tags: [playbook, development, testing, docker] +timestamp: 2026-06-23T00:00:00Z +--- + +# Local Development + +## Dev venv + +```shell +uv sync +.venv/bin/portable-python list +.venv/bin/portable-python build-report 3.13.2 +``` + +## Running the tests + +The suite mocks `runez.run()` and builds in `--dryrun` mode, so it asserts the *planned* commands without actually compiling. Target is 100% coverage. + +```shell +tox # all envs (py310-314, coverage, docs, style) +tox -e py313 # one python version +tox -e style # lint check (ruff) +tox -e reformat # auto-fix formatting + +.venv/bin/pytest tests/ # directly +.venv/bin/pytest tests/test_build.py::test_build_rc -vv # a single test +``` + +CI runs `uvx --with tox-uv tox -e py`. + +## Dryrun mode + +`--dryrun` (`-n`) prints what would be done without doing it — invaluable for fast iteration and for understanding the build: + +```shell +portable-python --dryrun build 3.13.2 +``` + +## Debugging + +`portable-python` can be run under a debugger (e.g. PyCharm: debug `.venv/bin/portable-python` with parameters like `build-report 3.13.2`). Set breakpoints in the [core classes](/architecture/index.md) or in any `tests/` file. + +## Building a Linux binary via Docker + +Real Linux builds (not dry runs) are easiest in a container: + +```shell +docker build -t portable-python-jammy . +docker run -it -v./:/src/ portable-python-jammy /bin/bash +# inside the container: +portable-python build 3.13.2 +``` + +## Key dependencies + +| Package | Purpose | +|---------|---------| +| `click` | CLI framework. | +| `pyyaml` | [Configuration](/configuration/portable-python-yml.md) parsing. | +| `requests` / `urllib3` | HTTP downloads. | +| `runez` | Foundational utilities — file ops, system info, CLI decorators, logging, `Version`/`PythonSpec`. Check `runez` before reimplementing anything. | + +# Citations + +[1] [DEVELOP.md](https://github.com/codrsquad/portable-python/blob/main/DEVELOP.md) +[2] [CLAUDE.md — Common Commands, Testing](https://github.com/codrsquad/portable-python/blob/main/CLAUDE.md) diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..7946b2d --- /dev/null +++ b/docs/index.md @@ -0,0 +1,30 @@ +--- +okf_version: "0.1" +--- + +# Portable Python — Knowledge Bundle + +A knowledge bundle (in [OKF](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) format) describing `portable-python`: a CLI and Python library for compiling statically-linked, relocatable CPython binaries from source. + +Start with the [Overview](/overview.md), then drill into the area you need. + +## Sections + +* [Overview](/overview.md) - What portable-python does, its guiding principles, and how a build flows end to end. +* [Concepts](/concepts/index.md) - Domain ideas: portability, static linking, telltale detection, folder masking, build layout. +* [Architecture](/architecture/index.md) - The core classes and how they collaborate during a build. +* [CLI](/cli/index.md) - The `portable-python` subcommands and their options. +* [External Modules](/modules/index.md) - The C libraries compiled and statically linked into CPython. +* [Configuration](/configuration/index.md) - The `portable-python.yml` file, its sections and overrides. +* [Guides](/guides/index.md) - Playbooks for common tasks: building, extending, local development. + +## Conventions + +* Concept IDs are file paths minus `.md` (e.g. `architecture/ppg`). +* Cross-links are bundle-relative, beginning with `/` (the bundle root is this `docs/` folder). +* `resource:` frontmatter points to the canonical source file on GitHub when a concept maps to code. +* Change history lives in [log.md](/log.md). + +## Relationship to the root-level docs + +This bundle is additive. The repository's hand-written `README.rst`, `ARCHITECTURE.md`, `CONFIGURATION.md`, and `DEVELOP.md` remain authoritative for now; this bundle reorganizes and expands that material into discrete, linkable concepts. See [Overview](/overview.md#citations) for pointers back to those files. diff --git a/docs/log.md b/docs/log.md new file mode 100644 index 0000000..a3c788c --- /dev/null +++ b/docs/log.md @@ -0,0 +1,12 @@ +# Knowledge Bundle Update Log + +## 2026-06-23 + +* **Creation**: Established the portable-python knowledge bundle in OKF format. +* **Creation**: Added the [Overview](/overview.md). +* **Creation**: Added the [Concepts](/concepts/index.md) section (portability, static linking, telltale detection, folder masking, build layout). +* **Creation**: Added the [Architecture](/architecture/index.md) section covering the core classes (`PPG`, `BuildSetup`, `ModuleBuilder`, `PythonBuilder`, `Cpython`, `Config`, `PythonInspector`). +* **Creation**: Added the [CLI](/cli/index.md) reference for all subcommands. +* **Creation**: Added the [External Modules](/modules/index.md) reference. +* **Creation**: Added the [Configuration](/configuration/index.md) section. +* **Creation**: Added the [Guides](/guides/index.md) section (building, adding a module, local development). diff --git a/docs/modules/external-modules.md b/docs/modules/external-modules.md new file mode 100644 index 0000000..7695539 --- /dev/null +++ b/docs/modules/external-modules.md @@ -0,0 +1,59 @@ +--- +type: Reference +title: External Modules Reference +description: The C libraries compiled and statically linked into CPython, with default versions, telltale markers, and Linux build constraints. +resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/external/xcpython.py +tags: [reference, modules, external, cpython] +timestamp: 2026-06-23T00:00:00Z +--- + +# External Modules Reference + +These are the candidate external modules for CPython, in the order declared by `Cpython.candidate_modules()`. Each is a [`ModuleBuilder`](/architecture/module-builder.md) that compiles a C library statically into the shared `build/deps/` prefix before CPython is built. Default versions are overridable via [config](/configuration/portable-python-yml.md) (e.g. `openssl-version`). + +# Schema + +| Module | CPython feature | Default version | Telltale (`{include}/…`) | Linux dev pkg | +|--------|-----------------|-----------------|--------------------------|----------------| +| `LibFFI` | `ctypes` | 3.5.2 | `ffi.h`, `ffi/ffi.h` | `!libffi-dev` | +| `Zlib` | `zlib` | 1.3.2 | `zlib.h` | `!zlib1g-dev` | +| `Zstd` | compression (zstd) | 1.5.7 | `zstd.h` | `!libzstd-dev` | +| `Xz` | `lzma` | 5.8.3 | `lzma.h` | — | +| `Bzip2` | `bz2` | 1.0.8 | `bzlib.h` | — | +| `Readline` | `readline` | 8.3 | `readline/readline.h` | `-libreadline-dev` | +| `Openssl` | `ssl`, `hashlib` | 3.5.6 | `openssl/ssl.h` | — | +| `Sqlite` | `sqlite3` | 3.51.3 | `sqlite3.h` | `+libsqlite3-dev` | +| `Bdb` | `dbm.ndbm` | 6.2.32 | `dbm.h` | `libgdbm-compat-dev` | +| `Gdbm` | `dbm.gnu` | 1.26 | `gdbm.h` | `libgdbm-dev` | +| `Uuid` | `uuid` | 1.0.3 | `uuid/uuid.h` | `+uuid-dev` | +| `TkInter` | `tkinter` | 8.6.17 | `tk`, `tk.h` | `-tk-dev` | +| `Mpdec` | `decimal` | 4.0.1 | `mpdecimal.h` | `!libmpdec-dev` | + +## Linux dev-package sigils + +The leading character on the Debian package encodes a build constraint (see [telltale detection](/concepts/telltale-detection.md)): + +| Sigil | Meaning | +|-------|---------| +| `!` | Required to compile **at all** (`LibFFI`, `Zlib`, `Zstd`, `Mpdec`). | +| `+` | Required when the module is **selected** for static compile (`Sqlite`, `Uuid`). | +| `-` | Dev package's presence **breaks** static compilation (`Readline`, `TkInter`). | +| (none) | Detection/reporting only. | + +## Sub-modules and toolchain + +Some modules bundle their own dependencies as sub-modules (their `candidate_modules()`): + +- `Readline` → `Ncurses` (`ncursesw`). +- `TkInter` → `Tcl`, `Tk`, `Tix` (defined in `external/xtkinter.py`). +- A separate `Toolchain` builds `GettextTiny` to prevent `libintl` from being picked up out of `/usr/local` — complementing the macOS [folder mask](/concepts/folder-masking.md). + +## Per-module config + +Every module honors these config keys (replace `{module}` with the lowercased name, e.g. `openssl`): `{module}-version`, `{module}-url`, `{module}-src-suffix`, `{module}-configure`, `{module}-http-headers`, `{module}-patches`. See [configuration](/configuration/portable-python-yml.md). + +# Citations + +[1] [src/portable_python/external/xcpython.py](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/external/xcpython.py) +[2] [src/portable_python/external/xtkinter.py](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/external/xtkinter.py) +[3] [src/portable_python/cpython.py — Cpython.candidate_modules](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cpython.py) diff --git a/docs/modules/index.md b/docs/modules/index.md new file mode 100644 index 0000000..9201cf8 --- /dev/null +++ b/docs/modules/index.md @@ -0,0 +1,7 @@ +# External Modules + +The C libraries that get compiled statically and linked into CPython, so that the corresponding standard-library modules work in a [portable](/concepts/portability.md) build. + +* [External modules reference](/modules/external-modules.md) - The full table: default versions, telltales, and Linux build constraints. + +Each module is a [`ModuleBuilder`](/architecture/module-builder.md) subclass. The set of candidates CPython can link is declared in `Cpython.candidate_modules()`; which ones are actually built depends on config, the `--modules` flag, [telltale detection](/concepts/telltale-detection.md), and auto-selection. diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 0000000..eebbb56 --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,69 @@ +--- +type: Overview +title: Portable Python Overview +description: A CLI and library that compiles statically-linked, relocatable CPython binaries from source, and validates their portability. +resource: https://github.com/codrsquad/portable-python +tags: [overview, cpython, portability, build] +timestamp: 2026-06-23T00:00:00Z +--- + +# Overview + +`portable-python` compiles CPython from source into a binary that is **statically linked** and **relocatable**: the resulting tarball can be unpacked into any folder and used immediately, with no installer and no dependency on system shared libraries. It targets **Linux and macOS** (Intel and Apple Silicon). Windows is not supported. + +It is both: + +- A **CLI** with one entry point, `portable-python` (see [CLI](/cli/index.md)). +- A **Python library** (`from portable_python import BuildSetup`) for driving builds or inspections programmatically. + +## What a build produces + +``` +dist/cpython-3.13.2-macos-arm64.tar.gz +``` + +Unpack and run, no installation step: + +```shell +tar -C ~/versions/ -xf dist/cpython-3.13.2-macos-arm64.tar.gz +~/versions/3.13.2/bin/python --version +``` + +## How a build flows + +1. **Resolve** the requested [`PythonSpec`](/architecture/build-setup.md) (family + version) and target platform via the [`PPG`](/architecture/ppg.md) singleton. +2. **Select** which [external modules](/modules/index.md) to compile, based on config and [telltale detection](/concepts/telltale-detection.md). +3. **Compile external C libraries** first (openssl, zlib, xz, …) into a shared `build/deps/` prefix — see [build layout](/concepts/build-layout.md). +4. **Compile CPython** ([`Cpython`](/architecture/cpython.md)), pointing its `configure`/`make` at `build/deps/` via injected environment variables (CPATH, LDFLAGS, …). +5. **Finalize**: clean test files, byte-compile, [relativize paths](/concepts/static-linking.md) so the install is relocatable. +6. **Validate** portability with the [`PythonInspector`](/architecture/python-inspector.md), then compress into `dist/`. + +The whole pipeline is coordinated by [`BuildSetup`](/architecture/build-setup.md). + +## Guiding principles + +- **One job, done well.** Compile a portable python, validate it really is portable, drop the result in `dist/`. +- **No source patches.** C compilation relies solely on `configure`/`make` flags (e.g. `--enable-shared=no`), never on modifying upstream source. +- **Validated builds.** A large part of the effort is the [`inspect`](/cli/inspect.md) capability that proves an installation is portable (and explains why if it isn't). +- **Only recent, non-EOL Pythons.** No historical support; older versions are dropped without notice as the tool evolves. +- **Pure Python, testable.** No shell scripts. 100% test coverage, a `--dryrun` mode for fast iteration, and the ability to run under a debugger. + +# Examples + +Dry-run a build to see exactly what would happen without doing it: + +```shell +portable-python --dryrun build 3.13.2 +``` + +Inspect any existing python for portability: + +```shell +portable-python inspect /usr/bin/python3 +``` + +# Citations + +[1] [README.rst](https://github.com/codrsquad/portable-python/blob/main/README.rst) - Project README and guiding principles. +[2] [ARCHITECTURE.md](https://github.com/codrsquad/portable-python/blob/main/ARCHITECTURE.md) - Class hierarchy and design patterns. +[3] [CLAUDE.md](https://github.com/codrsquad/portable-python/blob/main/CLAUDE.md) - Repository guidance and architecture summary. diff --git a/requirements.txt b/requirements.txt index 12ab1aa..812d78a 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,20 +1,20 @@ # This file was autogenerated by uv via the following command: # uv pip compile pyproject.toml --universal -o requirements.txt --python-version 3.10 --group dev -certifi==2026.4.22 +certifi==2026.6.17 # via requests charset-normalizer==3.4.7 # via requests -click==8.3.3 +click==8.4.1 # via portable-python (pyproject.toml) colorama==0.4.6 ; sys_platform == 'win32' # via # click # pytest -coverage==7.13.5 +coverage==7.14.2 # via pytest-cov exceptiongroup==1.3.1 ; python_full_version < '3.11' # via pytest -idna==3.13 +idna==3.18 # via requests iniconfig==2.3.0 # via pytest @@ -26,13 +26,13 @@ pluggy==1.6.0 # pytest-cov pygments==2.20.0 # via pytest -pytest==9.0.3 +pytest==9.1.1 # via pytest-cov pytest-cov==7.1.0 # via portable-python (pyproject.toml:dev) pyyaml==6.0.3 # via portable-python (pyproject.toml) -requests==2.33.1 +requests==2.34.2 # via portable-python (pyproject.toml) runez==5.9.1 # via portable-python (pyproject.toml) @@ -42,7 +42,7 @@ tomli==2.4.1 ; python_full_version <= '3.11' # pytest typing-extensions==4.15.0 ; python_full_version < '3.11' # via exceptiongroup -urllib3==2.6.3 +urllib3==2.7.0 # via # portable-python (pyproject.toml) # requests diff --git a/scripts/check_okf.py b/scripts/check_okf.py new file mode 100755 index 0000000..b889b87 --- /dev/null +++ b/scripts/check_okf.py @@ -0,0 +1,124 @@ +#!/usr/bin/env python3 +""" +Check an OKF (Open Knowledge Format) knowledge bundle for conformance. + +OKF spec: https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md + +Verifies: + - every non-reserved .md file has a parseable YAML frontmatter block with a non-empty `type` + - index.md files carry no frontmatter, except the bundle-root index.md (which declares `okf_version`) + - log.md carries no frontmatter + - markdown links to local .md files (bundle-relative "/..." or relative "./...") resolve to real files + +Exits non-zero if any conformance error or broken local link is found. + +Usage: + scripts/check_okf.py [BUNDLE_DIR] # default: docs +""" + +import argparse +import collections +import os +import re +import sys + +# Matches markdown links ending in .md, with an optional #anchor (external http(s)/mailto links are skipped below) +LINK_RE = re.compile(r"\]\((?P[^)\s#]+\.md)(?:#[^)]*)?\)") +TYPE_RE = re.compile(r"^type:\s*(?P\S.*)$", re.M) + + +def frontmatter(text): + """Return the YAML frontmatter block (without the --- fences), or None if absent.""" + if not text.startswith("---\n"): + return None + + end = text.find("\n---", 4) + if end == -1: + return None + + return text[4:end] + + +def check_bundle(root): + """Yield human-readable problem strings for the OKF bundle rooted at `root`.""" + md_files = [] + for dirpath, _, filenames in os.walk(root): + for name in filenames: + if name.endswith(".md"): + md_files.append(os.path.join(dirpath, name)) + + md_files.sort() + present = {os.path.relpath(p, root).replace(os.sep, "/") for p in md_files} + types = collections.Counter() + problems = [] + + for path in md_files: + rel = os.path.relpath(path, root).replace(os.sep, "/") + name = os.path.basename(path) + with open(path, encoding="utf-8") as fh: + text = fh.read() + + fm = frontmatter(text) + if name == "index.md": + if rel == "index.md": # bundle root: the one index.md allowed to have frontmatter + if not fm or "okf_version" not in fm: + problems.append(f"{rel}: bundle-root index.md must declare okf_version") + elif fm is not None: + problems.append(f"{rel}: non-root index.md must not have frontmatter") + elif name == "log.md": + if fm is not None: + problems.append(f"{rel}: log.md must not have frontmatter") + elif fm is None: + problems.append(f"{rel}: missing frontmatter") + else: + m = TYPE_RE.search(fm) + if m: + types[m.group("type").strip()] += 1 + else: + problems.append(f"{rel}: missing non-empty 'type' field") + + for m in LINK_RE.finditer(text): + target = m.group("target") + if target.startswith(("http://", "https://", "mailto:")): + continue # external link, not ours to resolve + + if target.startswith("/"): + resolved = target.lstrip("/") + else: + resolved = os.path.normpath(os.path.join(os.path.dirname(rel), target)).replace(os.sep, "/") + + if resolved not in present: + problems.append(f"{rel}: broken link -> {target}") + + return md_files, types, problems + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("bundle", nargs="?", default="docs", help="Path to the bundle root (default: docs)") + args = parser.parse_args(argv) + + if not os.path.isdir(args.bundle): + sys.exit(f"Not a directory: {args.bundle}") + + md_files, types, problems = check_bundle(args.bundle) + + print(f"Bundle: {args.bundle}") + print(f"Markdown files: {len(md_files)}") + if types: + print("Concept types:") + for concept_type, count in sorted(types.items()): + print(f" {count:2d} {concept_type}") + + if problems: + print(f"\n{len(problems)} problem(s):") + for problem in problems: + print(f" ERROR {problem}") + return 1 + + print("\nOK: bundle is OKF-conformant, all local links resolve.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From acaae77b88fc67551d39f44c4a7f37230947761c Mon Sep 17 00:00:00 2001 From: Zoran Simic Date: Tue, 23 Jun 2026 08:20:29 -0700 Subject: [PATCH 2/5] More docs --- ARCHITECTURE.md | 74 --------------- CLAUDE.md | 86 ++++------------- CONFIGURATION.md | 53 ----------- DEVELOP.md | 111 ---------------------- docs/architecture/build-setup.md | 5 - docs/architecture/config.md | 13 +-- docs/architecture/cpython.md | 8 +- docs/architecture/index.md | 3 +- docs/architecture/module-builder.md | 6 -- docs/architecture/ppg.md | 10 +- docs/architecture/python-builder.md | 5 - docs/architecture/python-inspector.md | 7 +- docs/cli/build-report.md | 5 - docs/cli/build.md | 5 - docs/cli/diagnostics.md | 5 - docs/cli/inspect.md | 6 -- docs/cli/lib-auto-correct.md | 6 -- docs/cli/list.md | 6 -- docs/cli/recompress.md | 5 - docs/concepts/build-layout.md | 9 +- docs/concepts/folder-masking.md | 7 +- docs/concepts/index.md | 1 + docs/concepts/portability.md | 4 - docs/concepts/ppp-marker.md | 24 +++++ docs/concepts/static-linking.md | 5 - docs/concepts/telltale-detection.md | 5 - docs/configuration/index.md | 2 +- docs/configuration/portable-python-yml.md | 9 +- docs/guides/add-a-config-option.md | 20 ++++ docs/guides/add-an-external-module.md | 5 - docs/guides/build-a-portable-python.md | 4 - docs/guides/bump-python-support.md | 18 ++++ docs/guides/ci-cd.md | 31 ++++++ docs/guides/fix-a-portability-issue.md | 18 ++++ docs/guides/index.md | 6 +- docs/guides/local-development.md | 17 ++-- docs/index.md | 12 +-- docs/log.md | 8 ++ docs/modules/external-modules.md | 43 ++++----- docs/overview.md | 7 -- 40 files changed, 191 insertions(+), 483 deletions(-) delete mode 100644 ARCHITECTURE.md delete mode 100644 CONFIGURATION.md delete mode 100644 DEVELOP.md create mode 100644 docs/concepts/ppp-marker.md create mode 100644 docs/guides/add-a-config-option.md create mode 100644 docs/guides/bump-python-support.md create mode 100644 docs/guides/ci-cd.md create mode 100644 docs/guides/fix-a-portability-issue.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md deleted file mode 100644 index ab3cfb4..0000000 --- a/ARCHITECTURE.md +++ /dev/null @@ -1,74 +0,0 @@ -## Architecture - -### Core Classes Hierarchy - -``` -ModuleBuilder (abstract base) -├── PythonBuilder (abstract, extends ModuleBuilder) -│ └── Cpython (concrete implementation in cpython.py) -└── External modules (in external/xcpython.py, xtkinter.py) - ├── Bdb, Bzip2, Gdbm, LibFFI, Mpdec, Openssl, Readline - ├── Sqlite, Uuid, Xz, Zlib, Zstd, TkInter - -BuildSetup (coordinates overall compilation) -├── BuildContext (handles isolation hacks on macOS) -├── Folders (manages build directory structure) -└── ModuleCollection (manages module selection and dependencies) - -PPG (Global state singleton) -├── config: Config (YAML configuration) -├── target: PlatformId (target OS/arch) -├── cpython: CPythonFamily (available versions) -└── families: dict (extensible family support) - -Config -├── Folders (build/dist/sources/logs directories) -├── target platform detection -├── YAML config merging (platform-specific overrides) - -PythonInspector -├── Inspect Python installation for portability -├── LibAutoCorrect (rewrite paths to be relative) -└── Report system and shared lib detection - -Tracker/Trackable -├── Categorizes found issues/objects by type -└── Provides detailed reports -``` - -### Key Design Patterns - -1. **Hierarchical Module Building**: External modules are compiled first, then Python itself, using the same build framework. - -2. **Environment Variable Injection**: `xenv_*` methods dynamically provide environment variables (CPATH, LDFLAGS, PATH, etc.) for compilation. - -3. **Platform Abstraction**: `PPG.target` (PlatformId) encapsulates platform logic. Compile methods named `_do__compile()` dispatch to platform-specific implementations. - -4. **Configuration Precedence**: YAML config supports platform-specific overrides (windows.ext, macos.env, etc.). Most specific setting wins. - -5. **Folder Masking (macOS)**: On macOS, `/usr/local` is temporarily masked with a RAM disk to prevent accidental dynamic linking. - -6. **Build Isolation**: All external modules compiled to a shared `build/deps/` folder, Python finds them via CPATH/LDFLAGS. - -7. **Lazy Version Fetching**: `VersionFamily` caches available versions, fetching from python.org on first access. - -8. **Telltale Detection**: Modules check for marker files (`m_telltale`) to determine if they're already available on the system (as shared libs). - -9. **Log Aggregation**: Each module logs to a separate file (`01-openssl.log`, `02-cpython.log`, etc.) under `build/logs/`. - - -## CI/CD - -### GitHub Actions - -**tests.yml** (main branch & PRs): -- Matrix test on py3.10, 3.11, 3.12, 3.13, 3.14 -- Runs: `uvx --with tox-uv tox -e py` -- Coverage upload to coveralls.io (parallel, then finish) -- Linter job: docs + style checks on 3.14 - -**release.yml** (version tags v*): -- Triggers on `v[0-9]*` tags -- Runs all tests + docs + style -- Builds distribution with `uv build` -- Publishes to PyPI via trusted publishing (OIDC) diff --git a/CLAUDE.md b/CLAUDE.md index 23d363d..f57975e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,78 +1,24 @@ # CLAUDE.md -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. +Guidance for Claude Code working in this repository. -## What This Project Does +`portable-python` is a CLI and Python library that compiles portable (statically-linked, relocatable) CPython binaries from source — see [`docs/overview.md`](docs/overview.md) for the full picture. -portable-python is a CLI tool and Python library for compiling portable CPython binaries from source. -Binaries are statically linked so they can be extracted to any folder and used without installation. -Supports Linux and macOS (not Windows). +## Documentation lives in `docs/` -## Common Commands +The authoritative, self-contained documentation is the OKF knowledge bundle under **`docs/`** — start at [`docs/index.md`](docs/index.md). **Don't duplicate it in this file — link to it.** When you change behavior, update the matching `docs/` entry and add a line to [`docs/log.md`](docs/log.md). -```bash -# Run all tests (tox manages envs for py310-314, coverage, docs, style) -tox +| Looking for… | Go to | +|--------------|-------| +| What it does, guiding principles, end-to-end build flow | [`docs/overview.md`](docs/overview.md) | +| Core classes & how they collaborate | [`docs/architecture/`](docs/architecture/index.md) | +| Concepts: portability, static linking, telltale detection, folder masking, ppp-marker, build layout | [`docs/concepts/`](docs/concepts/index.md) | +| CLI commands & options | [`docs/cli/`](docs/cli/index.md) | +| External modules (the statically-linked C libraries) | [`docs/modules/`](docs/modules/index.md) | +| Configuration (`portable-python.yml`) | [`docs/configuration/`](docs/configuration/index.md) | +| Dev setup, tests, code style, debugging, Docker, CI/CD, and common tasks | [`docs/guides/`](docs/guides/index.md) | -# Run tests for a single Python version -tox -e py313 +## Working in this repo -# Run a single test -.venv/bin/pytest tests/test_build.py::test_build_rc -vv - -# Lint check / auto-fix -tox -e style -tox -e reformat - -# Run tests directly (from venv) -.venv/bin/pytest tests/ - -# CI uses: uvx --with tox-uv tox -e py -``` - -## Architecture - -**Key classes:** - -- `PPG` (versions.py) — Global singleton holding config, target platform, and version families. All modules access shared state through this. -- `BuildSetup` (__init__.py) — Coordinates overall compilation: downloads sources, builds external modules, then CPython. -- `ModuleBuilder` (__init__.py) — Abstract base for anything that gets compiled. Both external C modules and Python itself extend this. -- `PythonBuilder` (__init__.py) — Extends ModuleBuilder for Python implementations. -- `Cpython` (cpython.py) — Concrete PythonBuilder that handles CPython's configure/make/install, optimization, and finalization. -- `Config` (config.py) — Loads and merges YAML configuration (portable-python.yml) with platform-specific overrides. -- `PythonInspector` (inspector.py) — Validates portability of a built Python by checking shared library dependencies and paths. - -**External modules** (external/xcpython.py): `Bdb`, `Bzip2`, `Gdbm`, `LibFFI`, `Mpdec`, `Openssl`, `Readline`, `Sqlite`, `Uuid`, `Xz`, `Zlib`, `Zstd` — each is a ModuleBuilder subclass that compiles a C library statically before CPython is built. - -**Key patterns:** - -- Platform-specific compile logic uses `_do_linux_compile()` / `_do_macos_compile()` method dispatch. -- Environment injection: `xenv_*` methods provide CPATH, LDFLAGS, PATH etc. for compilation. -- On macOS, `/usr/local` is masked with a RAM disk (`FolderMask`) to prevent accidental dynamic linking. -- External modules compile to a shared `build/deps/` prefix; CPython finds them via CPATH/LDFLAGS. -- Telltale detection: modules check for marker files (`m_telltale`) to determine if system already has the library. -- No patches to upstream CPython source — relies solely on configure flags. - -**runez** is the foundational utility library (file ops, system info, CLI decorators, logging, Version/PythonSpec types). Check runez before reimplementing anything. - -**Additional pointers:** - -- `ModuleCollection.selected` contains only the modules chosen for a build — not all candidates. -- Build logs go to `build/logs/NN-modulename.log` (e.g. `01-openssl.log`, `02-cpython.log`). -- YAML config supports platform-specific overrides and path templates — see CONFIGURATION.md. -- See ARCHITECTURE.md for class hierarchy and design patterns, DEVELOP.md for common tasks and dependencies. - -## Testing - -- pytest with 100% code coverage target -- Tests mock `runez.run()` to avoid actual compilation — uses `--dryrun` mode -- `conftest.py` provides a `cli` fixture (from runez) and forbids HTTP calls (`GlobalHttpCalls.forbid()`) -- Sample YAML configs in `tests/sample-config*.yml` for testing configuration parsing - -## Linting - -Ruff handles all linting and formatting. Key settings in pyproject.toml: -- Line length: 140 -- McCabe complexity: max 18 -- Security checks (S rules) disabled in tests -- Numpy-style docstrings +- Check **runez** before reimplementing file / system / CLI / logging / version helpers — see [`docs/guides/local-development.md`](docs/guides/local-development.md). +- The docs use the OKF format; keep the bundle conformant (`scripts/check_okf.py docs`) when you edit it. diff --git a/CONFIGURATION.md b/CONFIGURATION.md deleted file mode 100644 index 5dcc93d..0000000 --- a/CONFIGURATION.md +++ /dev/null @@ -1,53 +0,0 @@ -# Configuration (portable-python.yml) - -portable-python is configured via a YAML file (default: `portable-python.yml` in current directory, override with `--config`). - - -## Key Sections - -### folders - -- `build`, `dist`, `sources`, `logs`, `destdir`, `ppp-marker` -- All support path templates: `{build}`, `{version}`, `{abi_suffix}` - -### cpython-modules (CSV) - -Which external modules to auto-select. -Default is configured in `DEFAULT_CONFIG` (see `src/portable_python/config.py`). -Examples: openssl, zlib, xz, sqlite, bzip2, gdbm, libffi, readline, uuid - -### cpython-configure (list) - -Extra `./configure` args for CPython. -Default includes: `--enable-optimizations`, `--with-lto`, `--with-ensurepip=upgrade` - -### cpython-clean-1st-pass (list) - -Files to remove before `compileall` — removes test files, idle, 2to3 (~94 MB savings). - -### cpython-clean-2nd-pass (list) - -Files to remove after `compileall` — removes pycaches for seldom-used libs (~1.8 MB savings). - - -## Per-module config - -Each external module can be customized with these keys (replace `{module}` with the module name, e.g. `openssl`): - -| Key | Purpose | -|-----|---------| -| `{module}-version` | Version to use | -| `{module}-url` | URL to download from | -| `{module}-src-suffix` | File extension if not in URL | -| `{module}-configure` | Custom configure args | -| `{module}-http-headers` | HTTP headers for download | -| `{module}-patches` | File patches to apply | -| `{module}-debian` | Package name on Debian (for dependency detection) | - - -## Platform-specific overrides - -Configuration supports platform-specific sections (e.g. `windows.ext`, `macos.env`, etc.). -The most specific setting wins. - -`MACOSX_DEPLOYMENT_TARGET` defaults to 13 (Ventura). diff --git a/DEVELOP.md b/DEVELOP.md deleted file mode 100644 index 08ec5f1..0000000 --- a/DEVELOP.md +++ /dev/null @@ -1,111 +0,0 @@ -# Local development - -Create a dev venv: - -```shell -uv sync -``` - -You can then run `portable-python` from that venv: - -```shell -.venv/bin/portable-python list -.venv/bin/portable-python build-report 3.10.5 -``` - - -# Run the tests - -If you have tox, just run: `tox` to run all the tests. You can also run: -- `tox -e py313` to run with just one python version -- `tox -e style` to check PEP8 formatting -- etc - -If you don't have tox, you can run the tests with: `.venv/bin/pytest tests/` - -You can also run any of the `tests/` in IDEs such as PyCharm or VSCode. - -For example in PyCharm, just make sure that `pytest` is selected as "Default test runner" -in Preferences -> Tools -> Python Integrated Tools. -Then right-click on any file in `tests/` -(or in any function `test_...` function within a `test_*` file) -and select "Debug pytest in ..." - -You can set breakpoints as well during such test runs. - - -# Running in the debugger - -You can easily run `portable-python` in a debugger. -In PyCharm for example, you would simply browse to `.venv/bin/portable-python` -then right-click and select "Debug portable-python". -You can then edit the build/run configuration in PyCharm, add some "Parameters" to it, -like for example `build-report 3.13.2`, and then set breakpoints wherever you like. - -There is a `--dryrun` mode that can come in very handy for rapid iterations. - - -# Building a linux binary via docker - -Build a docker image, for example using the provided sample `Dockerfile`: - -```shell -docker build -t portable-python-jammy . -``` - -Run the docker image, with a folder `/src/` mounted to point to: - -```shell -docker run -it -v./:/src/ portable-python-jammy /bin/bash -``` - -Now inside docker, you run a build: - -```shell -portable-python build 3.13.2 -``` - - -# Key Dependencies - -| Package | Version | Purpose | -|---------|---------|---------| -| click | <9 | CLI framework | -| pyyaml | <7 | Configuration parsing | -| requests | <3 | HTTP downloads | -| runez | <6 | Utilities (file ops, system, logging) | -| urllib3 | <3 | HTTP transport | -| pytest-cov | - | Coverage reporting (dev only) | - -**runez** is central: provides file ops (`ls_dir`, `touch`, `compress`/`decompress`), system info (platform detection), CLI decorators (`@click.group`), logging, and version handling (`Version`, `PythonSpec`). - - -# Common Tasks - -## Add a New External Module - -1. Create class in `src/portable_python/external/xcpython.py` extending `ModuleBuilder` -2. Set `m_name`, `m_telltale`, `m_debian`, `version` property -3. Implement `url` property (or override `_do_linux_compile()` / `_do_macos_compile()`) -4. Add to `Cpython.candidate_modules()` if it's a CPython sub-module -5. Add tests in `tests/test_setup.py` - -## Add a New Config Option - -1. Update `DEFAULT_CONFIG` in `src/portable_python/config.py` -2. Use `PPG.config.get_value("key")` to retrieve it in code -3. Add tests to `tests/test_setup.py` - -## Fix a Portability Issue - -1. Run `portable-python inspect ` to diagnose -2. If lib is being dynamically linked, add to module list or update isolation -3. Use `LibAutoCorrect.run()` logic (or extend it) to fix paths -4. Add test case to `tests/test_inspector.py` - -## Bump Python Support - -1. Update `pyproject.toml` classifiers and `requires-python` -2. Update `.github/workflows/tests.yml` matrix -3. Update `CPythonFamily.min_version` if needed -4. Run full test matrix with `tox` diff --git a/docs/architecture/build-setup.md b/docs/architecture/build-setup.md index ef291ae..f025954 100644 --- a/docs/architecture/build-setup.md +++ b/docs/architecture/build-setup.md @@ -2,7 +2,6 @@ type: Class title: BuildSetup description: Coordinates the overall compilation — resolves the python spec, prepares folders, builds external modules then CPython, validates, and compresses the result. -resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py tags: [class, build, coordinator] timestamp: 2026-06-23T00:00:00Z --- @@ -45,7 +44,3 @@ setup.compile() ## Spec validation `BuildSetup` insists on a **full** version: a spec like `3.13` is rejected — you must give `3.13.2`. `"latest"` (or empty) resolves to `PPG.cpython.latest`. Invalid specs abort early with a red error. - -# Citations - -[1] [src/portable_python/__init__.py — BuildSetup](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py) diff --git a/docs/architecture/config.md b/docs/architecture/config.md index 5ed4074..353314a 100644 --- a/docs/architecture/config.md +++ b/docs/architecture/config.md @@ -2,7 +2,6 @@ type: Class title: Config & Folders description: Config loads and merges YAML configuration with platform-specific overrides; Folders resolves the templated build/dist/sources paths. -resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/config.py tags: [class, configuration, folders, yaml] timestamp: 2026-06-23T00:00:00Z --- @@ -29,13 +28,15 @@ timestamp: 2026-06-23T00:00:00Z | Member | Purpose | |--------|---------| | `build_folder`, `components`, `deps`, `sources`, `logs`, `dist`, `destdir` | Resolved component paths — see [build layout](/concepts/build-layout.md). | -| `ppp-marker` | Template for the in-progress install folder name. | +| `ppp-marker` | The [ppp-marker](/concepts/ppp-marker.md) install prefix; equals `--prefix` for non-portable builds. | | `formatted(text)` | Expand `{build}`, `{version}`, `{abi_suffix}`, … placeholders. | | `resolved_destdir(relative_path)` | Compose a path under the marker'd destdir. | ## Precedence & overrides -Configuration merges multiple sources. The built-in `DEFAULT_CONFIG` provides sane defaults; user files (default `portable-python.yml`, plus any `include:`d files) layer on top. Within a file, **platform-specific** sections override generic ones: +Configuration merges multiple sources. The built-in `DEFAULT_CONFIG` provides sane defaults; +user files (default `portable-python.yml`, plus any `include:`d files) layer on top. +Within a file, **platform-specific** sections override generic ones: ```yaml ext: gz # generic default @@ -47,9 +48,3 @@ macos: ``` `get_value(..., by_platform=True)` returns the most specific match for the current `PPG.target`. - -# Citations - -[1] [src/portable_python/config.py — Config, DEFAULT_CONFIG](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/config.py) -[2] [src/portable_python/versions.py — Folders](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/versions.py) -[3] [CONFIGURATION.md](https://github.com/codrsquad/portable-python/blob/main/CONFIGURATION.md) diff --git a/docs/architecture/cpython.md b/docs/architecture/cpython.md index 4e16028..23ad159 100644 --- a/docs/architecture/cpython.md +++ b/docs/architecture/cpython.md @@ -2,7 +2,6 @@ type: Class title: Cpython description: The concrete PythonBuilder that compiles CPython — configure/make/install, optimization flags, and the finalize step that makes the install relocatable. -resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cpython.py tags: [class, cpython, build, finalize] timestamp: 2026-06-23T00:00:00Z --- @@ -29,7 +28,7 @@ CPython is configured with, by default, `--enable-optimizations`, `--with-lto`, ## Finalize: making it relocatable -After `make install` the `_finalize()` step is what turns a normal install into a [portable](/concepts/portability.md) one: +After `make install` the `_finalize()` step is what turns a normal install into a [portable](/concepts/portability.md) one. For a portable build the interpreter's prefix is still the placeholder [`/ppp-marker/{version}`](/concepts/ppp-marker.md), so finalize rewrites those baked-in absolute paths to be relative: - **Clean passes** — remove tests/idle/2to3 before byte-compiling (`cpython-clean-1st-pass`, ~94 MB) and prune seldom-used pycaches after (`cpython-clean-2nd-pass`, ~1.8 MB). - **Byte-compile** the standard library (`cpython-compile-all`). @@ -37,8 +36,3 @@ After `make install` the `_finalize()` step is what turns a normal install into - `_relativize_sysconfig()` (`RelSysConf`) — rewrite absolute paths in the `sysconfig` data so the install resolves relative to its own location. - `_apply_pep668()` — mark the environment per PEP 668. - `_validate_venv_creation()` / `_validate_venv_module()` — sanity-check that `venv` works in the finished build. - -# Citations - -[1] [src/portable_python/cpython.py](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cpython.py) -[2] [src/portable_python/config.py — cpython-* defaults](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/config.py) diff --git a/docs/architecture/index.md b/docs/architecture/index.md index cffa058..8f80a8d 100644 --- a/docs/architecture/index.md +++ b/docs/architecture/index.md @@ -1,7 +1,6 @@ # Architecture -The core classes and how they collaborate during a build. See [ARCHITECTURE.md](https://github.com/codrsquad/portable-python/blob/main/ARCHITECTURE.md) for the original hierarchy diagram. - +The core classes and how they collaborate during a build. ## Global state & configuration * [PPG](/architecture/ppg.md) - Global singleton holding config, target platform, and version families. diff --git a/docs/architecture/module-builder.md b/docs/architecture/module-builder.md index 827a305..acbf75f 100644 --- a/docs/architecture/module-builder.md +++ b/docs/architecture/module-builder.md @@ -2,7 +2,6 @@ type: Class title: ModuleBuilder description: Abstract base for everything that gets compiled — external C libraries and CPython itself — providing the common download/configure/make/install flow and environment injection. -resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py tags: [class, abstract, build, module] timestamp: 2026-06-23T00:00:00Z --- @@ -52,8 +51,3 @@ Each `xenv_*` method supplies one environment variable for the compile, pointing ## Adding a module To add a new external module you subclass `ModuleBuilder`, set the `m_*` attributes, implement `url`/`version`, and implement `_do_linux_compile()` (macOS reuses it unless overridden). See the [guide](/guides/add-an-external-module.md). - -# Citations - -[1] [src/portable_python/__init__.py — ModuleBuilder](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py) -[2] [src/portable_python/external/xcpython.py — concrete modules](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/external/xcpython.py) diff --git a/docs/architecture/ppg.md b/docs/architecture/ppg.md index 47cef10..9749241 100644 --- a/docs/architecture/ppg.md +++ b/docs/architecture/ppg.md @@ -2,14 +2,15 @@ type: Class title: PPG description: Global singleton holding shared configuration, the target platform, and the registry of supported python version families. -resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/versions.py tags: [class, global-state, singleton, versions] timestamp: 2026-06-23T00:00:00Z --- # PPG -`PPG` is the global state holder for portable-python. Every module reaches shared state — configuration, the target platform, and version families — through `PPG`'s class attributes and classmethods. It is defined in `versions.py`. +`PPG` ("**P**ortable **P**ython **G**lobals") is the global state holder for portable-python. Every module reaches shared state — configuration, the target platform, and version families — through `PPG`'s class attributes and classmethods. It is defined in `versions.py`. + +Like [`ppp-marker`](/concepts/ppp-marker.md), the name is a deliberately uncommon character sequence chosen for searchability: grepping the codebase for `PPG` surfaces only its own usages, with no collisions in upstream CPython source or third-party libraries. # Schema @@ -36,8 +37,3 @@ A `VersionFamily` (e.g. `CPythonFamily`) knows how to **list available versions* ## Why a singleton The build touches global, process-wide facts: which config file is active, what platform we are targeting, and which python versions exist. Centralizing them in `PPG` avoids threading that state through every constructor, and gives the CLI a single place ([`main`](/cli/index.md)) to initialize it. - -# Citations - -[1] [src/portable_python/versions.py](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/versions.py) -[2] [CLAUDE.md — Key classes](https://github.com/codrsquad/portable-python/blob/main/CLAUDE.md) diff --git a/docs/architecture/python-builder.md b/docs/architecture/python-builder.md index 4ed1796..9fe8837 100644 --- a/docs/architecture/python-builder.md +++ b/docs/architecture/python-builder.md @@ -2,7 +2,6 @@ type: Class title: PythonBuilder description: A ModuleBuilder specialization for python implementations — adds module selection, the python install layout, and helpers to run the freshly built interpreter. -resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py tags: [class, abstract, python, build] timestamp: 2026-06-23T00:00:00Z --- @@ -32,7 +31,3 @@ A `PythonBuilder` owns a `ModuleCollection`, which models the set of candidate e - `selected` — only the modules chosen for *this* build (config + `--modules` + auto-selection). This is the list that actually gets compiled. - `auto_selected` — modules force-selected because a build can't succeed without them (each module's `auto_select_reason()`). - `report()` / `report_rows()` — the human-readable table shown by [`build-report`](/cli/build-report.md). - -# Citations - -[1] [src/portable_python/__init__.py — PythonBuilder, ModuleCollection](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py) diff --git a/docs/architecture/python-inspector.md b/docs/architecture/python-inspector.md index 8d5ccda..24f4d4f 100644 --- a/docs/architecture/python-inspector.md +++ b/docs/architecture/python-inspector.md @@ -2,7 +2,6 @@ type: Class title: PythonInspector description: Validates the portability of a python installation by parsing the dynamic-library dependencies of every executable and .so, and reporting any non-portable references. -resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/inspector.py tags: [class, inspection, validation, portability] timestamp: 2026-06-23T00:00:00Z --- @@ -40,6 +39,7 @@ if problem: | `LibType` | Enum classifying a reference (system, relative/self, problematic, …). | | `FullSoReport` | The complete scan and its `get_problem()` verdict. | | `LibAutoCorrect` | Rewrites absolute lib references in binaries to **relative** form (`@loader_path` / `$ORIGIN`), making an install relocatable. Exposed via [`lib-auto-correct`](/cli/lib-auto-correct.md). | +| `Tracker` / `Trackable` | Categorize the found objects (libs, problems) by type and render the detailed reports (`tracking.py`); `SoInfo` and `CLibInfo` are `Trackable`. | ## How it detects problems @@ -50,8 +50,3 @@ For each binary it lists dynamic dependencies, then classifies every reference: - **Absolute, non-system** reference → a portability problem; for a portable build this fails the inspection. `LibAutoCorrect` is also used **during** the build's finalize step to convert references to relative form before the portable check runs. - -# Citations - -[1] [src/portable_python/inspector.py](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/inspector.py) -[2] [src/portable_python/tracking.py — Tracker/Trackable](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/tracking.py) diff --git a/docs/cli/build-report.md b/docs/cli/build-report.md index fd76c32..9a1db91 100644 --- a/docs/cli/build-report.md +++ b/docs/cli/build-report.md @@ -2,7 +2,6 @@ type: CLI Command title: build-report description: Show the status of buildable modules — which will be auto-compiled, which are present on the system, and which would fail — without building anything. -resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py tags: [cli, build, report, command] timestamp: 2026-06-23T00:00:00Z --- @@ -32,7 +31,3 @@ For each candidate module the report shows its [telltale](/concepts/telltale-det portable-python build-report 3.13.2 portable-python build-report 3.13.2 -m openssl,sqlite ``` - -# Citations - -[1] [src/portable_python/cli.py — build_report](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py) diff --git a/docs/cli/build.md b/docs/cli/build.md index 0f9dfb0..8d4ffc0 100644 --- a/docs/cli/build.md +++ b/docs/cli/build.md @@ -2,7 +2,6 @@ type: CLI Command title: build description: Build a portable (or --prefix) python binary from source for the given version spec. -resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py tags: [cli, build, command] timestamp: 2026-06-23T00:00:00Z --- @@ -42,7 +41,3 @@ portable-python build 3.13.2 --prefix /apps/python3.13 # See exactly what would happen, without compiling portable-python --dryrun build 3.13.2 ``` - -# Citations - -[1] [src/portable_python/cli.py — build](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py) diff --git a/docs/cli/diagnostics.md b/docs/cli/diagnostics.md index dd0a5b5..ce94e12 100644 --- a/docs/cli/diagnostics.md +++ b/docs/cli/diagnostics.md @@ -2,7 +2,6 @@ type: CLI Command title: diagnostics description: Show system diagnostics (invoker python, platform info) alongside the active configuration. -resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py tags: [cli, diagnostics, command] timestamp: 2026-06-23T00:00:00Z --- @@ -18,7 +17,3 @@ portable-python diagnostics # Schema Takes no arguments. The left column comes from `runez.SYS_INFO` (invoker python, platform, etc.); the right column is `PPG.config.represented()` — the merged config and the files it was loaded from. - -# Citations - -[1] [src/portable_python/cli.py — diagnostics](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py) diff --git a/docs/cli/inspect.md b/docs/cli/inspect.md index 6f66bd8..ac883d8 100644 --- a/docs/cli/inspect.md +++ b/docs/cli/inspect.md @@ -2,7 +2,6 @@ type: CLI Command title: inspect description: Inspect a python installation for non-portable dynamic library usage, reporting any references that would break portability. -resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py tags: [cli, inspect, portability, command] timestamp: 2026-06-23T00:00:00Z --- @@ -41,8 +40,3 @@ portable-python inspect -v ~/versions/3.13.2 # Inspect the interpreter running this CLI portable-python inspect invoker ``` - -# Citations - -[1] [src/portable_python/cli.py — inspect](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py) -[2] [src/portable_python/inspector.py — PythonInspector](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/inspector.py) diff --git a/docs/cli/lib-auto-correct.md b/docs/cli/lib-auto-correct.md index bebd37a..e2ac553 100644 --- a/docs/cli/lib-auto-correct.md +++ b/docs/cli/lib-auto-correct.md @@ -2,7 +2,6 @@ type: CLI Command title: lib-auto-correct description: Scan a python installation and rewrite executables/libraries to use relative paths — the same relativization the build applies internally, runnable standalone. -resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py tags: [cli, lib-auto-correct, relativize, command] timestamp: 2026-06-23T00:00:00Z --- @@ -34,8 +33,3 @@ portable-python lib-auto-correct ~/versions/3.13.2 # Apply them portable-python lib-auto-correct --commit ~/versions/3.13.2 ``` - -# Citations - -[1] [src/portable_python/cli.py — lib_auto_correct](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py) -[2] [src/portable_python/inspector.py — LibAutoCorrect](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/inspector.py) diff --git a/docs/cli/list.md b/docs/cli/list.md index 300bda4..6db7e16 100644 --- a/docs/cli/list.md +++ b/docs/cli/list.md @@ -2,7 +2,6 @@ type: CLI Command title: list description: List the latest available versions for a python family (default cpython), optionally as JSON. -resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py tags: [cli, list, versions, command] timestamp: 2026-06-23T00:00:00Z --- @@ -28,8 +27,3 @@ portable-python list [OPTIONS] [FAMILY] portable-python list portable-python list cpython --json ``` - -# Citations - -[1] [src/portable_python/cli.py — list_cmd](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py) -[2] [src/portable_python/versions.py — CPythonFamily.get_available_versions](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/versions.py) diff --git a/docs/cli/recompress.md b/docs/cli/recompress.md index 7397007..014279d 100644 --- a/docs/cli/recompress.md +++ b/docs/cli/recompress.md @@ -2,7 +2,6 @@ type: CLI Command title: recompress description: Re-compress an existing binary tarball or folder into another format, mainly to compare resulting sizes. -resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py tags: [cli, recompress, command] timestamp: 2026-06-23T00:00:00Z --- @@ -29,7 +28,3 @@ After re-compressing, the command prints the size of both the source and the res ```shell portable-python recompress dist/cpython-3.13.2-linux-x86_64.tar.gz tar.zst ``` - -# Citations - -[1] [src/portable_python/cli.py — recompress](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cli.py) diff --git a/docs/concepts/build-layout.md b/docs/concepts/build-layout.md index 2665031..c914b15 100644 --- a/docs/concepts/build-layout.md +++ b/docs/concepts/build-layout.md @@ -28,7 +28,7 @@ build/ 00-portable-python.log 01-openssl.log 02-cpython.log - ppp-marker/{version}/ # Full installation after build completes + ppp-marker/{version}/ # The built python (its baked-in prefix is /ppp-marker/{version}) dist/ cpython-3.13.2-macos-arm64.tar.gz # Ready-to-go portable tarball ``` @@ -41,14 +41,9 @@ dist/ | `components/` | Scratch space where each component's source is unpacked and compiled. | | `deps/` | The shared install prefix passed to every external module's `./configure --prefix=build/deps`. CPython finds these via injected env vars — see [static linking](/concepts/static-linking.md). | | `logs/` | One log file per component, numbered so they sort in compile order. | -| `ppp-marker/{version}/` | The completed installation, named via the `ppp-marker` template before compression. | +| `ppp-marker/{version}/` | The built python installation, under the [ppp-marker](/concepts/ppp-marker.md) prefix. | | `dist/` | Final portable tarball(s). | ## Configurability All of these paths are templated and overridable in [configuration](/configuration/portable-python-yml.md) under the `folders:` key, using placeholders like `{build}`, `{version}`, and `{abi_suffix}`. The [`Folders`](/architecture/config.md) helper resolves them. For example, the dev config keeps builds per-version with `build: "build/{family}-{version}{abi_suffix}"`. - -# Citations - -[1] [README.rst — Build folder structure](https://github.com/codrsquad/portable-python/blob/main/README.rst) -[2] [src/portable_python/config.py — DEFAULT_CONFIG folders](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/config.py) diff --git a/docs/concepts/folder-masking.md b/docs/concepts/folder-masking.md index ee7e908..2dc3358 100644 --- a/docs/concepts/folder-masking.md +++ b/docs/concepts/folder-masking.md @@ -8,7 +8,7 @@ timestamp: 2026-06-23T00:00:00Z # Folder Masking (macOS) -On macOS, `/usr/local` is where Homebrew and other package managers install libraries and headers. If CPython's `configure` discovers those during a build, it may **dynamically link** against them — silently breaking [portability](/concepts/portability.md), because the resulting binary would depend on libraries that won't exist on another machine. +On macOS, `/usr/local` is historically where Homebrew and other package managers install libraries and headers. If CPython's `configure` discovers those during a build, it may **dynamically link** against them — silently breaking [portability](/concepts/portability.md), because the resulting binary would depend on libraries that won't exist on another machine. ## The mask @@ -24,8 +24,3 @@ Because it is a RAM disk mounted over the path (not a deletion), the real `/usr/ ## Scope This mask applies to **macOS only**. On Linux, isolation is achieved through telltale-driven module selection and the injected `CPATH`/`LDFLAGS` pointing at `build/deps/` first. - -# Citations - -[1] [src/portable_python/__init__.py — FolderMask, BuildContext](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py) -[2] [ARCHITECTURE.md — Folder Masking pattern](https://github.com/codrsquad/portable-python/blob/main/ARCHITECTURE.md) diff --git a/docs/concepts/index.md b/docs/concepts/index.md index 9475836..31c1c9c 100644 --- a/docs/concepts/index.md +++ b/docs/concepts/index.md @@ -7,3 +7,4 @@ The domain ideas behind portable-python. Read these to understand *why* the buil * [Telltale detection](/concepts/telltale-detection.md) - How the tool decides whether a library must be compiled or already exists. * [Folder masking](/concepts/folder-masking.md) - The macOS `/usr/local` RAM-disk mask that prevents accidental dynamic linking. * [Build layout](/concepts/build-layout.md) - The `build/` and `dist/` folder structure shared by every component. +* [ppp-marker prefix](/concepts/ppp-marker.md) - The placeholder install prefix that fills in for `--prefix` and is relativized for portable builds. diff --git a/docs/concepts/portability.md b/docs/concepts/portability.md index 435c643..3a847d4 100644 --- a/docs/concepts/portability.md +++ b/docs/concepts/portability.md @@ -33,7 +33,3 @@ See the [`build`](/cli/build.md) command for how to choose. ## Validation Portability is not assumed — it is **verified**. The [`PythonInspector`](/architecture/python-inspector.md) walks every executable and `.so` file, parses their dynamic dependencies (`ldd` on Linux, `otool` on macOS), and classifies each reference as a system library, a relative reference, or a problem. The [`inspect`](/cli/inspect.md) command surfaces this report and fails if a non-portable reference is found. - -# Citations - -[1] [README.rst — Supported operating systems](https://github.com/codrsquad/portable-python/blob/main/README.rst) diff --git a/docs/concepts/ppp-marker.md b/docs/concepts/ppp-marker.md new file mode 100644 index 0000000..7bee80d --- /dev/null +++ b/docs/concepts/ppp-marker.md @@ -0,0 +1,24 @@ +--- +type: Concept +title: The ppp-marker Prefix +description: "ppp-marker" (portable python path marker) is the install prefix handed to CPython's configure — the real --prefix for non-portable builds, or a searchable placeholder that gets relativized for portable builds. +tags: [ppp-marker, prefix, concept, portability, relativize] +timestamp: 2026-06-23T00:00:00Z +--- + +# The ppp-marker Prefix + +`ppp-marker` stands for **portable python path marker**. It is the install prefix the build hands to CPython's `configure`, and its role depends on the build kind: + +- **Non-portable (`--prefix`) builds** — `ppp-marker` is simply the `--prefix` you asked for; the python is built to live there for real. +- **Portable builds** — there is no real install location, so `ppp-marker` is a *dummy* placeholder folder that fills in for `--prefix`. CPython is configured with `--prefix=/ppp-marker/{version}`, so the freshly built interpreter initially believes it lives at `/ppp-marker/{version}`. The [finalize step](/architecture/cpython.md) then rewrites those baked-in absolute paths to be relative (see [static linking](/concepts/static-linking.md) and [portability](/concepts/portability.md)), making the install relocatable. + +## Why "ppp" + +The `ppp` token is chosen to be **easy to search and impossible to collide with**: grepping the codebase for `ppp` turns up only these usages, and `ppp-marker` appears nowhere in upstream CPython source. That uniqueness is what lets the finalize step find and rewrite every occurrence of the placeholder prefix unambiguously. ([`PPG`](/architecture/ppg.md) — "Portable Python Globals" — is named on the same principle.) + +## Where it shows up + +- Configured via the `ppp-marker` key (default `/ppp-marker/{version}{abi_suffix}`) — see [configuration](/configuration/portable-python-yml.md) and [`Folders`](/architecture/config.md). +- On disk the install lands under it, at `build/ppp-marker/{version}/` — see [build layout](/concepts/build-layout.md). +- Visible in `--dryrun` output as `./configure --prefix=/ppp-marker/{version}`. diff --git a/docs/concepts/static-linking.md b/docs/concepts/static-linking.md index 58a1b9b..e0bec04 100644 --- a/docs/concepts/static-linking.md +++ b/docs/concepts/static-linking.md @@ -29,8 +29,3 @@ Static linking is achieved **only** through `configure`/`make` flags — never b ## Relativization Static linking removes runtime library dependencies, but absolute *paths* can still leak into the install (shebang lines, `sysconfig` variables). The [`Cpython`](/architecture/cpython.md) finalize step rewrites these to be relative, and [`LibAutoCorrect`](/architecture/python-inspector.md) rewrites dynamic-library references in binaries to relative form (`@loader_path`/`$ORIGIN` style). Together these make the install relocatable. - -# Citations - -[1] [src/portable_python/__init__.py — ModuleBuilder.xenv_* methods](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py) -[2] [src/portable_python/cpython.py — finalize / relativize steps](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cpython.py) diff --git a/docs/concepts/telltale-detection.md b/docs/concepts/telltale-detection.md index 6d9c4c8..bb69ac7 100644 --- a/docs/concepts/telltale-detection.md +++ b/docs/concepts/telltale-detection.md @@ -47,8 +47,3 @@ On Linux, `m_debian` names the Debian dev package and a leading sigil encodes a | (none) | Used for detection/reporting only. | `libgdbm-dev` | This is why [`build-report`](/cli/build-report.md) can warn that a module is "broken" on the current host before a build is even attempted. - -# Citations - -[1] [src/portable_python/versions.py — PPG.find_telltale](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/versions.py) -[2] [src/portable_python/__init__.py — ModuleBuilder.linker_outcome](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/__init__.py) diff --git a/docs/configuration/index.md b/docs/configuration/index.md index 43a90d5..765b175 100644 --- a/docs/configuration/index.md +++ b/docs/configuration/index.md @@ -4,4 +4,4 @@ How portable-python is configured, and how settings are merged and overridden. * [portable-python.yml](/configuration/portable-python-yml.md) - The config file: sections, per-module keys, path templates, and platform overrides. -Configuration is loaded and merged by [`Config`](/architecture/config.md). The built-in `DEFAULT_CONFIG` provides defaults; user files layer on top, with platform-specific sections winning over generic ones. See also the root [CONFIGURATION.md](https://github.com/codrsquad/portable-python/blob/main/CONFIGURATION.md). +Configuration is loaded and merged by [`Config`](/architecture/config.md). The built-in `DEFAULT_CONFIG` provides defaults; user files layer on top, with platform-specific sections winning over generic ones. \ No newline at end of file diff --git a/docs/configuration/portable-python-yml.md b/docs/configuration/portable-python-yml.md index abcfe35..a726839 100644 --- a/docs/configuration/portable-python-yml.md +++ b/docs/configuration/portable-python-yml.md @@ -2,7 +2,6 @@ type: Configuration title: portable-python.yml description: The YAML configuration file controlling folders, module selection, configure flags, cleanup, and per-module overrides — with platform-specific sections. -resource: https://github.com/codrsquad/portable-python/blob/main/portable-python.yml tags: [configuration, yaml, config] timestamp: 2026-06-23T00:00:00Z --- @@ -39,7 +38,7 @@ All support path templates: `{build}`, `{family}`, `{version}`, `{abi_suffix}`. | `dist` | `dist` | Final tarball folder. | | `logs` | `{build}/logs` | Per-component logs. | | `sources` | `build/sources` | Downloaded tarball cache. | -| `ppp-marker` | `/ppp-marker/{version}{abi_suffix}` | In-progress install folder name. | +| `ppp-marker` | `/ppp-marker/{version}{abi_suffix}` | The [ppp-marker](/concepts/ppp-marker.md) placeholder install prefix. | See [build layout](/concepts/build-layout.md) for how these map onto disk. @@ -81,9 +80,3 @@ openssl-http-headers: macos: openssl-configure: --with-terminfo-dirs=/usr/share/terminfo ``` - -# Citations - -[1] [portable-python.yml — sample dev config](https://github.com/codrsquad/portable-python/blob/main/portable-python.yml) -[2] [src/portable_python/config.py — DEFAULT_CONFIG](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/config.py) -[3] [CONFIGURATION.md](https://github.com/codrsquad/portable-python/blob/main/CONFIGURATION.md) diff --git a/docs/guides/add-a-config-option.md b/docs/guides/add-a-config-option.md new file mode 100644 index 0000000..b955985 --- /dev/null +++ b/docs/guides/add-a-config-option.md @@ -0,0 +1,20 @@ +--- +type: Playbook +title: Add a Config Option +description: Add a new configuration key by extending DEFAULT_CONFIG and reading it via PPG.config.get_value(). +tags: [playbook, configuration, contributing] +timestamp: 2026-06-23T00:00:00Z +--- + +# Add a Config Option + +## Steps + +1. **Declare the default** in `DEFAULT_CONFIG` (in `src/portable_python/config.py`) so the key always has a value. +2. **Read it** where needed with [`PPG.config.get_value("your-key")`](/architecture/config.md) — pass `by_platform=False` if it should ignore platform-specific overrides. +3. **Add tests** in `tests/test_setup.py` (the suite runs in `--dryrun`). + +## Notes + +- Values support platform-specific overrides (`linux:` / `macos:` / `windows:` sections); the most specific match for [`PPG.target`](/architecture/ppg.md) wins. See [configuration](/configuration/portable-python-yml.md). +- For a per-module key, follow the `{module}-*` convention (`{module}-version`, `{module}-configure`, …) that the [`ModuleBuilder`](/architecture/module-builder.md) `cfg_*` helpers already read. diff --git a/docs/guides/add-an-external-module.md b/docs/guides/add-an-external-module.md index 14f30f6..69ae7c2 100644 --- a/docs/guides/add-an-external-module.md +++ b/docs/guides/add-an-external-module.md @@ -57,8 +57,3 @@ To statically link a new C library into CPython, add a [`ModuleBuilder`](/archit - Prefer config flags over source patches — see the [no-patches principle](/overview.md). - Use `--dryrun` to confirm the configure/make commands look right before a real build. - If the library is only needed when explicitly selected, leave `auto_select_reason()` unset; if a build cannot succeed without it, return a short reason there. - -# Citations - -[1] [DEVELOP.md — Add a New External Module](https://github.com/codrsquad/portable-python/blob/main/DEVELOP.md) -[2] [src/portable_python/external/xcpython.py — existing modules](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/external/xcpython.py) diff --git a/docs/guides/build-a-portable-python.md b/docs/guides/build-a-portable-python.md index 4142002..adb35b4 100644 --- a/docs/guides/build-a-portable-python.md +++ b/docs/guides/build-a-portable-python.md @@ -59,7 +59,3 @@ portable-python inspect ~/versions/3.13.2 ``` A clean exit means no non-portable dynamic-library references — see [`inspect`](/cli/inspect.md). - -# Citations - -[1] [README.rst — Building a portable cpython](https://github.com/codrsquad/portable-python/blob/main/README.rst) diff --git a/docs/guides/bump-python-support.md b/docs/guides/bump-python-support.md new file mode 100644 index 0000000..4b9a5c8 --- /dev/null +++ b/docs/guides/bump-python-support.md @@ -0,0 +1,18 @@ +--- +type: Playbook +title: Bump Python Support +description: Update the supported CPython versions across project metadata, the CI matrix, and the minimum version floor. +tags: [playbook, versions, cpython, maintenance] +timestamp: 2026-06-23T00:00:00Z +--- + +# Bump Python Support + +portable-python deliberately tracks only the recent, non-EOL CPython versions (see [overview](/overview.md)). To move the supported window: + +## Steps + +1. **Project metadata** — update the classifiers and `requires-python` in `pyproject.toml`. +2. **CI matrix** — update the Python versions in `.github/workflows/tests.yml` (see [CI/CD](/guides/ci-cd.md)). +3. **Minimum version** — bump `CPythonFamily.min_version` (in `versions.py`) if the floor moved; it gates which versions [`list`](/cli/list.md) and [`build`](/cli/build.md) accept. See [`PPG`](/architecture/ppg.md). +4. **Verify** — run the full matrix with `tox` (see [local development](/guides/local-development.md)). diff --git a/docs/guides/ci-cd.md b/docs/guides/ci-cd.md new file mode 100644 index 0000000..d1ad8b3 --- /dev/null +++ b/docs/guides/ci-cd.md @@ -0,0 +1,31 @@ +--- +type: Reference +title: CI/CD +description: The GitHub Actions workflows — tests on every push/PR across the Python matrix, and trusted PyPI publishing on version tags. +tags: [ci, cd, github-actions, release, reference] +timestamp: 2026-06-23T00:00:00Z +--- + +# CI/CD + +Two GitHub Actions workflows live in `.github/workflows/`. Both drive [tox](/guides/local-development.md) through `uv` / `tox-uv`. + +## Tests (`tests.yml`) + +Runs on every push to `main` and every pull request targeting `main`, with `cancel-in-progress` concurrency so superseded runs are cancelled. + +| Job | What it does | +|-----|--------------| +| `test` | Matrix over Python `3.10`–`3.14` on `ubuntu-latest` (`fail-fast: false`). Runs `uvx --with tox-uv tox -e py`, then uploads coverage to Coveralls as a **parallel** build, flagged per version. | +| `coveralls-finish` | After all `test` jobs finish, signals Coveralls that the parallel build is complete. | +| `linters` | On Python `3.14`, runs `uvx --with tox-uv tox -e style` (ruff check + format diff). | + +## Release (`release.yml`) + +Triggers on pushing a tag matching `v[0-9]*`. The single `publish` job (Python `3.14`, GitHub environment `release`): + +1. `uvx --with tox-uv tox -e py,style` — full test + lint gate. +2. `uv build` — build the sdist + wheel. +3. `pypa/gh-action-pypi-publish` — publish to PyPI via **trusted publishing** (OIDC, `id-token: write` — no API token is stored). + +See [bump python support](/guides/bump-python-support.md) for keeping the test matrix in sync with supported versions. diff --git a/docs/guides/fix-a-portability-issue.md b/docs/guides/fix-a-portability-issue.md new file mode 100644 index 0000000..bce706e --- /dev/null +++ b/docs/guides/fix-a-portability-issue.md @@ -0,0 +1,18 @@ +--- +type: Playbook +title: Fix a Portability Issue +description: Diagnose a non-portable build with inspect, then fix it by selecting the right module or relativizing lib references. +tags: [playbook, portability, inspect, debugging] +timestamp: 2026-06-23T00:00:00Z +--- + +# Fix a Portability Issue + +When a built python is flagged as non-[portable](/concepts/portability.md) — a dynamic dependency on a non-system library, or an absolute path baked in: + +## Steps + +1. **Diagnose** — run [`portable-python inspect `](/cli/inspect.md) to see exactly which `.so` or executable carries the offending reference, via the [`PythonInspector`](/architecture/python-inspector.md). +2. **If a library is being dynamically linked** — make sure its [external module](/modules/index.md) is selected so it gets compiled statically, or tighten build isolation so `configure` can't discover the system copy. On macOS check the [folder mask](/concepts/folder-masking.md); see also [telltale detection](/concepts/telltale-detection.md). +3. **If an absolute path is baked in** — the [`LibAutoCorrect`](/architecture/python-inspector.md) logic rewrites lib references to relative form. Run [`lib-auto-correct`](/cli/lib-auto-correct.md) to reproduce/iterate on it standalone, and extend it if a new case isn't handled. +4. **Add a test case** in `tests/test_inspector.py`. diff --git a/docs/guides/index.md b/docs/guides/index.md index cd28233..697e7c8 100644 --- a/docs/guides/index.md +++ b/docs/guides/index.md @@ -4,6 +4,8 @@ Task-oriented playbooks for using and extending portable-python. * [Build a portable python](/guides/build-a-portable-python.md) - Install the tool, build a binary, and use it. * [Add an external module](/guides/add-an-external-module.md) - Add a new C library to the static-link set. +* [Add a config option](/guides/add-a-config-option.md) - Introduce a new configuration key. +* [Fix a portability issue](/guides/fix-a-portability-issue.md) - Diagnose and correct a non-portable build. +* [Bump python support](/guides/bump-python-support.md) - Move the supported CPython version window. * [Local development](/guides/local-development.md) - Dev venv, tests, dryrun, and building Linux binaries via Docker. - -See also the root [DEVELOP.md](https://github.com/codrsquad/portable-python/blob/main/DEVELOP.md) for the original notes these guides are based on. +* [CI/CD](/guides/ci-cd.md) - The GitHub Actions test and release workflows. diff --git a/docs/guides/local-development.md b/docs/guides/local-development.md index 4df3732..8eddc90 100644 --- a/docs/guides/local-development.md +++ b/docs/guides/local-development.md @@ -21,7 +21,7 @@ uv sync The suite mocks `runez.run()` and builds in `--dryrun` mode, so it asserts the *planned* commands without actually compiling. Target is 100% coverage. ```shell -tox # all envs (py310-314, coverage, docs, style) +tox # all envs (py310-314, coverage, style) tox -e py313 # one python version tox -e style # lint check (ruff) tox -e reformat # auto-fix formatting @@ -32,6 +32,8 @@ tox -e reformat # auto-fix formatting CI runs `uvx --with tox-uv tox -e py`. +Test fixtures: a `cli` fixture (from runez) drives the CLI, `conftest.py` forbids real HTTP calls (`GlobalHttpCalls.forbid()`), and sample configs for parsing tests live in `tests/sample-config*.yml`. + ## Dryrun mode `--dryrun` (`-n`) prints what would be done without doing it — invaluable for fast iteration and for understanding the build: @@ -55,6 +57,14 @@ docker run -it -v./:/src/ portable-python-jammy /bin/bash portable-python build 3.13.2 ``` +## Code style + +Ruff handles both linting and formatting — `tox -e style` checks, `tox -e reformat` auto-fixes. The key settings (line length, max McCabe complexity, numpy-style docstrings, and relaxed security `S` rules in tests) live in `pyproject.toml`. + +## Documentation + +The project's documentation is the OKF knowledge bundle in `docs/` (start at [the bundle index](/index.md)). Validate its structure and cross-links with `scripts/check_okf.py docs`. + ## Key dependencies | Package | Purpose | @@ -63,8 +73,3 @@ portable-python build 3.13.2 | `pyyaml` | [Configuration](/configuration/portable-python-yml.md) parsing. | | `requests` / `urllib3` | HTTP downloads. | | `runez` | Foundational utilities — file ops, system info, CLI decorators, logging, `Version`/`PythonSpec`. Check `runez` before reimplementing anything. | - -# Citations - -[1] [DEVELOP.md](https://github.com/codrsquad/portable-python/blob/main/DEVELOP.md) -[2] [CLAUDE.md — Common Commands, Testing](https://github.com/codrsquad/portable-python/blob/main/CLAUDE.md) diff --git a/docs/index.md b/docs/index.md index 7946b2d..8953514 100644 --- a/docs/index.md +++ b/docs/index.md @@ -4,7 +4,8 @@ okf_version: "0.1" # Portable Python — Knowledge Bundle -A knowledge bundle (in [OKF](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) format) describing `portable-python`: a CLI and Python library for compiling statically-linked, relocatable CPython binaries from source. +A knowledge bundle (in [OKF](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) format) describing `portable-python`: a CLI and Python library for +compiling statically-linked, relocatable CPython binaries from source. Start with the [Overview](/overview.md), then drill into the area you need. @@ -21,10 +22,9 @@ Start with the [Overview](/overview.md), then drill into the area you need. ## Conventions * Concept IDs are file paths minus `.md` (e.g. `architecture/ppg`). -* Cross-links are bundle-relative, beginning with `/` (the bundle root is this `docs/` folder). -* `resource:` frontmatter points to the canonical source file on GitHub when a concept maps to code. -* Change history lives in [log.md](/log.md). +* Cross-links are bundle-relative, beginning with `/` (the bundle root is this `docs/` folder).* Change history lives in [log.md](/log.md). -## Relationship to the root-level docs +## Authority -This bundle is additive. The repository's hand-written `README.rst`, `ARCHITECTURE.md`, `CONFIGURATION.md`, and `DEVELOP.md` remain authoritative for now; this bundle reorganizes and expands that material into discrete, linkable concepts. See [Overview](/overview.md#citations) for pointers back to those files. +This bundle is the **authoritative** documentation for portable-python's internals, configuration, CLI, and development workflows. +The repository's `README.rst` remains the project's landing page (install, quick start, motivation). diff --git a/docs/log.md b/docs/log.md index a3c788c..7288726 100644 --- a/docs/log.md +++ b/docs/log.md @@ -2,6 +2,14 @@ ## 2026-06-23 +* **Update**: Removed the `resource:` frontmatter field from all concept files — unrendered, unreferenced by any tooling, and brittle (GitHub URLs). Source is pointed to inline in the prose instead. The bundle now contains no `github.com` repo links. +* **Update**: Removed the `# Citations` sections from all concept files — they duplicated the `resource:` frontmatter and inline source mentions, and added brittle GitHub-URL boilerplate that nothing referenced. Source pointers now live only in `resource:` and inline prose. +* **Update**: Inverted the `docs/` ↔ `CLAUDE.md` relationship — `CLAUDE.md` is now a thin pointer into the bundle. Folded the test-harness and code-style notes into [local development](/guides/local-development.md), and removed all back-references to `CLAUDE.md` from `docs/`. +* **Update**: Promoted `docs/` to the authoritative documentation; folded in and removed the root `ARCHITECTURE.md`, `CONFIGURATION.md`, and `DEVELOP.md`. +* **Creation**: Added guides for [CI/CD](/guides/ci-cd.md), [adding a config option](/guides/add-a-config-option.md), [fixing a portability issue](/guides/fix-a-portability-issue.md), and [bumping python support](/guides/bump-python-support.md). +* **Update**: Dropped the default-version column from the [external modules reference](/modules/external-modules.md) to avoid drift; it now points at `cfg_version()` in source and [`build-report`](/cli/build-report.md). +* **Creation**: Added the [ppp-marker prefix](/concepts/ppp-marker.md) concept. +* **Update**: Centralized the `ppp-marker` explanation there; build layout, PPG, Cpython, and configuration now link to it instead of repeating it. * **Creation**: Established the portable-python knowledge bundle in OKF format. * **Creation**: Added the [Overview](/overview.md). * **Creation**: Added the [Concepts](/concepts/index.md) section (portability, static linking, telltale detection, folder masking, build layout). diff --git a/docs/modules/external-modules.md b/docs/modules/external-modules.md index 7695539..2b14b2c 100644 --- a/docs/modules/external-modules.md +++ b/docs/modules/external-modules.md @@ -1,33 +1,34 @@ --- type: Reference title: External Modules Reference -description: The C libraries compiled and statically linked into CPython, with default versions, telltale markers, and Linux build constraints. -resource: https://github.com/codrsquad/portable-python/blob/main/src/portable_python/external/xcpython.py +description: The C libraries compiled and statically linked into CPython, with their telltale markers and Linux build constraints. tags: [reference, modules, external, cpython] timestamp: 2026-06-23T00:00:00Z --- # External Modules Reference -These are the candidate external modules for CPython, in the order declared by `Cpython.candidate_modules()`. Each is a [`ModuleBuilder`](/architecture/module-builder.md) that compiles a C library statically into the shared `build/deps/` prefix before CPython is built. Default versions are overridable via [config](/configuration/portable-python-yml.md) (e.g. `openssl-version`). +These are the candidate external modules for CPython, in the order declared by `Cpython.candidate_modules()`. Each is a [`ModuleBuilder`](/architecture/module-builder.md) that compiles a C library statically into the shared `build/deps/` prefix before CPython is built. + +Default versions are intentionally **not** listed here, to avoid drifting from the code: each module's default is the `cfg_version()` default in its source class (overridable via the `{module}-version` [config](/configuration/portable-python-yml.md) key). To see the versions that would actually be used for a build, run [`build-report`](/cli/build-report.md). # Schema -| Module | CPython feature | Default version | Telltale (`{include}/…`) | Linux dev pkg | -|--------|-----------------|-----------------|--------------------------|----------------| -| `LibFFI` | `ctypes` | 3.5.2 | `ffi.h`, `ffi/ffi.h` | `!libffi-dev` | -| `Zlib` | `zlib` | 1.3.2 | `zlib.h` | `!zlib1g-dev` | -| `Zstd` | compression (zstd) | 1.5.7 | `zstd.h` | `!libzstd-dev` | -| `Xz` | `lzma` | 5.8.3 | `lzma.h` | — | -| `Bzip2` | `bz2` | 1.0.8 | `bzlib.h` | — | -| `Readline` | `readline` | 8.3 | `readline/readline.h` | `-libreadline-dev` | -| `Openssl` | `ssl`, `hashlib` | 3.5.6 | `openssl/ssl.h` | — | -| `Sqlite` | `sqlite3` | 3.51.3 | `sqlite3.h` | `+libsqlite3-dev` | -| `Bdb` | `dbm.ndbm` | 6.2.32 | `dbm.h` | `libgdbm-compat-dev` | -| `Gdbm` | `dbm.gnu` | 1.26 | `gdbm.h` | `libgdbm-dev` | -| `Uuid` | `uuid` | 1.0.3 | `uuid/uuid.h` | `+uuid-dev` | -| `TkInter` | `tkinter` | 8.6.17 | `tk`, `tk.h` | `-tk-dev` | -| `Mpdec` | `decimal` | 4.0.1 | `mpdecimal.h` | `!libmpdec-dev` | +| Module | CPython feature | Telltale (`{include}/…`) | Linux dev pkg | +|--------|-----------------|--------------------------|----------------| +| `LibFFI` | `ctypes` | `ffi.h`, `ffi/ffi.h` | `!libffi-dev` | +| `Zlib` | `zlib` | `zlib.h` | `!zlib1g-dev` | +| `Zstd` | compression (zstd) | `zstd.h` | `!libzstd-dev` | +| `Xz` | `lzma` | `lzma.h` | — | +| `Bzip2` | `bz2` | `bzlib.h` | — | +| `Readline` | `readline` | `readline/readline.h` | `-libreadline-dev` | +| `Openssl` | `ssl`, `hashlib` | `openssl/ssl.h` | — | +| `Sqlite` | `sqlite3` | `sqlite3.h` | `+libsqlite3-dev` | +| `Bdb` | `dbm.ndbm` | `dbm.h` | `libgdbm-compat-dev` | +| `Gdbm` | `dbm.gnu` | `gdbm.h` | `libgdbm-dev` | +| `Uuid` | `uuid` | `uuid/uuid.h` | `+uuid-dev` | +| `TkInter` | `tkinter` | `tk`, `tk.h` | `-tk-dev` | +| `Mpdec` | `decimal` | `mpdecimal.h` | `!libmpdec-dev` | ## Linux dev-package sigils @@ -51,9 +52,3 @@ Some modules bundle their own dependencies as sub-modules (their `candidate_modu ## Per-module config Every module honors these config keys (replace `{module}` with the lowercased name, e.g. `openssl`): `{module}-version`, `{module}-url`, `{module}-src-suffix`, `{module}-configure`, `{module}-http-headers`, `{module}-patches`. See [configuration](/configuration/portable-python-yml.md). - -# Citations - -[1] [src/portable_python/external/xcpython.py](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/external/xcpython.py) -[2] [src/portable_python/external/xtkinter.py](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/external/xtkinter.py) -[3] [src/portable_python/cpython.py — Cpython.candidate_modules](https://github.com/codrsquad/portable-python/blob/main/src/portable_python/cpython.py) diff --git a/docs/overview.md b/docs/overview.md index eebbb56..ccdaa9f 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -2,7 +2,6 @@ type: Overview title: Portable Python Overview description: A CLI and library that compiles statically-linked, relocatable CPython binaries from source, and validates their portability. -resource: https://github.com/codrsquad/portable-python tags: [overview, cpython, portability, build] timestamp: 2026-06-23T00:00:00Z --- @@ -61,9 +60,3 @@ Inspect any existing python for portability: ```shell portable-python inspect /usr/bin/python3 ``` - -# Citations - -[1] [README.rst](https://github.com/codrsquad/portable-python/blob/main/README.rst) - Project README and guiding principles. -[2] [ARCHITECTURE.md](https://github.com/codrsquad/portable-python/blob/main/ARCHITECTURE.md) - Class hierarchy and design patterns. -[3] [CLAUDE.md](https://github.com/codrsquad/portable-python/blob/main/CLAUDE.md) - Repository guidance and architecture summary. From 9f72fed4d781280e8132b638908cfe6fee6b7674 Mon Sep 17 00:00:00 2001 From: Zoran Simic Date: Tue, 23 Jun 2026 10:55:49 -0700 Subject: [PATCH 3/5] Shortened docs to avoid drift risk from code --- docs/architecture/build-setup.md | 45 +++++++++------------------ docs/architecture/config.md | 36 +++++---------------- docs/architecture/cpython.md | 32 ++++++------------- docs/architecture/module-builder.md | 45 ++++++--------------------- docs/architecture/ppg.md | 28 +++-------------- docs/architecture/python-builder.md | 26 ++++++---------- docs/architecture/python-inspector.md | 45 ++++++--------------------- docs/log.md | 1 + 8 files changed, 64 insertions(+), 194 deletions(-) diff --git a/docs/architecture/build-setup.md b/docs/architecture/build-setup.md index f025954..904bb27 100644 --- a/docs/architecture/build-setup.md +++ b/docs/architecture/build-setup.md @@ -1,46 +1,29 @@ --- type: Class title: BuildSetup -description: Coordinates the overall compilation — resolves the python spec, prepares folders, builds external modules then CPython, validates, and compresses the result. +description: Orchestrates a build end to end — resolves the spec, builds external modules then CPython, validates, and compresses the result. tags: [class, build, coordinator] timestamp: 2026-06-23T00:00:00Z --- # BuildSetup -`BuildSetup` drives a build from end to end. The CLI's [`build`](/cli/build.md) command (and library users) construct one and call `compile()`. Defined in `__init__.py`. +The build orchestrator (`__init__.py`). The [`build`](/cli/build.md) command — and library users — construct one from a python spec and call `compile()`: ```python from portable_python import BuildSetup -setup = BuildSetup("cpython:3.13.2") -setup.compile() +BuildSetup("cpython:3.13.2").compile() ``` -# Schema - -| Member | Kind | Purpose | -|--------|------|---------| -| `__init__(python_spec, modules, prefix)` | constructor | Resolve the [`PythonSpec`](/cli/build.md), set up [`Folders`](/architecture/config.md), pick the extension, and instantiate the family's `python_builder`. | -| `python_spec` | attribute | The resolved family + version (full `X.Y.Z` required). | -| `folders` | attribute (`Folders`) | Resolved build/dist/sources/logs/destdir paths. | -| `prefix` | attribute | Optional `--prefix`; when set the build is **not** [portable](/concepts/portability.md). | -| `tarball_name` | attribute | Composed output name, e.g. `cpython-3.13.2-macos-arm64.tar.gz`. | -| `python_builder` | attribute (`PythonBuilder`) | The concrete builder, e.g. [`Cpython`](/architecture/cpython.md). | -| `validate_module_selection(fatal)` | method | Check every selected/candidate module's `linker_outcome`; abort on a `failed` outcome. | -| `compile()` | method | The full pipeline (see below). | - -## The `compile()` pipeline - -1. **Clean** the build folder (and logs); set up file logging to `logs/00-portable-python.log`. -2. `python_builder.validate_setup()`. -3. Enter a [`BuildContext`](/concepts/folder-masking.md) — applies macOS `/usr/local` masking and other isolation. -4. Log the platform, config files in use, and the build report. -5. `validate_module_selection()` (fatal unless dry-run / debug). -6. Clean `components/` and `deps/`. -7. `build_context.compile()` then `python_builder.compile()` — external modules first, then CPython. -8. If a `dist/` folder is configured, **compress** the install into `dist/{tarball_name}`. - -## Spec validation - -`BuildSetup` insists on a **full** version: a spec like `3.13` is rejected — you must give `3.13.2`. `"latest"` (or empty) resolves to `PPG.cpython.latest`. Invalid specs abort early with a red error. +## Mental model + +A conductor: it compiles nothing itself, it sequences the pieces. `compile()` is the spine — it cleans the [build folders](/concepts/build-layout.md), enters a [`BuildContext`](/concepts/folder-masking.md) (macOS isolation), builds the [external modules](/modules/index.md) into `build/deps/`, then builds CPython through its [`PythonBuilder`](/architecture/python-builder.md) (a [`Cpython`](/architecture/cpython.md)), and finally compresses the install into `dist/`. **External libraries first, CPython last.** + +It owns the [`Folders`](/architecture/config.md) (resolved paths) and the python builder; shared config, target platform, and version family come from [`PPG`](/architecture/ppg.md). + +## Worth knowing + +- **Start reading at `compile()`** — it calls everything else in order. +- A `--prefix` makes the build **non-portable**; without it the result is relocatable (see [ppp-marker](/concepts/ppp-marker.md) and [portability](/concepts/portability.md)). +- It requires a full `X.Y.Z` spec — a bare `3.13` is rejected; `"latest"` (or empty) resolves to the newest known version. diff --git a/docs/architecture/config.md b/docs/architecture/config.md index 353314a..bce9ae1 100644 --- a/docs/architecture/config.md +++ b/docs/architecture/config.md @@ -8,35 +8,11 @@ timestamp: 2026-06-23T00:00:00Z # Config & Folders -`Config` (in `config.py`) loads, merges, and queries the YAML [configuration](/configuration/portable-python-yml.md). `Folders` (in `versions.py`) turns templated path settings into concrete filesystem paths. Both are reached through [`PPG`](/architecture/ppg.md). +`Config` (`config.py`) loads, merges, and queries the YAML [configuration](/configuration/portable-python-yml.md). `Folders` (`versions.py`) turns its templated path settings into concrete filesystem paths. Both are reached through [`PPG`](/architecture/ppg.md). -# Schema +## Merge & precedence -## Config - -| Member | Purpose | -|--------|---------| -| `__init__(paths, target)` | Layer the built-in `DEFAULT_CONFIG` under any user config files, for the given target platform. | -| `get_value(*key, by_platform=True)` | Fetch a config value, honoring platform-specific overrides (most specific wins). | -| `get_entry(*key, by_platform=True)` | Like `get_value` but returns the raw entry. | -| `resolved_path(*key)` | Fetch a value and resolve it to a path. | -| `config_files_report()` / `represented()` | Human-readable summary of which config files are in effect (used by [`diagnostics`](/cli/diagnostics.md)). | -| `cleanup_globs(...)` / `symlink_duplicates(...)` / `ensure_main_file_symlinks(...)` | Finalize-time file housekeeping in the install. | - -## Folders - -| Member | Purpose | -|--------|---------| -| `build_folder`, `components`, `deps`, `sources`, `logs`, `dist`, `destdir` | Resolved component paths — see [build layout](/concepts/build-layout.md). | -| `ppp-marker` | The [ppp-marker](/concepts/ppp-marker.md) install prefix; equals `--prefix` for non-portable builds. | -| `formatted(text)` | Expand `{build}`, `{version}`, `{abi_suffix}`, … placeholders. | -| `resolved_destdir(relative_path)` | Compose a path under the marker'd destdir. | - -## Precedence & overrides - -Configuration merges multiple sources. The built-in `DEFAULT_CONFIG` provides sane defaults; -user files (default `portable-python.yml`, plus any `include:`d files) layer on top. -Within a file, **platform-specific** sections override generic ones: +The mental model is layering. A built-in `DEFAULT_CONFIG` provides sane defaults; user files (default `portable-python.yml`, plus any `include:`d files) layer on top. Within a file, **platform-specific** sections override generic ones, and the most specific match for the [target platform](/architecture/ppg.md) wins: ```yaml ext: gz # generic default @@ -47,4 +23,8 @@ macos: MACOSX_DEPLOYMENT_TARGET: 13 ``` -`get_value(..., by_platform=True)` returns the most specific match for the current `PPG.target`. +A lookup returns that most-specific value (with the option to ignore platform overrides). + +## Folders + +`Folders` resolves the templated `folders:` settings (placeholders like `{build}`, `{version}`, `{abi_suffix}`) into the concrete `build/`, `deps/`, `dist/`, … paths every component uses — including the [ppp-marker](/concepts/ppp-marker.md) install prefix. See [build layout](/concepts/build-layout.md) for the resulting tree. diff --git a/docs/architecture/cpython.md b/docs/architecture/cpython.md index 23ad159..6d642bd 100644 --- a/docs/architecture/cpython.md +++ b/docs/architecture/cpython.md @@ -8,31 +8,17 @@ timestamp: 2026-06-23T00:00:00Z # Cpython -`Cpython` is the concrete [`PythonBuilder`](/architecture/python-builder.md) that actually compiles CPython. It is returned by `CPythonFamily.get_builder()` (see [`PPG`](/architecture/ppg.md)) and lives in `cpython.py`. +The concrete [`PythonBuilder`](/architecture/python-builder.md) that compiles CPython (`cpython.py`); `CPythonFamily.get_builder()` returns it (see [`PPG`](/architecture/ppg.md)). Most of its work is two things: configuring the build, and finalizing the install so it's relocatable. -# Schema +## Configure -| Member | Kind | Purpose | -|--------|------|---------| -| `candidate_modules()` | classmethod | The external modules CPython can statically link (the [external modules](/modules/index.md) set). | -| `url` | property | CPython source URL (`python.org/ftp/...Python-X.Y.Z.tar.xz`), overridable via config. | -| `c_configure_args()` | method | Assemble `./configure` args (from `cpython-configure` config + computed flags). | -| `has_configure_opt(name, *variants)` | method | Test whether a configure option is in effect. | -| `xenv_LDFLAGS_NODIST`, `xenv_LIBZSTD_*`, `xenv_LIBMPDEC_*` | methods | Environment for linking specific static deps (zstd, mpdec). | -| `_do_linux_compile()` | method | configure → make → make install (DESTDIR). | -| `_finalize()` | method | Clean, byte-compile, relativize — see below. | +CPython is configured purely through flags — never source patches (the [no-patches principle](/overview.md)). Defaults come from `cpython-configure` in [config](/configuration/portable-python-yml.md) (optimizations, LTO, ensurepip), plus computed flags that point the build at the statically-compiled [deps](/concepts/static-linking.md). -## Configure arguments +## Finalize — making it relocatable -CPython is configured with, by default, `--enable-optimizations`, `--with-lto`, and `--with-ensurepip=upgrade` (from `cpython-configure` in [config](/configuration/portable-python-yml.md)), plus computed flags pointing at the static deps. Following the [no-patches principle](/overview.md), **all** behavior comes from configure flags — never source edits. +After `make install`, a portable build's interpreter still thinks it lives at the placeholder [`/ppp-marker/{version}`](/concepts/ppp-marker.md). Finalizing turns it into a [portable](/concepts/portability.md) install: -## Finalize: making it relocatable - -After `make install` the `_finalize()` step is what turns a normal install into a [portable](/concepts/portability.md) one. For a portable build the interpreter's prefix is still the placeholder [`/ppp-marker/{version}`](/concepts/ppp-marker.md), so finalize rewrites those baked-in absolute paths to be relative: - -- **Clean passes** — remove tests/idle/2to3 before byte-compiling (`cpython-clean-1st-pass`, ~94 MB) and prune seldom-used pycaches after (`cpython-clean-2nd-pass`, ~1.8 MB). -- **Byte-compile** the standard library (`cpython-compile-all`). -- `_relativize_shebangs()` — rewrite `bin/` script shebangs to be relative. -- `_relativize_sysconfig()` (`RelSysConf`) — rewrite absolute paths in the `sysconfig` data so the install resolves relative to its own location. -- `_apply_pep668()` — mark the environment per PEP 668. -- `_validate_venv_creation()` / `_validate_venv_module()` — sanity-check that `venv` works in the finished build. +- **Trim** — remove tests / idle / 2to3 before byte-compiling, then prune seldom-used pycaches after (driven by the `cpython-clean-*` config). +- **Byte-compile** the standard library. +- **Relativize** — rewrite `bin/` shebangs and the absolute paths baked into `sysconfig` so everything resolves relative to the install's own location. +- **Verify** — sanity-check that `venv` works in the finished build. diff --git a/docs/architecture/module-builder.md b/docs/architecture/module-builder.md index acbf75f..cd3e44a 100644 --- a/docs/architecture/module-builder.md +++ b/docs/architecture/module-builder.md @@ -8,46 +8,19 @@ timestamp: 2026-06-23T00:00:00Z # ModuleBuilder -`ModuleBuilder` is the abstract base for anything compiled by the tool. Both [external C modules](/modules/index.md) and [`PythonBuilder`](/architecture/python-builder.md) (hence [`Cpython`](/architecture/cpython.md)) extend it, so every component shares the same [build layout](/concepts/build-layout.md) and flow. +The abstract base for everything the tool compiles (`__init__.py`). Both the [external C modules](/modules/index.md) and [`PythonBuilder`](/architecture/python-builder.md) (hence [`Cpython`](/architecture/cpython.md)) extend it, so every component is built the same way and lands in the same [build layout](/concepts/build-layout.md). -# Schema +## Mental model -## Declarative class attributes +A subclass is mostly **declarative**: it sets a few `m_*` class attributes (name, [telltale](/concepts/telltale-detection.md) marker, Debian package, include subfolder) and provides a source `url` + `version`. The base class runs the rest of the flow — download → unpack → patch → configure → make → install — into the shared `build/deps/` prefix. Platform differences are isolated to `_do_linux_compile()` / `_do_macos_compile()`, dispatched on [`PPG.target`](/architecture/ppg.md). -| Attribute | Purpose | -|-----------|---------| -| `m_name` | Module name (derived from the class name). | -| `m_telltale` | Marker file(s) indicating the lib is present on the system. See [telltale detection](/concepts/telltale-detection.md). | -| `m_debian` | Debian dev package, with `!` / `+` / `-` sigils encoding build constraints. | -| `m_include` | Optional subfolder to add to `CPATH` when active (e.g. `openssl`). | -| `m_build_cwd` | Optional subfolder (relative to unpacked source) to run configure/make from. | +Two mechanisms recur everywhere and are worth understanding: -## Source resolution (overridable) +- **Environment injection** (`xenv_*` methods) — each contributes one variable (CPATH, LDFLAGS, PATH, …) pointing the compiler at `build/deps/`. This is how a statically-built dependency gets found by the next build; see [static linking](/concepts/static-linking.md). +- **Per-module config overrides** — `cfg_*` helpers read `{module}-version`, `{module}-url`, `{module}-configure`, … from [configuration](/configuration/portable-python-yml.md), so any default can be overridden without code changes. -| Member | Purpose | -|--------|---------| -| `url` | Download URL of the source tarball. | -| `version` | Version to build (default per module, overridable via config). | -| `headers` / `src_suffix` | HTTP headers and archive suffix when the URL lacks an extension. | -| `cfg_version`, `cfg_url`, `cfg_configure`, `cfg_patches`, `cfg_http_headers` | Read per-module overrides from [config](/configuration/portable-python-yml.md) (keys like `openssl-version`, `openssl-url`, …). | +`linker_outcome()` decides, per module and platform, whether it will be linked `static` / `shared`, is `absent`, or would `fail` — the verdict surfaced by [`build-report`](/cli/build-report.md). -## Environment injection (`xenv_*`) +## Adding one -Each `xenv_*` method supplies one environment variable for the compile, pointing tools at the shared `deps/` prefix — the mechanism behind [static linking](/concepts/static-linking.md): - -`xenv_CPATH`, `xenv_LDFLAGS`, `xenv_PATH`, `xenv_LD_LIBRARY_PATH`, `xenv_PKG_CONFIG_PATH`. `_find_all_env_vars()` gathers every `xenv_*` on the instance just before running. - -## Build flow - -| Method | Role | -|--------|------| -| `compile()` | Download → unpack → patch → `_prepare()` → platform compile → `_finalize()`. | -| `run_configure(program, *args, prefix)` | Run `./configure` with the deps prefix. | -| `run_make(*args, cpu_count)` | Run `make` (parallelized by CPU count). | -| `_do_linux_compile()` / `_do_macos_compile()` | Platform-specific compile, dispatched by `PPG.target`. | -| `linker_outcome(is_selected)` | Decide `static` / `shared` / `absent` / `failed` for this module. | -| `captured_logs()` | Context manager routing this module's output to its own numbered log file. | - -## Adding a module - -To add a new external module you subclass `ModuleBuilder`, set the `m_*` attributes, implement `url`/`version`, and implement `_do_linux_compile()` (macOS reuses it unless overridden). See the [guide](/guides/add-an-external-module.md). +Subclass `ModuleBuilder`, set the `m_*` attributes, implement `url` / `version` and `_do_linux_compile()` — see the [guide](/guides/add-an-external-module.md). diff --git a/docs/architecture/ppg.md b/docs/architecture/ppg.md index 9749241..9a6cde7 100644 --- a/docs/architecture/ppg.md +++ b/docs/architecture/ppg.md @@ -8,32 +8,14 @@ timestamp: 2026-06-23T00:00:00Z # PPG -`PPG` ("**P**ortable **P**ython **G**lobals") is the global state holder for portable-python. Every module reaches shared state — configuration, the target platform, and version families — through `PPG`'s class attributes and classmethods. It is defined in `versions.py`. +`PPG` ("**P**ortable **P**ython **G**lobals") is the global state holder (`versions.py`) — the one place the rest of the code reaches for three process-wide facts: the merged [configuration](/architecture/config.md), the target platform (OS/arch), and the registry of python version families. The CLI's `main` initializes it once; everything else reads from it. -Like [`ppp-marker`](/concepts/ppp-marker.md), the name is a deliberately uncommon character sequence chosen for searchability: grepping the codebase for `PPG` surfaces only its own usages, with no collisions in upstream CPython source or third-party libraries. +Like [`ppp-marker`](/concepts/ppp-marker.md), the name is a deliberately uncommon, greppable token — searching for `PPG` surfaces only its own usages, with no collisions in upstream CPython or third-party source. -# Schema +## Why a singleton -| Member | Kind | Purpose | -|--------|------|---------| -| `config` | attribute (`Config`) | The active merged [configuration](/architecture/config.md). | -| `target` | attribute (`PlatformId`) | Target OS/arch; drives platform dispatch and telltale expansion. | -| `families` | attribute (`dict`) | Registry mapping family name → `VersionFamily` (default: `{"cpython": ...}`). | -| `cpython` | attribute (`CPythonFamily`) | The built-in CPython family implementation. | -| `grab_config(paths, target)` | classmethod | (Re)load config from the given paths and set `target`. Called by the CLI's `main`. | -| `get_folders(base, family, version, abi_suffix)` | classmethod | Build a [`Folders`](/architecture/config.md) for a given family/version. | -| `family(name, fatal=True)` | classmethod | Look up a `VersionFamily` by name. | -| `find_python(spec)` | classmethod | Resolve a python on `PATH` via a cached `runez` `PythonDepot`. | -| `find_telltale(*telltales)` | classmethod | Expand `{include}` placeholders against `target.sys_include` and return the first existing path. See [telltale detection](/concepts/telltale-detection.md). | +The build depends on global facts — which config is active, what platform we target, which python versions exist — so centralizing them avoids threading that state through every constructor and gives the CLI one initialization point. Shared helpers hang off it too: resolving a python on `PATH`, building [`Folders`](/architecture/config.md) for a version, and expanding [telltale](/concepts/telltale-detection.md) markers against the target's system includes. ## Version families -A `VersionFamily` (e.g. `CPythonFamily`) knows how to **list available versions** and **provide a builder**: - -- `available_versions` / `latest` — lazily fetched and cached (CPython fetches from `python.org/ftp`, or GitHub tags when `cpython-use-github` is set). This lazy fetch is why [`list`](/cli/list.md) hits the network on first use. -- `get_builder()` — returns the concrete builder class ([`Cpython`](/architecture/cpython.md) for the cpython family). -- `min_version` — earliest non-EOL version known to compile well (currently `3.9`). - -## Why a singleton - -The build touches global, process-wide facts: which config file is active, what platform we are targeting, and which python versions exist. Centralizing them in `PPG` avoids threading that state through every constructor, and gives the CLI a single place ([`main`](/cli/index.md)) to initialize it. +A `VersionFamily` (e.g. `CPythonFamily`) lists the available versions and hands back the right builder ([`Cpython`](/architecture/cpython.md) for cpython). Versions are fetched lazily and cached — from `python.org/ftp`, or GitHub tags when configured — which is why [`list`](/cli/list.md) hits the network on first use. A family also pins the minimum supported version (non-EOL only). diff --git a/docs/architecture/python-builder.md b/docs/architecture/python-builder.md index 9fe8837..ea246ff 100644 --- a/docs/architecture/python-builder.md +++ b/docs/architecture/python-builder.md @@ -8,26 +8,18 @@ timestamp: 2026-06-23T00:00:00Z # PythonBuilder -`PythonBuilder` extends [`ModuleBuilder`](/architecture/module-builder.md) with behavior specific to building a python interpreter. It is still abstract — the concrete implementation is [`Cpython`](/architecture/cpython.md). Defined in `__init__.py`. +The abstract [`ModuleBuilder`](/architecture/module-builder.md) specialization for building a python interpreter (`__init__.py`); the concrete implementation is [`Cpython`](/architecture/cpython.md). -# Schema +## What it adds over ModuleBuilder -| Member | Kind | Purpose | -|--------|------|---------| -| `modules` | attribute (`ModuleCollection`) | The external modules selected/available for this build. | -| `selected_modules()` | method | Builds the `ModuleCollection` from candidates + the desired list. | -| `bin_python` | property | Path to the built interpreter (`.../bin/python`). | -| `version` | property | The python version being built. | -| `validate_setup()` | method | Pre-flight checks before compilation begins. | -| `run_python(*args)` | method | Invoke the freshly built interpreter (used during finalize/validation). | -| `xenv_LDFLAGS()` | method | Python-specific linker flags layered on the base `ModuleBuilder` behavior. | -| `_prepare()` | method | Hook run before the platform compile. | +Beyond the base compile flow, a `PythonBuilder` owns the **module selection** (which [external modules](/modules/index.md) to build) and can **run the freshly-built interpreter** — used during finalize and validation. ## ModuleCollection -A `PythonBuilder` owns a `ModuleCollection`, which models the set of candidate external modules and resolves which are actually built: +A `PythonBuilder` owns a `ModuleCollection` that resolves which external modules actually get built — a distinction worth keeping straight: -- `candidates` — every possible module for this builder (from `candidate_modules()`). -- `selected` — only the modules chosen for *this* build (config + `--modules` + auto-selection). This is the list that actually gets compiled. -- `auto_selected` — modules force-selected because a build can't succeed without them (each module's `auto_select_reason()`). -- `report()` / `report_rows()` — the human-readable table shown by [`build-report`](/cli/build-report.md). +- **candidates** — every module this builder *could* compile. +- **selected** — the ones chosen for *this* build (config + `--modules` + auto-selection). This is what's actually compiled, not all candidates. +- **auto-selected** — modules force-included because the build can't succeed without them (each module's `auto_select_reason()`). + +It's what [`build-report`](/cli/build-report.md) renders. diff --git a/docs/architecture/python-inspector.md b/docs/architecture/python-inspector.md index 24f4d4f..cf1ff8f 100644 --- a/docs/architecture/python-inspector.md +++ b/docs/architecture/python-inspector.md @@ -8,45 +8,18 @@ timestamp: 2026-06-23T00:00:00Z # PythonInspector -`PythonInspector` (in `inspector.py`) answers the question the whole project is built around: *is this python installation [portable](/concepts/portability.md), and if not, why not?* It powers the [`inspect`](/cli/inspect.md) command and the post-build validation in [`BuildSetup.compile()`](/architecture/build-setup.md). +`PythonInspector` (`inspector.py`) answers the question the whole project exists for: *is this python [portable](/concepts/portability.md), and if not, why not?* It powers the [`inspect`](/cli/inspect.md) command and the portability gate at the end of [`BuildSetup.compile()`](/architecture/build-setup.md). -```python -from portable_python.inspector import PythonInspector +## How it works -inspector = PythonInspector("/usr/bin/python3") -print(inspector.represented()) -problem = inspector.full_so_report.get_problem(portable=True) -if problem: - print("not portable: %s" % problem) -``` +It walks every executable and `.so` in an installation, lists each one's dynamic-library dependencies (parsing `ldd` on Linux / `otool` on macOS), and classifies every reference: -# Schema +- **system / standard** library → fine; +- **relative / self** reference → fine (what a relocatable build should have); +- **absolute, non-system** reference → a portability problem. -| Member | Kind | Purpose | -|--------|------|---------| -| `__init__(spec, modules)` | constructor | Resolve the python to inspect and the module set to report. | -| `full_so_report` | property (`FullSoReport`) | Aggregated scan of every `.so`/executable and its lib references. | -| `module_info` | property | Per-module info (versions, extra detail). | -| `represented(verbose)` | method | Render the human-readable report. | -| `get_problem(portable)` | method (on `FullSoReport`) | Return a description of the first portability problem, or empty if clean. | +For a portable build, any problem fails the inspection (the report exposes a single "first problem" verdict that callers check). -## Supporting types +## Relativization -| Type | Role | -|------|------| -| `SoInfo` | One `.so`/executable; parses `ldd` (Linux) / `otool` (macOS) output into references. | -| `CLibInfo` | One referenced C library, classified by `LibType`. | -| `LibType` | Enum classifying a reference (system, relative/self, problematic, …). | -| `FullSoReport` | The complete scan and its `get_problem()` verdict. | -| `LibAutoCorrect` | Rewrites absolute lib references in binaries to **relative** form (`@loader_path` / `$ORIGIN`), making an install relocatable. Exposed via [`lib-auto-correct`](/cli/lib-auto-correct.md). | -| `Tracker` / `Trackable` | Categorize the found objects (libs, problems) by type and render the detailed reports (`tracking.py`); `SoInfo` and `CLibInfo` are `Trackable`. | - -## How it detects problems - -For each binary it lists dynamic dependencies, then classifies every reference: - -- **System / standard** library → acceptable. -- **Relative / self** reference → acceptable (this is what `LibAutoCorrect` produces). -- **Absolute, non-system** reference → a portability problem; for a portable build this fails the inspection. - -`LibAutoCorrect` is also used **during** the build's finalize step to convert references to relative form before the portable check runs. +The fix lives in `LibAutoCorrect`: it rewrites absolute library references to relative form (`@loader_path` / `$ORIGIN`). It runs **during** the build's finalize step (so the result passes the portable check), and is also exposed standalone as [`lib-auto-correct`](/cli/lib-auto-correct.md) for iterating without a full rebuild. diff --git a/docs/log.md b/docs/log.md index 7288726..d785d14 100644 --- a/docs/log.md +++ b/docs/log.md @@ -2,6 +2,7 @@ ## 2026-06-23 +* **Update**: Reshaped the six architecture `type: Class` pages — dropped the `# Schema` member tables (which mirrored code and risked drift) for a condensed mental-model + "worth knowing" framing; the code stays authoritative for the API. * **Update**: Removed the `resource:` frontmatter field from all concept files — unrendered, unreferenced by any tooling, and brittle (GitHub URLs). Source is pointed to inline in the prose instead. The bundle now contains no `github.com` repo links. * **Update**: Removed the `# Citations` sections from all concept files — they duplicated the `resource:` frontmatter and inline source mentions, and added brittle GitHub-URL boilerplate that nothing referenced. Source pointers now live only in `resource:` and inline prose. * **Update**: Inverted the `docs/` ↔ `CLAUDE.md` relationship — `CLAUDE.md` is now a thin pointer into the bundle. Folded the test-harness and code-style notes into [local development](/guides/local-development.md), and removed all back-references to `CLAUDE.md` from `docs/`. From da663e5c4fbf8c93c81598eac9c3ab459c81f292 Mon Sep 17 00:00:00 2001 From: Zoran Simic Date: Tue, 23 Jun 2026 11:17:31 -0700 Subject: [PATCH 4/5] Simplified CLI usage docs --- docs/cli/build-report.md | 13 ++----------- docs/cli/build.md | 31 +++++++++---------------------- docs/cli/diagnostics.md | 6 ++---- docs/cli/index.md | 14 ++------------ docs/cli/inspect.md | 29 ++++++++--------------------- docs/cli/lib-auto-correct.md | 19 ++++--------------- docs/cli/list.md | 9 ++------- docs/cli/recompress.md | 11 ++--------- docs/log.md | 1 + 9 files changed, 32 insertions(+), 101 deletions(-) diff --git a/docs/cli/build-report.md b/docs/cli/build-report.md index 9a1db91..991b657 100644 --- a/docs/cli/build-report.md +++ b/docs/cli/build-report.md @@ -8,22 +8,13 @@ timestamp: 2026-06-23T00:00:00Z # build-report -Show which modules would be compiled for a given spec, and validate that the selection can actually build on this host — without running a build. It prints the [`ModuleCollection.report()`](/architecture/python-builder.md) table and then calls `validate_module_selection()`. +Show which [modules](/modules/index.md) would be compiled for a spec, and whether the selection can actually build on this host — **without** building. The fastest way to catch a "broken" module before committing to a full compile. ``` portable-python build-report [OPTIONS] [PYTHON_SPEC] ``` -# Schema - -| Argument / Option | Purpose | -|-------------------|---------| -| `PYTHON_SPEC` (optional) | Version to report on; defaults to the latest known version. | -| `--modules, -m CSV` | Specific modules to check. | - -## What it tells you - -For each candidate module the report shows its [telltale](/concepts/telltale-detection.md) status and its `linker_outcome`: whether it will be compiled `static`, linked `shared`, is `absent`, or is `failed` (a combination that cannot build on this host — e.g. a `-`-sigil Debian dev package being present). This is the fastest way to catch a "broken" module before committing to a full build. +For each candidate module it shows the [telltale](/concepts/telltale-detection.md) status and the linker outcome — `static` (will be compiled), `shared` (system copy used), `absent`, or `failed` (can't build here, e.g. a `-`-sigil Debian dev package is present). `PYTHON_SPEC` defaults to the latest known version. # Examples diff --git a/docs/cli/build.md b/docs/cli/build.md index 8d4ffc0..f490909 100644 --- a/docs/cli/build.md +++ b/docs/cli/build.md @@ -8,36 +8,23 @@ timestamp: 2026-06-23T00:00:00Z # build -Build a portable python binary. Constructs a [`BuildSetup`](/architecture/build-setup.md) and calls `compile()`. +Build a portable python binary from source — constructs a [`BuildSetup`](/architecture/build-setup.md) and runs it. The output tarball lands in `dist/` (e.g. `dist/cpython-3.13.2-macos-arm64.tar.gz`; see [build layout](/concepts/build-layout.md)). ``` portable-python build [OPTIONS] PYTHON_SPEC ``` -# Schema +Worth knowing (`--help` for the full flag list): -| Argument / Option | Purpose | -|-------------------|---------| -| `PYTHON_SPEC` (required) | Full version to build, e.g. `3.13.2` or `cpython:3.13.2`. A bare `X.Y` is rejected — give `X.Y.Z`. `latest` resolves to the newest known version. | -| `--modules, -m CSV` | External modules to include, e.g. `-m openssl,zlib`. Default comes from config + auto-selection. | -| `--prefix, -p PATH` | Build for a fixed install location. **Disables portability** — the result must live at `PATH`. | - -## Output - -The finished tarball lands in `dist/`, named for the family, version, platform, and arch, e.g. `dist/cpython-3.13.2-macos-arm64.tar.gz`. See [build layout](/concepts/build-layout.md). +- The spec must be a **full** `X.Y.Z` (a bare `3.13` is rejected); `latest` resolves to the newest known version. +- `--prefix` builds for a fixed location and **disables portability** (see [ppp-marker](/concepts/ppp-marker.md)); without it you get a relocatable build. +- `--modules/-m` overrides which [external modules](/modules/index.md) are included (default: config + auto-selection). # Examples ```shell -# Portable build, modules auto-selected from config -portable-python build 3.13.2 - -# Restrict to specific modules -portable-python build 3.13.2 -m openssl,zlib,xz - -# Non-portable build pinned to a prefix -portable-python build 3.13.2 --prefix /apps/python3.13 - -# See exactly what would happen, without compiling -portable-python --dryrun build 3.13.2 +portable-python build 3.13.2 # portable, modules auto-selected +portable-python build 3.13.2 -m openssl,zlib,xz # restrict modules +portable-python build 3.13.2 --prefix /apps/python3.13 # non-portable, fixed location +portable-python --dryrun build 3.13.2 # show the plan, compile nothing ``` diff --git a/docs/cli/diagnostics.md b/docs/cli/diagnostics.md index ce94e12..a103835 100644 --- a/docs/cli/diagnostics.md +++ b/docs/cli/diagnostics.md @@ -8,12 +8,10 @@ timestamp: 2026-06-23T00:00:00Z # diagnostics -Print system diagnostics next to the resolved [configuration](/architecture/config.md), as a two-column table. Useful when a build behaves unexpectedly and you need to confirm which config files and platform are in effect. +Print system diagnostics next to the resolved [configuration](/architecture/config.md). Reach for it when a build behaves unexpectedly and you want to confirm which config files and target platform are actually in effect. ``` portable-python diagnostics ``` -# Schema - -Takes no arguments. The left column comes from `runez.SYS_INFO` (invoker python, platform, etc.); the right column is `PPG.config.represented()` — the merged config and the files it was loaded from. +It shows the invoker python and platform info on one side, and the merged config — plus the files it was loaded from — on the other. diff --git a/docs/cli/index.md b/docs/cli/index.md index 70ef93a..076c090 100644 --- a/docs/cli/index.md +++ b/docs/cli/index.md @@ -1,22 +1,12 @@ # CLI -`portable-python` is a single-entry-point CLI built with `click` (via `runez.click`). All commands are defined in `cli.py`. +`portable-python` is a single-entry-point CLI (built with `click` via `runez.click`); every command lives in `cli.py`. ``` portable-python [GLOBAL OPTIONS] COMMAND [ARGS] ``` -## Global options - -| Option | Purpose | -|--------|---------| -| `--config, -c PATH` | Config file to use (default: `portable-python.yml`). | -| `--quiet, -q` | Turn off DEBUG logging. | -| `--dryrun, -n` | Show what would be done without doing it. | -| `--target, -t` | Override the detected platform (internal/testing). | -| `--version` / `--color` | Standard `runez` options. | - -The `main` group initializes logging and calls [`PPG.grab_config()`](/architecture/ppg.md) before any subcommand runs. +Two global options are worth knowing: `--config/-c` (which [config](/configuration/portable-python-yml.md) file to use) and `--dryrun/-n` (print what *would* happen without doing it — the fastest way to understand a build). `portable-python --help` lists the rest. The top-level group loads config (via [`PPG`](/architecture/ppg.md)) before any subcommand runs. ## Commands diff --git a/docs/cli/inspect.md b/docs/cli/inspect.md index ac883d8..ce20c80 100644 --- a/docs/cli/inspect.md +++ b/docs/cli/inspect.md @@ -8,35 +8,22 @@ timestamp: 2026-06-23T00:00:00Z # inspect -Inspect any python installation and report whether it is [portable](/concepts/portability.md). Backed by the [`PythonInspector`](/architecture/python-inspector.md); aborts non-zero if a portability problem is found (unless `--prefix` is given). +Inspect any python installation and report whether it is [portable](/concepts/portability.md) — powered by the [`PythonInspector`](/architecture/python-inspector.md). ``` portable-python inspect [OPTIONS] PATH ``` -# Schema +Worth knowing (`--help` for the full flag list): -| Argument / Option | Purpose | -|-------------------|---------| -| `PATH` (required) | Path to a python installation or interpreter. The special value `invoker` inspects the python running the CLI. | -| `--modules, -m MODULES` | Which modules to inspect (`all` to check everything). | -| `--verbose, -v` | Show the full `.so` report. | -| `--prefix, -p` | The build used `--prefix` (not portable) — relaxes the portability check accordingly. | -| `--skip-so, -s` | Don't scan every `.so` file. | - -## Exit behavior - -After printing the report, if scanning `.so` files (and not skipped), it asks `full_so_report.get_problem(portable=not prefix)`. A non-empty problem aborts with a non-zero exit — making `inspect` usable as a CI gate. +- It **exits non-zero** when it finds a portability problem, so it works as a CI gate. +- `PATH` can be the special value `invoker` to inspect the python running the CLI itself. +- `--prefix` says the build was pinned to a prefix (not portable), relaxing the check; `--verbose/-v` shows the full `.so` report. # Examples ```shell -# Is the system python portable? (it usually is not) -portable-python inspect /usr/bin/python3 - -# Full detail -portable-python inspect -v ~/versions/3.13.2 - -# Inspect the interpreter running this CLI -portable-python inspect invoker +portable-python inspect /usr/bin/python3 # system python — usually NOT portable +portable-python inspect -v ~/versions/3.13.2 # full detail +portable-python inspect invoker # the python running this CLI ``` diff --git a/docs/cli/lib-auto-correct.md b/docs/cli/lib-auto-correct.md index e2ac553..2fd3deb 100644 --- a/docs/cli/lib-auto-correct.md +++ b/docs/cli/lib-auto-correct.md @@ -8,28 +8,17 @@ timestamp: 2026-06-23T00:00:00Z # lib-auto-correct -Scan a python installation and auto-correct its executables and libraries to reference dependencies by **relative** path. This exercises the [`LibAutoCorrect`](/architecture/python-inspector.md) logic that the build runs internally — handy for iterating on relativization without waiting for a full build. +Rewrite a python installation's executables and libraries to reference their dependencies by **relative** path — the same [`LibAutoCorrect`](/architecture/python-inspector.md) step the build runs internally, exposed standalone so you can iterate on relativization without a full rebuild. See [static linking](/concepts/static-linking.md) for why it's needed. ``` portable-python lib-auto-correct [OPTIONS] PATH ``` -# Schema - -| Argument / Option | Purpose | -|-------------------|---------| -| `PATH` (required) | The python installation to scan. | -| `--commit` | Actually perform the changes. Without it, the command runs in dry-run mode. | -| `--prefix, -p PATH` | The `--prefix` the program was built with. Defaults to the scanned python's own `sysconfig` prefix. | - -By default this is **dry-run**: it shows what it would rewrite. Pass `--commit` to apply the changes. See [static linking](/concepts/static-linking.md) for why relativization is needed. +It is **dry-run by default** — it shows what it would rewrite; pass `--commit` to actually apply the changes. `--prefix` overrides the build prefix (default: the scanned python's own `sysconfig` prefix). # Examples ```shell -# Preview corrections (dry-run) -portable-python lib-auto-correct ~/versions/3.13.2 - -# Apply them -portable-python lib-auto-correct --commit ~/versions/3.13.2 +portable-python lib-auto-correct ~/versions/3.13.2 # preview (dry-run) +portable-python lib-auto-correct --commit ~/versions/3.13.2 # apply ``` diff --git a/docs/cli/list.md b/docs/cli/list.md index 6db7e16..182688e 100644 --- a/docs/cli/list.md +++ b/docs/cli/list.md @@ -8,18 +8,13 @@ timestamp: 2026-06-23T00:00:00Z # list -List the latest available versions for a [version family](/architecture/ppg.md). Versions are fetched lazily (from `python.org/ftp`, or GitHub tags when `cpython-use-github` is configured) and cached. +List the latest available versions for a [version family](/architecture/ppg.md) (default `cpython`). ``` portable-python list [OPTIONS] [FAMILY] ``` -# Schema - -| Argument / Option | Purpose | -|-------------------|---------| -| `FAMILY` (optional) | Family to list; defaults to `cpython`. Unknown families abort with a message. | -| `--json` | Emit the available versions as JSON instead of a table. | +Versions are fetched lazily from the network (python.org, or GitHub tags when `cpython-use-github` is set) and cached, so the first run may pause briefly. `--json` emits machine-readable output. # Examples diff --git a/docs/cli/recompress.md b/docs/cli/recompress.md index 014279d..f0f8444 100644 --- a/docs/cli/recompress.md +++ b/docs/cli/recompress.md @@ -8,20 +8,13 @@ timestamp: 2026-06-23T00:00:00Z # recompress -Re-compress an existing build (folder or tarball) into a different compression format. Mildly useful for comparing the size trade-offs of different compressions. +Re-compress an existing build (folder or tarball) into a different format and print the before/after sizes — handy for comparing compression trade-offs. A utility, not part of a normal build. ``` portable-python recompress PATH EXT ``` -# Schema - -| Argument | Purpose | -|----------|---------| -| `PATH` (required) | An existing folder or tarball. Resolved against the configured base/build/dist/destdir folders if not absolute. | -| `EXT` (required) | Target compression extension. Constrained to the platform's `supported_compression` choices. | - -After re-compressing, the command prints the size of both the source and the result so they can be compared. +`PATH` is an existing folder or tarball (resolved against the configured build/dist folders if relative); `EXT` is the target compression, limited to the platform's supported formats (`--help` lists them). # Examples diff --git a/docs/log.md b/docs/log.md index d785d14..8b74349 100644 --- a/docs/log.md +++ b/docs/log.md @@ -2,6 +2,7 @@ ## 2026-06-23 +* **Update**: Trimmed the CLI pages — dropped the per-command option tables (which mirror `--help`) for purpose + non-obvious gotchas + examples, pointing at `--help` for the full flag list. * **Update**: Reshaped the six architecture `type: Class` pages — dropped the `# Schema` member tables (which mirrored code and risked drift) for a condensed mental-model + "worth knowing" framing; the code stays authoritative for the API. * **Update**: Removed the `resource:` frontmatter field from all concept files — unrendered, unreferenced by any tooling, and brittle (GitHub URLs). Source is pointed to inline in the prose instead. The bundle now contains no `github.com` repo links. * **Update**: Removed the `# Citations` sections from all concept files — they duplicated the `resource:` frontmatter and inline source mentions, and added brittle GitHub-URL boilerplate that nothing referenced. Source pointers now live only in `resource:` and inline prose. From e26fe5ba1e55b3ab2d41f95db4a23bcbe5070c0c Mon Sep 17 00:00:00 2001 From: Zoran Simic Date: Tue, 23 Jun 2026 13:48:08 -0700 Subject: [PATCH 5/5] Integrated info from wiki --- docs/architecture/python-inspector.md | 4 +++- docs/configuration/portable-python-yml.md | 2 +- docs/log.md | 1 + 3 files changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/architecture/python-inspector.md b/docs/architecture/python-inspector.md index cf1ff8f..8cb6005 100644 --- a/docs/architecture/python-inspector.md +++ b/docs/architecture/python-inspector.md @@ -22,4 +22,6 @@ For a portable build, any problem fails the inspection (the report exposes a sin ## Relativization -The fix lives in `LibAutoCorrect`: it rewrites absolute library references to relative form (`@loader_path` / `$ORIGIN`). It runs **during** the build's finalize step (so the result passes the portable check), and is also exposed standalone as [`lib-auto-correct`](/cli/lib-auto-correct.md) for iterating without a full rebuild. +The fix lives in `LibAutoCorrect`, which rewrites the absolute library paths (rpaths) baked into binaries so they resolve relative to the install — `patchelf` on Linux (an `$ORIGIN`-relative rpath), `install_name_tool` on macOS (`@rpath` / `@loader_path`). It runs **during** finalize (so the result passes the portable check) and is exposed standalone as [`lib-auto-correct`](/cli/lib-auto-correct.md). + +The same step also supports **non-portable** builds: with `--prefix` or `--enable-shared`, it additionally keeps the real prefix's `lib/` in the rpath, so a build pinned to a fixed location still finds its libraries. (CPython is built with a deliberately long rpath — unlike `chrpath`, `patchelf` can set a fresh rpath of any length, leaving room to rewrite it.) diff --git a/docs/configuration/portable-python-yml.md b/docs/configuration/portable-python-yml.md index a726839..a07dd99 100644 --- a/docs/configuration/portable-python-yml.md +++ b/docs/configuration/portable-python-yml.md @@ -49,7 +49,7 @@ Replace `{module}` with the lowercased module name (e.g. `openssl`): | Key | Purpose | |-----|---------| | `{module}-version` | Version to build. | -| `{module}-url` | Source URL (`$version` is substituted). | +| `{module}-url` | Source URL (`$version` is substituted; may carry a `#sha256=…` fragment, verified on download). | | `{module}-src-suffix` | Archive extension when the URL lacks one. | | `{module}-configure` | **Replace** the default configure args (`$deps_lib` substituted). | | `{module}-http-headers` | Headers for the download (env vars expanded). | diff --git a/docs/log.md b/docs/log.md index 8b74349..b650c3d 100644 --- a/docs/log.md +++ b/docs/log.md @@ -2,6 +2,7 @@ ## 2026-06-23 +* **Update**: Pulled two verified facts in from the project wiki — the relativization tools (`patchelf` / `install_name_tool`) and `--prefix` / `--enable-shared` support ([PythonInspector](/architecture/python-inspector.md)), and the `#sha256=` download-checksum URL fragment ([configuration](/configuration/portable-python-yml.md)). Skipped the wiki's `sources:` download-url design — it's an unimplemented draft. * **Update**: Trimmed the CLI pages — dropped the per-command option tables (which mirror `--help`) for purpose + non-obvious gotchas + examples, pointing at `--help` for the full flag list. * **Update**: Reshaped the six architecture `type: Class` pages — dropped the `# Schema` member tables (which mirrored code and risked drift) for a condensed mental-model + "worth knowing" framing; the code stays authoritative for the API. * **Update**: Removed the `resource:` frontmatter field from all concept files — unrendered, unreferenced by any tooling, and brittle (GitHub URLs). Source is pointed to inline in the prose instead. The bundle now contains no `github.com` repo links.