Generate centrally managed Python serialization libraries

[dellaert]

Adds an opt-in shared library that owns Boost registrations and text-archive operations for multiple Python extensions, with static or shared Boost. Existing wrapping stays unchanged. Part of #211; no GTSAM-specific code or downstream changes.

Reviewing guide — start with these three files

1. gtwrap/serialization_library.py — The archive owner. Check the generated typed save/load API, private export registrations, exception translation, and symbol visibility. Archive construction must stay inside this library, particularly with static Boost.
2. gtwrap/pybind_wrapper.py — The consumer path. Check that managed .serialize(), .deserialize(), and pickle bindings call the owner and suppress local registrations, while unselected types retain existing behavior.
3. cmake/PybindWrap.cmake — The integration contract. Check pybind_add_serialization_library, the trailing SERIALIZATION_LIBRARY option, private Boost linkage, generation dependencies, and source-tree/installed-package paths.

Contract

Consumers supply the complete polymorphic serialization domain, use compatible C++ ABIs/runtimes, and own installation and runtime search paths. Legacy archive identifiers and pickle's one-element tuple are preserved.

Windows static-Boost exception: precompiled MSVC libraries force some helper exports. CI permits only 28 exact reviewed names for this configuration; they are unsupported implementation details. Unknown exports and missing entry points still fail, and isolation from an unrelated Boost runtime remains mandatory. Other configurations require exactly the generated entry points.

Validation

• All six runtime jobs pass: Linux/macOS/Windows × static/shared Boost. Coverage includes both consumer load orders, pickle, polymorphic and legacy archive exchange, malformed input, runtime isolation, incremental regeneration, and installed loading.
• All eight existing Python-version CI jobs pass. Local py312: 143 tests + 9 subtests pass, including existing Python/MATLAB generation tests.

No MATLAB runtime or GTSAM extension build was performed.

[dellaert]

Still making changes on this

[dellaert]

This pretends to implement what you wanted in the other PR. This took several prompts and I even looked at the code a bit :-). Please let me know what you think. I'll kick off a copilot review in parallel.

[Copilot]

🔵 Needs a closer look

Cross-library C++ ABI and Boost isolation require human review, and a blocking finding remains unresolved.

1 open finding

• Reject generated header name collisions with included headers · New

What changed in this PR

Adds an opt-in shared serialization owner for Python extensions, addressing #211 without downstream-specific changes.

Changes:

• Generates centralized Boost registrations and typed save/load functions.
• Connects managed bindings through CMake and a generated manifest.
• Adds documentation and cross-platform runtime validation.

File	Description
tests/​test_serialization_runtime.py	Validates runtime behavior, installation, and exports.
tests/​test_serialization_library.py	Checks generated API and registrations.
tests/​test_serialization_discovery.py	Tests selection and identifier collisions.
tests/​test_serialization_bindings.py	Tests managed and legacy bindings.
tests/​fixtures/​serialization_library/​windows_static_boost_exports.txt	Lists permitted Windows helper exports.
tests/​fixtures/​serialization_library/​unrelated.cpp	Exercises independent Boost runtime isolation.
tests/​fixtures/​serialization_library/​types.i	Declares serializable fixture types.
tests/​fixtures/​serialization_library/​types.h	Implements polymorphic fixture types.
tests/​fixtures/​serialization_library/​module.tpl	Provides the fixture binding template.
tests/​fixtures/​serialization_library/​legacy.cpp	Tests legacy archive compatibility.
tests/​fixtures/​serialization_library/​consumer_b.i	Defines the second consumer.
tests/​fixtures/​serialization_library/​consumer_a.i	Defines the first consumer.
tests/​fixtures/​serialization_library/​CMakeLists.txt	Builds and installs integration fixtures.
tests/​fixtures/​serialization_library/​check.py	Checks installed consumers and archive exchange.
scripts/​pybind_wrap.py	Accepts serialization manifests.
scripts/​pybind_serialization.py	Adds the owner-generation command.
gtwrap/​serialization.py	Discovers serialization declarations.
gtwrap/​serialization_library.py	Generates the archive owner and API.
gtwrap/​pybind_wrapper.py	Routes managed bindings through the owner.
DOCS.md	Documents integration and migration requirements.
CMakeLists.txt	Installs the generator and adds test targets.
cmake/​PybindWrap.cmake	Adds owner and consumer integration.
cmake/​gtwrapConfig.cmake.in	Configures installed generator paths.
.github/​workflows/​serialization.yml	Adds platform and Boost-linkage coverage.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.