lance 13 / arrow 58 / datafusion 54

[AdaWorldAPI]

Summary

This moves ladybug-rs onto the workspace's lance 13 line. It is the migration the manifest comment from 2026-09-25 says "lands together with that migration, never alone".

• Dependencies:
  • lance 2.0 → 13.*
  • arrow*, parquet and arrow-flight 57 → 58
  • datafusion 51 → 54
  • sqlparser 0.60 → 0.62 (the version datafusion-sql 54 uses)
• lzma link conflict: cargo allows only one package per build to link the native lzma library. The n8n vendor crates (datafusion 51) link it through xz2, and datafusion 54's default compression feature links it through liblzma. I took datafusion 54 with its default features minus compression. ladybug never reads compressed files through DataFusion, and lance 13 doesn't enable compression either.
• n8n-rs stays on arrow 57 / datafusion 51 / lance 2.0. No arrow type crosses the n8n boundary: ladybug only calls n8n_grpc::handle_api_{post,get}, which take strings and bytes. So cargo builds both arrow lines side by side.
• lance 13 storage code: storage/lance*.rs and query/datafusion.rs compiled unchanged.
• DataFusion 54 API port, in query/{cognitive_udfs,dn_tree_provider,fingerprint_table,graph_provider,scent_scan}.rs:
  • as_any was removed from ScalarUDFImpl, ExecutionPlan and TableProvider (they now take Any as a supertrait). I removed the 22 impls and the std::any::Any imports they left unused.
  • ExecutionPlan::properties now returns &Arc<PlanProperties>, so the four plan structs store Arc<PlanProperties>.

Test plan

Rust 1.99, debug info off. cargo test --lib on this branch and on main (lance 2.0 / DF 51), run with RUST_MIN_STACK=64MiB; with the default stack, both runs abort part-way with a stack overflow, also on main.

	passed	failed
main	1455	27
this PR	1455	27

• The failure sets are identical: no test fails only on this branch.
• The lance storage and DataFusion query tests account for 86 passes. Their one failure, query::dn_tree_provider::tests::test_dn_tree_provider_with_data, panics at storage/bind_space.rs:451 on main as well.
• cargo check --all-targets: the library and lib tests compile. tests/proof_level_a_gaps.rs does not compile because it imports ladybug::core::simd, which does not exist on main either.

The first commit's message says "WIP / not yet compiled". That was its state when pushed; the second commit completes the port, and everything above was verified on the final head.

🤖 Generated with Claude Code

https://claude.ai/code/session_017YPZhKpSxN7eSxk7VH2vzB

---

Generated by Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-10T08:41:59.289831Z	924ad53	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[AdaWorldAPI]

CI status on a6e1b51. This PR now fixes two things that used to stop CI before any test could run:

• protoc missing (ddee1fa). Lance 13's lance-encoding compiles .proto files in its build script. The proof.yml and ci-master.yml jobs now install protobuf-compiler and libprotobuf-dev.
• tests/proof_level_a_gaps.rs did not compile on main (a6e1b51). It still imported core::simd, which ced28ba removed. The test now imports the current core::hamming_distance and keeps a local scalar reference to compare against. Locally it passes 6/6.

Unit Tests (all modules) is still red, but those failures are not caused by this PR:

• The library tests abort with a stack overflow in spo::codebook_training::tests::test_concept_learning. That happens on main too when run without RUST_MIN_STACK.
• spectroscopy::detector::tests::test_different_containers_can_yield_different_styles is one of the failures that main also has.
• I ran main and this branch locally, with RUST_MIN_STACK=64MiB. Both gave 1455 passed / 27 failed, and the failing tests were the same on both.
• main's CI never reached these tests, because it stopped at the compile errors above.

Fixing the 27 failing tests that main already has would widen this Lance 13 / DataFusion 54 PR, so I'm leaving them for a separate change. I'll keep watching this PR.

---

Generated by Claude Code