Skip to content

fix(swift): preserve declaration identity and compatible call edges - #2436

Open
DavidHLP wants to merge 78 commits into
DeusData:mainfrom
DavidHLP:feat/issue-2061-swift-identity
Open

DavidHLP wants to merge 78 commits into
DeusData:mainfrom
DavidHLP:feat/issue-2061-swift-identity

Conversation

@DavidHLP

@DavidHLP DavidHLP commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Refs #2061. Give Swift overloads declaration-qualified identities, preserve bare node names, and use the uncapped selector inside receiver-typed candidates. Keep static/class, async, and explicit return markers. Qualified types retain their last segment, and the associated-type spelling in where T.Element is retained pending maintainer follow-up.

  • Match extensions to owners in visible SwiftPM targets using the registry's type-path owner map. Preserve both declaration and extension markers when they share a node, and include extension initializers in typed lookup.
  • Honor literal SwiftPM target paths and nested package boundaries. Changes to Package.swift invalidate source-only repair and trigger a full reindex.
  • Continue call chains when all compatible candidates agree on the return type. Preserve declaration ownership through conditional compilation, wrapped trailing closures, lexical closure alias shapes, and serial argument keywords.
  • Keep the index at format 5 with a single migration entry. The branch includes main through e0c7a1da, including fix(extraction): attribute Swift lifecycle calls #2395.
  • Cover receiver collisions, two same-named types in separate targets, cross-file extensions, manifest changes, lexical aliases, and serial, parallel, and incremental paths in the regression suites.

Validation

AI assistance: This update was prepared with an AI agent acting on behalf of DavidHLP.

  • Unmodified PR production build at 1a6b2923 passed the default production regression guards. On Alamofire bda9ed57, the chain tails at Tests/SessionTests.swift:1632 and Tests/DataStreamTests.swift:582 each resolved to the expected method. On swift-nio eabcc14f, the NIOFS and _NIOFileSystem cross-target CALLS check returned zero.
  • In the Alamofire checkout, changing the target path from Source to Source/Core triggered incremental.force_full. The rebuilt and fresh graphs of that same modified checkout matched on node and edge rows, including properties. Earlier general incremental comparisons recorded SIMILAR_TO and decorator_tags differences separately.
  • Local combined validation tree used 1a6b2923 plus the remaining test-environment and baseline fixes from draft fix: isolate validation and handle empty initialization state #2809, retaining main's mimalloc 3.5.3. The default scripts/test.sh passed with GCC, ASan and UBSan, and UBSAN_OPTIONS=halt_on_error=1. It completed 192 suites with 10,363 passed, zero failed, and 10 skips. The incremental suite ran all 164 tests using FastAPI 0.99.1 at dd4e78ca, with no skips. The fix: isolate validation and handle empty initialization state #2809 changes remain separate from this PR.
  • GitHub CI at 1a6b2923 passed Linux, macOS, Windows CLANG64, TSan, lint, CodeQL, DCO, shard completeness, and ci-ok. PR workflow · DCO

DeusData and others added 8 commits September 25, 2026 18:49
…Data#2061)

Overloads that share a base QN (Java/C#/C++/Kotlin/Swift/Scala/ObjC) merge
into one graph node today, and that node collects the CALLS of every
overload. The agreed cure is one signature-qualified callable identity for
all overloading languages:

    callable_qn := base_qn [<tparams>] (params) [cvref]

This change lands the plumbing with every language still at
CBM_CALLABLE_ID_NONE, so the graph is byte-identical; each language is
switched on later in its own change (with its index-format bump).

- internal/cbm/callable_sig.{c,h}: the per-language mode side table
  (NONE|TYPED|LABELED_TYPED|LABELED|ARITY; kept out of CBMLangSpec like
  cbm_string_dispatch_suffixes, since ~160 positional rows would otherwise
  all need the member under -Wmissing-field-initializers), the shared
  builder cbm_callable_sig[_mode] for the def and the call-scope QN, and
  the inverse cbm_qn_callable_base_len[_named]. Suffix contract: no '.',
  '::' or '->'; whitespace-normalized; comments/annotations/attributes
  dropped; qualified types keep their last segment; '->' spelled '=>';
  variadic '[]' (Java) / '~'; C++ top-level cv dropped, arrays decay,
  cvref kept; <= 200 bytes, longer ones keep whole leading entries and end
  in '#<fnv1a64 of the full suffix>)'. The builder is iterative (cursor
  walk, bounded declarator loop) and self-checks against the inverse.
- The inverse is anchored to the node's bare name wherever the name is
  known: real QNs already end in ')' without being callables (a Java field
  whose name is its declarator `X = f()`, make `$(VAR)` targets), and only
  the anchor makes the inverse exact on them.
- Leaf splitters strip a suffix first (no-op today): registry by-name key
  (indexed by the bare name for a suffixed callable), cbm_lsp_bare_segment,
  cbm_pipeline_qn_class_method_tail, pxc_last_component, pxc_qn_leaf,
  go_index_visit_node, collect_prod_defs, sg_qn_prefix_len.
- New unused fields: CBMDefinition.qn_sig_off, CBMResolvedCall.callee_sig
  (relocated by result compaction), CBMRegisteredFunc.sig, CBMLSPDef.sig.
  The lsp_surface codec is unchanged (no format bump in this change).
- get_code_snippet and trace_path gain base-QN tiers
  (cbm_store_find_nodes_by_qn_base) that find nothing until suffixes exist.
- store: cbm_store_find_nodes_by_qn_suffix built its LIKE pattern in a
  512-byte buffer (a long suffix was truncated into a pattern that matched
  nothing) and did not escape '_'/'%' (`my_func` also matched `myXfunc`).
  The pattern is now heap-built, escaped, and matched with ESCAPE '\'.

Proof: full sorted node+edge+lsp_surface dumps are byte-identical before
and after on 7 corpora (this repo, elasticsearch libs, Exposed, dotnet
System.Linq, System.Collections, coreclr/gc, a C corpus).

Refs DeusData#2061

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Enable typed, labeled Swift callable identities on the shared DeusData#2061 plumbing. Resolve calls by compatible labels, defaulted parameters, and trailing closures; retain all compatible overload edges with candidate counts. Cover registry, pipeline, and signature behavior.

Refs DeusData#2061

Signed-off-by: DavidHLP <lysf15520112973@163.com>
Signed-off-by: DavidHLP <lysf15520112973@163.com>
Force the parallel call pass on a Swift fixture and assert both label-compatible type overloads retain CALLS edges with the ambiguity count.

Refs DeusData#2061

Signed-off-by: DavidHLP <lysf15520112973@163.com>
Signed-off-by: DavidHLP <lysf15520112973@163.com>
A Swift call that shares its bare name with project symbols but matches
none of their overloads was handled differently by the two resolvers:
the parallel path cleared the bare-name match and fell through as
unresolved (keeping each resolver's empty-resolution service
fallbacks), while the sequential path dropped the call outright. Skip
the bare-name registry bind on the sequential path too — it would bind
a wrong overload — and continue as unresolved, so count < 0 behaves
exactly like any other unresolved call on each resolver.

Pipeline regressions pin the global-fetch fallback (DeusData#856) on both
paths: the incompatible local overload gets no wrong CALLS edge, and
the URL call keeps its HTTP_CALLS edge. The route-registration fallback
differs between the paths on main (sequential additionally requires a
'/'-leading first argument); that pre-existing divergence is out of
scope here.

Refs DeusData#2061

Signed-off-by: DavidHLP <lysf15520112973@163.com>
Signed-off-by: DavidHLP <lysf15520112973@163.com>
@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

Signed-off-by: DavidHLP <lysf15520112973@163.com>
Signed-off-by: DavidHLP <lysf15520112973@163.com>
@DeusData
DeusData force-pushed the feat/callable-identity-plumbing branch 2 times, most recently from 3441da4 to 5538355 Compare September 30, 2026 22:34
Signed-off-by: DavidHLP <lysf15520112973@163.com>
@DeusData DeusData added enhancement New feature or request parsing/quality Graph extraction bugs, false positives, missing edges priority/normal Standard review queue; useful PR with ordinary maintainer urgency. labels Oct 1, 2026
@DeusData

DeusData commented Oct 1, 2026

Copy link
Copy Markdown
Owner

Thank you for the detailed status update and for clearly distinguishing the actual PR head from unpublished work and unverified results. The changed-file scope includes callable identity, resolution and persisted metadata, so we need more time for a complete review together with the parent change. This remains queued for deeper review; this comment is not an approval or a request for another routine rebase. The queue is full, and we appreciate your patience and the care taken to document the remaining work.

Swift-scope checkpoint for DeusData#2061: capture bounded trailing-closure labels, match Swift overload candidates conservatively, and cover extraction, registry, compact/spill, serial, parallel, and incremental behavior.

Validation: ASan passed on the combined WIP source only; this A-only checkpoint has not been independently verified. GCC TSan, smoke, and memory gates remain failed/open. This commit is a scope checkpoint, not acceptance.

Signed-off-by: David HLP <lysf15520112973@163.com>
Signed-off-by: David HLP <lysf15520112973@163.com>
Signed-off-by: David HLP <lysf15520112973@163.com>
Signed-off-by: David HLP <lysf15520112973@163.com>
Signed-off-by: David HLP <lysf15520112973@163.com>
Signed-off-by: David HLP <lysf15520112973@163.com>
Signed-off-by: David HLP <lysf15520112973@163.com>
@DavidHLP
DavidHLP force-pushed the feat/issue-2061-swift-identity branch from 4dbfd27 to eea3143 Compare October 3, 2026 09:49
Signed-off-by: DavidHLP <lysf15520112973@163.com>
Signed-off-by: DavidHLP <lysf15520112973@163.com>
Signed-off-by: DavidHLP <lysf15520112973@163.com>
DavidHLP and others added 4 commits October 15, 2026 01:08
Signed-off-by: DavidHLP <lysf15520112973@163.com>
Signed-off-by: DavidHLP <lysf15520112973@163.com>
Preserve Swift identity and upstream image migration coverage. Advance the combined index format to 6 so pre-merge format-5 indexes also rebuild invented image hosts. Refs DeusData#2061.

Signed-off-by: DavidHLP <lysf15520112973@163.com>
@DeusData

Copy link
Copy Markdown
Owner

Thanks for the Swift identity update and for distinguishing current-head checks from earlier implementation validation. I have read your request for feedback on identity spelling and the migration boundary; those decisions remain on the maintainer side. Please hold off on further routine rebases while that review is pending.

@DeusData

Copy link
Copy Markdown
Owner

Thank you for the Swift identity measurements and the migration questions. Those decisions remain on our side; please continue to hold off on routine rebases while that review is pending. Sorry for the delay—I’ll follow up soon.

@DeusData DeusData left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @DavidHLP, for carrying the Swift identity on top of #2342 exactly as we proposed on #2061, for your patience while we held the spelling and migration questions on our side, and for the care in this branch. Keeping the selector uncapped and separate from the bounded identity, carrying it through serial, parallel and incremental indexing, and filtering candidates before the output limit are all the right calls. Here are our decisions on the questions you asked, and what we need before this can land. The hold on updates is lifted: please bring the branch up to current main (merging main in is fine).

We built your current head (4885230) and indexed Alamofire (bda9ed57) with it. The core fix works: on the #2061 repro, work(flag:Bool) and work(name:String) are separate nodes, onlyCallsOverloadB reaches only work(name:String), and the fake path to target() is gone. On Alamofire, 74 overload sets that were one node each are split into 404 nodes, including all 13 Session.upload overloads, and repeated runs give an identical graph.

  1. Identity spelling. Please keep the static/class and async markers and the explicit =>Return. They separate real Swift overloads: Alamofire's three encode(_:any Encodable) differ only in their return type. For qualified types, please use the last segment, as in the #2342 contract, instead of / paths. Two declarations that differ only in the scope of a same-named parameter type then stay one node, as they are on main today. That case is rare, and it keeps Swift QNs consistent with the other languages. One detail we will confirm separately: associated types of a generic parameter in a where-requirement (for example where T.Element == Int). Please keep your current spelling for that one case until we follow up on it.

  2. Keep the receiver's type in charge. The label block in resolve_single_call (and its twin in pass_parallel.c) runs before the registry's receiver-typed strategies and returns early, so the label shape can override the receiver's declared type. On Alamofire, swift_labels resolves 3,650 CALLS while field_type_hint resolutions fell from 884 to 1 compared with main at the time. In Tests/CachedResponseHandlerTests.swift:41, session is a Session, but session.request(.default) resolves to DataResponsePublisher.Inner.request(_:Subscribers/Demand) with candidates:1. That one pattern adds 175 edges from tests to the Combine Inner.request methods, the same kind of fake caller #2061 is about. Calls on standard-library values get bound too: output.sorted() on a [String] in URLEncodedFormEncoder.swift:1076 points at HTTPHeaders.sorted(). Please apply the label and arity filter inside the receiver-typed candidates rather than in front of them, and leave a call unresolved when the receiver is a known non-project type. A regression test with a typed receiver whose method shares its label shape with an unrelated class would pin this.

  3. Chained calls with trailing closures. Some calls that main resolves are dropped, for example session.request(.default).responseDecodable(of:) { ... } (Tests/SessionTests.swift:1632) and session.streamRequest(...).responseStream { ... } (Tests/DataStreamTests.swift:582). These are common Swift idioms, so a test for each would be very welcome.

  4. Language-neutral names for the new fields. CBMDefinition and CBMCall are shared by every language, and Kotlin, Scala and C# will need default-argument and trailing-lambda matching when their identity changes land. Please give the new struct fields and the persisted property keys neutral names (for example param_default_mask, param_count, call_selector and trailing_closure_*), so those changes can reuse them.

  5. Index format 5. main is at index format 4 (#1913, receiver-qualified Go methods), so this change becomes format 5 with a single history entry. The format-6 step and the format-5 image migration test cover a state that only existed on this branch (main already stopped inventing image hosts in format 3), so please drop both. When you update the branch you will see conflicts in pass_calls.c (main now resolves through cbm_registry_resolve_call_scoped), pass_definitions.c and pass_parallel.c (main reworked the def-properties sizing into per-field helpers, so the selector becomes one more term there, and registration now goes through cbm_registry_add_lang_in_module), pipeline_incremental.c, pipeline.h and store.h.

  6. Initializers. #2395 is about to land and makes Swift init and deinit callable definitions (Box.init becomes a Method node). Because the identity applies to every callable, your branch will give them signature QNs too, for example Box.init(name:String), and Box.deinit() under the zero-argument rule, which is what we want. Please update the #2395 tests that look up the unsuffixed Box.init and Deinitable.deinit definitions, and add tests where two init overloads stay separate nodes, a super.init call reaches the matching Base.init overload, and an external UIViewController.init(nibName:bundle:) with a label-compatible project init forms no edge. #2395 makes self.init form no edge because overloads used to share one node; we will revisit that separately once your change is in, so please leave it as it is.

  7. Three smaller points from reading the code:
    a. When the signature cannot be built, extract_func_def, push_method_def and compute_func_qn now return without a definition or QN. Please fall back to the plain base QN instead, so a definition is never lost.
    b. cbm_service_pattern_match now strips a callable suffix for every language with cbm_qn_callable_base_len. Other languages' QNs can end in ')' for other reasons, so please use the name-anchored form or limit it to Swift.
    c. cbm_registry_swift_candidates scans the whole same-name bucket and allocates on every call. With initializers in the graph, init becomes by far the largest Swift bucket, so the cost grows with initializers times calls. Please look up the named type's overloads directly when the call has a type receiver, and drop a small Swift fixture into tests/fixtures/complexity/ so the complexity suite covers it (no test code change is needed there).

Before merging we will run a Swift call-graph census on the updated branch ourselves (Alamofire, one larger Swift project and a non-Swift control), so you do not need to produce those numbers.

To be clear about what is already good here: outside the session.request cluster, a random sample of call sites whose target changed came out better with your branch than with main (3 fixed, none broken, the rest wrong either way), and most of the 372 calls that only your branch resolves are correct, for example Interceptor(adapter:retrier:) and self.state.write. So the label matching itself is a real gain; it just must not override the receiver.

Thank you again for the very careful work on this, and sorry it took us a while to come back with the decisions.

Keep receiver types ahead of overload labels, preserve chained trailing closures, and index typed overload buckets directly. Use neutral call metadata, retain base identities on signature failure, and publish index format 5. Preserve current main registration and properties sizing.

Signed-off-by: DavidHLP <lysf15520112973@163.com>
…ritance

Signed-off-by: DavidHLP <lysf15520112973@163.com>
Signed-off-by: DavidHLP <lysf15520112973@163.com>
@DavidHLP

DavidHLP commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor Author

Following up on your review: I have addressed the requested contributor-side changes at debefaff45f4c77b9ba17a87d1d707eacc8d8345.

  1. Identity spelling: kept static/class, async, and explicit =>Return markers; qualified types now use the last segment. The generic associated-type where T.Element case retains its existing spelling pending your follow-up.
  2. Receiver precedence: label/arity/default/trailing-closure matching now filters receiver-typed candidates. Known non-project receivers stay unresolved. Regressions cover unrelated classes with the same label shape, standard-library values, and lexical shadowing.
  3. Chained trailing closures: added coverage for request/responseDecodable and streamRequest/responseStream chains, including both Swift AST locations for trailing closures.
  4. Shared metadata: new fields and persisted keys are language-neutral. selector_param_count is distinct from the existing complexity metric param_count.
  5. Integration and index format: merged main at 8530c8cd, retaining scoped resolution, module-aware registration, and per-field property sizing. The index format is 5 with one history entry; the branch-only format-6 step and format-5 image migration test are removed.
  6. Initializers: integrated fix(extraction): attribute Swift lifecycle calls #2395's b5b16e09 head for compatibility testing and updated its signature-QN lookups. Tests cover separate init overloads, matching super.init, and no edge to a label-compatible project initializer from an external UIViewController initializer. self.init retains the requested no-edge policy.
  7. Smaller fixes: signature-construction failure falls back to the base QN; service suffix stripping is name-anchored; typed receivers use their owner's overload bucket instead of scanning the full same-name bucket. A Swift fixture now participates in the existing complexity suite.

Validation on this PR head: 196 focused tests passed under ASan/UBSan with UBSAN_OPTIONS=halt_on_error=1; the separate registry suite passed 112 tests (overlapping coverage). Current-head Linux, macOS, Windows, TSan, lint, security, DCO, and the aggregate ci-ok check have now passed.

The local full clang-tidy sweep still reports pre-existing warnings reproduced on main; I have not changed its thresholds or suppressions. Other local full-test problems were reproduced against main's implementations and isolated in #2809, rather than added to this Swift PR. Those findings are now tracked in #2814 for a maintainer decision, with #2809 retained as a draft candidate. Its full-test results are not being presented as validation of this PR's head.

I believe the requested code changes are complete and this is ready for your Swift call-graph census and final review. As you specified, the census of Alamofire, a larger Swift project, and a non-Swift control remains on the maintainer side.

@DeusData

Copy link
Copy Markdown
Owner

Thank you, @DavidHLP, for the very thorough update and the point-by-point reply. We checked all seven points against debefaf and each one is addressed in the source, with a test for every case, which made this easy to verify. The next step is ours: we will run the Swift call-graph census on the merged tree (Alamofire, a larger Swift project and a non-Swift control) and post the numbers here. We also plan to land #2395 first, since your branch already carries it; after that, merging main into this branch drops the diff back to the identity change itself. Nothing is needed from you in the meantime. Thank you again for the care in this branch.

@DavidHLP

Copy link
Copy Markdown
Contributor Author

Thank you for the careful review and for verifying all seven points. The plan to land #2395 first and then run the census makes sense. I appreciate the time you’re putting into validating this across projects, and I look forward to seeing the results.

@DeusData

DeusData commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Thank you, @DavidHLP, for this very thorough update. We checked all seven points of our review against debefaf and every one is addressed, with a test for each, which made this easy to verify. Thank you also for keeping the #2395 integration and the index-format history so tidy: your branch merged cleanly with current main (including #2395), and the format 5 step rebuilds an existing index exactly once.

We ran the Swift call-graph census on Alamofire (bda9ed57), swift-nio (eabcc14f) and helm as a non-Swift control, each indexed from scratch with your branch merged into current main and with main itself:

  • helm: the graph is identical to main, as it should be.
  • Alamofire: 82 overload sets now get their own nodes (430 nodes), 3,016 calls resolve through argument labels, and no call has more than 7 candidates any more (main went up to 25).
  • swift-nio: 478 overload sets split into 1,244 nodes, 15,331 calls resolve through argument labels, and the worst candidate count drops from 91 to 32.
  • Repeated runs are byte-identical, the sequential and parallel indexers now produce the same call edges, and an incremental update after an edit matches a full index.

The receiver-first resolution clearly works: in URLEncodedFormEncoderTests, encoder.encode(...) now goes to URLEncodedFormEncoder.encode instead of JSONParameterEncoder.encode, in swift-nio context.close(promise: nil) binds to ChannelHandlerContext.close(mode:promise:) through the defaulted label, and many wrong guesses such as metrics.append -> MultipartFormData.append are gone.

One remaining case: when a receiver's type is known, the lookup only checks the members filed under that one type QN and its base classes. In Swift a type's members often live in an extension Type { ... } in another file, and the graph files those under that file's own QN, for example DataRequest in one file and extension DataRequest { func responseDecodable(...) } in ResponseSerialization.swift. The typed lookup then misses them, the call counts as incompatible, and it stays unresolved where main resolves it (about 400 calls on Alamofire and about 2,200 on swift-nio, including SessionTests.swift:1632 and DataStreamTests.swift:582, where session.request(.default) and session.streamRequest(.bytes(1)) target the Endpoint overloads in extension Session in Tests/TestHelpers.swift, and swift-nio calls such as self.buf.writeString(...) into ByteBuffer-aux.swift). Could you include the members of every same-named type's extensions in the owner lookup (the same for initializers declared in extensions), and add a test with the type in one file, an extension in a second and the caller in a third?

Thank you again for the care in this branch. We are close.

Include matching extension owners in typed Swift method and initializer lookup. Preserve owner metadata for incremental indexing and cover three-file resolution in sequential, parallel, and incremental runs.

Refs DeusData#2061

Signed-off-by: DavidHLP <lysf15520112973@163.com>
Signed-off-by: DavidHLP <lysf15520112973@163.com>
@DeusData

DeusData commented Nov 4, 2026

Copy link
Copy Markdown
Owner

Thank you, @DavidHLP, for turning this around so quickly. d33c6ac is exactly the shape we asked for: the extension members are now part of the typed owner lookup, initializers included, and the new test puts the type, its extension and the caller in three separate files and checks the sequential, parallel and incremental paths. Thank you also for the careful merge in 05c7ba7: you carried main's unresolved-call bookkeeping into both Swift label paths, which we would otherwise have had to ask for.

We re-ran the same census as last time (Alamofire bda9ed57, swift-nio eabcc14f, helm as the non-Swift control, every run indexed from scratch), comparing your previous head (debefaf merged with current main) to 05c7ba7:

  • helm: still identical to main.
  • Alamofire: 407 more call edges, all resolved through argument labels (3,016 -> 3,423), and 181 more call sites resolve. Calls into members declared in an extension that main resolves and the branch still missed went from about 400 to 283.
  • swift-nio: 2,591 more call edges (15,331 -> 17,922) and 1,818 more call sites resolve. The extension-member misses went from about 2,200 to about 1,680, and the test-to-code links that follow those calls are back to main's count.
  • Repeated runs are byte-identical, sequential and parallel indexing still produce the same call edges, the candidate maximum is unchanged (7 and 32), and the indexing time is the same as main.
  • In a sample of the newly resolved edges almost all were right, for example buf.readUTF8ValidatedString(length:) into ByteBuffer-aux.swift, group.next() into the MultiThreadedEventLoopGroup extension, self.responseSerializerDidComplete into the Request extension, and session.download(endpoint) into the Endpoint overloads in TestHelpers.swift.

On the two sites from last time: session.request(.default) at SessionTests.swift:1632 now resolves to the three Endpoint overloads, and session.streamRequest(.bytes(1)) at DataStreamTests.swift:582 to its two. The chained .responseDecodable and .responseStream after them are still unresolved. All candidates of each call return the same type (DataRequest and DataStreamRequest), so if every compatible candidate agrees on the return type, the chain could continue with that type.

One new problem comes from the lookup key. Extensions are matched by the type's path inside its module (swift_type_path / swift_extension, for example "Service"), so every type with that name anywhere in the repository shares its extensions:

  1. swift-nio ships two copies of its file-system module, Sources/NIOFS and Sources/_NIOFileSystem, which do not import each other (and the NIOFS tests import only NIOFS). With d33c6ac, 431 call edges cross between the two copies (debefaf: 0). For example, 296 call sites that resolved to one copy now also point into the other, such as Tests/NIOFSTests/FileOpenOptionsTests.swift:68 OpenOptions.Write.newFile. Another 28 sites pick up a same-named type from a different target, for example IntegerBitPacking, which exists in both NIOCore and NIOPosix. And 10 sites lose their edge, almost all of them correct ones. At Sources/_NIOFileSystem/BufferedReader.swift:145, self.readFromFile now has two candidates, the chain loses the type, and chunk.moveReaderIndex(forwardBy:) gets no edge.
  2. The same pattern grows quadratically. In a small generated repository with k directories, each holding struct Service, an extension Service in a second file and a typed caller in a third, each caller gets 121 call edges at k=60 and 241 at k=120 instead of 3, so the edge count grows about 4x when the repository doubles. The Swift fixture in tests/fixtures/complexity has no extension, so the complexity suite does not see this yet.
  3. When one file contains both the declaration and an extension of a type (very common), the two share one node, and that node ends up marked swift_extension instead of swift_type_path. On Alamofire that is 53 of the 140 extension-marked nodes, on swift-nio 322 of 683, for example BufferedReader, EventLoop and MultiThreadedEventLoopGroup. Then nothing distinguishes the declaration from an extension, which also feeds point 1.

Could you:

  1. keep the declaration marker when a declaration and an extension share a node, for example by storing both keys;
  2. only pool extensions that can actually extend the receiver's type: when several declarations share the path, take the extensions from the declaring module and the modules that can see it, not every same-named type in the repository;
  3. build the path -> owners map once when the registry is built, instead of parsing node properties for every call (it costs nothing measurable on these two repositories, but it is per call times per same-named type);
  4. add a test with two same-named types in two directories, each with its own extension and caller (no edges across), and an extension to tests/fixtures/complexity/Swift/ so the complexity suite covers this path?
    Continuing a chain after an ambiguous call whose candidates all share a return type (the two sites above) would be a welcome addition, but it is not required for this PR.

A small pre-existing gap explains the few wrong new edges we saw on Alamofire, and it is not caused by this change: the open func request(...) / upload(...) overloads in Source/Core/Session.swift are extracted without their Session owner. So for session.request(urlString) or session.upload(InputStream(...), to: url), the only label-compatible candidates are the Endpoint/Data overloads in TestHelpers.swift.

We are also looking at a few further points and will follow up on those separately. Thank you again for the quick and careful work; the receiver-first resolution keeps getting better.

Signed-off-by: DavidHLP <lysf15520112973@163.com>
Signed-off-by: DavidHLP <lysf15520112973@163.com>
Keep extension owners within visible targets and respect literal custom
SwiftPM roots, nested package boundaries and manifest invalidation.
Continue compatible call chains through declared return types and bounded
initializer inference. Recover conditional declaration ownership and
wrapped trailing closures, and preserve serial argument keywords.

Add extraction, registry and serial/parallel/incremental regressions.

Signed-off-by: DavidHLP <lysf15520112973@163.com>
Signed-off-by: DavidHLP <lysf15520112973@163.com>
Signed-off-by: DavidHLP <lysf15520112973@163.com>
@DavidHLP

DavidHLP commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Thank you, @DeusData, for rerunning the census and documenting the exact call sites. The two filesystem targets in swift-nio and the generated Service example made the extension lookup problem easy to reproduce. The follow-up is ready for review at 1a6b2923.

  1. Declaration and extension markers. When a declaration and an extension share a graph node, the node now retains both swift_type_path and swift_extension. The registry can still identify the declaration as an owner.
  2. Target visibility. Extension lookup now considers the declaring target and targets that can see it. The same rule applies to initializers declared in extensions. The lookup also respects literal SwiftPM target paths and nested package boundaries. On swift-nio eabcc14f, I found zero cross-target CALLS between NIOFS and _NIOFileSystem, including calls from NIOFSTests into _NIOFileSystem.
  3. Registry lookup and regression coverage. The registry builds the type-path-to-owners map once and uses it during call resolution. The regression suite covers same-named types with separate extensions and callers, and the Swift complexity fixture now exercises extension lookup.
  4. Call chains. When all compatible overloads agree on a return type, resolution carries that type into the next call. On Alamofire bda9ed57, the two sites you flagged now each resolve to one expected chain tail. SessionTests.swift:1632 reaches DataRequest.responseDecodable, and DataStreamTests.swift:582 reaches DataStreamRequest.responseStream. The extraction changes also preserve ownership for declarations inside conditional compilation blocks, including the Session methods you flagged.

I also changed Alamofire’s target path from Source to Source/Core to check manifest invalidation. The existing index took the full rebuild path, and its node and edge rows, including properties, matched a fresh index of the same edited checkout. That comparison covers this specific manifest change.

The PR head passed the default production regression checks. The local validation tree, which included the separate fixes from draft #2809, completed all 192 suites under ASan and UBSan with 10,363 passed, 0 failed, and 10 skipped. All 164 FastAPI incremental tests ran and passed. GitHub CI for the PR head passed Linux, macOS, Windows CLANG64, TSan, CodeQL, DCO, and ci-ok. [CI run](https://github.com/DeusData/codebase-memory-mcp/actions/runs/41848187305)

I kept the real-repository rerun focused on the reported call sites and manifest invalidation. I haven’t rerun the complete census or the generated 60/120-directory benchmark. Thanks again for the detailed review.

@DeusData

DeusData commented Nov 4, 2026

Copy link
Copy Markdown
Owner

Thank you, @DavidHLP, for another fast and careful round, and for the clear point-by-point reply. 1a6b292 closes the three structural problems from this morning, and the conditional-compilation work goes further than we asked: it fixes the missing Session owner we mentioned as a pre-existing gap.

We re-ran the same census as this morning (Alamofire bda9ed57, swift-nio eabcc14f, helm 9b88efd6 as the non-Swift control, every run indexed from scratch), comparing 05c7ba7 to 1a6b292:

  • Same-named types in different targets: call edges between the swift-nio NIOFS and _NIOFileSystem copies went from 431 to 0 (main has 244), and call sites whose targets span two targets went from 351 to 0. Tests/NIOFSTests/FileOpenOptionsTests.swift:68 now resolves only into NIOFS.
  • The generated repository with k directories (struct Service, an extension in a second file, a typed caller in a third): 3 call edges per caller at both k=60 and k=120 (this morning 121 and 241), with no edge into another directory. That holds with and without a Package.swift.
  • Declaration and extension in one file: nodes marked only as an extension although their file declares the type went from 53 to 1 on Alamofire and from 322 to 8 on swift-nio; 52 and 317 nodes now carry both markers.
  • Chains: SessionTests.swift:1632 now reaches DataRequest.responseDecodable and DataStreamTests.swift:582 reaches DataStreamRequest.responseStream, each with one candidate.
  • Conditional compilation: lines the parser could not read went from 652 to 32 on Alamofire and from 6,188 to 2,907 on swift-nio, and many members that used to be loose functions are now methods of their types (Alamofire 459 functions / 1,961 methods, was 632 / 1,755).
  • Totals: resolved call sites 4,995 -> 5,362 on Alamofire and 23,290 -> 24,655 on swift-nio; single-candidate edges 3,661 -> 3,891 and 18,922 -> 20,138; the candidate maximum is unchanged (7 and 32).
  • Repeated runs are byte-identical, sequential and parallel indexing produce the same call edges, the indexing time matches main, and helm is still identical to main.
  • In a sample of the newly resolved call sites almost all were right, for example ServerBootstrap(...).serverChannelOption(...).childChannelOption(...), buffer.readString(length:) into ByteBuffer-aux.swift, and session.request(url, headers:).cURLDescription into Request.

One thing went the other way: 1,976 call sites on swift-nio (103 on Alamofire) that 05c7ba7 resolved now get no edge, and for 1,362 of them (61 on Alamofire) main resolves the same target. In a sample about half of these were right before, so this is a real loss. It also shows up as test links: TESTS on swift-nio drops from 52 to 30. A few shapes we saw:

  • Tests/NIOPosixTests/SyscallAbstractionLayerContext.swift:97, result.salWait(context:), where let result = try self.runSALOnEventLoop(...) and the declared return type of runSALOnEventLoop is its generic parameter Result. It looks like the chain takes Result as a concrete type, finds no salWait on it, and drops the call.
  • IntegrationTests/tests_04_performance/test_01_resources/test_future_assume_isolated_lots_of_callbacks.swift:26, p.futureResult.assumeIsolated().flatMap { ... }.map { ... } with let p = loop.makePromise(of: Int.self) and loop: EventLoop: the whole chain, including assumeIsolated() itself, has no edge any more. This is where the TESTS links go.
  • Sources/_NIOFileSystem/BufferedReader.swift:145, chunk.moveReaderIndex(forwardBy:): var chunk = try await self.readFromFile(self.capacity) now resolves to one readFromFile returning ByteBuffer, but the type does not reach chunk.
  • Tests/NIOPosixTests/VsockAddressTest.swift:93-96: let socket = try ServerSocket(...) now gets a correct initializer edge, but socket.close(), socket.bind(to:) and socket.getLocalVsockAddress() get none (close() is declared on ServerSocket itself; bind(to:) and getLocalVsockAddress() are on its base class BaseSocket and its extension in VsockAddress.swift).
  • Tests/NIOHTTP1Tests/HTTPDecoderTesting.swift:273 and Tests/NIOPosixTests/FileRegionTest.swift:251, buffer.writeString(...) / buffer.writeBytes(...) with var buffer = channel.allocator.buffer(capacity: ...): these methods are now correctly members of ByteBuffer, and with the receiver type unknown the call is dropped instead of being matched by its labels as before.
  • Alamofire Source/Core/Session.swift:1389, mutableState.read { ... } on a Protected<MutableState>.

Could you:

  1. when the receiver's type cannot be established, or the inferred type has no member with that name and labels, fall back to the label match as 05c7ba7 did instead of dropping the call, and treat a callee's generic parameter used as its return type (like Result above) as unknown rather than as a type name;
  2. carry the type through try await initializers, and include members of the base class and its extensions for a typed receiver (the BufferedReader and ServerSocket sites above);
  3. keep the re-parsed tree from the conditional-compilation retry only when it has fewer error regions than the first parse. Two files got worse: Alamofire Source/Core/Protected.swift (3 -> 14 unreadable lines, now 84-95 and 97, so the Protected initializer node is gone; most likely the #error(...) line from the #else branch stays in the class body after masking) and swift-nio Tests/NIOPosixTests/NonBlockingFileIOTest.swift (12 -> 15);
  4. add a regression test for two or three of these shapes (a generic return type, a try await initializer, a base-class extension member on a subclass receiver)?

Two smaller observations, not required for this PR: 288 Alamofire call sites such as session.request(.default) now get 6 targets (the three Endpoint overloads in TestHelpers.swift plus the three URLConvertible overloads in Session.swift), which is expected from labels alone now that those overloads have their owner. And struct X: Error conformances in swift-nio resolve to project types named Error (IOError.Error in parallel, the newly visible NonBlockingFileIO.Error in sequential). That is older resolver behavior, but it widens the sequential/parallel INHERITS difference from 35 to 86 entries.

We are also looking at a few further points and will follow up on those separately. Thank you again; the structural part is in very good shape now, and the census shows how much the conditional-compilation fix adds.

@DeusData

DeusData commented Nov 4, 2026

Copy link
Copy Markdown
Owner

Hi @DavidHLP, a quick follow-up to our comment from earlier today: to save you another round, we will make the four changes from that list ourselves on your branch (maintainer edits are enabled), and re-run the same census on the result. If you have already started on any of them, just say so here and we will build on your work instead of duplicating it. Otherwise, please hold off pushing to this branch until we post the result, or pull first, so neither of us loses commits.

Thank you again for all the work on this series - the structural part is in very good shape now.

@DavidHLP

DavidHLP commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Thank you, @DeusData . I haven’t started working on these fixes yet, so there are no changes from my side to coordinate. I appreciate you handling the fixes and rerunning the census.

@DavidHLP

DavidHLP commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Hi @DeusData,

Thank you again for taking this on. Following my earlier reply, I ran some verification in an isolated Docker environment, comparing 05c7ba75 with 1a6b2923. I haven’t started implementing fixes or made any additional commits or pushes, and I’ll continue holding off while you work on the branch.

I reproduced the generic-return and try-wrapped constructor regressions, as well as the swift-nio TESTS drop from 52 to 30. One finding that may help: the diagnostic traces suggest that failed initializer inference is cached as an empty string, which then follows the incompatible-receiver path rather than falling back to label matching.

There was one difference in my run: Sources/_NIOFileSystem/BufferedReader.swift:145 resolved successfully, with chunk inferred as ByteBuffer. I haven’t established why our results differ on that particular site, but wanted to mention it in case it helps when comparing runs.

These observations are specific to 1a6b2923; I haven’t reproduced the complete per-call-site census or run the full C test matrix.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request parsing/quality Graph extraction bugs, false positives, missing edges priority/normal Standard review queue; useful PR with ordinary maintainer urgency.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants