diff --git a/CHANGELOG.md b/CHANGELOG.md index ac0545a765..3fe35074fe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -487,20 +487,15 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - `codegraph sync` and the file watcher now move an import written without an extension, like `import { sizeOf } from './button'`, to a file added beside the one it found before when the new file comes first, such as `button.ts` next to `button.vue`. Before, the import and the names it binds stayed linked to `button.vue` until the importing file changed or the project was re-indexed. JavaScript, TypeScript, Vue, Svelte and Astro. - In C, C++ and Objective-C, a struct or class member declared as a pointer, reference, array or function, like `SharedState* shared;`, `jv elements[];` or `virtual Status Put(…) = 0;`, no longer makes the struct look like it inherits from the member's type. Before, `codegraph_explore` and impact listed those types as base classes, every struct holding a pointer to a type as one of its subclasses, and a method of that struct as an override of the type's method with the same name. Real base classes like `class Derived : public Base`, Objective-C superclasses and Go's embedded fields are unchanged. Re-index C, C++ and Objective-C projects after upgrading. -- In C++, a call on a variable, parameter or member declared through a `typedef` or `using` alias now links to the method of the type the alias names, with the alias looked up the way C++ does it: in the calling function, its class and the classes that class inherits from, then the namespaces around it. In google/leveldb, `table_.Insert(buf)` on a `Table table_;` declared next to `typedef SkipList Table;` now reaches `SkipList::Insert` instead of the unrelated `HandleTable::Insert`, and `Table::Iterator iter(&table_);` reaches `SkipList::Iterator`'s constructor. A class that only shares the alias's name is no longer taken for it, and when the aliased type has no such method, as with an alias of `std::vector` or of a template parameter, no method is guessed from the receiver's name. Re-index C++ projects after upgrading. - - In C++, a call a method makes with no receiver, or through `this->`, now links to the method C++ itself would call: one of the method's own class, else one the class inherits from a base class, else a static member of a class it is nested in, which a nested class can name without an object. The same goes for a call in a lambda inside the method, and of a class's overloads the one the call's arguments fit is taken. Before, the call was matched by its name alone, so another class's method of that name could win because it was declared nearer or found first: in protocolbuffers/protobuf, every generated message's `Swap` and move assignment calling `InternalSwap(…)` reached `Any::InternalSwap`, and the `GetArena()` messages inherit from `MessageLite` reached an arena helper's `GetArena`, while googletest's `~linked_ptr() { depart(); }`, vendored in facebook/rocksdb, reached `linked_ptr_internal::depart`. Calls C++ doesn't look up in the class are matched as before: a free function's, a call to a parameter or local of that name, and a template's bare call into a base class that depends on its template parameters. Re-index C++ projects after upgrading. Thanks @danusha2345 for reporting the C++ case in #1952. - In C++, a call on the variable of a range-based `for` loop over `auto` elements, like `cfd->Ref()` inside facebook/rocksdb's `for (auto cfd : cfds)`, now links to the method of the type the loop goes over. That type is read from what the loop iterates: a `std::vector`, `std::set`, `absl::Span` or similar container, protocolbuffers/protobuf's `RepeatedPtrField` or RocksDB's `autovector`, whether a variable, parameter or member holds it, an `auto` variable is set to it or a function or method returns it, and a project class you loop over through its `begin()`, like RocksDB's `ColumnFamilySet`. Before, the call was matched by the variable's name, or by an earlier variable of that name: in RocksDB, `blob_file.GetBlobFileNumber()` over a list of `BlobFileGarbage` reached `BlobFileAddition::GetBlobFileNumber`, and in protobuf's Objective-C generator, calls over a list of `std::unique_ptr` reached other generators' methods. A call on an element of a standard-library type, like `file.find(…)` over a `std::vector`, no longer links to a project method, and `->` on an element held in a `std::unique_ptr` or `std::shared_ptr` reaches the type it holds. Re-index C++ projects after upgrading. -- In C++, a call on a variable, parameter or member declared as one of your classes, like `iter->Valid()` on an `Iterator* iter` or `it.Valid()` on a `const Iterator& it`, now links to that method of the class C++ itself would find for the type where the call is written, or of a class it inherits from. Before, the method was picked by the type's last name, so a class of the same name nested elsewhere could win: in facebook/rocksdb, calls on an `Iterator` reached `MemTableRep::Iterator`, which declares `Valid` and `Next` itself while the `Iterator` the code means inherits them, google/leveldb's reached `SkipList::Iterator`, and in protocolbuffers/protobuf the Java generator's `context->options()` reached the hpb generator's `Context` and `text_parser.Parse(…)` on a `TextFormat::Parser` reached the `.proto` parser. When that class and the classes it derives from have no method of that name, the call no longer links to an unrelated class's method that shares it, as protobuf's `field->full_name()` on a `FieldDescriptor*` did. A type is also found through a `using` declaration the file writes, like protobuf's `using google::protobuf::Descriptor;`, before a class of the same name declared at global scope elsewhere in the repository; this also applies to base classes and to the type a smart pointer holds. Re-index C++ projects after upgrading. - In this fork, the two C++ changes above run in the native resolver. A call on a variable whose type is a template parameter, like `T* iter`, stays unlinked rather than reaching a class that happens to share the parameter's name. A call stays linked as before when the class has no method of that name but one of its base classes comes from an outside library other than the standard library, such as googletest's `testing::Test`. -- In C++, a call on an `auto` variable now links to the method of the class C++ finds for the type its initializer gives it, as a call on a variable declared with that type does. The type is read from a cast or smart pointer written out, like facebook/rocksdb's `auto iter = std::unique_ptr(…)` or `auto* mutex = static_cast(arg)`, from `new` or `std::make_unique` of a class, and from what a function or method is declared to return, like the `Iterator*` of `txn->GetIterator(…)`, read to the end of an initializer that goes on to the next line. Before, such a variable took the type of an earlier variable of the same name, or none: in RocksDB, `iter->SeekToFirst()` reached `MemTableRep::Iterator` and `mutex->Lock()` reached `port::Mutex`. A chained call like `cfd->current()->storage_info()` now looks up the class the inner call is declared to return the same way, through a smart pointer too, so protocolbuffers/protobuf's Java generator no longer reaches the C# generator's `EnumGenerator`. A type from outside the project no longer links to a class of the project that shares its name: protobuf's `status.ok()` on an `absl::Status` reached `upb::Status::ok`, and fmtlib/fmt's `os.flush()` on a `std::ostream&` reached fmt's own `ostream::flush`. Re-index C++ projects after upgrading. - In C++, a call on a data member of the calling method's class, like `versions_->GetColumnFamilySet()` in facebook/rocksdb's `db_impl_write.cc`, now links to the method of the member's type wherever the class is declared. Before, only the call's own file and the header of the same name were read for the member, and RocksDB declares `DBImpl` in `db_impl.h` while its methods are spread over `db_impl_write.cc`, `db_impl_compaction_flush.cc`, `db_impl_open.cc` and more, so such calls were guessed by the member's name. A member declared below an inline method that uses it, one a base class declares, and, in a gtest `TEST_F` or `TEST_P` body, one of the test's fixture, like RocksDB's `db_`, are found the same way, and a member of a class nested in the caller's class is no longer taken for the caller's own. A call on a member of a standard-library type, like `std::atomic shutting_down_`, no longer links to a project method that only shares the method's name. Re-index C++ projects after upgrading. - In C++, a variable, parameter or member whose declaration puts a qualifier or an annotation around its name is now read for its type, so calls on it link to the declared class's methods: a `const` after the `*` or `&`, like facebook/rocksdb's `Iterator* const iter = iter_guard.get();` or google/leveldb's `Env* const env_;`, a type written before its `const`, like `Slice const& key`, a nullability annotation after the `*`, like protocolbuffers/protobuf's `io::EpsCopyOutputStream* PROTOBUF_NONNULL stream`, and a thread-safety annotation after the name, like LevelDB's `MemTable* imm_ GUARDED_BY(mutex_);`. Before, such a declaration was skipped, or the annotation was taken for its type, so the call took the type of an earlier variable of the same name or was guessed by the method's name: RocksDB's `iter->SeekToFirst()` reached the unrelated `ArenaWrappedDBIter`, and LevelDB's `env_->NewWritableFile(…)` reached `InMemoryEnv`. Re-index C++ projects after upgrading. - In C++, a call on a name a structured binding declares now links to the method of the type C++ gives it. That covers a range-based `for` over pairs or a map, like facebook/rocksdb's `for (auto& [cfh, iter] : cfh_iter_pairs_) { iter->Valid(); }`, and a declaration like protocolbuffers/protobuf's `auto [dst_ext, is_new] = Insert(…);`. A call on what a `std::function` or another function wrapper returns, like `create_iterator()->Valid()` on a `std::function`, links the same way, and so does a call on a class member whose type is written on the line above its name. Before, these calls were guessed by the method's name, and calls on a library container declared that way, like a `std::priority_queue` member's `pop()`, reached a project class's method of that name. Re-index C++ projects after upgrading. - In C++, a call written without a receiver in a gtest `TEST_F` or `TEST_P` body now links to the test fixture's method, its own or one it inherits, as C++ finds it: facebook/rocksdb's tests call `DBTestBase`'s `Put(…)`, `Flush()` and `NewIterator(…)` this way. A variable such a call initializes, like `auto iter = NewIterator(…)`, takes the type the method returns, so `iter->SeekToFirst()` reaches the iterator class. Before, those calls were guessed by name and often reached a namesake in another class, like RocksDB's `ColumnFamilyTestBase::Put` or protocolbuffers/protobuf's `EncodeDecodeTest::ExpectNoErrors`, and `iter->SeekToFirst()` reached the unrelated `MemTableRep::Iterator`. A `->` call on a value of a type from outside the project, like protobuf's `m->value()` on an `absl::StatusOr`, no longer links to a class of the project that shares the type's name, like protobuf's own `hpb::StatusOr`. Re-index C++ projects after upgrading. - In C++, a call made with `->` on an `absl::StatusOr`, an `absl::optional` or an `absl::NoDestructor`, like protocolbuffers/protobuf's `resolver->MergeFeatures(…)` on an `absl::StatusOr`, now links to the method of the class it holds, as one on a `std::unique_ptr` or `std::optional` does. A call written without a receiver in a class template now also finds a member of a base that depends on the template's parameters when the class brings it in with a using-declaration, like facebook/rocksdb's `using BasicTypedCacheHelperFns::DownCastValue;`, and so does a call on such a member. Re-index C++ projects after upgrading. - In C and C++, a call or a type name no longer links to something only another source file can see: a class, struct or alias a `.c` or `.cc` file declares for itself, its members, or anything inside that file's unnamed `namespace { … }`. In facebook/rocksdb, other test files' bare `Put(…)`, `Flush()` and `dbfull()` calls had linked to the methods of a test fixture declared in `column_family_test.cc`, and calls on an `Env*` had linked to a test-only environment class in `memenv.cc`. When a header and a source file each declare a class of the same name, like RocksDB's `SharedState`, a call now links to the header's. A source file's own declarations are still found from the file itself, from a file that `#include`s it, and from a header it includes below them. Re-index C and C++ projects after upgrading. -- In C++, a call to an overloaded method now links to the overload its arguments fit: by how many there are, and then by their types. That covers a call on a variable or member of a known class, like facebook/rocksdb's `db->Get(ro, key, &value)` and `db->Put(write_opts, key, value)`, as well as a call written without a receiver, like `Insert(ch)` on a `char ch`. Before, a call on a variable always linked to the first overload declared, so RocksDB's convenience overloads of `DB::Get`, `DB::Put` and `DB::NewIterator` and protocolbuffers/protobuf's `io::Printer::Print`, `Emit` and `Annotate` had almost no callers, and a call without a receiver picked by argument count alone. A literal, a variable the calling function declares, `&variable`, `container[i]` and the result of a call are read for their types; a parameter's default value is read from the method's declaration in its class, and comments in argument lists are skipped. Re-index C++ projects after upgrading. - In C++, a nested class defined outside the class it belongs to, like google/leveldb's `struct SkipList::Node { … }` or facebook/rocksdb's `struct BlockBasedTable::Rep { … }`, is now indexed under its own name, `SkipList::Node`, with its methods and members. Before, it was indexed as an unnamed class, so it was missing from search and `codegraph_explore`, its methods belonged to no class you could name, and a call like `x->Next(level)` on a `Node*` was matched by the method's name alone. Creating an object, as in `new log::Writer(…)` or `Writer w(…)`, now links to the class C++ itself finds for the type as written there: leveldb's `new log::Writer(…)` in `DBImpl` methods no longer links to `DBImpl`'s own `Writer`, RocksDB's tests' `Random rnd(301)` links to RocksDB's `Random` instead of googletest's, an object created through an alias links to the class the alias names, and one links to its class rather than to the class's constructor. An object of a template parameter's type, or of a type the function names with its own `using`, like protocolbuffers/protobuf's `using Map = std::map<…>`, no longer links to a project class that only shares the name. Re-index C++ projects after upgrading. - In C++, a call whose receiver's type can't be read no longer links to a method of a class nested in the class the receiver's name suggests, or of a class whose name only contains that name. Before, such a guess took any method of the call's name whose qualified name contained the class's name: protocolbuffers/protobuf's `descriptor->file()` in a template reached `Descriptor::ExtensionRange::file`, and facebook/rocksdb's `CompositeEnv::IsInstanceOf(name)` reached `CompositeEnvWrapper::IsInstanceOf`. A call written through a class, like `PessimisticTransaction::PopSavePoint()` in an override, now links to the method that class inherits, here `TransactionBaseImpl::PopSavePoint`. Re-index C++ projects after upgrading. - In C++, an alias that names a class through a class derived from the one that declares it, like protocolbuffers/protobuf's `using ConformanceRequestSetting = BinaryAndJsonConformanceSuite::ConformanceRequestSetting;`, now leads to that class, `ConformanceTestSuite::ConformanceRequestSetting`. Before, an alias was followed only to a class declared under exactly the name it spells, so objects created through this one linked to the alias rather than the class. Re-index C++ projects after upgrading. @@ -539,8 +534,6 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In C++, a class, struct or method now keeps the namespaces and the class it is declared in when the parser misreads something earlier in the file, such as an unknown macro in front of a member or in a class header. Before, everything after the misread code could lose them, or land inside a class it isn't in: in protocolbuffers/protobuf, `FieldDescriptor` was indexed without `google::protobuf::`, and in RocksDB, the `Opts` struct declared inside a cache table class was indexed outside it, so finding a class's base classes and the methods called on its objects fell back to guessing by name. Methods that had come loose from their class are its members again, and classes the misread code had hidden, like leveldb's POSIX file and environment classes, are now indexed. Re-index C++ projects after upgrading. -- In C++, a call on a variable, parameter or member declared as a standard-library or other outside type with a lowercase name, like `std::string`, `std::vector`, `std::set` or `absl::flat_hash_set`, no longer links to a project method that only shares the method's name. In google/leveldb, `saved_key_.clear()` on a `std::string saved_key_;` reached `Slice::clear`, and in protocolbuffers/protobuf, `proto.append(…)` on a `std::string` reached an unrelated `LeftoverBuffer::append`. The same goes for `->` through a raw pointer like `std::string* out`, while `->` through a smart pointer, iterator or optional still links to the type it holds. Comments also no longer pass for a variable's declaration, so more calls now link to the method of the type the variable is really declared with. Re-index C++ projects after upgrading. - - In a repository that mixes languages, a web framework's naming conventions now link only code written in that framework's own languages. Before, Express's also linked Go and Python code: in etcd a Go method's `*zap.Logger` result type linked to the method itself, and in Harbor a Python `self.jaeger.validate()` call linked to another class's `validate`. Links that cross languages on purpose, like React Native and Expo native modules, Swift and Objective-C, Drupal's `routing.yml`, SvelteKit's `$lib/…` imports in `.ts` files and a Razor page's `@model`, are unchanged. - In Go, working out which structs implement an interface now counts the methods of the interfaces it embeds, and the methods a struct gets from the types it embeds. Before, only each type's own methods counted: etcd's `AuthReadTx`, which embeds `UnsafeAuthReader` and adds `RLock` and `RUnlock`, was listed as implemented by every read-write lock in the project, an interface that only embeds others had no implementations at all, and a struct that gets its methods from an embedded base, like prometheus's service discoveries embedding `refresh.Discovery` or gin's `Engine` embedding `RouterGroup`, was missing. `codegraph_explore`, impact and the type hierarchy now list the right implementations, and a call through the interface reaches a method the embedding struct overrides, like gin's `Engine.Use`. Re-index Go projects after upgrading. @@ -554,7 +547,6 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - React Router routes kept in a table of their own are now indexed: the ASP.NET Core React template's `AppRoutes` array that `App.js` maps into ``, a `routes` array imported into `useRoutes(routes)` or `createBrowserRouter(routes)` or returned by a function as in `useRoutes(routes(isLoggedIn))`, route objects kept one per file and listed in `createBrowserRouter([MainRoutes, LoginRoutes])`, and routes written in place in `useRoutes([…])`. Before, route objects were read only from a file that itself creates a data router, so these apps had no routes, and their `navigate('/login')` calls and `` links led nowhere. Each route links to the page its `element` renders, past a guard like `` or ``, and through `lazy(() => import(…))` to the page it loads rather than a same-named page elsewhere in the repository. An `index: true` route is the page at its parent's address, and a parent route around others counts as their layout. A `{ path, element }` list that nothing hands to the router, such as a menu, still makes no routes. Re-index React projects after upgrading. - In `.jsx` files, the components and functions that read a constant or variable declared at the top of the file, like a page loaded with `const LoginPage = Loadable(lazy(…))` and rendered as ``, a route table, or a messages or style object, now show up in its impact, as they already did in `.js` and `.tsx` files. Before, `.jsx` files were left out, so `codegraph impact` and the impact trail in `codegraph_explore` and `codegraph_node` showed nothing depending on such a value. Re-index projects with `.jsx` files after upgrading. -- In Go, a call through a parameter or local variable named like a standard-library package, such as `context`, `trace`, `printer` or `token`, now links to the method it calls. Before, every call written `context.…` or `printer.…` was taken for a call into the standard library and linked to nothing: gin's `context.AbortWithError(…)` inside `func(context *Context)`, harbor's `context.IsAuthenticated()` after `context := NewSecurityContext(…)`, etcd's `printer.DBHashKV(…)` and `trace.Step(…)`. The variable's declaration decides the method: the type it is declared as, or the package of the project function its value comes from. A variable holding a value from outside your project, like `scanner := bufio.NewScanner(r)` or `url, err := url.Parse(raw)`, still links to nothing, and one whose type can't be told links only to a method no other type in the project has, never to one picked because it matches the variable's name. Re-index Go projects after upgrading. - In Go, a local declared from a function of a project package that shares its name, like harbor's `log := log.G(ctx)`, now links `log.G` to that function and the later `log.Errorf` to the method of the logger it returns. Before, both linked to nothing: the new local hid the package from its own line, and a project package named like a standard-library one, such as `log` or `context`, was taken for the standard library. A direct call into such a project package, `log.G(ctx)` with no local, now links too. Re-index Go projects after upgrading. - React Router routes written as JSX now read `` the way route objects do: inside `}>`, `} />` is the page at `/`, and `Layout` counts as the layout of every route nested in it, as does a guard with no path of its own like `}>`. Before, the index route was never read, so `/` led to the layout itself and a `` holding only an index route led to nothing. A route's `element` written on lines of its own, the way Prettier formats a long one, or wrapped in `}>`, now links to the page it shows rather than to nothing or to the fallback, and a path written in braces, like `path={"agents"}`, is read like any other. Re-index React projects after upgrading. @@ -611,8 +603,6 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In C++, a call on a range-based `for` loop's variable, like `suite->SetVerbose(verbose)` inside `for (ConformanceTestSuite *suite : suites)`, now links to the method of the type the loop declares. Before, the loop's declaration was never read, so the call was matched by the variable's name alone: in RocksDB, `log->file()` on a `log::Writer` went to `BlobLogWriter::file`, and a loop variable named like a member of its class was taken for that member. The variable counts only inside its loop. Re-index C++ projects after upgrading. -- In C++, a call made with `->` on a variable, parameter or member declared as a `std::unique_ptr`, `std::shared_ptr` or `std::optional`, like `iter->Valid()` on a `std::unique_ptr iter`, now links to that method of the type it holds, or of a class that type inherits from, with the type looked up the way C++ does it, so a class of the same name nested elsewhere is not mistaken for it. Before, the call went to whichever project method shared its name: in facebook/rocksdb, `iter->Valid()` reached `ArenaWrappedDBIter::Valid` and `fs->GetFileSize(…)` on a `std::shared_ptr` reached a test's stand-in file system, and calls like `it->Next()` were often not linked at all. When what it holds is a template parameter or a standard-library type, like `std::unique_ptr>`, or a class that has no method of that name and neither do the classes it derives from, the call no longer links to a project method; a class derived from something that can't be followed, like a `decltype(…)`, may still have it, so that call is left unlinked. Re-index C++ projects after upgrading. - - `codegraph sync` and the file watcher now move a React Router route to its page when the router file loads that page lazily into a value of its own, like `const Docs = lazy(() => import('./pages/Docs'))` rendered by `} />`, and the page's file is added after the router was indexed, comes back after being deleted, or gets or changes its default export in a later edit. Before, the route kept pointing at the `Docs` declaration, or at the page's old component, until the router file itself changed or the project was re-indexed, unless the page's component happened to share the declaration's name. The same now holds for a `lazy: () => import('./pages/Docs')` route whose page switches its default export to another component. Re-index React projects after upgrading to correct routes an earlier version left behind. - `codegraph sync` and the file watcher now link a component to the JSX that renders it, like ``, when the component is added in a file with no JSX of its own, as in `export default function Team() { return null; }`. Before, the link waited for an unrelated change to a file with JSX, or for a re-index, so the new component looked unused. @@ -627,6 +617,32 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Kotlin, a call on a typed receiver now links to an extension function declared in a package, like `value.shout()` or the infix `a join b` with `value: Foo` and `fun Foo.shout()` under `package demo`, including one declared in another file of the package. Before, it linked only when the file had no `package` line, so the usual Kotlin project got no edge. On JetBrains Exposed, 1,020 such calls now link (`Table.selectAll()` to `FieldSet.selectAll`, `transaction.throwUnsupportedException(…)`), and none that linked before is lost. A member of the receiver's class still wins over an extension of the same name. +- In C++, a call on a variable, parameter or member declared through a `typedef` or `using` alias now links to the method of the type the alias names, with the alias looked up the way C++ does it: in the calling function, its class and the classes that class inherits from, then the namespaces around it. In google/leveldb, `table_.Insert(buf)` on a `Table table_;` declared next to `typedef SkipList Table;` now reaches `SkipList::Insert` instead of the unrelated `HandleTable::Insert`, and `Table::Iterator iter(&table_);` reaches `SkipList::Iterator`'s constructor. A class that only shares the alias's name is no longer taken for it, and when the aliased type has no such method, as with an alias of `std::vector` or of a template parameter, no method is guessed from the receiver's name. An alias of a type written from the global scope names that global type, also inside a namespace of the same name: protocolbuffers/protobuf's `using Arena = ::upb::Arena;` in `hpb::internal::backend::upb`, so hpb's `arena_.Fuse(…)` on a `backend::Arena` now reaches `upb::Arena::Fuse` instead of the calling method itself. Re-index C++ projects after upgrading. + +- In C++, a call on a variable, parameter or member declared as a standard-library or other outside type with a lowercase name, like `std::string`, `std::vector`, `std::set` or `absl::flat_hash_set`, no longer links to a project method that only shares the method's name, even one of a project class in another namespace with the type's name, like your own `util::flat_hash_set`. A variable declared outside the calling function, like another function's or a file's `std::string text`, no longer links its calls to such a class either. In google/leveldb, `saved_key_.clear()` on a `std::string saved_key_;` reached `Slice::clear`, and in protocolbuffers/protobuf, `proto.append(…)` on a `std::string` reached an unrelated `LeftoverBuffer::append`. The same goes for `->` through a raw pointer like `std::string* out`, while `->` through a smart pointer, iterator or optional still links to the type it holds. Comments also no longer pass for a variable's declaration, so more calls now link to the method of the type the variable is really declared with. Re-index C++ projects after upgrading. + +- In C++, a call made with `->` on a variable, parameter or member declared as a `std::unique_ptr`, `std::shared_ptr` or `std::optional`, like `iter->Valid()` on a `std::unique_ptr iter`, now links to that method of the type it holds, or of a class that type inherits from, with the type looked up the way C++ does it, so a class of the same name nested elsewhere is not mistaken for it. Before, the call went to whichever project method shared its name: in facebook/rocksdb, `iter->Valid()` reached `ArenaWrappedDBIter::Valid` and `fs->GetFileSize(…)` on a `std::shared_ptr` reached a test's stand-in file system, and calls like `it->Next()` were often not linked at all. When what it holds is a template parameter or a standard-library type, like `std::unique_ptr>`, or a class that has no method of that name and neither do the classes it derives from, the call no longer links to a project method; a class derived from something that can't be followed, like a `decltype(…)`, may still have it, so that call is left unlinked. The same goes for `->` on a container's iterator, like `it->data()` on a `std::deque::const_iterator it` or `loc->path_size()` on protocolbuffers/protobuf's `RepeatedPtrField::iterator loc`: the call links to that method of the container's element type, and to no project method when the element is a standard-library type like `std::string`. That includes an iterator an `auto` variable gets from a standard-library container's `begin()`, `find()` and the like, or from a standard search like `std::lower_bound`, as in facebook/rocksdb's `auto it = logs_.begin(); it->IsSyncing()`. A call on a dereferenced iterator or pointer, like `(*it)->Visit()` over a `std::vector` or `(*it).Weight()` over a `std::vector`, and on a variable a dereference sets, like `auto first = *nodes.begin()`, links the same way, where before such a call went to whichever class had a method of that name. So does a call on the key or value a map's iterator points at, like `it->second.Flush()`, `it->second->Flush()` or a range-based `for` loop's `kv.second.Flush()`, and on the `first` or `second` of a `std::pair` the code declares. So does a call on an element read by subscript, like `statuses[i].IsNotFound()` over a `std::vector`, `buffers[key].Flush()` over a map, `iters[i]->Valid()` over owning pointers, or an element of an array or of a class's `operator[]`. So does a call on what a container's `front()`, `back()` or `at()` hands out, like facebook/rocksdb's `edits.back()->MarkAtomicGroup(…)` or `wals.at(log_number).HasSyncedSize()` over a map, on a `std::stack`'s `top()` or a `std::queue`'s `front()`, and on what a smart pointer's `get()` or an optional's or `absl::StatusOr`'s `value()` returns, like `mock_clock.get()->GetCurrentTime(…)`, where before such calls linked to nothing; an overloaded function given such a value, like `test::PerThreadDBPath(env_.get(), …)`, now takes the overload for its type. Re-index C++ projects after upgrading. + +- In C++, a call a method makes with no receiver, or through `this->`, now links to the method C++ itself would call: one of the method's own class, else one the class inherits from a base class, else one of a class it is nested in. The same goes for a call in a lambda inside the method, and of a class's overloads the one the call's arguments fit is taken. Before, the call was matched by its name alone, so another class's method of that name could win because it was declared nearer or found first: in protocolbuffers/protobuf, every generated message's `Swap` and move assignment calling `InternalSwap(…)` reached `Any::InternalSwap`, and the `GetArena()` messages inherit from `MessageLite` reached an arena helper's `GetArena`, while googletest's `~linked_ptr() { depart(); }`, vendored in facebook/rocksdb, reached `linked_ptr_internal::depart`. Calls C++ doesn't look up in the class are matched as before: a free function's, a call to a parameter or local of that name, and a template's bare call into a base class that depends on its template parameters. Re-index C++ projects after upgrading. Thanks @danusha2345 for reporting the C++ case in #1952. + +- In C++, a call on a variable, parameter or member declared as one of your classes, like `iter->Valid()` on an `Iterator* iter` or `it.Valid()` on a `const Iterator& it`, now links to that method of the class C++ itself would find for the type where the call is written, or of a class it inherits from. Before, the method was picked by the type's last name, so a class of the same name nested elsewhere could win: in facebook/rocksdb, calls on an `Iterator` reached `MemTableRep::Iterator`, which declares `Valid` and `Next` itself while the `Iterator` the code means inherits them, google/leveldb's reached `SkipList::Iterator`, and in protocolbuffers/protobuf the Java generator's `context->options()` reached the hpb generator's `Context` and `text_parser.Parse(…)` on a `TextFormat::Parser` reached the `.proto` parser. When that class and the classes it derives from have no method of that name, the call no longer links to an unrelated class's method that shares it, as protobuf's `field->full_name()` on a `FieldDescriptor*` did. A type is also found through a `using` declaration the file writes, like protobuf's `using google::protobuf::Descriptor;`, before a class of the same name declared at global scope elsewhere in the repository; this also applies to base classes and to the type a smart pointer holds. A type written from the global scope, like `::upb::Arena& arena`, is looked up there, so a namespace's own `upb::Arena` is not taken for it, nor is an alias of that name the calling function declares. A type written as a member of a class template, like `SkipList::Iterator iter` in leveldb's and RocksDB's skip-list tests, is now read whole, so its calls reach the skip list's own iterator rather than the general `Iterator`, and RocksDB's skip-list memtable iterator no longer links its calls on the skip list's iterator to its own methods; a standard container's `iterator` written this way is the library's, never a class of yours named `iterator`. Re-index C++ projects after upgrading. + +- In C++, a method defined by a macro of the same file is now indexed where the file invokes the macro, so calls to it link. protocolbuffers/protobuf's descriptor classes only declare most of their accessors and define them further down with macros like `PROTOBUF_DEFINE_ACCESSOR(FieldDescriptor, number, int)`, and its reflection defines `GetInt32`, `SetInt32` and the like by pasting names together in a macro. Those methods had no node at all, so calls like `field->number()`, `descriptor->full_name()` or `reflection->GetInt32(…)` linked to nothing. Calls through what such a method returns, like `field->containing_type()->full_name()`, now link too. A macro defined in another file, and a method declared but defined nowhere in the repository, are still not indexed. Re-index C++ projects after upgrading. + +- In C++, an explicit specialization of a function or member function template, like protocolbuffers/protobuf's `template <> bool WireFormatLite::ReadPrimitive(…)` or facebook/rocksdb's `template <> int64_t Deserialize(…)`, is now indexed under its template's name, so search finds it and calls link to it. Before, it kept its template arguments in its name, and one with a `::` among them was named after the last part, like `TYPE_INT32>`. A call that writes template arguments, like `Deserialize(src, offset)`, links to the specialization written with the same ones, and otherwise to the template itself, as fmt's `num_bits()` does; a call that writes none links to a specialization only when nothing else has that name. Re-index C++ projects after upgrading. + +- In C++, a call to an overloaded function outside a class now links to the overload its arguments fit, by how many it passes and the types they have where the call writes them, as a call to a method already did: protocolbuffers/protobuf's `ParseWithLimit(limit, data, result)` on an `int32_t result` reaches the overload taking an `int32_t&`, and facebook/rocksdb's `NewLRUCache(cache_opts)` the one taking an `LRUCacheOptions`. Before, the overload nearest the call was taken. Such a call is never linked to an overload only another source file can see, like another `.cc` file's `static` function: it now links to the one its own file sees, where before it often linked to nothing. A call to a name that both a function and a class have reaches the function, which hides the class, and a parameter named like the function it belongs to, like protobuf's `std::string u64(uint64_t u64)`, now has its own type. Of two overloads alike but for a `const`, like protobuf's `DynamicCastMessage(const MessageLite* from)` and `DynamicCastMessage(MessageLite* from)`, a call now reaches the one C++ picks: the non-const overload for a pointer to a non-const object, or for the object itself, and the const one for a const object or a temporary, where before the overload declared first won. An argument declared with `const` in front, like `const Buffer* b`, counts as const, and an array as a pointer to its element, so facebook/rocksdb's `Add(level, file, smallest, largest, …)` on `char` arrays reaches the overload taking `const char*`. An object is no longer matched to a pointer parameter, nor a character array to a pointer of another type, so protobuf's `printer->WithVars(EnumVars(…))` reaches `WithVars(Map&& vars)`, not `WithVars(const Map* vars)`. Re-index C++ projects after upgrading. + +- In C, a call now links to the function of that name its own file can see: its own, else one a header it includes defines, like a `static inline` helper, else one another source file defines without `static`. Before, the function nearest the call was taken, and when that was another source file's `static` function, which no other file can call, the call linked to nothing: in redis, the calls to `server.c`'s `mstime()` and `ustime()` from the rest of the server were lost to the `static` copies other files keep for themselves, and jemalloc's tests' calls to the `do_arena_create` their shared header defines to a neighbouring test's own. A call to a `static inline` function that several headers define reaches the copy in a header the file includes, so protocolbuffers/protobuf's upb code no longer links to the copies in its PHP extension's combined `php-upb.h`. A source file that `#include`s another, like redis's `ae.c` with its event-loop backend or hiredis's `async.c` with `dict.c`, now calls that file's `static` functions, and a header a file includes through an include directory, like jemalloc's `#include "jemalloc/internal/bit_util.h"`, counts as included. A function a header defines only for a build that turns on a flag, like lua-cjson's `fpconv_init` under `#ifdef USE_INTERNAL_FPCONV`, gives way to another source file's definition when nothing in the project's code or build files sets that flag, so redis's calls to `fpconv_init()` reach the `fpconv.c` its build compiles. Re-index C projects after upgrading. + +- In C++, a call on an `auto` variable now links to the method of the class C++ finds for the type its initializer gives it, as a call on a variable declared with that type does. The type is read from a cast or smart pointer written out, like facebook/rocksdb's `auto iter = std::unique_ptr(…)` or `auto* mutex = static_cast(arg)`, from `new` or `std::make_unique` of a class, and from what a function or method is declared to return, like the `Iterator*` of `txn->GetIterator(…)`, read to the end of an initializer that goes on to the next line. Before, such a variable took the type of an earlier variable of the same name, or none: in RocksDB, `iter->SeekToFirst()` reached `MemTableRep::Iterator` and `mutex->Lock()` reached `port::Mutex`. A chained call like `cfd->current()->storage_info()` now looks up the class the inner call is declared to return the same way, through a smart pointer too, so protocolbuffers/protobuf's Java generator no longer reaches the C# generator's `EnumGenerator`. A type from outside the project no longer links to a class of the project that shares its name: protobuf's `status.ok()` on an `absl::Status` reached `upb::Status::ok`, and fmtlib/fmt's `os.flush()` on a `std::ostream&` reached fmt's own `ostream::flush`. The same holds for a variable declared in a macro body, or typed by what a function in another file returns, and for a class the index doesn't hold whose name only another class shares: protobuf's `_status.ok()` in its `RETURN_IF_ERROR` macro reached `upb::Status::ok`, and `range.options()` on its generated `DescriptorProto::ExtensionRange` reached `Descriptor::ExtensionRange::options`. A namespace a macro stands for is the project's under either name: facebook/rocksdb declares its classes in `namespace ROCKSDB_NAMESPACE` with `#define ROCKSDB_NAMESPACE rocksdb`, so `db->Put(…)` on a `rocksdb::DB*` links to RocksDB's own `DB::Put`. Re-index C++ projects after upgrading. + +- In C++, a call to an overloaded method now links to the overload its arguments fit: by how many there are, and then by their types. That covers a call on a variable or member of a known class, like facebook/rocksdb's `db->Get(ro, key, &value)` and `db->Put(write_opts, key, value)`, as well as a call written without a receiver, like `Insert(ch)` on a `char ch`. Before, a call on a variable always linked to the first overload declared, so RocksDB's convenience overloads of `DB::Get`, `DB::Put` and `DB::NewIterator` and protocolbuffers/protobuf's `io::Printer::Print`, `Emit` and `Annotate` had almost no callers, and a call without a receiver picked by argument count alone. A literal, a variable the calling function declares, `&variable`, an element read by subscript, like `container[i]` or a map's `values[key]`, and the result of a call are read for their types; a parameter's default value is read from the method's declaration in its class, and comments in argument lists are skipped. Re-index C++ projects after upgrading. + +- In this fork, the C++ changes above run in the native resolver, and a call whose receiver's type they cannot read stays unlinked where upstream matches the method by its name alone: a call on an element only a class template's own `operator*` writes, like `it->Run()` on a `ParamGenerator::iterator`; on the key or value of a map whose type a template parameter writes, like `p.first->GetUncompressData()` over facebook/rocksdb's `CoreLocalArray`; on a member whose type a template argument decides, like a `typename PolicyT::Payload`; and on a variable another function declares. + +- In Go, a call through a parameter or local variable named like a standard-library package, such as `context`, `log`, `trace`, `printer` or `token`, now links to the method it calls. Before, every call written `context.…` or `printer.…` was taken for a call into the standard library and linked to nothing: gin's `context.AbortWithError(…)` inside `func(context *Context)`, harbor's `context.IsAuthenticated()` after `context := NewSecurityContext(…)` and `log.Errorf(…)` after `log := log.G(ctx)`, etcd's `printer.DBHashKV(…)` and `trace.Step(…)`. The variable's declaration decides the method: the type it is declared as, or the package of the project function its value comes from. A variable holding a value from outside your project, like `scanner := bufio.NewScanner(r)` or `url, err := url.Parse(raw)`, still links to nothing, and one whose type can't be told links only to a method no other type in the project has, never to one picked because it matches the variable's name. Re-index Go projects after upgrading. + #### MCP / indexing - Full indexes of large projects no longer sometimes run two to three times slower than usual. A timing clash between two background database maintenance steps could switch reference resolution from several threads to one partway through. diff --git a/README.md b/README.md index 7174c632e6..dffbacdf5a 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ Follow [@getcodegraph](https://x.com/getcodegraph) on X for updates. ## About this fork -This is **bompus/codegraph**, a fork of [colbymchenry/codegraph](https://github.com/colbymchenry/codegraph). Its default branch, `fork/consolidated`, contains upstream `main` through [`eec9288a`](https://github.com/colbymchenry/codegraph/commit/eec9288a) (after v1.6.2) plus the fork's own work, and it takes upstream changes as they land. Changes that suit upstream are also offered there as pull requests. +This is **bompus/codegraph**, a fork of [colbymchenry/codegraph](https://github.com/colbymchenry/codegraph). Its default branch, `fork/consolidated`, contains upstream `main` through [`19813544`](https://github.com/colbymchenry/codegraph/commit/19813544) (after v1.6.2) plus the fork's own work, and it takes upstream changes as they land. Changes that suit upstream are also offered there as pull requests. The fork publishes no releases. The install scripts, npm package, badges and `codegraph upgrade` further down this page install **upstream's** releases. To run the fork, build it from source (below). @@ -154,7 +154,7 @@ Call resolution the fork adds: - Kotlin multi-dollar strings keep their interpolation threshold. - C++ typedef and using aliases follow their declaration scope, including nested owners and inherited aliases. Pointer, array and function members are not supertypes. Visible class-scoped aliases and declared complex receivers keep their method owners; unsubstituted template parameters and ambiguous owners stay unresolved. - C++ `->` calls on a `std::unique_ptr`, `std::shared_ptr` or `std::optional` reach the type it holds, looked up through the caller's enclosing classes and their bases, its namespaces, `using` directives and aliases. A held type the project does not declare leaves the call unresolved, as a plain pointer to one does; upstream guesses a callee from the receiver's name there. A partial class template specialization is indexed as an anonymous class, so a call to a member only it declares stays unresolved. -- C++ calls on an `auto` local, on a chained call and on a data member of the caller's class (one a base class declares, or a gtest `TEST_F` / `TEST_P` fixture's) take the type its declaration writes, as upstream's do. Where that type cannot be read (a template parameter's, a `std::shared_ptr` of one, a trait of one, what a function template returns), the call stays unlinked, as does a local of a type the file does not declare; upstream guesses a callee from the receiver's name at 0.7. A chain of up to four calls is read from its declarations, where upstream reads its first inner call; a longer one is looked up by the last name, as before. +- C++ calls on an `auto` local, on a chained call and on a data member of the caller's class (one a base class declares, or a gtest `TEST_F` / `TEST_P` fixture's) take the type its declaration writes, as upstream's do. Where that type cannot be read (a template parameter's, a `std::shared_ptr` of one, a trait of one, what a function template returns, an iterator's element or a map's key or value that a class template writes with its parameter), the call stays unlinked, as does a local of a type the file does not declare; upstream guesses a callee from the receiver's name at 0.7. A chain of up to four calls is read from its declarations, where upstream reads its first inner call; a longer one is looked up by the last name, as before. - C++ namespace aliases and declarations in macro-opened namespaces are looked up only in the caller's include closure; upstream pools them across all files. - C++ object creation sees a header under a test directory, such as nlohmann/json's old-release copy in `tests/abi/`, only from inside that directory's tree or through an `#include`; upstream sees every header. A class declared in an included `.cc` file is the includer's too, and of two headers declaring one class the creation takes the one in its include closure. - C# namespace `using` directives and `using` aliases apply only inside their enclosing namespace, not to sibling namespaces in the same file (`using static` is still file-wide, as upstream). diff --git a/__tests__/c-feature-macro-variants.test.ts b/__tests__/c-feature-macro-variants.test.ts new file mode 100644 index 0000000000..4e8bf90d16 --- /dev/null +++ b/__tests__/c-feature-macro-variants.test.ts @@ -0,0 +1,186 @@ +/** + * A C call the file's headers answer only when the build sets a flag nothing + * in the project sets reaches the definition the build without it calls. + * lua-cjson's fpconv.h defines `fpconv_init` under `#ifdef + * USE_INTERNAL_FPCONV` and otherwise declares it for fpconv.c to define; the + * included header's definition was taken, though no build of redis compiles + * it. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; +import type { Node } from '../src/types'; + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +describe('a C call past a definition only a flag nothing sets compiles', () => { + let root: string; + let cg: CodeGraph; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-c-feature-variants-')); + const write = (file: string, text: string): void => { + fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true }); + fs.writeFileSync(path.join(root, file), text); + }; + // lua-cjson in redis: the header's definition is for a build that sets USE_INTERNAL_FPCONV. + write('deps/lua/src/fpconv.h', lines( + '#ifndef FPCONV_H', + '#define FPCONV_H', + '#ifdef USE_INTERNAL_FPCONV', + 'static inline void fpconv_init()', + '{', + '}', + '#else', + 'extern void fpconv_init();', + '#endif', + '#endif', + )); + write('deps/lua/src/fpconv.c', lines('#include "fpconv.h"', 'void fpconv_init()', '{', '}')); + // ...nor the internal one's own definition. + write('deps/lua/src/dtoa_fpconv.c', lines('#ifdef USE_INTERNAL_FPCONV', 'void fpconv_init()', '{', '}', '#endif')); + write('deps/lua/src/lua_cjson.c', lines('#include "fpconv.h"', 'int luaopen_cjson(void) { fpconv_init(); return 0; }')); + // hiredis: the header's definition is for the build that leaves the flag off. + write('deps/hiredis/alloc.h', lines( + '#ifndef HIREDIS_ALLOC_H', + '#define HIREDIS_ALLOC_H', + '#ifndef HIREDIS_EXTERN_ALLOC', + 'static inline void hi_free(void *ptr) {', + '}', + '#else', + 'void hi_free(void *ptr);', + '#endif', + '#endif', + )); + write('deps/hiredis/alloc_ext.c', lines('void hi_free(void *ptr) {', '}')); + write('deps/hiredis/sds.c', lines('#include "alloc.h"', 'void sdsfree(void *s) { hi_free(s); }')); + // A flag the project defines may be set. + write('include/config.h', lines('#define USE_FAST_CRC 1')); + write('include/crc.h', lines( + '#ifdef USE_FAST_CRC', + 'static inline unsigned crc32c(unsigned crc) { return crc; }', + '#else', + 'unsigned crc32c(unsigned crc);', + '#endif', + )); + write('src/crc.c', lines('unsigned crc32c(unsigned crc) { return ~crc; }')); + write('src/store.c', lines('#include "include/config.h"', '#include "include/crc.h"', 'unsigned checksum(void) { return crc32c(0); }')); + // So may one a build file names. + write('lib/Makefile', lines('CFLAGS += -DUSE_SIMD_HASH')); + write('lib/hash.h', lines('#ifdef USE_SIMD_HASH', 'static inline unsigned hash_bytes(const char *p) { return 0; }', '#endif')); + write('lib/hash_scalar.c', lines('unsigned hash_bytes(const char *p) { return 1; }')); + write('lib/table.c', lines('#include "hash.h"', 'unsigned slot(const char *k) { return hash_bytes(k); }')); + // And one of the platform's. + write('include/clock.h', lines('#ifdef __linux__', 'static inline long clock_ns(void) { return 0; }', '#else', 'long clock_ns(void);', '#endif')); + write('src/clock_posix.c', lines('long clock_ns(void) { return 1; }')); + write('src/timer.c', lines('#include "include/clock.h"', 'long tick(void) { return clock_ns(); }')); + // `#if` expressions, `#elif` and `#else`. + write('include/codec.h', lines( + '#if defined(CODEC_ZSTD) && !defined(CODEC_DISABLE)', + 'static inline int codec_level(void) { return 3; }', + '#elif CODEC_LZ4', + 'static inline int codec_bound(int n) { return n; }', + '#else', + 'static inline int codec_name(void) { return 0; }', + '#endif', + )); + write('src/codec_impl.c', lines( + 'int codec_level(void) { return 1; }', + 'int codec_bound(int n) { return n + 1; }', + 'int codec_name(void) { return 1; }', + )); + write('src/pack.c', lines('#include "include/codec.h"', 'int pack(void) { return codec_level() + codec_bound(1) + codec_name(); }')); + // An `#else` after a test that holds. + write('include/log.h', lines( + '#ifndef LOG_EXTERNAL', + 'static inline void log_write(const char *m) {}', + '#else', + 'static inline void log_flush(void) {}', + '#endif', + )); + write('src/log_ext.c', lines('void log_flush(void) {}')); + write('src/server.c', lines('#include "include/log.h"', 'void shutdown_server(void) { log_flush(); }')); + // A header that defines the function for either build. + write('include/prof.h', lines( + '#ifdef PROF_ENABLED', + 'static inline int prof_tick(void) { return 1; }', + '#else', + 'static inline int prof_tick(void) { return 0; }', + '#endif', + )); + write('src/prof_impl.c', lines('int prof_tick(void) { return 2; }')); + write('src/loop.c', lines('#include "include/prof.h"', 'int loop_tick(void) { return prof_tick(); }')); + // redis's quicklist.c: a call compiled only with the flag reaches what the flag compiles with it. + write('src/quicklist.c', lines( + '#ifdef REDIS_TEST', + 'static long long ustime(void) { return 0; }', + 'int quicklistTest(void) { return (int)ustime(); }', + '#endif', + )); + write('src/server_time.c', lines('long long ustime(void) { return 1; }')); + // ...in another file too. + write('include/simd.h', lines('#ifdef USE_SIMD_CRC', 'static inline unsigned crc_simd(unsigned c) { return c; }', '#endif')); + write('src/crc_soft.c', lines('unsigned crc_simd(unsigned c) { return ~c; }')); + write('src/checksum_simd.c', lines('#include "include/simd.h"', '#ifdef USE_SIMD_CRC', 'unsigned fast_sum(void) { return crc_simd(0); }', '#endif')); + // A flag another component defines is not this one's. + write('fpx/Makefile', lines('all: fpx.o use.o')); + write('fpx/fpx.h', lines('#ifdef FPX_FAST', 'static inline int fpx_round(int v) { return v; }', '#endif')); + write('fpx/fpx.c', lines('int fpx_round(int v) { return v + 1; }')); + write('fpx/use.c', lines('#include "fpx.h"', 'int use_round(void) { return fpx_round(1); }')); + write('other/config.h', lines('#define FPX_FAST 1')); + // A test in a comment is no test. + write('include/api.h', lines('/*', '#ifdef OLD_API', '*/', 'static inline int api_version(void) { return 2; }')); + write('src/api_v1.c', lines('int api_version(void) { return 1; }')); + write('src/client.c', lines('#include "include/api.h"', 'int client_version(void) { return api_version(); }')); + cg = await CodeGraph.init(root, { index: true }); + }, 120_000); + + afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); + }); + + /** `:` of what the function `name` calls, sorted. */ + const calls = (name: string): string[] => { + const caller = cg.getNodesByKind('function').find((n: Node) => n.name === name)!; + expect(caller, name).toBeDefined(); + return cg.getCallees(caller.id).filter((r) => r.edge.kind === 'calls').map((r) => `${r.node.filePath}:${r.node.startLine}`).sort(); + }; + + it('the definition the build without the flag compiles', () => { + expect(calls('luaopen_cjson')).toEqual(['deps/lua/src/fpconv.c:2']); + expect(calls('use_round')).toEqual(['fpx/fpx.c:1']); + }); + + it("the header's own, when the build without the flag compiles it", () => { + expect(calls('sdsfree')).toEqual(['deps/hiredis/alloc.h:4']); + }); + + it("the header's own, when the project or its build may set the flag", () => { + expect(calls('checksum')).toEqual(['include/crc.h:2']); + expect(calls('slot')).toEqual(['lib/hash.h:2']); + expect(calls('tick')).toEqual(['include/clock.h:2']); + }); + + it('reads `#if` expressions, `#elif` and `#else`', () => { + expect(calls('pack')).toEqual(['include/codec.h:6', 'src/codec_impl.c:1', 'src/codec_impl.c:2']); + expect(calls('shutdown_server')).toEqual(['src/log_ext.c:1']); + }); + + it('the definition the flag compiles with a call it compiles too', () => { + expect(calls('quicklistTest')).toEqual(['src/quicklist.c:2']); + expect(calls('fast_sum')).toEqual(['include/simd.h:2']); + }); + + it("the header's own, when it defines the function for either build", () => { + const callees = calls('loop_tick'); + expect(callees).toHaveLength(1); + expect(callees[0]).toMatch(/^include\/prof\.h:/); + }); + + it('reads no test in a comment', () => { + expect(calls('client_version')).toEqual(['include/api.h:4']); + }); +}); diff --git a/__tests__/c-static-call-rescue.test.ts b/__tests__/c-static-call-rescue.test.ts new file mode 100644 index 0000000000..a175a3f334 --- /dev/null +++ b/__tests__/c-static-call-rescue.test.ts @@ -0,0 +1,129 @@ +/** + * A C call reaches the function of its name the call's own file sees: its + * own, else one a header it includes defines, else one with external + * linkage. jemalloc's tests call the `static inline do_arena_create` of the + * arena_util.h they include; by directory, the name strategies picked a + * neighbouring test's own `static do_arena_create`, which no other file can + * call, and the call was left unlinked. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; +import type { Node } from '../src/types'; + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +describe("a C call reaches the function its file sees, not another file's static", () => { + let root: string; + let cg: CodeGraph; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-c-static-rescue-')); + const write = (file: string, text: string): void => { + fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true }); + fs.writeFileSync(path.join(root, file), text); + }; + // jemalloc: a header's static inline helper, and two tests with a static helper of its name. + write('test/include/test/arena_util.h', lines( + 'static inline unsigned', + 'do_arena_create(long dirty_decay_ms, long muzzy_decay_ms) {', + ' return 0;', + '}', + )); + write('test/unit/arena_reset.c', lines( + 'static unsigned', + 'do_arena_create(void *hooks) {', + ' return 1;', + '}', + 'void test_reset(void) { do_arena_create(0); }', + )); + write('test/unit/arena_decay.c', lines( + '#include "test/include/test/arena_util.h"', + 'void test_decay(void) { unsigned arena = do_arena_create(0, 0); }', + )); + // A function declared in a header and defined in another source file. + write('lib/helper.h', lines('int helper(int x);')); + write('lib/helper.c', lines('#include "lib/helper.h"', 'int helper(int x) { return x; }')); + write('app/other.c', lines('static int helper(int x) { return -x; }', 'int other(void) { return helper(1); }')); + write('app/main.c', lines('#include "lib/helper.h"', 'int run(void) { return helper(2); }')); + // Two builds' variants of one function: C has no overloads to choose among. + write('src/posix/log.c', lines('void log_msg(const char* msg) {}')); + write('src/win/log.c', lines('void log_msg(int level, const char* msg) {}')); + write('src/posix/app.c', lines('void start(void) { log_msg(1, "up"); }')); + // The header's function, over one nearer the call that no include brings in. + write('include/timer.h', lines('static inline long now_ns(void) { return 0; }')); + write('bench/unit/clock.c', lines('static long now_ns(void) { return 1; }')); + write('bench/other/fake_clock.c', lines('long now_ns(void) { return 2; }')); + write('bench/unit/latency.c', lines('#include "include/timer.h"', 'long elapsed(void) { return now_ns(); }')); + // ...and over one with external linkage the call could reach, where the file's header defines one. + write('include/ticks.h', lines('static inline long ticks(void) { return 0; }')); + write('bench/unit/ticks.c', lines('long ticks(void) { return 1; }')); + write('bench/unit/loop.c', lines('#include "include/ticks.h"', 'long spin(void) { return ticks(); }')); + // hiredis: a source file that includes another one calls its static functions. + write('deps/hiredis/dict.c', lines('static unsigned int dictGenHashFunction(const char* key) { return 0; }')); + write('deps/hiredis/async.c', lines('#include "deps/hiredis/dict.c"', 'unsigned int callbackHash(const char* key) { return dictGenHashFunction(key); }')); + write('server/dict.c', lines('unsigned int dictGenHashFunction(const char* key) { return 1; }')); + // jemalloc's sources include `jemalloc/internal/…` headers that only the end of their path names. + write('deps/jemalloc/include/jemalloc/internal/bit_util.h', lines('static inline unsigned pow2_ceil_u32(unsigned x) { return x; }')); + write('deps/jemalloc/src/arena.c', lines('#include "jemalloc/internal/bit_util.h"', 'unsigned arena_size(unsigned n) { return pow2_ceil_u32(n); }')); + write('deps/jemalloc/src/bitmap.c', lines('unsigned pow2_ceil_u32(unsigned x) { return x + 1; }')); + // ...and its tests a `test/math.h`, which no `#include ` names. + write('deps/jemalloc/test/include/test/math.h', lines('static inline double ln_gamma(double x) { return x; }')); + write('src/sort.c', lines('#include ', 'int sort_nan(double a) { return a != a; }')); + // A file's own definition, under another build's configuration than its header's. + write('include/fast.h', lines('#ifndef HAVE_FAST', 'static inline int fast_add(int a, int b) { return a + b; }', '#endif')); + write('src/fast.c', lines( + '#include "include/fast.h"', + '#ifdef HAVE_FAST', + 'static int fast_add(int a, int b) { return a - b; }', + '#endif', + 'int use_fast(void) { return fast_add(1, 2); }', + )); + cg = await CodeGraph.init(root, { index: true }); + }, 120_000); + + afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); + }); + + /** `:` of what the function `name` calls. */ + const calls = (name: string): string[] => { + const caller = cg.getNodesByKind('function').find((n: Node) => n.name === name)!; + expect(caller, name).toBeDefined(); + return cg.getCallees(caller.id).filter((r) => r.edge.kind === 'calls').map((r) => `${r.node.filePath}:${r.node.startLine}`); + }; + + it("the header's function, past a neighbouring file's static", () => { + expect(calls('test_decay')).toEqual(['test/include/test/arena_util.h:1']); + // The file's own static stays its own. + expect(calls('test_reset')).toEqual(['test/unit/arena_reset.c:1']); + }); + + it('a function another source file defines, when no include defines one', () => { + expect(calls('run')).toEqual(['lib/helper.c:2']); + expect(calls('other')).toEqual(['app/other.c:1']); + }); + + it('the function picked by where it is, when the file sees it', () => { + expect(calls('start')).toEqual(['src/posix/log.c:1']); + expect(calls('use_fast')).toEqual(['src/fast.c:3']); + }); + + it('the static function of a source file the file includes', () => { + expect(calls('callbackHash')).toEqual(['deps/hiredis/dict.c:1']); + }); + + it('one a header the file includes defines, over a nearer one', () => { + expect(calls('elapsed')).toEqual(['include/timer.h:1']); + expect(calls('spin')).toEqual(['include/ticks.h:1']); + }); + + it('one a header defines that the file includes by the end of its path', () => { + expect(calls('arena_size')).toEqual(['deps/jemalloc/include/jemalloc/internal/bit_util.h:1']); + // A standard header is not the project's header of its name. + expect(cg.getFileDependencies('src/sort.c')).not.toContain('deps/jemalloc/test/include/test/math.h'); + }); +}); diff --git a/__tests__/cpp-alias-owner-controls.test.ts b/__tests__/cpp-alias-owner-controls.test.ts index 1f1b76d05d..b9d7096387 100644 --- a/__tests__/cpp-alias-owner-controls.test.ts +++ b/__tests__/cpp-alias-owner-controls.test.ts @@ -63,11 +63,14 @@ it('keeps an inner free call separate from an outer same-named method', async () const targets=graph.getOutgoingEdgesFrom(graph.getNodesInFile('use.cpp').map(n=>n.id)).filter(e=>e.kind==='calls').map(e=>graph!.getNode(e.target)!.qualifiedName).sort(); expect(targets).toEqual(['Json::dump','dump']); }); -it('refuses an ambiguous C++ method overload set on a subscript result', async () => { +it('picks the C++ method overload its argument count fits on a subscript result', async () => { root=fs.mkdtempSync(path.join(os.tmpdir(),'cg-cpp-alias-')); fs.writeFileSync(path.join(root,'use.cpp'),'class Json {public: using reference=Json&; reference operator[](int){return *this;} void dump(){} void dump(int){} };\nvoid run(){Json j; j[0].dump(); j[0].dump(1);}\n'); graph=await CodeGraph.init(root,{index:true}); - expect(graph.getOutgoingEdgesFrom(graph.getNodesInFile('use.cpp').map(n=>n.id)).filter(e=>e.kind==='calls')).toEqual([]); + // The subscript reads `Json&` (upstream #2602), so each call reaches the overload of its arity: + // `dump()` at column 84, `dump(int)` at column 98. + const targets=graph.getOutgoingEdgesFrom(graph.getNodesInFile('use.cpp').map(n=>n.id)).filter(e=>e.kind==='calls').map(e=>`${graph!.getNode(e.target)!.qualifiedName}:${graph!.getNode(e.target)!.startColumn}`).sort(); + expect(targets).toEqual(['Json::dump:84','Json::dump:98']); }); it('does not resolve an unsubstituted generic field to a same-named global class', async () => { root=fs.mkdtempSync(path.join(os.tmpdir(),'cg-cpp-alias-')); diff --git a/__tests__/cpp-auto-iterator-arrow.test.ts b/__tests__/cpp-auto-iterator-arrow.test.ts new file mode 100644 index 0000000000..0d79542b38 --- /dev/null +++ b/__tests__/cpp-auto-iterator-arrow.test.ts @@ -0,0 +1,248 @@ +/** + * A `->` call on a C++ `auto` local that a library container's member or a + * standard search hands an iterator, like rocksdb's `auto it = + * logs_.begin(); it->IsSyncing()`, calls a member of the container's + * element, as on an iterator declared with its type. The local had no type, + * so the call was matched by the method's name alone. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +async function indexed(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-auto-iterator-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +function callable(cg: CodeGraph, qualifiedName: string): Node { + const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName); + if (!node) throw new Error(`no function or method ${qualifiedName}`); + return node; +} + +/** `calls` callees of a function or method, as `qualifiedName`. */ +function calls(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(callable(cg, caller).id) + .filter((r) => r.edge.kind === 'calls') + .map((r) => r.node.qualifiedName) + .sort(); +} + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +describe('a `->` call on a C++ `auto` local holding a library container\'s iterator', () => { + it('reproduction: rocksdb\'s `auto it = logs_.begin(); it->IsSyncing()`', async () => { + const cg = await indexed({ + 'db/log_writer.h': lines( + 'class Writer {', + ' public:', + ' bool IsSyncing() const { return false; }', + '};', + ), + 'db/db_impl.h': lines( + '#include ', + 'class DBImpl {', + ' public:', + ' struct LogWriterNumber {', + ' bool IsSyncing() const { return true; }', + ' void ReleaseWriter() {}', + ' };', + ' void MarkLogsSynced();', + ' private:', + ' std::deque logs_;', + '};', + ), + 'db/db_impl.cc': lines( + '#include "db/db_impl.h"', + '#include "db/log_writer.h"', + 'void DBImpl::MarkLogsSynced() {', + ' auto it = logs_.begin();', + ' if (it->IsSyncing()) it->ReleaseWriter();', + '}', + ), + }); + try { + expect(calls(cg, 'DBImpl::MarkLogsSynced')).toEqual(['DBImpl::LogWriterNumber::IsSyncing', 'DBImpl::LogWriterNumber::ReleaseWriter']); + } finally { + cg.close(); + } + }); + + it('from a standard search, and a reverse iterator', async () => { + const cg = await indexed({ + 'util/entries.h': lines( + 'class Entry {', + ' public:', + ' const char* name() const { return nullptr; }', + '};', + 'class Column {', + ' public:', + ' const char* name() const { return nullptr; }', + '};', + ), + 'util/lookup.cc': lines( + '#include ', + '#include ', + '#include "util/entries.h"', + 'const char* Find(const std::vector& entries, const Entry& key) {', + ' auto it = std::lower_bound(entries.begin(), entries.end(), key);', + ' return it->name();', + '}', + 'const char* Last(std::vector& entries) {', + ' auto it = entries.rbegin();', + ' return it->name();', + '}', + ), + }); + try { + expect(calls(cg, 'Find')).toEqual(['Entry::name']); + expect(calls(cg, 'Last')).toEqual(['Entry::name']); + } finally { + cg.close(); + } + }); + + it('the element is looked up where the container\'s type is written', async () => { + const cg = await indexed({ + 'db/version_set.h': lines( + '#include ', + 'struct ObsoleteFile {', + ' void DeleteFile() {}', + '};', + 'class VersionSet {', + ' public:', + ' struct ObsoleteFile {', + ' void DeleteFile() {}', + ' };', + ' std::vector obsolete_files_;', + '};', + ), + 'db/purge.cc': lines( + '#include "db/version_set.h"', + 'void Purge(VersionSet* vs) {', + ' auto it = vs->obsolete_files_.begin();', + ' it->DeleteFile();', + '}', + ), + }); + try { + expect(calls(cg, 'Purge')).toEqual(['VersionSet::ObsoleteFile::DeleteFile']); + } finally { + cg.close(); + } + }); + + it('a smart pointer member\'s type is still looked up where the call is: a pimpl defined in the source file', async () => { + const cg = await indexed({ + 'db/database.h': lines( + '#include ', + 'class SimpleDatabase {', + ' public:', + ' class DescriptorIndex {', + ' public:', + ' bool AddFile() { return false; }', + ' };', + '};', + 'class EncodedDatabase {', + ' public:', + ' bool Add();', + ' private:', + ' class DescriptorIndex;', + ' std::unique_ptr index_;', + '};', + ), + 'db/database.cc': lines( + '#include "db/database.h"', + 'class EncodedDatabase::DescriptorIndex {', + ' public:', + ' bool AddFile() { return true; }', + '};', + 'bool EncodedDatabase::Add() {', + ' return index_->AddFile();', + '}', + ), + }); + try { + expect(calls(cg, 'EncodedDatabase::Add')).toEqual(['EncodedDatabase::DescriptorIndex::AddFile']); + } finally { + cg.close(); + } + }); + + it('an element of a library type has no project method to call', async () => { + const cg = await indexed({ + 'util/library.h': lines( + 'class ObjectLibrary {', + ' public:', + ' ObjectLibrary* get() { return this; }', + '};', + ), + 'util/registry.cc': lines( + '#include ', + '#include ', + '#include "util/library.h"', + 'class Registry {', + ' public:', + ' bool Newest() const {', + ' auto iter = libraries_.crbegin();', + ' return iter->get() != nullptr;', + ' }', + ' private:', + ' std::vector> libraries_;', + '};', + ), + }); + try { + expect(calls(cg, 'Registry::Newest')).toEqual([]); + } finally { + cg.close(); + } + }); + + it('a project class named like a library container keeps what its own `begin()` returns', async () => { + const cg = await indexed({ + 'util/span.h': lines( + 'class Foo {', + ' public:', + ' void Bar() {}', + '};', + 'class Baz {', + ' public:', + ' void Bar() {}', + '};', + 'namespace util {', + 'class span {', + ' public:', + ' Foo* begin() { return nullptr; }', + '};', + '} // namespace util', + ), + 'util/use.cc': lines( + '#include "util/span.h"', + 'void Use(util::span s) {', + ' auto it = s.begin();', + ' it->Bar();', + '}', + ), + }); + try { + expect(calls(cg, 'Use')).toEqual(['Foo::Bar', 'util::span::begin']); + } finally { + cg.close(); + } + }); +}); diff --git a/__tests__/cpp-container-accessor-calls.test.ts b/__tests__/cpp-container-accessor-calls.test.ts new file mode 100644 index 0000000000..827b4d67da --- /dev/null +++ b/__tests__/cpp-container-accessor-calls.test.ts @@ -0,0 +1,137 @@ +/** + * A C++ call on what a library container or holder hands out, like + * `bufs.front().Flush()` over a `std::vector`, `by_id.at(1)->Flush()` + * over a map, `pending.top()->Valid()` over a `std::stack`, `iter.get()->Valid()` + * on a `std::unique_ptr` or `table.value().Size()` on a `std::optional`, calls + * a member of the element's class. The library's member returned nothing the + * index could read, so such calls linked to nothing. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +describe('a C++ call on what a library container or holder hands out', () => { + let root: string; + let cg: CodeGraph; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-container-accessors-')); + const write = (file: string, text: string): void => { + fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true }); + fs.writeFileSync(path.join(root, file), text); + }; + // `Other` comes first: a call matched by its bare name lands on it. + write('src/types.h', lines( + 'class Other {', + ' public:', + ' void Flush() {}', + ' bool Valid() const { return false; }', + ' int Size() const { return 0; }', + '};', + 'class Buffer {', + ' public:', + ' void Flush() {}', + '};', + 'class Iterator {', + ' public:', + ' bool Valid() const { return true; }', + '};', + 'class Table {', + ' public:', + ' int Size() const { return 1; }', + '};', + // A project class named like a standard container is that class. + 'class list {', + ' public:', + ' Other front() { return Other(); }', + '};', + )); + write('src/autovector.h', lines( + 'namespace rocksdb {', + 'template ', + 'class autovector {', + ' public:', + ' T& front() { return values_[0]; }', + ' T& back() { return values_[kSize - 1]; }', + ' private:', + ' T values_[kSize];', + '};', + '} // namespace rocksdb', + )); + write('src/use.cc', lines( + '#include ', + '#include ', + '#include ', + '#include ', + '#include ', + '#include ', + '#include ', + '#include "src/autovector.h"', + '#include "src/types.h"', + 'void Front(std::vector& bufs) { bufs.front().Flush(); }', + 'void Back(std::vector* iters) { iters->back()->Valid(); }', + 'void At(const std::vector& tables) { tables.at(0).Size(); }', + 'void MapAt(std::map& by_id) { by_id.at(1)->Flush(); }', + 'void Top(std::stack& pending) { pending.top()->Valid(); }', + 'void QueueFront(std::queue& queue) { queue.front().Flush(); }', + 'void Get(const std::unique_ptr& iter) { iter.get()->Valid(); }', + 'void Value(const std::optional
& table) { table.value().Size(); }', + 'void StatusOrValue(absl::StatusOr& buf) { buf.value().Flush(); }', + 'void Owned(std::vector>& iters) { iters.back()->Valid(); }', + 'void Local(std::vector& bufs) { auto& last = bufs.back(); last.Flush(); }', + 'void ProjectSequence(rocksdb::autovector& bufs) { bufs.front()->Flush(); }', + 'void ProjectList(list& l) { l.front().Flush(); }', + 'void Strings(std::vector& names) { names.front().size(); }', + 'template ', + 'void Generic(std::vector& items) { items.front().Flush(); }', + )); + cg = await CodeGraph.init(root, { index: true }); + }, 120_000); + + afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); + }); + + /** `calls` callees of the function `name`, as `qualifiedName`. */ + const calls = (name: string): string[] => { + const caller = cg.getNodesByKind('function').find((n: Node) => n.name === name); + expect(caller, name).toBeDefined(); + return cg.getCallees(caller!.id).filter((r) => r.edge.kind === 'calls').map((r) => r.node.qualifiedName).sort(); + }; + + it("a sequence's front(), back() and at(i)", () => { + expect(calls('Front')).toEqual(['Buffer::Flush']); + expect(calls('Back')).toEqual(['Iterator::Valid']); + expect(calls('At')).toEqual(['Table::Size']); + expect(calls('Owned')).toEqual(['Iterator::Valid']); + }); + + it("a map's at(key), a stack's top() and a queue's front()", () => { + expect(calls('MapAt')).toEqual(['Buffer::Flush']); + expect(calls('Top')).toEqual(['Iterator::Valid']); + expect(calls('QueueFront')).toEqual(['Buffer::Flush']); + }); + + it("a smart pointer's get() and an optional's or a StatusOr's value()", () => { + expect(calls('Get')).toEqual(['Iterator::Valid']); + expect(calls('Value')).toEqual(['Table::Size']); + expect(calls('StatusOrValue')).toEqual(['Buffer::Flush']); + }); + + it('a local the accessor initializes, and a project sequence', () => { + expect(calls('Local')).toEqual(['Buffer::Flush']); + expect(calls('ProjectSequence')).toEqual(['Buffer::Flush', 'rocksdb::autovector::front']); + }); + + it("a project class of a container's name, a library element and a template's own", () => { + expect(calls('ProjectList')).toEqual(['Other::Flush', 'list::front']); + expect(calls('Strings')).toEqual([]); + expect(calls('Generic')).toEqual([]); + }); +}); diff --git a/__tests__/cpp-dereferenced-iterator-calls.test.ts b/__tests__/cpp-dereferenced-iterator-calls.test.ts new file mode 100644 index 0000000000..d05ffb428c --- /dev/null +++ b/__tests__/cpp-dereferenced-iterator-calls.test.ts @@ -0,0 +1,176 @@ +/** + * A C++ call on a dereferenced variable, like `(*it)->Visit()` over a + * `std::vector`, or on a local an iterator's dereference initializes, + * like `auto first = *nodes.begin(); first->Visit()`, calls a member of the + * class the dereferenced value has. The extractor records the first by its + * bare name, which went to whichever class had a method of that name, and + * the second's local had no type. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +async function indexed(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-deref-iterator-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +function callable(cg: CodeGraph, qualifiedName: string): Node { + const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName); + if (!node) throw new Error(`no function or method ${qualifiedName}`); + return node; +} + +/** `calls` callees of a function or method, as `qualifiedName`. */ +function calls(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(callable(cg, caller).id) + .filter((r) => r.edge.kind === 'calls') + .map((r) => r.node.qualifiedName) + .sort(); +} + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +/** `Other` comes first: a call matched by its bare name lands on it. */ +const NODES = { + 'src/node.h': lines( + 'class Other {', + ' public:', + ' int Visit() { return 2; }', + ' int Weight() const { return 3; }', + '};', + 'class Node {', + ' public:', + ' int Visit() { return 0; }', + ' int Weight() const { return 1; }', + ' Node* Child() { return this; }', + '};', + ), +}; + +describe('a C++ call on a dereferenced iterator or pointer', () => { + it('reproduction: `(*it)->Visit()` over a `std::vector`, and `(*it).Weight()` over a `std::vector`', async () => { + const cg = await indexed({ + ...NODES, + 'src/walk.cc': lines( + '#include ', + '#include "src/node.h"', + 'int Walk(std::vector& nodes) {', + ' int sum = 0;', + ' for (std::vector::iterator it = nodes.begin(); it != nodes.end(); ++it) sum += (*it)->Visit();', + ' return sum;', + '}', + 'int Weigh(std::vector& values) {', + ' int sum = 0;', + ' for (auto vit = values.begin(); vit != values.end(); ++vit) sum += (*vit).Weight();', + ' return sum;', + '}', + ), + }); + try { + expect(calls(cg, 'Walk')).toEqual(['Node::Visit']); + expect(calls(cg, 'Weigh')).toEqual(['Node::Weight']); + } finally { + cg.close(); + } + }); + + it('a local the dereference initializes, an element held in a smart pointer, and a raw pointer', async () => { + const cg = await indexed({ + ...NODES, + 'src/first.cc': lines( + '#include ', + '#include ', + '#include "src/node.h"', + 'int First(std::vector& nodes) {', + ' auto first = *nodes.begin();', + ' return first->Visit();', + '}', + 'int Owned(std::vector>& owned) {', + ' auto oit = owned.begin();', + ' return (*oit)->Weight();', + '}', + 'int Pointed(Node* node) {', + ' return (*node).Weight();', + '}', + ), + }); + try { + expect(calls(cg, 'First')).toEqual(['Node::Visit']); + expect(calls(cg, 'Owned')).toEqual(['Node::Weight']); + expect(calls(cg, 'Pointed')).toEqual(['Node::Weight']); + } finally { + cg.close(); + } + }); + + it('`->` through an iterator in a chain, and in what initializes a local', async () => { + const cg = await indexed({ + ...NODES, + 'src/chain.cc': lines( + '#include ', + '#include "src/node.h"', + 'int Chain(std::vector& values) {', + ' std::vector::iterator it = values.begin();', + ' return it->Child()->Visit();', + '}', + 'int Local(std::vector& values) {', + ' std::vector::iterator it = values.begin();', + ' auto child = it->Child();', + ' return child->Visit();', + '}', + ), + }); + try { + expect(calls(cg, 'Chain')).toEqual(['Node::Child', 'Node::Visit']); + expect(calls(cg, 'Local')).toEqual(['Node::Child', 'Node::Visit']); + } finally { + cg.close(); + } + }); + + it('an element of a library type has no project method to call, nor does a free function of the name', async () => { + const cg = await indexed({ + ...NODES, + 'src/names.cc': lines( + '#include ', + '#include ', + '#include "src/node.h"', + 'class Label {', + ' public:', + ' int size() const { return 0; }', + '};', + 'int Visit() { return 9; }', + 'int Total(std::vector& names) {', + ' int n = 0;', + ' for (auto sit = names.begin(); sit != names.end(); ++sit) n += (*sit).size();', + ' return n;', + '}', + 'int Each(std::vector& nodes) {', + ' std::vector::iterator it = nodes.begin();', + ' return (*it)->Visit();', + '}', + ), + }); + try { + expect(calls(cg, 'Total')).toEqual([]); + expect(calls(cg, 'Each')).toEqual(['Node::Visit']); + } finally { + cg.close(); + } + }); +}); diff --git a/__tests__/cpp-element-access-calls.test.ts b/__tests__/cpp-element-access-calls.test.ts new file mode 100644 index 0000000000..0689af627c --- /dev/null +++ b/__tests__/cpp-element-access-calls.test.ts @@ -0,0 +1,231 @@ +/** + * A C++ call on a subscripted element, like `statuses[i].IsNotFound()` over + * a `std::vector`, `m[k].Flush()` over a map, or `iters[i]->Valid()` + * over owning pointers, calls a member of the element's class. The + * extractor records such a call by its bare name, which went to whichever + * class had a method of that name, and a local an element initializes + * (`auto& s = statuses[i]`) had no type. An array's element is the type the + * array is declared with (`Row rows[2]`), however many dimensions it has. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +async function indexed(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-element-access-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +function callable(cg: CodeGraph, qualifiedName: string): Node { + const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName); + if (!node) throw new Error(`no function or method ${qualifiedName}`); + return node; +} + +/** `calls` callees of a function or method, as `qualifiedName`. */ +function calls(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(callable(cg, caller).id) + .filter((r) => r.edge.kind === 'calls') + .map((r) => r.node.qualifiedName) + .sort(); +} + +/** `calls` callees of a function or method, as `qualifiedName:startLine`, which tells overloads apart. */ +function callsAt(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(callable(cg, caller).id) + .filter((r) => r.edge.kind === 'calls') + .map((r) => `${r.node.qualifiedName}:${r.node.startLine}`) + .sort(); +} + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +/** `Other` comes first: a call matched by its bare name lands on it. */ +const TYPES = { + 'src/types.h': lines( + 'class Other {', + ' public:', + ' bool IsNotFound() const { return false; }', + ' bool ok() const { return false; }', + ' void Flush() {}', + ' bool Valid() const { return false; }', + ' int size() const { return 0; }', + ' void Print() const {}', + ' int Size() const { return 0; }', + '};', + 'class Status {', + ' public:', + ' bool IsNotFound() const { return true; }', + ' bool ok() const { return true; }', + '};', + 'class Buffer {', + ' public:', + ' void Flush() {}', + '};', + 'class Iterator {', + ' public:', + ' bool Valid() const { return true; }', + '};', + 'class Cell {', + ' public:', + ' void Print() const {}', + '};', + 'class Row {', + ' public:', + ' Cell& operator[](int i) { return cells_[i]; }', + ' int Size() const { return 4; }', + ' private:', + ' Cell cells_[4];', + '};', + ), +}; + +describe('a C++ call on a subscripted element', () => { + it('reproduction: a vector\'s, a map\'s, and owning pointers\' elements', async () => { + const cg = await indexed({ + ...TYPES, + 'src/access.cc': lines( + '#include ', + '#include ', + '#include ', + '#include "src/types.h"', + 'bool Missing(std::vector& statuses, int i) { return statuses[i].IsNotFound(); }', + 'void FlushOne(std::map& buffers, int k) { buffers[k].Flush(); }', + 'bool AnyValid(std::vector>& iters, int i) { return iters[i]->Valid(); }', + ), + }); + try { + expect(calls(cg, 'Missing')).toEqual(['Status::IsNotFound']); + expect(calls(cg, 'FlushOne')).toEqual(['Buffer::Flush']); + expect(calls(cg, 'AnyValid')).toEqual(['Iterator::Valid']); + } finally { + cg.close(); + } + }); + + it('a raw array, a pointer, two subscripts, a pair\'s member, and a local the element initializes', async () => { + const cg = await indexed({ + ...TYPES, + 'src/more.cc': lines( + '#include ', + '#include ', + '#include "src/types.h"', + 'bool Array() { Status statuses[3]; return statuses[0].ok(); }', + 'bool Pointer(Status* statuses, int i) { return statuses[i].ok(); }', + 'void Grid(std::vector>& table) { table[0][1].Flush(); }', + 'void Pairs(std::vector>& cfs, int i) { cfs[i].first.Flush(); }', + 'bool Bound(std::vector& statuses, int i) {', + ' auto& s = statuses[i];', + ' return s.ok();', + '}', + ), + }); + try { + expect(calls(cg, 'Array')).toEqual(['Status::ok']); + expect(calls(cg, 'Pointer')).toEqual(['Status::ok']); + expect(calls(cg, 'Grid')).toEqual(['Buffer::Flush']); + expect(calls(cg, 'Pairs')).toEqual(['Buffer::Flush']); + expect(calls(cg, 'Bound')).toEqual(['Status::ok']); + } finally { + cg.close(); + } + }); + + it('a class\'s `operator[]`, a key with a `]` in it, and an element of a library type', async () => { + const cg = await indexed({ + ...TYPES, + 'src/rows.cc': lines( + '#include ', + '#include ', + '#include ', + '#include "src/types.h"', + 'void Show(Row& row) { row[0].Print(); }', + 'void Named(std::map& buffers) { buffers["a]b"].Flush(); }', + 'int Length(std::vector& values, int i) { return values[i].size(); }', + ), + }); + try { + expect(calls(cg, 'Show')).toEqual(['Cell::Print']); + expect(calls(cg, 'Named')).toEqual(['Buffer::Flush']); + expect(calls(cg, 'Length')).toEqual([]); + } finally { + cg.close(); + } + }); + + it('an array is of the type it is declared with, however many dimensions it has', async () => { + const cg = await indexed({ + ...TYPES, + 'src/arrays.cc': lines( + '#include ', + '#include "src/types.h"', + 'class Shards {', + ' public:', + ' bool Second() { return statuses_[1].ok(); }', + ' private:', + ' Status statuses_[2];', + '};', + 'int Rows() { Row rows[2]; return rows[1].Size(); }', + 'using Line = Row;', + 'int Lines() { Line lines[2]; return lines[1].Size(); }', + 'bool Grid() { Status grid[2][3]; return grid[1][2].ok(); }', + 'void Pools() { std::vector pools[2]; pools[1][0].Flush(); }', + 'bool First() { Status statuses[2]; return (*statuses).ok(); }', + 'bool Each() {', + ' Status statuses[2];', + ' for (auto& s : statuses) {', + ' if (!s.ok()) return false;', + ' }', + ' return true;', + '}', + ), + }); + try { + expect(calls(cg, 'Shards::Second')).toEqual(['Status::ok']); + expect(calls(cg, 'Rows')).toEqual(['Row::Size']); + expect(calls(cg, 'Lines')).toEqual(['Row::Size']); + expect(calls(cg, 'Grid')).toEqual(['Status::ok']); + expect(calls(cg, 'Pools')).toEqual(['Buffer::Flush']); + expect(calls(cg, 'First')).toEqual(['Status::ok']); + expect(calls(cg, 'Each')).toEqual(['Status::ok']); + } finally { + cg.close(); + } + }); + + it('an element read by subscript picks the overload it is passed to', async () => { + const cg = await indexed({ + ...TYPES, + 'src/sink.cc': lines( + '#include ', + '#include "src/types.h"', + 'class Sink {', + ' public:', + ' void Put(const Other& o) {}', + ' void Put(const Buffer& b) {}', + '};', + 'void Drain(Sink& sink, std::map& buffers) { sink.Put(buffers[1]); }', + ), + }); + try { + expect(callsAt(cg, 'Drain')).toEqual(['Sink::Put:6']); + } finally { + cg.close(); + } + }); +}); diff --git a/__tests__/cpp-free-function-overloads.test.ts b/__tests__/cpp-free-function-overloads.test.ts new file mode 100644 index 0000000000..c48af450b2 --- /dev/null +++ b/__tests__/cpp-free-function-overloads.test.ts @@ -0,0 +1,257 @@ +/** + * A C++ call to a free function picks among the function's overloads by its + * arguments, as a call to a method already does: their count, then the + * types they have where the call writes them. The name strategies pick a + * function by where it is, so protobuf's + * + * int32_t result; + * ParseWithLimit(limit, data, result); + * + * reached the `ParseWithLimit(…, int64_t& res)` overload defined nearer the + * call, not the `int32_t&` one. Only overloads the call's file can see + * compete: a `static` one in another source file never does. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; +import type { Node } from '../src/types'; + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +describe('a C++ call picks the free-function overload its arguments fit', () => { + let root: string; + let cg: CodeGraph; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-free-overloads-')); + const write = (file: string, text: string): void => { + fs.mkdirSync(path.dirname(path.join(root, 'src', file)), { recursive: true }); + fs.writeFileSync(path.join(root, 'src', file), text); + }; + // protobuf's varint_shuffle_test.cc: each call sits nearer the overload it doesn't take. + write('varint.cc', lines( + 'namespace pb {', + 'const char* ParseWithLimit(int rtlimit, const char* data, int32_t& res) { return data; }', + 'const char* ParseWithLimit(int rtlimit, const char* data, int64_t& res) { return data; }', + 'void Parse32(const char* data) { int32_t result; ParseWithLimit(1, data, result); }', + 'int Scale(int x, int y) { return x * y; }', + 'int Scale(int x) { return x; }', + 'int Twice(int v) { return Scale(v, 2); }', + 'void Put(const char* text) {}', + 'void Put(int value) {}', + 'void PutText() { Put("text"); }', + 'template void Store(T value) {}', + 'void Store(int value) {}', + 'void StoreName(const std::string& name) { Store(name); }', + 'void StoreCount() { Store(1); }', + '} // namespace pb', + )); + // Arguments both overloads fit alike; a namesake in another namespace, no overload. + write('render.cc', lines( + 'namespace pb {', + 'struct Circle {}; struct Square {}; struct Shape {};', + 'void Render(const Circle& c) {}', + 'void Render(const Square& s) {}', + 'void Draw(const Shape& shape) { Render(shape); }', + '} // namespace pb', + )); + write('print.cc', lines( + 'namespace a {', + 'struct Message { Message(const char* text) {} };', + 'void Print(const Message& m) {}', + 'void Hello() { Print("hi"); }', + '} // namespace a', + 'namespace b {', + 'void Print(const char* text) {}', + '} // namespace b', + )); + // Qualified calls: from another file, from the overloads' own file, from the global scope. + write('codec.h', lines( + 'namespace codec {', + 'inline void Encode(const char* text, char* out) {}', + 'inline void Encode(long value, char* out) {}', + 'inline void EncodeHere(char* out) { codec::Encode(7L, out); }', + '} // namespace codec', + )); + write('codec.cc', lines( + '#include "src/codec.h"', + 'void EncodeThere(char* out) { codec::Encode(7L, out); }', + 'void EncodeGlobal(char* out) { ::codec::Encode(7L, out); }', + )); + // A function with only specializations: deduction picks one. + write('serialize.h', lines( + 'namespace ns {', + 'template void Serialize(T val, char* dest);', + 'template <> inline void Serialize(char t, char* dest) {}', + 'template <> inline void Serialize(long t, char* dest) {}', + '} // namespace ns', + )); + write('serialize.cc', lines( + '#include "src/serialize.h"', + 'void WriteLong(char* dest) { ns::Serialize(1L, dest); }', + )); + // An overload only another source file sees, which the arguments fit + // better: indexed after the one the call sees (`Log`), and before it (`Warn`). + write('log.h', lines( + 'namespace lg {', + 'struct Status { Status(int code) {} };', + 'void Log(const Status& status);', + 'void Warn(const Status& status);', + '} // namespace lg', + )); + write('log.cc', lines( + '#include "src/log.h"', + 'namespace lg {', + 'void Log(const Status& status) {}', + 'void Warn(const Status& status) {}', + '} // namespace lg', + )); + write('log_private.cc', lines( + 'namespace lg {', + 'static void Log(int level) {}', + '} // namespace lg', + )); + write('a_private.cc', lines( + 'namespace lg {', + 'static void Warn(int level) {}', + '} // namespace lg', + )); + write('log_use.cc', lines( + '#include "src/log.h"', + 'void LogCode() { lg::Log(3); }', + 'void WarnCode() { lg::Warn(3); }', + )); + // protobuf's conformance suite: a parameter named like its function. + write('wire.h', lines( + 'namespace wire {', + 'struct Wire {};', + 'Wire Fixed64(uint64_t x);', + 'template Wire Fixed64(T x) { return Fixed64(static_cast(x)); }', + '} // namespace wire', + )); + write('wire.cc', lines( + '#include "src/wire.h"', + 'namespace wire {', + 'Wire Fixed64(uint64_t x) { return Wire(); }', + '} // namespace wire', + )); + write('wire_use.cc', lines( + '#include "src/wire.h"', + 'std::string u64(uint64_t u64) {', + ' wire::Fixed64(u64);', + ' return "";', + '}', + 'struct Counter { Counter(int n) {} int value() const { return 0; } };', + 'struct Gauge { int value() const { return 1; } };', + 'int count(int n) {', + ' Counter count(n);', + ' return count.value();', + '}', + )); + // A struct and a function of one name: the function hides the struct in a call. + write('stat.h', lines( + 'namespace sys {', + 'struct stat { long size; };', + 'inline int stat(const char* path, struct stat* out) { return 0; }', + '} // namespace sys', + )); + write('stat.cc', lines( + '#include "src/stat.h"', + 'long FileSize(const char* path) { sys::stat st; sys::stat(path, &st); return st.size; }', + 'struct Sized : sys::stat {};', + )); + // protobuf's Rust bindings: a Rust call to the C function its `extern` block declares. + write('rust/upb/lib.rs', lines('mod sys;', 'mod wire;')); + write('rust/upb/sys/mod.rs', lines('pub mod wire;')); + write('rust/upb/sys/wire/mod.rs', lines('pub mod wire;')); + write('rust/upb/sys/wire/wire.rs', lines('extern "C" {', ' pub fn upb_encode(msg: *const u8, len: usize) -> i32;', '}')); + write('rust/upb/wire.rs', lines( + 'use super::sys::wire::wire::{upb_encode};', + 'pub fn encode(msg: *const u8) -> i32 {', + ' unsafe { upb_encode(msg, 0) }', + '}', + )); + write('upb/wire/encode.c', lines('int upb_encode(const char* msg, unsigned long len) { return 0; }')); + // fmt's namespace, opened by a macro: the overloads take as many arguments. + write('fmtlike.h', lines( + '#define MY_BEGIN_NAMESPACE namespace mylib {', + '#define MY_END_NAMESPACE }', + 'MY_BEGIN_NAMESPACE', + 'inline void write(char* out, const char* text) {}', + 'inline void write(char* out, int value) {}', + 'MY_END_NAMESPACE', + )); + write('fmtlike.cc', lines( + '#include "src/fmtlike.h"', + 'void WriteCount(char* out, int count) { mylib::write(out, count); }', + )); + cg = await CodeGraph.init(root, { index: true }); + }, 120_000); + + afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); + }); + + /** `@:` of what the function `name` calls. */ + const calls = (name: string): string[] => { + const caller = cg.getNodesByKind('function').find((n: Node) => n.qualifiedName === name)!; + expect(caller, name).toBeDefined(); + return cg.getCallees(caller.id).filter((r) => r.edge.kind === 'calls') + .map((r) => `${r.node.qualifiedName}@${path.basename(r.node.filePath)}:${r.node.startLine}`); + }; + + it('by the types of its arguments', () => { + expect(calls('pb::Parse32')).toEqual(['pb::ParseWithLimit@varint.cc:2']); + expect(calls('pb::StoreName')).toEqual(['pb::Store@varint.cc:11']); + expect(calls('pb::StoreCount')).toEqual(['pb::Store@varint.cc:12']); + // The parameter, not the function it belongs to; a local declared like a call, as before. + expect(calls('u64')).toEqual(['wire::Fixed64@wire.cc:3']); + expect(calls('count')).toContain('Counter::value@wire_use.cc:6'); + }); + + it('a function, not the struct of its name; the struct where a type is named', () => { + expect(calls('FileSize')).toEqual(['sys::stat@stat.h:3']); + const sized = cg.getNodesByKind('struct').find((n: Node) => n.qualifiedName === 'Sized')!; + expect(cg.getOutgoingEdgesFrom([sized.id], ['extends']).map((e) => `${cg.getNode(e.target)?.kind} ${cg.getNode(e.target)?.qualifiedName}`)) + .toEqual(['struct sys::stat']); + }); + + it('by how many arguments it passes, and its string literals', () => { + expect(calls('pb::Twice')).toEqual(['pb::Scale@varint.cc:5']); + expect(calls('pb::PutText')).toEqual(['pb::Put@varint.cc:8']); + }); + + it('through a qualified name', () => { + expect(calls('EncodeThere')).toEqual(['codec::Encode@codec.h:3']); + expect(calls('codec::EncodeHere')).toEqual(['codec::Encode@codec.h:3']); + expect(calls('EncodeGlobal')).toEqual(['codec::Encode@codec.h:3']); + expect(calls('WriteCount')).toEqual(['write@fmtlike.h:5']); + }); + + it('among the specializations deduction chooses from', () => { + expect(calls('WriteLong')).toEqual(['ns::Serialize@serialize.h:4']); + }); + + it('the one picked by where it is, when the arguments fit both alike', () => { + expect(calls('pb::Draw')).toEqual(['pb::Render@render.cc:4']); + }); + + it("never another namespace's function of the name", () => { + expect(calls('a::Hello')).toEqual(['a::Print@print.cc:3']); + }); + + it('only for a C++ call: a Rust call keeps the declaration it names', () => { + const encode = cg.getNodesByKind('function').find((n: Node) => n.name === 'encode')!; + expect(encode).toBeDefined(); + expect(cg.getCallees(encode.id).filter((r) => r.edge.kind === 'calls') + .map((r) => `${r.node.name}@${r.node.filePath}:${r.node.startLine}`)).toEqual(['upb_encode@src/rust/upb/sys/wire/wire.rs:2']); + }); + + it('never an overload only another source file sees', () => { + expect(calls('LogCode')).toEqual(['lg::Log@log.cc:3']); + expect(calls('WarnCode')).toEqual(['lg::Warn@log.cc:4']); + }); +}); diff --git a/__tests__/cpp-global-alias-target.test.ts b/__tests__/cpp-global-alias-target.test.ts new file mode 100644 index 0000000000..5c6754c981 --- /dev/null +++ b/__tests__/cpp-global-alias-target.test.ts @@ -0,0 +1,216 @@ +/** + * A C++ type written from the global scope (`::upb::Arena`) is looked up there + * alone, also where an alias names it and where a class's base clause or a + * declaration writes it. + * + * protocolbuffers/protobuf's hpb declares `class Arena { … backend::Arena + * arena_; }`, where `namespace backend = hpb::internal::backend::upb;` and + * that namespace says `using Arena = ::upb::Arena;`. The alias was looked up + * as if written `upb::Arena`, from its own namespace, whose nearest `upb` is + * that namespace itself: the alias named itself, no class was found, and + * `arena_.Fuse(other.arena_)` fell back to the type's last name, landing on + * the calling method `hpb::Arena::Fuse`. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +async function indexed(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-global-alias-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +function callable(cg: CodeGraph, qualifiedName: string): Node { + const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName); + if (!node) throw new Error(`no function or method ${qualifiedName}`); + return node; +} + +/** `calls` callees of a function or method, as `qualifiedName`. */ +function calls(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(callable(cg, caller).id) + .filter((r) => r.edge.kind === 'calls') + .map((r) => r.node.qualifiedName) + .sort(); +} + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +const UPB_ARENA = lines( + 'namespace upb {', + 'class Arena {', + ' public:', + ' bool Fuse(Arena& other) { return true; }', + ' bool IsFused(Arena& other) const { return false; }', + ' void RefArena(const Arena& to) {}', + '};', + '} // namespace upb', +); + +describe('a C++ alias of a type written from the global scope', () => { + it('reproduction: protobuf hpb\'s `backend::Arena` is `upb::Arena`, not the calling method\'s own class', async () => { + const cg = await indexed({ + 'upb/mem/arena.hpp': UPB_ARENA, + 'hpb/multibackend.h': lines( + '#define HPB_INTERNAL_BACKEND_UPB 1', + '#define HPB_INTERNAL_BACKEND_CPP 2', + 'namespace hpb {', + 'namespace internal {', + 'namespace backend {', + 'namespace upb {}', + 'namespace cpp {}', + '} // namespace backend', + '} // namespace internal', + '#if HPB_INTERNAL_BACKEND == HPB_INTERNAL_BACKEND_UPB', + 'namespace backend = hpb::internal::backend::upb;', + '#elif HPB_INTERNAL_BACKEND == HPB_INTERNAL_BACKEND_CPP', + 'namespace backend = hpb::internal::backend::cpp;', + '#endif', + '} // namespace hpb', + ), + 'hpb/backend/types.h': lines( + '#include "hpb/multibackend.h"', + '#include "upb/mem/arena.hpp"', + 'namespace hpb {', + 'namespace internal {', + 'namespace backend {', + '#if HPB_INTERNAL_BACKEND == HPB_INTERNAL_BACKEND_UPB', + 'namespace upb {', + 'using Arena = ::upb::Arena;', + '}', + '#elif HPB_INTERNAL_BACKEND == HPB_INTERNAL_BACKEND_CPP', + 'namespace cpp {', + 'using Arena = google::protobuf::Arena;', + '}', + '#endif', + '} // namespace backend', + '} // namespace internal', + '} // namespace hpb', + ), + 'hpb/arena.h': lines( + '#include "hpb/backend/types.h"', + 'namespace hpb {', + 'class Arena {', + ' public:', + ' bool Fuse(Arena& other) { return arena_.Fuse(other.arena_); };', + ' bool IsFused(Arena& other) { return arena_.IsFused(other.arena_); };', + ' void RefArena(const Arena& to) { arena_.RefArena(to.arena_); }', + ' private:', + ' backend::Arena arena_;', + '};', + '} // namespace hpb', + ), + }); + try { + expect(calls(cg, 'hpb::Arena::Fuse')).toEqual(['upb::Arena::Fuse']); + expect(calls(cg, 'hpb::Arena::IsFused')).toEqual(['upb::Arena::IsFused']); + expect(calls(cg, 'hpb::Arena::RefArena')).toEqual(['upb::Arena::RefArena']); + } finally { + cg.close(); + } + }); + + it('names the global class, not a namesake in the scope around the alias', async () => { + const cg = await indexed({ + 'upb/mem/arena.hpp': UPB_ARENA, + 'hpb/arena.cc': lines( + '#include "upb/mem/arena.hpp"', + 'namespace hpb {', + 'namespace upb {', + 'class Arena {', + ' public:', + ' bool Fuse(Arena& other) { return false; }', + '};', + '} // namespace upb', + 'using GlobalArena = ::upb::Arena;', + 'bool Join(GlobalArena& a, GlobalArena& b) {', + ' return a.Fuse(b);', + '}', + '} // namespace hpb', + ), + }); + try { + expect(calls(cg, 'hpb::Join')).toEqual(['upb::Arena::Fuse']); + } finally { + cg.close(); + } + }); + + it('through an `auto` variable the alias types', async () => { + const cg = await indexed({ + 'upb/mem/arena.hpp': UPB_ARENA, + 'hpb/arena.cc': lines( + '#include "upb/mem/arena.hpp"', + 'namespace hpb {', + 'namespace upb {', + 'using Arena = ::upb::Arena;', + '} // namespace upb', + 'class Arena {', + ' public:', + ' bool Fuse(Arena& other) { return false; }', + '};', + 'upb::Arena MakeArena() { return upb::Arena(); }', + 'bool Join(upb::Arena& other) {', + ' auto made = MakeArena();', + ' return made.Fuse(other);', + '}', + '} // namespace hpb', + ), + }); + try { + expect(calls(cg, 'hpb::Join').filter((c) => c.endsWith('::Fuse'))).toEqual(['upb::Arena::Fuse']); + } finally { + cg.close(); + } + }); +}); + +describe('other C++ names written from the global scope', () => { + it('a base class: `::lib::Options`, not the namespace\'s own `lib::Options`', async () => { + const cg = await indexed({ + 'src/config.cc': lines( + 'namespace app {', + 'struct Inner {', + ' void Run() {}', + '};', + '} // namespace app', + 'namespace lib {', + 'struct Options {', + ' struct Inner {', + ' void Run() {}', + ' };', + '};', + '} // namespace lib', + 'namespace app {', + 'namespace lib {', + 'struct Options {};', + '} // namespace lib', + 'class Config : public ::lib::Options {};', + 'void Use() {', + ' Config::Inner inner;', + ' inner.Run();', + '}', + '} // namespace app', + ), + }); + try { + expect(calls(cg, 'app::Use')).toEqual(['lib::Options::Inner::Run']); + } finally { + cg.close(); + } + }); +}); diff --git a/__tests__/cpp-iterator-arrow.test.ts b/__tests__/cpp-iterator-arrow.test.ts new file mode 100644 index 0000000000..dac980659b --- /dev/null +++ b/__tests__/cpp-iterator-arrow.test.ts @@ -0,0 +1,228 @@ +/** + * A `->` call on a C++ receiver declared as a container's iterator, like + * `std::deque::const_iterator it; it->data()`, calls a member of the + * container's element, as `->` on a smart pointer calls one of what it + * holds. The call was matched by the method's name alone, so it could reach + * any class with a method of that name. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +async function indexed(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-iterator-arrow-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +function callable(cg: CodeGraph, qualifiedName: string): Node { + const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName); + if (!node) throw new Error(`no function or method ${qualifiedName}`); + return node; +} + +/** `calls` callees of a function or method, as `qualifiedName`. */ +function calls(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(callable(cg, caller).id) + .filter((r) => r.edge.kind === 'calls') + .map((r) => r.node.qualifiedName) + .sort(); +} + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +describe('a `->` call on a C++ receiver declared as a container\'s iterator', () => { + it('reproduction: `std::deque::const_iterator it; it->data()` calls the element\'s method', async () => { + const cg = await indexed({ + 'util/buffer.h': lines( + 'class Buffer {', + ' public:', + ' const char* data() const { return nullptr; }', + '};', + ), + 'include/slice.h': lines( + 'class Slice {', + ' public:', + ' const char* data() const { return nullptr; }', + '};', + ), + 'util/append.cc': lines( + '#include ', + '#include "util/buffer.h"', + '#include "include/slice.h"', + 'void Append(const std::deque& operands) {', + ' for (std::deque::const_iterator it = operands.begin(); it != operands.end(); ++it) {', + ' it->data();', + ' }', + '}', + ), + }); + try { + expect(calls(cg, 'Append')).toEqual(['Slice::data']); + } finally { + cg.close(); + } + }); + + it('an element of a library type has no project method to call', async () => { + const cg = await indexed({ + 'util/name.h': lines( + 'class Name {', + ' public:', + ' const char* c_str() const { return nullptr; }', + '};', + ), + 'tools/print.cc': lines( + '#include ', + '#include ', + '#include "util/name.h"', + 'void Print(const std::vector& names) {', + ' for (std::vector::const_iterator itr = names.begin(); itr != names.end(); ++itr) {', + ' itr->c_str();', + ' }', + '}', + ), + }); + try { + expect(calls(cg, 'Print')).toEqual([]); + } finally { + cg.close(); + } + }); + + it('the iterator a project container\'s own alias names: protobuf\'s `RepeatedPtrField::iterator`', async () => { + const cg = await indexed({ + 'src/repeated_ptr_field.h': lines( + 'namespace google {', + 'namespace protobuf {', + 'namespace internal {', + 'template ', + 'class RepeatedPtrIterator {', + ' public:', + ' Element& operator*() const { return *it_; }', + ' Element* operator->() const { return it_; }', + ' private:', + ' Element* it_;', + '};', + '} // namespace internal', + 'template ', + 'class RepeatedPtrField {', + ' public:', + ' typedef internal::RepeatedPtrIterator iterator;', + ' iterator begin() { return iterator(); }', + '};', + '} // namespace protobuf', + '} // namespace google', + ), + 'src/descriptor.pb.h': lines( + 'namespace google {', + 'namespace protobuf {', + 'class FileDescriptorProto {', + ' public:', + ' int path_size() const { return 1; }', + '};', + 'class SourceCodeInfo_Location {', + ' public:', + ' int path_size() const { return 0; }', + '};', + '} // namespace protobuf', + '} // namespace google', + ), + 'src/option_interpreter.cc': lines( + '#include "src/repeated_ptr_field.h"', + '#include "src/descriptor.pb.h"', + 'namespace google {', + 'namespace protobuf {', + 'int Count(RepeatedPtrField* locs) {', + ' RepeatedPtrField::iterator loc = locs->begin();', + ' return loc->path_size();', + '}', + '} // namespace protobuf', + '} // namespace google', + ), + }); + try { + expect(calls(cg, 'google::protobuf::Count')).toEqual([ + 'google::protobuf::RepeatedPtrField::begin', + 'google::protobuf::SourceCodeInfo_Location::path_size', + ]); + } finally { + cg.close(); + } + }); + + it('an element of the calling template\'s own parameter names no particular class', async () => { + const cg = await indexed({ + 'src/visit.cc': lines( + '#include ', + 'class Node {', + ' public:', + ' void Visit() {}', + '};', + 'template ', + 'void VisitAll(typename std::vector::iterator it) {', + ' it->Visit();', + '}', + ), + }); + try { + expect(calls(cg, 'VisitAll')).toEqual([]); + } finally { + cg.close(); + } + }); + + it('an element only a class template\'s own `operator*` names goes on as before', async () => { + const cg = await indexed({ + 'gtest/param.h': lines( + 'template ', + 'class ParamIterator {', + ' public:', + ' const T& operator*() const { return *value_; }', + ' const T* operator->() const { return value_; }', + ' private:', + ' const T* value_;', + '};', + 'template ', + 'class ParamGenerator {', + ' public:', + ' typedef ParamIterator iterator;', + ' iterator begin() const { return iterator(); }', + '};', + ), + 'gtest/run.cc': lines( + '#include "gtest/param.h"', + 'class Case {', + ' public:', + ' void Run() const {}', + '};', + 'void RunAll(const ParamGenerator& gen) {', + ' ParamGenerator::iterator it = gen.begin();', + ' it->Run();', + '}', + ), + }); + try { + // The element is `const T&`, which only the class template's own `operator*` writes, so + // its type can't be read here. Upstream then matches `Run` by its name alone; the fork's + // resolver leaves such a call unlinked (README, "About this fork": C++ calls on an `auto` + // local, ... a template parameter's type). + expect(calls(cg, 'RunAll')).toEqual(['ParamGenerator::begin']); + } finally { + cg.close(); + } + }); +}); diff --git a/__tests__/cpp-library-type-receiver.test.ts b/__tests__/cpp-library-type-receiver.test.ts index 19d82f6341..7f92f0fb33 100644 --- a/__tests__/cpp-library-type-receiver.test.ts +++ b/__tests__/cpp-library-type-receiver.test.ts @@ -233,7 +233,8 @@ describe('a C++ receiver declared as a library type with a lowercase name', () = }); try { expect(calls(cg, 'leveldb::DictSize')).toEqual([]); - expect(calls(cg, 'leveldb::Drop')).toEqual([]); + // A standard container's `value_type` is its element (upstream #2573), so `first` is a Slice. + expect(calls(cg, 'leveldb::Drop')).toEqual(['leveldb::Slice::remove_prefix']); } finally { cg.close(); } @@ -291,8 +292,9 @@ describe('a declaration that is not the receiver\'s does not rule a guess out', }); try { expect(calls(cg, 'leveldb::Trim')).toEqual(['leveldb::Slice::remove_prefix']); - // The closed block cannot type the new binding; its auto initializer is unproven. - expect(calls(cg, 'leveldb::Flush')).toEqual([]); + // The closed block's `buf.clear()` is the string's; the `auto` one holds `caches[0]`, an + // element of a `Cache*`, which the subscript reads (upstream #2602). + expect(calls(cg, 'leveldb::Flush')).toEqual(['leveldb::Cache::reset']); } finally { cg.close(); } diff --git a/__tests__/cpp-macro-defined-methods.test.ts b/__tests__/cpp-macro-defined-methods.test.ts new file mode 100644 index 0000000000..aca066388f --- /dev/null +++ b/__tests__/cpp-macro-defined-methods.test.ts @@ -0,0 +1,371 @@ +/** + * A C++ method a function-like macro defines, like protobuf's + * `PROTOBUF_DEFINE_ACCESSOR(FieldDescriptor, number, int)` expanding to + * `inline int FieldDescriptor::number() const { … }`, is a method node where + * the file invokes the macro. Its class only declares it (`int number() + * const;`), which mints no node, so it had none at all and no call to it + * linked. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { extractFromSource } from '../src/extraction/tree-sitter'; +import type { Node } from '../src/types'; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +async function indexed(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-macro-methods-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +function callable(cg: CodeGraph, qualifiedName: string): Node { + const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName); + if (!node) throw new Error(`no function or method ${qualifiedName}`); + return node; +} + +/** `calls` callees of a function or method, as `qualifiedName@file:line`. */ +function calls(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(callable(cg, caller).id) + .filter((r) => r.edge.kind === 'calls') + .map((r) => `${r.node.qualifiedName}@${r.node.filePath}:${r.node.startLine}`) + .sort(); +} + +/** The methods of a file, as `qualifiedName:line`. */ +function methodsIn(cg: CodeGraph, file: string): string[] { + return cg.getNodesByKind('method').filter((n) => n.filePath === file).map((n) => `${n.qualifiedName}:${n.startLine}`).sort(); +} + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +/** protobuf's descriptor.h, in small: accessors declared in the class, defined by macros below it. */ +const DESCRIPTOR_H = lines( + 'namespace google {', + 'namespace protobuf {', + 'class FieldDescriptor;', + 'class Descriptor {', + ' public:', + ' absl::string_view name() const;', + ' absl::string_view full_name() const;', + ' int field_count() const;', + ' const FieldDescriptor* field(int index) const;', + ' const Descriptor* containing_type() const;', + ' private:', + ' int field_count_;', + ' const FieldDescriptor* fields_;', + ' const Descriptor* containing_type_;', + '};', + 'class FieldDescriptor {', + ' public:', + ' int number() const;', + ' const Descriptor* containing_type() const;', + ' private:', + ' int number_;', + ' const Descriptor* containing_type_;', + '};', + '', + '#define PROTOBUF_DEFINE_ACCESSOR(CLASS, FIELD, TYPE) \\', + ' inline TYPE CLASS::FIELD() const { return FIELD##_; }', + '', + '// Name and full name are stored in a single array to save space.', + '#define PROTOBUF_DEFINE_NAME_ACCESSOR(CLASS) \\', + ' inline absl::string_view CLASS::name() const { return all_names_.name(); } \\', + ' inline absl::string_view CLASS::full_name() const { \\', + ' return all_names_.full_name(); \\', + ' }', + '', + '#define PROTOBUF_DEFINE_ARRAY_ACCESSOR(CLASS, FIELD, TYPE) \\', + ' inline TYPE CLASS::FIELD(int index) const { \\', + ' return FIELD##s_ + index; \\', + ' }', + '', + 'PROTOBUF_DEFINE_NAME_ACCESSOR(Descriptor)', + 'PROTOBUF_DEFINE_ACCESSOR(Descriptor, field_count, int)', + 'PROTOBUF_DEFINE_ARRAY_ACCESSOR(Descriptor, field, const FieldDescriptor*)', + 'PROTOBUF_DEFINE_ACCESSOR(Descriptor, containing_type,', + ' const Descriptor*)', + 'PROTOBUF_DEFINE_ACCESSOR(FieldDescriptor, number, int)', + 'PROTOBUF_DEFINE_ACCESSOR(FieldDescriptor, containing_type, const Descriptor*)', + '', + '#undef PROTOBUF_DEFINE_ACCESSOR', + '} // namespace protobuf', + '} // namespace google', +); + +describe('a C++ method a macro defines', () => { + it('reproduction: protobuf descriptor accessors, and calls through what they return', async () => { + const cg = await indexed({ + // A namesake of the class an accessor returns: the type is read as written. + 'src/legacy.h': lines( + 'namespace legacy {', + 'class Descriptor {', + ' public:', + ' absl::string_view full_name() const { return ""; }', + '};', + '} // namespace legacy', + ), + 'src/descriptor.h': DESCRIPTOR_H, + 'src/printer.cc': lines( + '#include "src/legacy.h"', + '#include "src/descriptor.h"', + 'namespace google {', + 'namespace protobuf {', + 'int Number(const FieldDescriptor* field) { return field->number(); }', + 'int Count(const Descriptor& d) { return d.field_count(); }', + 'absl::string_view Owner(const FieldDescriptor* field) { return field->containing_type()->full_name(); }', + 'int First(const Descriptor* d) { return d->field(0)->number(); }', + 'absl::string_view Parent(const FieldDescriptor* field) {', + ' auto parent = field->containing_type();', + ' return parent->full_name();', + '}', + '} // namespace protobuf', + '} // namespace google', + ), + }); + try { + expect(methodsIn(cg, 'src/descriptor.h')).toEqual([ + 'google::protobuf::Descriptor::containing_type:43', + 'google::protobuf::Descriptor::field:42', + 'google::protobuf::Descriptor::field_count:41', + 'google::protobuf::Descriptor::full_name:40', + 'google::protobuf::Descriptor::name:40', + 'google::protobuf::FieldDescriptor::containing_type:46', + 'google::protobuf::FieldDescriptor::number:45', + ]); + expect(calls(cg, 'google::protobuf::Number')).toEqual(['google::protobuf::FieldDescriptor::number@src/descriptor.h:45']); + expect(calls(cg, 'google::protobuf::Count')).toEqual(['google::protobuf::Descriptor::field_count@src/descriptor.h:41']); + expect(calls(cg, 'google::protobuf::Owner')).toEqual([ + 'google::protobuf::Descriptor::full_name@src/descriptor.h:40', + 'google::protobuf::FieldDescriptor::containing_type@src/descriptor.h:46', + ]); + expect(calls(cg, 'google::protobuf::First')).toEqual([ + 'google::protobuf::Descriptor::field@src/descriptor.h:42', + 'google::protobuf::FieldDescriptor::number@src/descriptor.h:45', + ]); + expect(calls(cg, 'google::protobuf::Parent')).toEqual([ + 'google::protobuf::Descriptor::full_name@src/descriptor.h:40', + 'google::protobuf::FieldDescriptor::containing_type@src/descriptor.h:46', + ]); + const number = callable(cg, 'google::protobuf::FieldDescriptor::number'); + expect(number.signature).toBe('inline int FieldDescriptor::number() const'); + expect(number.returnType).toBeUndefined(); + expect(callable(cg, 'google::protobuf::FieldDescriptor::containing_type').returnType).toBe('Descriptor'); + const holders = cg.getIncomingEdgesTo([number.id], ['contains']); + expect(holders.map((e) => cg.getNode(e.source)?.qualifiedName).sort()).toEqual(['google::protobuf::FieldDescriptor', 'src/descriptor.h']); + } finally { + cg.close(); + } + }); + + it('a macro of a source file that pastes the method name, for a class its header declares', async () => { + const cg = await indexed({ + 'src/reflection.h': lines( + 'class Reflection {', + ' public:', + ' int GetInt32(int index) const;', + ' void SetInt32(int index, int value) const;', + ' bool GetBool(int index) const;', + ' void SetBool(int index, bool value) const;', + '};', + ), + 'src/reflection.cc': lines( + '#include "src/reflection.h"', + '#define DEFINE_PRIMITIVE_ACCESSORS(TYPENAME, TYPE) \\', + ' TYPE Reflection::Get##TYPENAME(int index) const { \\', + ' return TYPE(); \\', + ' } \\', + ' void Reflection::Set##TYPENAME(int index, TYPE value) const {}', + '', + 'DEFINE_PRIMITIVE_ACCESSORS(Int32, int)', + 'DEFINE_PRIMITIVE_ACCESSORS(Bool, bool)', + '#undef DEFINE_PRIMITIVE_ACCESSORS', + 'namespace {', + 'class Local {', + ' public:', + ' int Get() const;', + '};', + '#define DEFINE_LOCAL_GET(C) inline int C::Get() const { return 1; }', + 'DEFINE_LOCAL_GET(Local)', + '} // namespace', + ), + 'src/use.cc': lines( + '#include "src/reflection.h"', + 'int Read(const Reflection& r) { r.SetBool(0, true); return r.GetInt32(1); }', + ), + }); + try { + // An anonymous namespace qualifies nothing, as for a definition written out. + expect(methodsIn(cg, 'src/reflection.cc')).toEqual([ + 'Local::Get:17', + 'Reflection::GetBool:9', + 'Reflection::GetInt32:8', + 'Reflection::SetBool:9', + 'Reflection::SetInt32:8', + ]); + expect(calls(cg, 'Read')).toEqual(['Reflection::GetInt32@src/reflection.cc:8', 'Reflection::SetBool@src/reflection.cc:9']); + } finally { + cg.close(); + } + }); + + it('a macro-defined overload is picked by the parameters its expansion writes, and is never a deleted one', async () => { + const cg = await indexed({ + 'src/getter.h': lines( + 'class Getter {', + ' public:', + ' int Get(int index) const;', + ' int Get(const char* key) const;', + '};', + '#define DEFINE_GET(PARAM) \\', + ' inline int Getter::Get(PARAM) const { return 0; }', + 'DEFINE_GET(int index)', + '// Between two invocations, another class deletes a method of the same name.', + 'class Blocked {', + ' public:', + ' int Get(double value) = delete;', + '};', + 'DEFINE_GET(const char* key)', + ), + 'src/use.cc': lines( + '#include "src/getter.h"', + 'int ByKey(const Getter& g) { return g.Get("key"); }', + 'int ByIndex(const Getter& g) { return g.Get(1); }', + ), + }); + try { + expect(calls(cg, 'ByKey')).toEqual(['Getter::Get@src/getter.h:14']); + expect(calls(cg, 'ByIndex')).toEqual(['Getter::Get@src/getter.h:8']); + } finally { + cg.close(); + } + }); + + it('an explicit specialization a macro defines is named without its template arguments', async () => { + const cg = await indexed({ + 'src/wire.h': lines( + 'class Wire {', + ' public:', + ' template static bool Read(T* value);', + '};', + '#define READ_PRIMITIVE(CPPTYPE, KIND) \\', + ' template <> inline bool Wire::Read(CPPTYPE* value) { return true; }', + 'READ_PRIMITIVE(int, 1)', + 'READ_PRIMITIVE(double, 2)', + ), + 'src/use.cc': lines( + '#include "src/wire.h"', + 'bool Use(int* value) { return Wire::Read(value); }', + ), + }); + try { + expect(methodsIn(cg, 'src/wire.h')).toEqual(['Wire::Read:7', 'Wire::Read:8']); + expect(calls(cg, 'Use')).toHaveLength(1); + expect(calls(cg, 'Use')[0]).toMatch(/^Wire::Read@src\/wire\.h:[78]$/); + } finally { + cg.close(); + } + }); + + it('what is not a method a macro defines', async () => { + const cg = await indexed({ + 'src/other.h': lines( + 'class Widget {', + ' public:', + ' Widget();', + ' void Draw() const;', + ' void Paint() const;', + ' int Size() const;', + ' static void Log(int x);', + '};', + '/*', + '#define COMMENTED(C) inline void C::Draw() const {}', + '*/', + '#define DEFINE_DRAW(C) inline void C::Draw() const {}', + '#define DEFINE_CTOR(C) inline C::C() {}', + '#define DEFINE_FREE(NAME) inline void NAME##_fn() {}', + '#define CALL_LOG(x) Widget::Log(x);', + '#define DECLARE_SIZE(C) int C::Size() const;', + 'COMMENTED(Widget)', + 'DEFINE_CTOR(Widget)', + 'DEFINE_FREE(widget)', + 'DECLARE_SIZE(Widget)', + 'inline void Use() {', + ' DEFINE_DRAW(Widget)', + ' CALL_LOG(1)', + '}', + 'struct Holder {', + ' DEFINE_DRAW(Widget)', + '};', + '#undef DEFINE_DRAW', + 'DEFINE_DRAW(Widget)', + '#define DEFINE_DRAW(C) inline void C::Paint() const {}', + 'DEFINE_DRAW(Widget)', + ), + }); + try { + // Only the invocation of the redefined macro, at namespace scope, defines one. + expect(methodsIn(cg, 'src/other.h')).toEqual(['Widget::Paint:31', 'Widget::Widget:3']); + } finally { + cg.close(); + } + }); + + it('keeps a default argument its class declares when a macro defines the method inside a namespace', async () => { + const cg = await indexed({ + 'src/widget.h': lines( + 'namespace shop {', + 'class Widget {', + ' public:', + ' int Get(int index = 0) const;', + '};', + '#define DEFINE_GET(C) inline int C::Get(int index) const { return index; }', + 'DEFINE_GET(Widget)', + '}', + ), + 'src/use.cpp': lines('#include "widget.h"', 'int Use(shop::Widget& w) {', ' return w.Get();', '}'), + }); + try { + expect(calls(cg, 'Use')).toEqual(['shop::Widget::Get@src/widget.h:7']); + } finally { + cg.close(); + } + }); + + it('is minted alike whether the kernel or the wasm extractor reads the file', () => { + const minted = (kernel: boolean): string[] => { + const saved = process.env.CODEGRAPH_KERNEL; + if (kernel) delete process.env.CODEGRAPH_KERNEL; + else process.env.CODEGRAPH_KERNEL = '0'; + try { + const result = extractFromSource('src/descriptor.h', DESCRIPTOR_H, 'cpp'); + const byId = new Map(result.nodes.map((n) => [n.id, n])); + return result.nodes + .filter((n) => n.kind === 'method' && n.signature?.includes('::')) + .map((n) => { + const holders = result.edges.filter((e) => e.kind === 'contains' && e.target === n.id).map((e) => byId.get(e.source)?.qualifiedName); + return `${n.id} ${n.qualifiedName} ${n.startLine}:${n.startColumn}-${n.endLine}:${n.endColumn} ${n.returnType ?? '-'} ${n.signature} <- ${holders.sort().join(',')}`; + }) + .sort(); + } finally { + if (saved === undefined) delete process.env.CODEGRAPH_KERNEL; + else process.env.CODEGRAPH_KERNEL = saved; + } + }; + const wasm = minted(false); + expect(wasm).toHaveLength(7); + expect(minted(true)).toEqual(wasm); + }); +}); diff --git a/__tests__/cpp-macro-namespace.test.ts b/__tests__/cpp-macro-namespace.test.ts index d6d76d9757..ca53c9fa96 100644 --- a/__tests__/cpp-macro-namespace.test.ts +++ b/__tests__/cpp-macro-namespace.test.ts @@ -28,6 +28,7 @@ OVERLOAD_END `, 'overloads.cc': `#include "overloads.h" int overload_call() { return overloaded::choose(1); } +int overload_unread(int n) { return overloaded::choose(mystery(n)); } `, 'alpha.h': `#define PROJECT_BEGIN namespace alpha { #define PROJECT_END } @@ -104,9 +105,16 @@ afterAll(() => { }); describe('C++ namespaces opened by a macro', () => { - it('declines a visible overload set without argument-type evidence', () => { - const ids = cg.getNodesInFile('overloads.cc').map(n => n.id); - expect(cg.getOutgoingEdgesFrom(ids).filter(e => e.kind === 'calls')).toEqual([]); + it('picks the visible overload its arguments fit, and declines one they leave undecided', () => { + // A literal `1` fits `choose(int)`, never `choose(const char*)` (upstream #2612's argument + // types); an argument whose type can't be read leaves the overload set undecided. + const calls = (fn: string) => { + const caller = cg.getNodesByKind('function').find(n => n.name === fn)!; + return cg.getOutgoingEdgesFrom([caller.id]).filter(e => e.kind === 'calls') + .map(e => `${cg.getNode(e.target)!.filePath}:${cg.getNode(e.target)!.startLine}`); + }; + expect(calls('overload_call')).toEqual(['overloads.h:4']); + expect(calls('overload_unread')).toEqual([]); }); it('uses the namespace macro definition visible in each translation unit', () => { for (const owner of ['alpha', 'beta']) { diff --git a/__tests__/cpp-map-iterator-calls.test.ts b/__tests__/cpp-map-iterator-calls.test.ts new file mode 100644 index 0000000000..7b828fa168 --- /dev/null +++ b/__tests__/cpp-map-iterator-calls.test.ts @@ -0,0 +1,219 @@ +/** + * A C++ call on a member of the `std::pair` a map's iterator points at, like + * `it->second.Flush()`, or a range-for's `kv.second.Flush()`, calls a member + * of the map's value (or key, for `first`). The extractor records such a + * call by its bare name, which went to whichever class had a method of that + * name, and a local the member initializes (`auto& v = it->second`) had no + * type. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +async function indexed(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-map-iterator-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +function callable(cg: CodeGraph, qualifiedName: string): Node { + const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName); + if (!node) throw new Error(`no function or method ${qualifiedName}`); + return node; +} + +/** `calls` callees of a function or method, as `qualifiedName`. */ +function calls(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(callable(cg, caller).id) + .filter((r) => r.edge.kind === 'calls') + .map((r) => r.node.qualifiedName) + .sort(); +} + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +/** `Other` comes first: a call matched by its bare name lands on it. */ +const TYPES = { + 'src/types.h': lines( + 'class Other {', + ' public:', + ' void Flush() {}', + ' void Print() const {}', + ' bool ok() const { return false; }', + ' int size() const { return 0; }', + '};', + 'class Writer {', + ' public:', + ' void Flush() {}', + '};', + 'class Buffer {', + ' public:', + ' void Flush() {}', + '};', + 'class Key {', + ' public:', + ' void Print() const {}', + '};', + 'class Status {', + ' public:', + ' bool ok() const { return true; }', + '};', + ), +}; + +describe('a C++ call on the member of the pair a map\'s iterator points at', () => { + it('reproduction: `it->second->Flush()` and `it->second.Flush()`', async () => { + const cg = await indexed({ + ...TYPES, + 'src/maps.cc': lines( + '#include ', + '#include ', + '#include "src/types.h"', + 'void FlushAll(std::map& writers) {', + ' for (auto it = writers.begin(); it != writers.end(); ++it) it->second->Flush();', + '}', + 'void FlushOne(std::map& buffers, int k) {', + ' auto it = buffers.find(k);', + ' it->second.Flush();', + '}', + ), + }); + try { + expect(calls(cg, 'FlushAll')).toEqual(['Writer::Flush']); + expect(calls(cg, 'FlushOne')).toEqual(['Buffer::Flush']); + } finally { + cg.close(); + } + }); + + it('a range-for\'s pair, a dereferenced iterator\'s, the key, and a local the member initializes', async () => { + const cg = await indexed({ + ...TYPES, + 'src/walk.cc': lines( + '#include ', + '#include "src/types.h"', + 'void Each(std::map& buffers) {', + ' for (auto& kv : buffers) kv.second.Flush();', + '}', + 'void Deref(std::map& buffers) {', + ' auto it = buffers.begin();', + ' (*it).second.Flush();', + '}', + 'void Keys(std::map& counts) {', + ' auto it = counts.begin();', + ' it->first.Print();', + '}', + 'void Bound(std::map& buffers, int k) {', + ' auto it = buffers.find(k);', + ' auto& v = it->second;', + ' v.Flush();', + '}', + ), + }); + try { + expect(calls(cg, 'Each')).toEqual(['Buffer::Flush']); + expect(calls(cg, 'Deref')).toEqual(['Buffer::Flush']); + expect(calls(cg, 'Keys')).toEqual(['Key::Print']); + expect(calls(cg, 'Bound')).toEqual(['Buffer::Flush']); + } finally { + cg.close(); + } + }); + + it('a pair the code declares, and a member of a library type, which has no project method', async () => { + const cg = await indexed({ + ...TYPES, + 'src/pairs.cc': lines( + '#include ', + '#include ', + '#include ', + '#include "src/types.h"', + 'bool Check() {', + ' std::pair status_and_path;', + ' return status_and_path.first.ok();', + '}', + 'int Length(std::map& names) {', + ' auto it = names.begin();', + ' return it->second.size();', + '}', + ), + }); + try { + expect(calls(cg, 'Check')).toEqual(['Status::ok']); + expect(calls(cg, 'Length')).toEqual([]); + } finally { + cg.close(); + } + }); + + it('a member another class template\'s parameter types goes on as before', async () => { + const cg = await indexed({ + 'util/core_local.h': lines( + '#include ', + '#include ', + 'template ', + 'class CoreLocalArray {', + ' public:', + ' std::pair AccessElementAndIndex() const { return {nullptr, 0}; }', + '};', + ), + 'util/cache.cc': lines( + '#include "util/core_local.h"', + 'class ZSTDCachedData {', + ' public:', + ' int GetUncompressData(long idx) { return 0; }', + '};', + 'class Rep {', + ' public:', + ' int Get() {', + ' auto p = per_core_.AccessElementAndIndex();', + ' return p.first->GetUncompressData(0);', + ' }', + ' private:', + ' CoreLocalArray per_core_;', + '};', + ), + }); + try { + // `p.first` is a `T*` that `CoreLocalArray` writes with its template parameter, so its + // type can't be read here. Upstream then matches `GetUncompressData` by its name alone; the + // fork's resolver leaves such a call unlinked (README, "About this fork": C++ calls on an + // `auto` local, ... a template parameter's type). + expect(calls(cg, 'Rep::Get')).toEqual(['CoreLocalArray::AccessElementAndIndex']); + } finally { + cg.close(); + } + }); + + it('a call on `this` is still the class\'s own, whatever its members are named', async () => { + const cg = await indexed({ + ...TYPES, + 'src/holder.cc': lines( + '#include "src/types.h"', + 'class Holder {', + ' public:', + ' void Flush() {}', + ' void Run() { this->Flush(); }', + '};', + ), + }); + try { + expect(calls(cg, 'Holder::Run')).toEqual(['Holder::Flush']); + } finally { + cg.close(); + } + }); +}); diff --git a/__tests__/cpp-outside-type-receiver.test.ts b/__tests__/cpp-outside-type-receiver.test.ts new file mode 100644 index 0000000000..fe13bf5dbc --- /dev/null +++ b/__tests__/cpp-outside-type-receiver.test.ts @@ -0,0 +1,439 @@ +/** + * A C++ receiver declared with a namespace-qualified type calls a member of a + * class that qualified name can be, never of a class that only shares its + * last name. Three cases read that wrong: + * + * - A namespace a macro stands for. facebook/rocksdb declares its types in + * `namespace ROCKSDB_NAMESPACE` under `#define ROCKSDB_NAMESPACE rocksdb`, + * so the index names them `ROCKSDB_NAMESPACE::DB`. Code that writes + * `rocksdb::DB* db; db->Put(…)` read as a call on an outside type and got no + * edge, and `ROCKSDB_NAMESPACE::Cache` did the same where a file declares + * `namespace rocksdb` itself. + * - A lowercase name a project type shares. `absl::flat_hash_set seen; + * seen.insert(…)` reached the project's own `util::flat_hash_set::insert`: + * a lowercase outside type was told apart from the project's only when no + * project type had its name at all. + * - A type whose last name finds only namesakes, where the receiver's + * declaration is none the call-site rules read (a macro body's, an `auto` + * variable's from another file's return type) or the type's namespace is + * one the project uses: protocolbuffers/protobuf's generated + * `DescriptorProto::ExtensionRange`, which the index doesn't hold, reached + * `Descriptor::ExtensionRange::options`, and `absl::Status _status` in a + * macro body `upb::Status::ok`. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +async function indexed(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-outside-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +function callable(cg: CodeGraph, qualifiedName: string): Node { + const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName); + if (!node) throw new Error(`no function or method ${qualifiedName}`); + return node; +} + +/** `calls` callees of a function or method, as `qualifiedName`. */ +function calls(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(callable(cg, caller).id) + .filter((r) => r.edge.kind === 'calls') + .map((r) => r.node.qualifiedName) + .sort(); +} + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +describe('a namespace a macro stands for is the project\'s', () => { + const ROCKSDB = { + 'include/rocksdb/rocksdb_namespace.h': lines( + '#ifndef ROCKSDB_NAMESPACE', + '#define ROCKSDB_NAMESPACE rocksdb', + '#endif', + ), + 'include/rocksdb/db.h': lines( + '#include "include/rocksdb/rocksdb_namespace.h"', + 'namespace ROCKSDB_NAMESPACE {', + 'class DB {', + ' public:', + ' int Put(int key) { return key; }', + '};', + '} // namespace ROCKSDB_NAMESPACE', + ), + 'upb/status.h': lines( + 'namespace upb {', + 'class Status {', + ' public:', + ' bool ok() const { return true; }', + '};', + '} // namespace upb', + ), + }; + + it('reproduction: `rocksdb::DB* db; db->Put(…)` is rocksdb\'s own DB', async () => { + const cg = await indexed({ + ...ROCKSDB, + 'java/rocksjni/put.cc': lines( + '#include "include/rocksdb/db.h"', + 'int Put(rocksdb::DB* db) {', + ' return db->Put(1);', + '}', + ), + }); + try { + expect(calls(cg, 'Put')).toEqual(['ROCKSDB_NAMESPACE::DB::Put']); + } finally { + cg.close(); + } + }); + + it('under either name, while a namespace the project declares nothing in stays outside', async () => { + // This tree declares its types in `namespace rocksdb` itself, and code + // names them through the macro. + const cg = await indexed({ + 'include/rocksdb/rocksdb_namespace.h': ROCKSDB['include/rocksdb/rocksdb_namespace.h'], + 'upb/status.h': ROCKSDB['upb/status.h'], + 'cache/cache.h': lines( + 'namespace rocksdb {', + 'class Cache {', + ' public:', + ' void Release(int handle) {}', + '};', + '} // namespace rocksdb', + ), + 'db/release.cc': lines( + '#include "include/rocksdb/rocksdb_namespace.h"', + '#include "cache/cache.h"', + '#include "upb/status.h"', + 'void Drop(ROCKSDB_NAMESPACE::Cache* cache) {', + ' cache->Release(1);', + '}', + 'bool Passed(const absl::Status& status) {', + ' return status.ok();', + '}', + ), + }); + try { + expect(calls(cg, 'Drop')).toEqual(['rocksdb::Cache::Release']); + expect(calls(cg, 'Passed')).toEqual([]); + } finally { + cg.close(); + } + }); +}); + +describe('a lowercase type from outside the project beside a namesake of the project\'s', () => { + const UTIL = { + 'util/hash_set.h': lines( + 'namespace util {', + 'template ', + 'class flat_hash_set {', + ' public:', + ' void insert(const T& v) {}', + '};', + '} // namespace util', + ), + }; + + it('reproduction: `absl::flat_hash_set seen; seen.insert(…)` is not util::flat_hash_set\'s', async () => { + const cg = await indexed({ + ...UTIL, + 'db/seen.cc': lines( + '#include "absl/container/flat_hash_set.h"', + '#include "util/hash_set.h"', + 'void Mark(int v) {', + ' absl::flat_hash_set seen;', + ' seen.insert(v);', + '}', + ), + }); + try { + expect(calls(cg, 'Mark')).toEqual([]); + expect(cg.getCallers(callable(cg, 'util::flat_hash_set::insert').id).filter((r) => r.edge.kind === 'calls')).toEqual([]); + } finally { + cg.close(); + } + }); + + it('the project\'s own, written with its namespace, a namespace alias, or one a macro opens', async () => { + const cg = await indexed({ + ...UTIL, + 'include/fmt/format.h': lines( + '#define FMT_BEGIN_NAMESPACE namespace fmt { inline namespace v11 {', + '#define FMT_END_NAMESPACE } }', + 'FMT_BEGIN_NAMESPACE', + 'namespace detail {', + 'template ', + 'class buffer {', + ' public:', + ' void push_back(const T& value) {}', + '};', + '} // namespace detail', + 'FMT_END_NAMESPACE', + ), + 'other/buffer.h': lines( + 'namespace other {', + 'class buffer {', + ' public:', + ' void push_back(int value) {}', + '};', + '} // namespace other', + ), + 'db/keep.cc': lines( + '#include "util/hash_set.h"', + '#include "include/fmt/format.h"', + 'namespace u = util;', + 'void Keep(int v) {', + ' util::flat_hash_set mine;', + ' mine.insert(v);', + '}', + 'void Alias(int v) {', + ' u::flat_hash_set aliased;', + ' aliased.insert(v);', + '}', + 'void Append(fmt::detail::buffer& buf) {', + ' buf.push_back(\'x\');', + '}', + ), + }); + try { + expect(calls(cg, 'Keep')).toEqual(['util::flat_hash_set::insert']); + expect(calls(cg, 'Alias')).toEqual(['util::flat_hash_set::insert']); + expect(calls(cg, 'Append')).toEqual(['detail::buffer::push_back']); + } finally { + cg.close(); + } + }); +}); + +describe('a qualified type whose last name finds only namesakes', () => { + /** protobuf's descriptor classes: `descriptor.pb.h` is too large to index, `descriptor.pb.cc` is not. */ + const DESCRIPTORS = { + 'src/google/protobuf/descriptor.h': lines( + 'namespace google {', + 'namespace protobuf {', + 'class Descriptor {', + ' public:', + ' class ExtensionRange {', + ' public:', + ' const int& options() const { return options_; }', + ' private:', + ' int options_;', + ' };', + '};', + 'namespace util {', + 'class SimpleFieldComparator {', + ' public:', + ' void set_treat_nan_as_equal(bool value) {}', + '};', + 'class DefaultFieldComparator : public SimpleFieldComparator {};', + '} // namespace util', + '} // namespace protobuf', + '} // namespace google', + ), + 'src/google/protobuf/descriptor.pb.cc': lines( + 'namespace google {', + 'namespace protobuf {', + 'class DescriptorProto::_Internal {', + ' public:', + ' using HasBits = int;', + '};', + 'DescriptorProto::DescriptorProto() {}', + 'void DescriptorProto::Clear() {}', + '} // namespace protobuf', + '} // namespace google', + ), + }; + + it('reproduction: a generated `DescriptorProto::ExtensionRange` is not `Descriptor::ExtensionRange`', async () => { + const cg = await indexed({ + ...DESCRIPTORS, + 'src/google/protobuf/compiler/importer_unittest.cc': lines( + '#include "src/google/protobuf/descriptor.h"', + 'namespace google {', + 'namespace protobuf {', + 'namespace compiler {', + 'int Options(const DescriptorProto::ExtensionRange& range) {', + ' return range.options();', + '}', + '} // namespace compiler', + '} // namespace protobuf', + '} // namespace google', + ), + }); + try { + expect(calls(cg, 'google::protobuf::compiler::Options')).toEqual([]); + } finally { + cg.close(); + } + }); + + it('a class whose methods alone are indexed, a base class\'s method, a template parameter\'s member', async () => { + const cg = await indexed({ + ...DESCRIPTORS, + // A namesake the written name can't be, beside the class only its methods show. + 'upb/reflection/descriptor_proto.h': lines( + 'namespace upb {', + 'class DescriptorProto {};', + '} // namespace upb', + ), + 'hpb/iterator.h': lines( + 'namespace hpb {', + 'namespace internal {', + 'struct ScalarIteratorPolicy {', + ' struct Payload {', + ' void AddOffset(int n) {}', + ' };', + '};', + 'template ', + 'class Iterator {', + ' private:', + ' typename PolicyT::Payload it_;', + ' public:', + ' void Next() { it_.AddOffset(1); }', + '};', + '} // namespace internal', + '} // namespace hpb', + ), + 'src/google/protobuf/compiler/reset.cc': lines( + '#include "src/google/protobuf/descriptor.h"', + 'namespace google {', + 'namespace protobuf {', + 'namespace compiler {', + 'void Reset(google::protobuf::DescriptorProto* proto) {', + ' proto->Clear();', + '}', + 'void Compare() {', + ' google::protobuf::util::DefaultFieldComparator comparator;', + ' comparator.set_treat_nan_as_equal(true);', + '}', + '} // namespace compiler', + '} // namespace protobuf', + '} // namespace google', + ), + }); + try { + expect(calls(cg, 'google::protobuf::compiler::Reset')).toEqual(['google::protobuf::DescriptorProto::Clear']); + expect(calls(cg, 'google::protobuf::compiler::Compare')).toEqual(['google::protobuf::util::SimpleFieldComparator::set_treat_nan_as_equal']); + // `typename PolicyT::Payload` is whatever the template argument makes it. Upstream then + // matches `AddOffset` by its name alone; the fork's resolver leaves such a call unlinked + // (README, "About this fork": where the type cannot be read, a template parameter's, the + // call stays unlinked). + expect(calls(cg, 'hpb::internal::Iterator::Next')).toEqual([]); + } finally { + cg.close(); + } + }); + + it('a nested type a derived class inherits, named through the derived class', async () => { + // Its namespace is a class of the project's, so the written name is never + // judged: `ns::Derived::Node` is `ns::Base::Node`. + const cg = await indexed({ + 'src/tree.h': lines( + 'namespace ns {', + 'struct Base {', + ' struct Node {', + ' void Visit() {}', + ' };', + '};', + 'struct Derived : Base {};', + '} // namespace ns', + ), + 'src/walk.cc': lines( + '#include "src/tree.h"', + 'void Root() {', + ' ns::Derived::Node node;', + '}', + 'void Walk() {', + ' node.Visit();', + '}', + ), + }); + try { + expect(calls(cg, 'Walk')).toEqual(['ns::Base::Node::Visit']); + } finally { + cg.close(); + } + }); + + it('an outside type no project class shares a name with stays past guessing', async () => { + // The declaration is another function's, so only the rule for a type no + // class has decides: a `->` on it is no guess at the one project `Read`. + const cg = await indexed({ + 'src/stream.h': lines( + 'class Stream {', + ' public:', + ' bool Read(int n) { return n > 0; }', + '};', + ), + 'src/client.cc': lines( + '#include "src/stream.h"', + 'void Open() {', + ' grpc::ClientReader reader;', + '}', + 'bool Pump() {', + ' return reader->Read(1);', + '}', + ), + }); + try { + expect(calls(cg, 'Pump')).toEqual([]); + } finally { + cg.close(); + } + }); + + it('a `->` on an outside holder of a class the project shares a name with reaches what it holds', async () => { + const cg = await indexed({ + 'util/optional.h': lines( + 'namespace util {', + 'template ', + 'class Optional {', + ' public:', + ' bool has_value() const { return true; }', + '};', + '} // namespace util', + ), + 'src/widget.h': lines( + 'class Widget {', + ' public:', + ' void Draw() {}', + '};', + ), + 'src/paint.cc': lines( + '#include "util/optional.h"', + '#include "src/widget.h"', + 'void Load() {', + ' folly::Optional widget;', + '}', + 'void Paint() {', + ' widget->Draw();', + '}', + ), + }); + try { + // `widget` is declared in `Load`, not in `Paint`, so nothing in scope gives it a type. + // Upstream matches `Draw` by its name alone; the fork's resolver leaves such a call unlinked + // (README, "About this fork": upstream guesses a callee from the receiver's name). + expect(calls(cg, 'Paint')).toEqual([]); + } finally { + cg.close(); + } + }); +}); diff --git a/__tests__/cpp-overload-constness.test.ts b/__tests__/cpp-overload-constness.test.ts new file mode 100644 index 0000000000..d8d2b11c66 --- /dev/null +++ b/__tests__/cpp-overload-constness.test.ts @@ -0,0 +1,195 @@ +/** + * A C++ call whose argument fits a function's const and non-const overloads + * alike — protobuf's `DynamicCastMessage(ptr.get())` beside `const T* + * DynamicCastMessage(const MessageLite*)` and `T* DynamicCastMessage(MessageLite*)` + * — reaches the one C++ picks: the non-const overload for a non-const object, + * the const one for a const object or for a value a call returns. Before, the + * overload declared first won the tie. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +describe("a C++ call picks between const and non-const overloads by its argument's constness", () => { + let root: string; + let cg: CodeGraph; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-overload-constness-')); + const write = (file: string, text: string): void => { + fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true }); + fs.writeFileSync(path.join(root, file), text); + }; + // The const overloads come first, as in protobuf's message_lite.h. + write('src/message_lite.h', lines( + '#include ', + 'class MessageLite {', + ' public:', + ' virtual ~MessageLite() {}', + '};', + 'class Message : public MessageLite {};', + 'template ', + 'const T* DynamicCastMessage(const MessageLite* from) { return nullptr; }', + 'template ', + 'T* DynamicCastMessage(MessageLite* from) { return nullptr; }', + 'template ', + 'const T& DynamicCastMessage(const MessageLite& from) { return *static_cast(&from); }', + 'template ', + 'T& DynamicCastMessage(MessageLite& from) { return *static_cast(&from); }', + 'class Buffer {};', + 'class Writer {', + ' public:', + ' void Write(const Buffer* buffer) {}', + ' void Write(Buffer* buffer) {}', + '};', + 'Message MakeMessage() { return Message(); }', + // The non-const overload first. + 'class Reader {', + ' public:', + ' void Read(Buffer* buffer) {}', + ' void Read(const Buffer* buffer) {}', + '};', + // Overloads that differ by more than a `const`, as fmt's `to_string` do. + 'template ', + 'int to_text(T value) { return 0; }', + 'template ', + 'int to_text(const T& value) { return 1; }', + // An array is a pointer to its element, as fmt's `Char buf[8]`. + 'template ', + 'OutputIt format_hex(OutputIt out, unsigned value) { return out; }', + 'template ', + 'Char* format_hex(Char* out, unsigned value) { return out; }', + // rocksdb's static_cast_with_check: the template's own `Src` takes the argument's `const`. + 'template ', + 'Dest* checked_cast(Src* x) { return nullptr; }', + 'template ', + 'std::shared_ptr checked_cast(std::shared_ptr&& x) { return nullptr; }', + // fmt's to_string_view: an argument of a template parameter's type could be a pointer. + 'template ', + 'int view_of(const T* s) { return 0; }', + 'template ', + 'int view_of(const T& s) { return 1; }', + // rocksdb's Put: characters are no ColumnFamilyHandle. + 'class Handle {};', + 'class Slice {', + ' public:', + ' Slice(const char* data) {}', + '};', + 'class Store {', + ' public:', + ' int Put(Handle* handle, const Slice& key) { return 0; }', + ' int Put(const Slice& key, const Slice& value) { return 1; }', + '};', + // ...but a template's own `T*` takes them. + 'template ', + 'int sink(T* p) { return 0; }', + 'int sink(const Slice& s) { return 1; }', + // A pointer is no characters, a `constexpr` array is const, and a long header is still a header. + 'int Emit(const char* text) { return 0; }', + 'int Emit(const Slice& text) { return 1; }', + 'int Take(char* p) { return 0; }', + 'int Take(const char* p) { return 1; }', + 'template ', + 'int peek(T* p) { return 0; }', + 'int peek(const Slice& s) { return 1; }', + )); + write('src/use.cc', lines( + '#include "src/message_lite.h"', + 'void NonConstHolder(std::shared_ptr ptr) { DynamicCastMessage(ptr.get()); }', + 'void ConstHolder(std::shared_ptr ptr) { DynamicCastMessage(ptr.get()); }', + 'void NonConstPointer(Message* m) { DynamicCastMessage(m); }', + 'void ConstPointer(const Message* m) { DynamicCastMessage(m); }', + 'void NonConstReference(Message& m) { DynamicCastMessage(m); }', + 'void ConstReference(const Message& m) { DynamicCastMessage(m); }', + 'void Temporary() { DynamicCastMessage(MakeMessage()); }', + 'void WriteNonConst(Writer& w, Buffer* b) { w.Write(b); }', + 'void WriteConst(Writer& w, const Buffer* b) { w.Write(b); }', + 'void WriteConstPointer(Writer& w, Buffer* const b) { w.Write(b); }', + 'void ReadNonConst(Reader& r, Buffer* b) { r.Read(b); }', + 'void ReadConst(Reader& r, const Buffer* b) { r.Read(b); }', + 'void Show(Buffer& b) { to_text(b); }', + 'template ', + 'void WriteHex(unsigned v) { Char buf[8]; format_hex(buf, v); }', + 'void Merge(const Buffer& other) { checked_cast(&other); }', + 'template ', + 'void Format(const S& fmt) { view_of(fmt); }', + // A header over several lines, as fmt's `join` writes. + 'template ', + 'void Join(It begin, Sentinel end, S&& sep) { view_of(sep); }', + 'void Save(Store& s) {', + ' char key[] = "key";', + ' char value[] = "value";', + ' s.Put(key, value);', + '}', + 'void Drain() { char buf[4]; sink(buf); }', + 'void Send(Buffer* b) { Emit(b); }', + 'void Keep() {', + ' constexpr char name[] = "name";', + ' Take(name);', + '}', + 'void Look(const Buffer* b) { peek(b); }', + )); + cg = await CodeGraph.init(root, { index: true }); + }, 120_000); + + afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); + }); + + /** The `name` overloads a function calls, as `name:startLine`. */ + const calls = (caller: string, name: string): string[] => { + const fn = cg.getNodesByKind('function').find((n: Node) => n.name === caller); + expect(fn, caller).toBeDefined(); + return cg.getCallees(fn!.id).filter((r) => r.edge.kind === 'calls' && r.node.name === name).map((r) => `${r.node.name}:${r.node.startLine}`); + }; + + it('a pointer to a non-const object takes the non-const overload, to a const one the const overload', () => { + expect(calls('NonConstHolder', 'DynamicCastMessage')).toEqual(['DynamicCastMessage:10']); + expect(calls('ConstHolder', 'DynamicCastMessage')).toEqual(['DynamicCastMessage:8']); + expect(calls('NonConstPointer', 'DynamicCastMessage')).toEqual(['DynamicCastMessage:10']); + expect(calls('ConstPointer', 'DynamicCastMessage')).toEqual(['DynamicCastMessage:8']); + }); + + it('a non-const object takes the non-const reference, a const object or a temporary the const one', () => { + expect(calls('NonConstReference', 'DynamicCastMessage')).toEqual(['DynamicCastMessage:14']); + expect(calls('ConstReference', 'DynamicCastMessage')).toEqual(['DynamicCastMessage:12']); + expect(calls('Temporary', 'DynamicCastMessage')).toEqual(['DynamicCastMessage:12']); + }); + + it("a class's own overloads, whichever comes first", () => { + expect(calls('WriteNonConst', 'Write')).toEqual(['Write:19']); + expect(calls('WriteConst', 'Write')).toEqual(['Write:18']); + // A pointer's own `const` is not its object's. + expect(calls('WriteConstPointer', 'Write')).toEqual(['Write:19']); + expect(calls('ReadNonConst', 'Read')).toEqual(['Read:24']); + expect(calls('ReadConst', 'Read')).toEqual(['Read:25']); + }); + + it('no overload that differs by more than a `const`, and an array as a pointer', () => { + expect(calls('Show', 'to_text')).toEqual(['to_text:28']); + expect(calls('WriteHex', 'format_hex')).toEqual(['format_hex:34']); + }); + + it("a template's own parameter, an argument of a template parameter's type, and characters for a class pointer", () => { + expect(calls('Merge', 'checked_cast')).toEqual(['checked_cast:36']); + expect(calls('Format', 'view_of')).toEqual(['view_of:40']); + expect(calls('Join', 'view_of')).toEqual(['view_of:40']); + expect(calls('Save', 'Put')).toEqual(['Put:51']); + expect(calls('Drain', 'sink')).toEqual(['sink:54']); + expect(calls('Send', 'Emit')).toEqual(['Emit:57']); + expect(calls('Keep', 'Take')).toEqual(['Take:59']); + expect(calls('Look', 'peek')).toEqual(['peek:64']); + }); +}); diff --git a/__tests__/cpp-receiver-global-far-library.test.ts b/__tests__/cpp-receiver-global-far-library.test.ts new file mode 100644 index 0000000000..ae8f877f4f --- /dev/null +++ b/__tests__/cpp-receiver-global-far-library.test.ts @@ -0,0 +1,278 @@ +/** + * Two ways a C++ receiver's declared type reached a class it doesn't name. + * + * - A type written from the global scope. The declaration a call's receiver + * was read from lost its leading `::`, so `::upb::Arena& a` inside `namespace + * hpb` was looked up from `hpb` and found a namespace-local `hpb::upb::Arena` + * first; and a function-local `using Foo = Bar;` was taken for `::Foo`. + * - A lowercase type from outside the project, declared outside the calling + * function (another function's variable of that name, a file's global): its + * last name still found a project class sharing it, as + * `util::flat_hash_set::insert` for an `absl::flat_hash_set`, or a + * global `string` class's `clear` for a `std::string`. Such a declaration + * may be another variable's, so the call is guessed for as when no class + * shares the type's name, never linked to the namesake. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +async function indexed(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-global-far-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +function callable(cg: CodeGraph, qualifiedName: string): Node { + const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName); + if (!node) throw new Error(`no function or method ${qualifiedName}`); + return node; +} + +/** `calls` callees of a function or method, as `qualifiedName`. */ +function calls(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(callable(cg, caller).id) + .filter((r) => r.edge.kind === 'calls') + .map((r) => r.node.qualifiedName) + .sort(); +} + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +describe('a C++ receiver declared with a type written from the global scope', () => { + const ARENAS = { + 'upb/mem/arena.hpp': lines( + 'namespace upb {', + 'class Arena {', + ' public:', + ' bool Fuse(Arena& other) { return true; }', + '};', + '} // namespace upb', + ), + 'hpb/upb_arena.h': lines( + 'namespace hpb {', + 'namespace upb {', + 'class Arena {', + ' public:', + ' bool Fuse(Arena& other) { return false; }', + '};', + '} // namespace upb', + '} // namespace hpb', + ), + }; + + it('reproduction: `::upb::Arena& a` inside `namespace hpb` is the global `upb::Arena`', async () => { + const cg = await indexed({ + ...ARENAS, + 'hpb/join.cc': lines( + '#include "upb/mem/arena.hpp"', + '#include "hpb/upb_arena.h"', + 'namespace hpb {', + 'bool Join(::upb::Arena& a, ::upb::Arena& b) {', + ' return a.Fuse(b);', + '}', + '} // namespace hpb', + ), + }); + try { + expect(calls(cg, 'hpb::Join')).toEqual(['upb::Arena::Fuse']); + } finally { + cg.close(); + } + }); + + it('declared over two lines, and as a range-based for loop\'s variable', async () => { + const cg = await indexed({ + ...ARENAS, + 'hpb/fuse.cc': lines( + '#include ', + '#include "upb/mem/arena.hpp"', + '#include "hpb/upb_arena.h"', + 'namespace hpb {', + 'bool Pair(::upb::Arena& b) {', + ' ::upb::Arena', + ' paired;', + ' return paired.Fuse(b);', + '}', + 'void FuseAll(std::vector<::upb::Arena>& arenas, ::upb::Arena& b) {', + ' for (::upb::Arena& each : arenas) {', + ' each.Fuse(b);', + ' }', + '}', + '} // namespace hpb', + ), + }); + try { + expect(calls(cg, 'hpb::Pair')).toEqual(['upb::Arena::Fuse']); + expect(calls(cg, 'hpb::FuseAll')).toEqual(['upb::Arena::Fuse']); + } finally { + cg.close(); + } + }); + + it('a `::` after a template\'s arguments is no global scope: `Outer::Iterator it` is Outer\'s', async () => { + const cg = await indexed({ + 'src/outer.h': lines( + 'class Iterator {', + ' public:', + ' bool Valid() const { return false; }', + '};', + 'namespace ns {', + 'template ', + 'class Outer {', + ' public:', + ' class Iterator {', + ' public:', + ' bool Valid() const { return true; }', + ' };', + ' bool Scan() {', + ' Outer::Iterator it;', + ' return it.Valid();', + ' }', + '};', + '} // namespace ns', + ), + }); + try { + expect(calls(cg, 'ns::Outer::Scan')).toEqual(['ns::Outer::Iterator::Valid']); + } finally { + cg.close(); + } + }); + + it('is no alias of that name the calling function declares', async () => { + const cg = await indexed({ + 'src/use.cc': lines( + 'class Foo {', + ' public:', + ' void Run() {}', + '};', + 'class Bar {', + ' public:', + ' void Run() {}', + '};', + 'void Use() {', + ' using Foo = Bar;', + ' ::Foo f;', + ' f.Run();', + '}', + ), + }); + try { + expect(calls(cg, 'Use')).toEqual(['Foo::Run']); + } finally { + cg.close(); + } + }); +}); + +describe('a lowercase outside type declared outside the calling function', () => { + it('reproduction: an `absl::flat_hash_set` is not the project\'s `util::flat_hash_set`', async () => { + const cg = await indexed({ + 'util/hash_set.h': lines( + 'namespace util {', + 'template ', + 'class flat_hash_set {', + ' public:', + ' void insert(const T& v) {}', + '};', + '} // namespace util', + ), + 'util/list.h': lines( + 'class List {', + ' public:', + ' void insert(int v) {}', + '};', + ), + 'db/seen.cc': lines( + '#include "util/hash_set.h"', + '#include "util/list.h"', + 'void Open() {', + ' absl::flat_hash_set seen;', + '}', + 'void Mark(int v) {', + ' seen.insert(v);', + '}', + ), + }); + try { + expect(calls(cg, 'Mark')).toEqual([]); + } finally { + cg.close(); + } + }); + + it('a `std::string` is not a global project class named `string`', async () => { + const cg = await indexed({ + 'src/string.h': lines( + 'class string {', + ' public:', + ' void clear() {}', + '};', + ), + 'src/buffer.h': lines( + 'class Buffer {', + ' public:', + ' void clear() {}', + '};', + ), + 'src/text.cc': lines( + '#include ', + '#include "src/string.h"', + '#include "src/buffer.h"', + 'void Load() {', + ' std::string text;', + '}', + 'void Reset() {', + ' text.clear();', + '}', + ), + }); + try { + expect(calls(cg, 'Reset')).toEqual([]); + } finally { + cg.close(); + } + }); + + it('the project\'s own lowercase type is still its own', async () => { + const cg = await indexed({ + 'util/hash_set.h': lines( + 'namespace util {', + 'template ', + 'class flat_hash_set {', + ' public:', + ' void insert(const T& v) {}', + '};', + '} // namespace util', + ), + 'db/keep.cc': lines( + '#include "util/hash_set.h"', + 'void Open() {', + ' util::flat_hash_set mine;', + '}', + 'void Keep(int v) {', + ' mine.insert(v);', + '}', + ), + }); + try { + expect(calls(cg, 'Keep')).toEqual(['util::flat_hash_set::insert']); + } finally { + cg.close(); + } + }); +}); diff --git a/__tests__/cpp-specialization-names.test.ts b/__tests__/cpp-specialization-names.test.ts new file mode 100644 index 0000000000..c51642fd13 --- /dev/null +++ b/__tests__/cpp-specialization-names.test.ts @@ -0,0 +1,465 @@ +/** + * A C++ explicit specialization of a function or member function template is + * named after its template, without its template arguments: + * + * template <> bool WireFormatLite::ReadPrimitive(…) { … } + * template <> int WireFormatLite::get() const { … } + * template <> void Swap(Foo& a, Foo& b) { … } + * + * The first was named `TYPE_INT32>` and qualified + * `…::WireFormatLite::ReadPrimitive::TYPE_INT32>`: the qualified id was split + * at every `::`, the one inside its template arguments too. The others were + * named `get` and `Swap`. A call writes the bare name, so none + * linked to them. Extraction runs against the native kernel (when built) + * and the wasm extractor, which must agree. + */ +import { describe, it, expect, beforeAll, afterAll, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; +import { extractFromSource } from '../src/extraction'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; +import { tryKernelExtract, resetKernelForTests } from '../src/extraction/kernel'; +import type { ExtractionResult, Node } from '../src/types'; + +const KERNEL_PATH = path.join( + __dirname, + '..', + 'codegraph-kernel', + 'prebuilds', + `${process.platform}-${process.arch}`, + 'codegraph-kernel.node' +); +const kernelAvailable = fs.existsSync(KERNEL_PATH) || process.env.CODEGRAPH_KERNEL_EXPECT === '1'; + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +/** protobuf's wire_format_lite.h, in small, with a free function template and a class template. */ +const WIRE_H = lines( + 'namespace pb {', + 'class WireFormatLite {', + ' public:', + ' enum FieldType { TYPE_INT32 = 1, TYPE_INT64 = 2 };', + ' template ', + ' static bool ReadPrimitive(int* input, CType* value);', + ' template T get() const;', + ' bool operator<(const WireFormatLite& other) const;', + '};', + '', + 'template <>', + 'inline bool WireFormatLite::ReadPrimitive(int* input, int* value) {', + ' return true;', + '}', + 'template <>', + 'inline bool WireFormatLite::ReadPrimitive(int* input, long* value) {', + ' return false;', + '}', + 'template <>', + 'inline int WireFormatLite::get() const { return 1; }', + 'inline bool WireFormatLite::operator<(const WireFormatLite& other) const { return false; }', + '', + 'template void Swap(T& a, T& b) {}', + 'template <> void Swap(WireFormatLite& a, WireFormatLite& b) {}', + 'template T* Make() { return nullptr; }', + 'template <> WireFormatLite* Make() { return nullptr; }', + '', + 'template struct Box { T get() const; };', + 'template <> int Box::get() const { return 0; }', + 'template T Box::get() const { return T(); }', + '} // namespace pb', +); + +const ENV_KEYS = ['CODEGRAPH_KERNEL', 'CODEGRAPH_KERNEL_LANGS'] as const; + +/** ` ` of every function and method, sorted. */ +function callables(result: ExtractionResult): string[] { + return result.nodes + .filter((n) => n.kind === 'method' || n.kind === 'function') + .map((n) => `${n.kind} ${n.name} ${n.qualifiedName} ${n.startLine}`) + .sort(); +} + +describe('a C++ explicit specialization is named after its template', () => { + let savedEnv: Record = {}; + + beforeAll(async () => { + await initGrammars(); + await loadGrammarsForLanguages(['cpp']); + }); + + beforeEach(() => { + savedEnv = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]])); + resetKernelForTests(); + }); + + afterEach(() => { + for (const k of ENV_KEYS) { + if (savedEnv[k] === undefined) delete process.env[k]; + else process.env[k] = savedEnv[k]; + } + resetKernelForTests(); + }); + + function extract(backend: 'kernel' | 'wasm', source: string): ExtractionResult { + if (backend === 'wasm') { + process.env.CODEGRAPH_KERNEL = '0'; + return extractFromSource('src/wire.h', source, 'cpp'); + } + delete process.env.CODEGRAPH_KERNEL; + process.env.CODEGRAPH_KERNEL_LANGS = 'all'; + const result = tryKernelExtract('src/wire.h', source, 'cpp'); + expect(result, 'kernel extraction of src/wire.h').not.toBeNull(); + return result!; + } + + const backends = kernelAvailable ? (['kernel', 'wasm'] as const) : (['wasm'] as const); + + for (const crlf of [false, true]) { + const source = crlf ? WIRE_H.replace(/\n/g, '\r\n') : WIRE_H; + const label = crlf ? ' (CRLF)' : ''; + + it.each(backends)(`member and function specializations, beside what was already right: %s${label}`, (backend) => { + const result = extract(backend, source); + expect(callables(result)).toEqual([ + 'function Make pb::Make 25', + 'function Make pb::Make 26', + 'function Swap pb::Swap 23', + 'function Swap pb::Swap 24', + 'method get pb::Box::get 29', + 'method get pb::Box::get 30', + 'method get pb::WireFormatLite::get 20', + 'method operator< pb::WireFormatLite::operator< 21', + 'method ReadPrimitive pb::WireFormatLite::ReadPrimitive 12', + 'method ReadPrimitive pb::WireFormatLite::ReadPrimitive 16', + ].sort()); + // The class the receiver names holds them, as it holds any out-of-line definition. + const byId = new Map(result.nodes.map((n) => [n.id, n])); + const held = result.edges + .filter((e) => e.kind === 'contains' && byId.get(e.source)?.name === 'WireFormatLite') + .map((e) => `${byId.get(e.target)?.name} ${byId.get(e.target)?.startLine}`) + .sort(); + expect(held).toEqual(expect.arrayContaining(['ReadPrimitive 12', 'ReadPrimitive 16', 'get 20'])); + }); + } +}); + +/** + * A specialization now shares its template's name, so a call meets both + * among that name's candidates. The template arguments a call writes pick + * between them: the specialization written with the same ones, else the + * template itself or an ordinary overload, else nothing. A call that writes + * none reaches a specialization only when there is nothing else. + */ +describe('a C++ call reaches the specialization its template arguments name', () => { + let root: string; + let cg: CodeGraph; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-specialization-calls-')); + const write = (file: string, text: string): void => fs.writeFileSync(path.join(root, 'src', file), text); + fs.mkdirSync(path.join(root, 'src')); + write('wire.h', WIRE_H); + write('use.cc', lines( + '#include "src/wire.h"', + 'namespace pb {', + 'bool Read(int* in, int* v) { return WireFormatLite::ReadPrimitive(in, v); }', + 'bool ReadLong(int* in, long* v) { return WireFormatLite::ReadPrimitive(in, v); }', + 'bool ReadLongQualified(int* in, long* v) { return WireFormatLite::ReadPrimitive(in, v); }', + 'int Get(const WireFormatLite& w) { return w.get(); }', + 'void SwapWire(WireFormatLite& a, WireFormatLite& b) { Swap(a, b); }', + 'void SwapInts(int& x, int& y) { Swap(x, y); }', + 'void SwapDeduced(int& x, int& y) { Swap(x, y); }', + 'int* MakeInt() { return Make(); }', + 'WireFormatLite* MakeWire() { return Make(); }', + '} // namespace pb', + )); + // Inside the class: a call written with no receiver, its arguments + // unqualified; one in a template, whose arguments name its parameters. + write('reader.h', lines( + '#include "src/wire.h"', + 'namespace pb {', + 'class Reader : public WireFormatLite {', + ' public:', + ' bool ReadLong(int* in) { return ReadPrimitive(in, nullptr); }', + ' template bool ReadAny(int* in, C* v) { return ReadPrimitive(in, v); }', + '};', + '} // namespace pb', + )); + // rocksdb's cassandra serialize.h: the template is declared, its specializations defined. + write('serialize.h', lines( + 'namespace ns {', + 'template void Serialize(T val, char* dest);', + 'template <> inline void Serialize(char t, char* dest) {}', + 'template <> inline void Serialize(long t, char* dest) {}', + 'template <> inline void Serialize(const char* t, char* dest) {}', + 'inline void WriteLocal(char* dest) { ns::Serialize(1, dest); }', + '} // namespace ns', + )); + write('format.cc', lines( + '#include "src/serialize.h"', + 'namespace ns {', + 'void Write(char* dest) { ns::Serialize(1, dest); }', + 'void WriteShort(char* dest) { ns::Serialize(1, dest); }', + "void WriteDeduced(char* dest) { ns::Serialize('c', dest); }", + "void WriteBrackets(char* dest) { ns::Serialize<>('c', dest); }", + 'void WriteText(char* dest) { ns::Serialize("x", dest); }', + 'void WriteGlobal(char* dest) { ::ns::Serialize(1, dest); }', + '} // namespace ns', + )); + // Only a specialization, of a template declared without a body. + write('parse.h', lines( + 'namespace fmt {', + 'template T parse(const char* text);', + 'template <> int parse(const char* text) { return 0; }', + 'inline long parse_long(const char* t) { return parse(t); }', + '} // namespace fmt', + )); + // fmt's namespace, opened by a macro. + write('digits.h', lines( + '#define MY_BEGIN_NAMESPACE namespace mylib {', + '#define MY_END_NAMESPACE }', + 'MY_BEGIN_NAMESPACE', + 'template constexpr int digits() { return 1; }', + 'template <> constexpr int digits() { return 19; }', + 'MY_END_NAMESPACE', + )); + write('digits.cc', lines( + '#include "src/digits.h"', + 'int long_digits() { return mylib::digits(); }', + 'int char_digits() { return mylib::digits(); }', + )); + // gmock's converting constructor, which names its class template in its parameter. + write('action.h', lines( + 'namespace gm {', + 'template class ActionInterface {};', + 'template class Action {', + ' public:', + ' explicit Action(ActionInterface* impl) {}', + ' template explicit Action(const Action& action) {}', + '};', + 'template Action Convert(const Action& from) { return Action(from); }', + '} // namespace gm', + )); + // A specialization whose return type begins with its name. + write('get.h', lines( + 'namespace kv {', + 'struct GetResult {};', + 'template GetResult Get(int key) { return GetResult(); }', + 'template <> GetResult Get(int key) { return GetResult(); }', + 'inline GetResult GetInt() { return Get(1); }', + '} // namespace kv', + )); + // protobuf's arena.h, whose tests specialize `DefaultConstruct` to hook + // it — with a parameter the call's argument fits as well as the template's. + write('arena.h', lines( + 'namespace pb {', + 'class Arena;', + 'typedef Arena* ArenaPtr;', + 'class Arena {', + ' public:', + ' template static void* DefaultConstruct(ArenaPtr arena);', + ' template static T* Create(Arena* arena) { return static_cast(DefaultConstruct(arena)); }', + '};', + 'template void* Arena::DefaultConstruct(ArenaPtr arena) { return nullptr; }', + '} // namespace pb', + )); + // A specialization whose parameter is written through a typedef: the + // call's argument fits the template's parameter better. + write('registry.h', lines( + 'namespace pb {', + 'class Handle;', + 'typedef Handle* HandlePtr;', + 'class Registry {', + ' public:', + ' template void Add(Handle* h) {}', + '};', + 'template <> inline void Registry::Add(HandlePtr h) {}', + 'class Pool : public Registry {', + ' public:', + ' void AddInt(Handle* h) { Add(h); }', + '};', + 'inline void Register(Registry& r, Handle* h) { r.Add(h); }', + '} // namespace pb', + )); + // fmt's format.h: a call beside the specializations, nearer them than the template. + write('bits.h', lines( + 'namespace fmt {', + 'template constexpr int num_bits() { return 8; }', + 'template <> constexpr int num_bits() { return 64; }', + 'template <> constexpr int num_bits() { return 16; }', + 'inline int char_bits() { return num_bits(); }', + 'inline int long_bits() { return num_bits(); }', + 'template int width(T value) { return 1; }', + 'template <> int width(long value) { return 8; }', + 'inline int int_width(int v) { return width(v); }', + 'inline int named_long_bits() { const char* unit = "größe"; return num_bits(); }', + 'template T decode(T v) { return v; }', + 'template <> int decode(int v) { return -v; }', + 'inline int plain(int v) { return decode(v); }', + 'template int scale(T v);', + 'template <> int scale(long v) { return 2; } template int scale(T v) { return 1; }', + 'inline int int_scale() { return scale(1); }', + 'template <> long decode(long v) { return 2 * v; }', + 'inline long zigzag(long v) { return decode(v); }', + '} // namespace fmt', + )); + // A member template's specialization, and one called through another call. + write('codec.h', lines( + '#include "src/wire.h"', + 'namespace pb {', + 'class Codec {', + ' public:', + ' template void Put(T v) {}', + '};', + 'class IntCodec : public Codec {', + ' public:', + ' template void Put(T v);', + ' WireFormatLite& target() { return w_; }', + ' WireFormatLite w_;', + '};', + 'template <> inline void IntCodec::Put(int v) {}', + 'inline void PutInt(IntCodec& c) { c.Put(1); }', + 'inline void PutLong(IntCodec& c) { c.Put(1L); }', + 'inline long LongFromCodec(IntCodec& codec) { return codec.target().get(); }', + '} // namespace pb', + )); + // protobuf's generated_message_reflection.cc: a macro defines the specializations. + write('reflection.h', lines( + 'namespace pb {', + 'class Reflection {', + ' public:', + ' template const T& GetField(int index) const;', + '};', + 'template const T& Reflection::GetField(int index) const { static T t; return t; }', + '#define DEFINE_FIELD_GETTER(TYPE) \\', + ' template <> const TYPE& Reflection::GetField(int index) const { static TYPE t; return t; }', + 'DEFINE_FIELD_GETTER(int)', + 'DEFINE_FIELD_GETTER(long)', + 'inline long GetLong(const Reflection& r) { return r.GetField(0); }', + 'inline double GetDouble(const Reflection& r) { return r.GetField(0); }', + '} // namespace pb', + )); + write('arena_test.cc', lines( + '#include "src/arena.h"', + 'namespace pb {', + 'struct Hooked {};', + 'template <> void* Arena::DefaultConstruct(Arena* arena) { return nullptr; }', + 'void* MakeHooked(Arena* a) { return Arena::DefaultConstruct(a); }', + '} // namespace pb', + )); + // A base class's member template, specialized, beside a derived class's namesake. + write('repeated.h', lines( + 'namespace pb {', + 'class FieldBase {', + ' public:', + ' template void MergeFrom(const FieldBase& from) {}', + ' virtual void Visit(int value) {}', + '};', + 'template <> inline void FieldBase::MergeFrom(const FieldBase& from) {}', + 'class Field : public FieldBase {', + ' public:', + ' void MergeFrom(const Field& other) {}', + ' template void Visit(T value) {}', + '};', + 'template <> inline void Field::Visit(int value) {}', + '} // namespace pb', + )); + cg = await CodeGraph.init(root, { index: true }); + }, 120_000); + + afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); + }); + + /** `@` of what the function or method `name` (at `line`, if given) calls. */ + const calls = (name: string, line?: number): string[] => { + const caller = [...cg.getNodesByKind('function'), ...cg.getNodesByKind('method')] + .find((n: Node) => n.qualifiedName === name && (line === undefined || n.startLine === line))!; + expect(caller, name).toBeDefined(); + return cg.getCallees(caller.id).filter((r) => r.edge.kind === 'calls').map((r) => `${r.node.qualifiedName}@${r.node.startLine}`); + }; + + it('the specialization the call writes the arguments of', () => { + expect(calls('pb::Read')).toEqual(['pb::WireFormatLite::ReadPrimitive@12']); + expect(calls('pb::ReadLong')).toEqual(['pb::WireFormatLite::ReadPrimitive@16']); + expect(calls('pb::Get')).toEqual(['pb::WireFormatLite::get@20']); + expect(calls('pb::SwapWire')).toEqual(['pb::Swap@24']); + expect(calls('pb::MakeWire')).toEqual(['pb::Make@26']); + expect(calls('ns::Write')).toEqual(['ns::Serialize@4']); + expect(calls('pb::MakeHooked')).toEqual(['pb::Arena::DefaultConstruct@4']); + expect(calls('fmt::long_bits')).toEqual(['fmt::num_bits@3']); + expect(calls('pb::Register')).toEqual(['pb::Registry::Add@8']); + expect(calls('pb::Pool::AddInt')).toEqual(['pb::Registry::Add@8']); + expect(calls('pb::PutInt')).toEqual(['pb::IntCodec::Put@13']); + // The call's column counts the bytes of the wider characters before it. + expect(calls('fmt::named_long_bits')).toEqual(['fmt::num_bits@3']); + // An argument qualified further than the specialization writes it, or spaced otherwise. + expect(calls('pb::ReadLongQualified')).toEqual(['pb::WireFormatLite::ReadPrimitive@16']); + expect(calls('ns::WriteText')).toEqual(['ns::Serialize@5']); + // From the global scope, and beside the specializations. + expect(calls('ns::WriteGlobal')).toEqual(['ns::Serialize@4']); + expect(calls('ns::WriteLocal')).toEqual(['ns::Serialize@4']); + // In a namespace a macro opens. + expect(calls('long_digits')).toEqual(['digits@5']); + }); + + it('the specializations a macro defines', () => { + expect(calls('pb::GetLong')).toEqual(['pb::Reflection::GetField@10']); + expect(calls('pb::GetDouble')).toEqual(['pb::Reflection::GetField@6']); + }); + + it('a specialization when nothing else is there', () => { + expect(calls('ns::WriteDeduced')).toEqual(['ns::Serialize@3']); + expect(calls('ns::WriteBrackets')).toEqual(['ns::Serialize@3']); + }); + + it('the template itself for arguments no specialization writes', () => { + expect(calls('pb::SwapInts')).toEqual(['pb::Swap@23']); + expect(calls('pb::MakeInt')).toEqual(['pb::Make@25']); + expect(calls('fmt::char_bits')).toEqual(['fmt::num_bits@2']); + // protobuf's `Arena::Create` calls `DefaultConstruct`, not the tests' hook. + expect(calls('pb::Arena::Create')).toEqual(['pb::Arena::DefaultConstruct@9']); + // `decode` leaves `zigzag` at its default; `decode` is `decode`. + expect(calls('fmt::plain')).toEqual(['fmt::decode@11']); + expect(calls('fmt::zigzag')).toEqual(['fmt::decode@11']); + // The template defined after its specialization on one line. + expect(calls('fmt::int_scale')).toEqual(['fmt::scale@15']); + expect(calls('char_digits')).toEqual(['digits@4']); + expect(calls('kv::GetInt')).toEqual(['kv::Get@3']); + }); + + it('a constructor is no specialization', () => { + expect(calls('gm::Convert')).toEqual(['gm::Action::Action@6']); + }); + + it('the template itself for a call that writes no arguments', () => { + expect(calls('pb::SwapDeduced')).toEqual(['pb::Swap@23']); + expect(calls('fmt::int_width')).toEqual(['fmt::width@7']); + }); + + it('a call inside the class, its arguments written unqualified', () => { + expect(calls('pb::Reader::ReadLong')).toEqual(['pb::WireFormatLite::ReadPrimitive@16']); + }); + + it('nothing when only specializations are there and none is the one called', () => { + expect(calls('pb::Reader::ReadAny')).toEqual([]); + expect(calls('ns::WriteShort')).toEqual([]); + // `IntCodec::Put` hides `Codec::Put`. + expect(calls('pb::PutLong')).toEqual([]); + expect(calls('pb::LongFromCodec')).toEqual(['pb::IntCodec::target@10']); + expect(calls('fmt::parse_long')).toEqual([]); + }); + + it("a derived class's namesake overrides the template, not its specialization", () => { + const overriders = (name: string, line: number): string[] => { + const method = cg.getNodesByKind('method').find((n: Node) => n.qualifiedName === name && n.startLine === line)!; + expect(method, `${name}@${line}`).toBeDefined(); + return cg.getCallers(method.id).filter((r) => r.edge.kind === 'calls').map((r) => `${r.node.qualifiedName}@${r.node.startLine}`); + }; + expect(overriders('pb::Field::MergeFrom', 10)).toEqual(['pb::FieldBase::MergeFrom@4']); + expect(overriders('pb::Field::Visit', 11)).toEqual(['pb::FieldBase::Visit@5']); + expect(overriders('pb::Field::Visit', 13)).toEqual([]); + }); +}); diff --git a/__tests__/cpp-template-member-types.test.ts b/__tests__/cpp-template-member-types.test.ts new file mode 100644 index 0000000000..f4d540dcaa --- /dev/null +++ b/__tests__/cpp-template-member-types.test.ts @@ -0,0 +1,246 @@ +/** + * A C++ receiver declared as a member type of a class template, like + * google/leveldb's and facebook/rocksdb's `SkipList::Iterator iter(&list);`. The declaration was read from its + * last name alone, `Iterator`, which C++ lookup from the calling function + * then found in the enclosing namespace: leveldb's skip-list tests linked + * `iter.Valid()` to `leveldb::Iterator::Valid`, and rocksdb's to + * `IteratorBase::Valid`, instead of the skip list's own iterator. The type is + * now read whole, and the member found in the class template it is written in. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +async function indexed(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-template-member-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +function callable(cg: CodeGraph, qualifiedName: string): Node { + const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName); + if (!node) throw new Error(`no function or method ${qualifiedName}`); + return node; +} + +/** `calls` callees of a function or method, as `qualifiedName`, without constructors. */ +function calls(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(callable(cg, caller).id) + .filter((r) => r.edge.kind === 'calls' && !/::(\w+)::\1$/.test(r.node.qualifiedName)) + .map((r) => r.node.qualifiedName) + .sort(); +} + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +const SKIPLIST = { + 'include/iterator.h': lines( + 'namespace leveldb {', + 'class Iterator {', + ' public:', + ' virtual ~Iterator() {}', + ' virtual bool Valid() const { return false; }', + ' virtual void SeekToFirst() {}', + '};', + '} // namespace leveldb', + ), + 'db/skiplist.h': lines( + '#include "include/iterator.h"', + 'namespace leveldb {', + 'template ', + 'class SkipList {', + ' public:', + ' class Iterator {', + ' public:', + ' explicit Iterator(const SkipList* list) : list_(list) {}', + ' bool Valid() const { return true; }', + ' void SeekToFirst() {}', + ' private:', + ' const SkipList* list_;', + ' };', + '};', + '} // namespace leveldb', + ), +}; + +describe('a C++ receiver declared as a member type of a class template', () => { + it('reproduction: `SkipList::Iterator iter` is the skip list\'s iterator', async () => { + const cg = await indexed({ + ...SKIPLIST, + 'db/skiplist_test.cc': lines( + '#include "db/skiplist.h"', + 'namespace leveldb {', + 'typedef unsigned long Key;', + 'struct Comparator {};', + 'void Empty() {', + ' SkipList list;', + ' SkipList::Iterator iter(&list);', + ' if (!iter.Valid()) iter.SeekToFirst();', + '}', + 'void Split() {', + ' SkipList list;', + ' SkipList::Iterator', + ' iter(&list);', + ' iter.Valid();', + '}', + '} // namespace leveldb', + ), + }); + try { + expect(calls(cg, 'leveldb::Empty')).toEqual(['leveldb::SkipList::Iterator::SeekToFirst', 'leveldb::SkipList::Iterator::Valid']); + expect(calls(cg, 'leveldb::Split')).toEqual(['leveldb::SkipList::Iterator::Valid']); + } finally { + cg.close(); + } + }); + + it('written with `typename` in a template, and through a pointer', async () => { + const cg = await indexed({ + ...SKIPLIST, + 'db/concurrent_test.cc': lines( + '#include "db/skiplist.h"', + 'namespace leveldb {', + 'template ', + 'class ConcurrentTest {', + ' public:', + ' void ReadStep() {', + ' typename SkipList::Iterator iter(&list_);', + ' iter.SeekToFirst();', + ' }', + ' bool Check(typename SkipList::Iterator* it) {', + ' return it->Valid();', + ' }', + ' private:', + ' SkipList list_;', + '};', + '} // namespace leveldb', + ), + }); + try { + expect(calls(cg, 'leveldb::ConcurrentTest::ReadStep')).toEqual(['leveldb::SkipList::Iterator::SeekToFirst']); + expect(calls(cg, 'leveldb::ConcurrentTest::Check')).toEqual(['leveldb::SkipList::Iterator::Valid']); + } finally { + cg.close(); + } + }); + + it('a standard container\'s iterator is the library\'s, not a project class named `iterator`', async () => { + const cg = await indexed({ + 'util/ring.h': lines( + 'class Ring {', + ' public:', + ' class iterator {', + ' public:', + ' int base() const { return 0; }', + ' };', + '};', + ), + 'src/touch.cc': lines( + '#include ', + '#include "util/ring.h"', + 'void Touch(std::vector::iterator it) {', + ' it.base();', + '}', + ), + }); + try { + expect(calls(cg, 'Touch')).toEqual([]); + } finally { + cg.close(); + } + }); + + it('a library template\'s member that names another type may be one of the project\'s', async () => { + const cg = await indexed({ + 'include/slice.h': lines( + 'class Slice {', + ' public:', + ' int size() const { return 0; }', + '};', + ), + 'src/keys.cc': lines( + '#include ', + '#include "include/slice.h"', + 'int First(const std::vector& keys) {', + ' std::vector::value_type key = keys[0];', + ' return key.size();', + '}', + ), + }); + try { + expect(calls(cg, 'First')).toEqual(['Slice::size']); + } finally { + cg.close(); + } + }); + + it('a member alias of a class template is the class it names', async () => { + const cg = await indexed({ + 'cache/clock_cache.h': lines( + 'class Shard {', + ' public:', + ' int GetTable() const { return 0; }', + '};', + 'namespace cache {', + 'template ', + 'class ClockCacheShard {', + ' public:', + ' int GetTable() const { return 1; }', + '};', + 'template ', + 'class BaseHyperClockCache {', + ' public:', + ' using Shard = ClockCacheShard
;', + ' int Report(const BaseHyperClockCache
::Shard* shard) const {', + ' return shard->GetTable();', + ' }', + '};', + '} // namespace cache', + ), + }); + try { + expect(calls(cg, 'cache::BaseHyperClockCache::Report')).toEqual(['cache::ClockCacheShard::GetTable']); + } finally { + cg.close(); + } + }); + it('a library map\'s `value_type` is a pair of its key and value, not its key', async () => { + const cg = await indexed({ + 'src/entries.cc': lines( + '#include ', + '#include ', + 'class Key { public: void swap(Key& o) {} int size() const { return 0; } };', + 'class Value {};', + 'void Swap(std::map& m, std::map::value_type& other) {', + ' std::map::value_type entry = *m.begin();', + ' entry.swap(other);', + '}', + 'int First(const std::vector& keys) {', + ' std::vector::value_type key = keys[0];', + ' return key.size();', + '}', + ), + }); + try { + expect(calls(cg, 'Swap')).toEqual([]); + // A sequence's `value_type` is still its element. + expect(calls(cg, 'First')).toEqual(['Key::size']); + } finally { + cg.close(); + } + }); +}); diff --git a/codegraph-kernel/src/ccpp/mod.rs b/codegraph-kernel/src/ccpp/mod.rs index 6db8c26309..375922828a 100644 --- a/codegraph-kernel/src/ccpp/mod.rs +++ b/codegraph-kernel/src/ccpp/mod.rs @@ -1234,6 +1234,16 @@ fn is_statement_keyword(text: &str) -> bool { matches!(text, "if" | "else" | "while" | "for" | "do" | "switch" | "return" | "catch") } +/// cppQualifiedIdName (languages/c-cpp.ts): the name a `qualified_identifier` +/// ends in, past its scopes. None when the tree lacks one. +fn cpp_qualified_id_name(qid: Node) -> Option { + let mut name = qid; + while name.kind() == "qualified_identifier" { + name = name.child_by_field_name("name")?; + } + Some(name) +} + /// findDeclaratorQualifiedId (languages/c-cpp.ts:13): BFS for the declarator's /// `qualified_identifier`, skipping parameter_list + trailing_return_type so a /// qualified PARAMETER type can't be mistaken for the method name. diff --git a/codegraph-kernel/src/ccpp/names.rs b/codegraph-kernel/src/ccpp/names.rs index 11395108d1..3999d444b1 100644 --- a/codegraph-kernel/src/ccpp/names.rs +++ b/codegraph-kernel/src/ccpp/names.rs @@ -88,6 +88,9 @@ impl<'t> Walker<'t> { if let Some(mut parts) = self.cpp_written_class_parts(node) { return parts.pop().unwrap_or_default(); } + if let Some(name) = self.cpp_specialized_function_name(node) { + return name; + } } if let Some(name_node) = node.child_by_field_name("declarator") { let mut resolved = name_node; @@ -127,18 +130,52 @@ impl<'t> Walker<'t> { "".to_string() } - /// extractCppQualifiedMethodName (languages/c-cpp.ts:75). + /// extractCppQualifiedMethodName (languages/c-cpp.ts): the name the + /// declarator's qualified id ends in, an explicit specialization's without + /// its template arguments (cppDefinedName). pub(super) fn extract_cpp_qualified_method_name(&self, node: Node) -> Option { if let Some(n) = self.recover_cpp_macro_defined_name(node) { return Some(n); } let declarator = node.child_by_field_name("declarator")?; let qid = find_declarator_qualified_id(declarator)?; + if let Some(name) = cpp_qualified_id_name(qid) { + return Some(self.cpp_defined_name(name)); + } let text = self.text(qid).trim(); let parts: Vec<&str> = text.split("::").filter(|p| !p.is_empty()).collect(); parts.last().map(|s| s.to_string()) } + /// cppDefinedName (languages/c-cpp.ts): a definition's name node as + /// named, a `template_function`'s without its template arguments + /// (`ReadPrimitive` is `ReadPrimitive`). + fn cpp_defined_name(&self, name: Node) -> String { + let own = if name.kind() == "template_function" { name.child_by_field_name("name") } else { None }; + self.text(own.unwrap_or(name)).trim().to_string() + } + + /// cppSpecializedFunctionName (languages/c-cpp.ts): the name of a + /// function an explicit specialization defines, without its template + /// arguments (`Swap` for `template <> void Swap(Foo& a, Foo& b)`). + fn cpp_specialized_function_name(&self, node: Node) -> Option { + if node.kind() != "function_definition" { + return None; + } + let mut declarator = node.child_by_field_name("declarator")?; + while matches!(declarator.kind(), "pointer_declarator" | "reference_declarator") { + declarator = declarator.child_by_field_name("declarator").or_else(|| declarator.named_child(0))?; + } + if declarator.kind() != "function_declarator" { + return None; + } + let inner = declarator.child_by_field_name("declarator")?; + if inner.kind() != "template_function" { + return None; + } + Some(self.cpp_defined_name(inner)) + } + /// recoverSingleArgMacroDefinedName: only a local macro definition proves /// that its argument is the function name, in either parser shape (#1373). pub(super) fn recover_single_arg_macro_defined_name(&self, node: Node) -> Option { @@ -328,16 +365,32 @@ impl<'t> Walker<'t> { Some(name.to_string()) } - /// extractCppReceiverType (languages/c-cpp.ts:86). + /// extractCppReceiverType (languages/c-cpp.ts): the scopes are what the + /// qualified id writes before its name, so a specialization's arguments + /// (`ReadPrimitive`) are no scope. pub(super) fn receiver_type_of(&self, node: Node) -> Option { let declarator = node.child_by_field_name("declarator")?; let qid = find_declarator_qualified_id(declarator)?; - let text = self.text(qid).trim(); - let parts: Vec<&str> = text.split("::").filter(|p| !p.is_empty()).collect(); - if parts.len() <= 1 { + let scopes = match cpp_qualified_id_name(qid) { + Some(name) => { + let written = self.src[qid.start_byte()..name.start_byte()].trim_end(); + // A `::` the parser supplied in recovery writes nothing: Metal's + // `vertex VertexOut vertexShader(…)` reads as `VertexOut::vertexShader`. + if !written.ends_with("::") { + return None; + } + written.to_string() + } + None => { + let parts: Vec<&str> = self.text(qid).trim().split("::").filter(|p| !p.is_empty()).collect(); + parts[..parts.len().saturating_sub(1)].join("::") + } + }; + let parts: Vec<&str> = scopes.split("::").filter(|p| !p.is_empty()).collect(); + if parts.is_empty() { return None; } - let receiver = strip_cpp_template_args(&parts[..parts.len() - 1].join("::")); + let receiver = strip_cpp_template_args(&parts.join("::")); if receiver.is_empty() { None } else { diff --git a/codegraph-kernel/src/resolve/bound.rs b/codegraph-kernel/src/resolve/bound.rs index 5f374acf5b..c1bbc24f96 100644 --- a/codegraph-kernel/src/resolve/bound.rs +++ b/codegraph-kernel/src/resolve/bound.rs @@ -261,6 +261,14 @@ if site.language == "dart" && site.reference_kind == "calls" { if let Some(m) = member { return Ok(Some(bound_member_cand(m))); } + // Specializations of the class's own member template, none the call's: what C++ finds is + // the template, which has no body here, not a base's namesake (`IntCodec::Put` hides + // `Codec::Put`). + if site.language == "cpp" && site.reference_kind == "calls" { + let family = self.nodes_by_qualified_name(&format!("{}::{}", owner.qualified_name, method))?; + let family: Vec<_> = family.iter().filter(|n| n.kind == "method" && n.language == "cpp").cloned().collect(); + if !family.is_empty() && self.cpp_specialization_pool(family, site).is_empty() { return Ok(None); } + } // A db that may lack supertype edges (a pool worker's snapshot taken // mid-prerequisite phase — unreachable since the snapshot refreshes // before the first calls batch) gets no walk: a miss, never a guess. @@ -726,6 +734,9 @@ if site.language == "dart" && site.reference_kind == "calls" { }) .cloned() .collect(); + // Of a member template's explicit specializations, the one the call's template arguments + // name (cpp_specialization_pool). + let members = self.cpp_specialization_pool(members, site); if members.len() > 1 && site.language == "cpp" && site.reference_kind == "calls" && !site.reference_name.contains("().") { // A C++ call on a typed receiver takes the overload its arguments fit, not the first // one declared (a chained call `inner(…).name(…)` keeps the first: the arguments at @@ -1293,6 +1304,10 @@ Ok(self.dart_member_of(type_name, method, r, false)? } return Ok(None); } + // Of a member template's explicit specializations, the one the call's template arguments + // name; none of them when it names another (cpp_specialization_pool). + let matches = self.cpp_specialization_pool(matches, r); + if matches.is_empty() { return Ok(None); } let matches = self.cpp_visible_first(matches, r)?; // A C++ call on a typed receiver takes the overload its arguments fit, not the first one // declared (a chained call `inner(…).name(…)` keeps the first: the arguments at its diff --git a/codegraph-kernel/src/resolve/c_feature_variants.rs b/codegraph-kernel/src/resolve/c_feature_variants.rs new file mode 100644 index 0000000000..f104bd278d --- /dev/null +++ b/codegraph-kernel/src/resolve/c_feature_variants.rs @@ -0,0 +1,327 @@ +//! Is a C function defined only for a build that turns on a flag nothing in the project turns on +//! (c-feature-variants.ts, upstream #2615)? lua-cjson's fpconv.h defines `static inline void +//! fpconv_init()` under `#ifdef USE_INTERNAL_FPCONV` and otherwise declares it `extern`, for +//! fpconv.c to define. Nothing in redis's tree sets USE_INTERNAL_FPCONV, so its build compiles +//! fpconv.c's. +//! +//! The overload choice asks this only when choosing among definitions of one name, where a call +//! has to reach one of them (cpp_function_overload): of a flag nothing sets, the build to follow +//! is the one that leaves it off. Whether a macro is visible at a call (cpp.rs) keeps such a flag +//! unknown instead, since there an unknown flag must not suppress anything. +//! +//! A branch is off only when flags decide its test: `#ifdef X`, `#ifndef X`, `#if defined(X)`, +//! `#if X`, `#if 0`, with `!`, `&&`, `||` and parentheses, and the `#elif` and `#else` after +//! them. A flag is set nowhere when the definition's own file doesn't `#define` it, no build file +//! in the directories from the project root down to the definition or the call names it (a +//! Makefile's `-DX`, CMake's `option(X …)`, configure's `AC_DEFINE(X)`) and no C or C++ file of +//! their components does, a component being the tree under the nearest directory with a build +//! file, redis's `deps/lua/src` for fpconv.h. A macro of the compiler or the platform, a reserved +//! name like `_WIN32` or `__linux__`, or `WIN32`, `linux`, `unix`, is unknown, and so is any +//! other test (`#if LEVEL > 1`). + +use super::cpp::{and, directive_lines, not, or, Truth}; +use super::*; + +#[derive(Clone, Copy, PartialEq)] +enum Op { If, Ifdef, Ifndef, Elifdef, Elifndef, Elif, Else, Endif } + +struct Directive { op: Op, expression: String, line: i64 } + +struct FileDirectives { conditionals: Vec, defines: HashSet } + +#[derive(Default)] +pub(super) struct VariantCache { + files: HashMap>, + /// Directory → the text of the build files directly in it. + build_text: HashMap>, + /// Directory → the nearest directory at or above it with a build file (`` for the project root). + components: HashMap, + /// The project's C and C++ files, once. + c_files: Option>>, + /// (flag, component) → whether a C or C++ file of the component `#define`s it. + defines: HashMap<(String, String), bool>, + answers: HashMap, +} + +const PLATFORM_FLAGS: [&str; 4] = ["WIN32", "WIN64", "linux", "unix"]; +const BUILD_FILES: [&str; 15] = [ + "Makefile", "makefile", "GNUmakefile", "Makefile.am", "Makefile.in", "CMakeLists.txt", "meson.build", + "meson_options.txt", "configure.ac", "configure.in", "config.h.in", "BUILD", "BUILD.bazel", "SConstruct", "SConscript", +]; + +fn dir_of(file: &str) -> &str { + file.rfind('/').map_or("", |at| &file[..at]) +} + +/// `a/b/c.h` → `a/b`, `a`, `` (the project root). +fn ancestors(file: &str) -> Vec { + let mut dirs = Vec::new(); + let mut dir = dir_of(file); + loop { + dirs.push(dir.to_string()); + if dir.is_empty() { return dirs; } + dir = dir_of(dir); + } +} + +impl KernelResolver { + /// Is `function`'s definition compiled only by a build that sets a flag nothing in the project + /// sets, for the call `r`? A branch around both the definition and the call, in the call's own + /// file, decides nothing. + pub(super) fn is_c_feature_variant(&mut self, function: &KNode, r: &ResolveRefIn) -> bool { + let same_file = function.file_path == r.file_path; + let key = format!("v\0{}\0{}", function.id, if same_file { r.line.to_string() } else { dir_of(&r.file_path).to_string() }); + if let Some(&hit) = self.c_variants.answers.get(&key) { return hit; } + let file = self.variant_directives(&function.file_path); + let mut groups = enclosing_tests(&file.conditionals, function.start_line); + if same_file { + let around = enclosing_tests(&file.conditionals, r.line); + let shared = groups.iter().zip(&around).take_while(|(a, b)| a[0] == b[0] && a.len() == b.len()).count(); + groups.drain(..shared); + } + let off = !groups.is_empty() && self.variant_off(&groups, &file, &[function.file_path.as_str(), r.file_path.as_str()]); + self.c_variants.answers.insert(key, off); + off + } + + /// Is line `line` of `path` compiled only by a build that sets a flag nothing in the project sets? + pub(super) fn is_off_by_default(&mut self, path: &str, line: i64) -> bool { + let key = format!("o\0{path}\0{line}"); + if let Some(&hit) = self.c_variants.answers.get(&key) { return hit; } + let file = self.variant_directives(path); + let groups = enclosing_tests(&file.conditionals, line); + let off = !groups.is_empty() && self.variant_off(&groups, &file, &[path]); + self.c_variants.answers.insert(key, off); + off + } + + /// Is a branch of `groups` off when no flag `scope`'s files see is set? + fn variant_off(&mut self, groups: &[Vec], file: &FileDirectives, scope: &[&str]) -> bool { + let mut dirs: Vec = Vec::new(); + for f in scope { + for dir in ancestors(f) { if !dirs.contains(&dir) { dirs.push(dir); } } + } + let mut components: Vec = Vec::new(); + for f in scope { + let component = self.variant_component(dir_of(f)); + if !components.contains(&component) { components.push(component); } + } + let mut flag = |name: &str| -> Truth { + if name.starts_with('_') || PLATFORM_FLAGS.contains(&name) || file.defines.contains(name) { return None; } + // A word of its own, or a compiler flag's `-DX` / `/DX`. + let Ok(named) = shared_regex(&format!(r"(?m)(?:^|[^\w]|[-/]D){}\b", regex::escape(name))) else { return None }; + if dirs.iter().any(|dir| named.is_match(&self.variant_build_text(dir))) { return None; } + if components.iter().any(|component| self.variant_defined_in(name, component)) { None } else { Some(false) } + }; + groups.iter().any(|tests| { + let tests: Vec<&Directive> = tests.iter().map(|&i| &file.conditionals[i]).collect(); + branch_truth(&tests, &mut flag) == Some(false) + }) + } + + fn variant_directives(&mut self, file: &str) -> Rc { + if let Some(hit) = self.c_variants.files.get(file) { return hit.clone(); } + let text = self.read_file(file).map(|s| s.text().to_string()).unwrap_or_default(); + let lines = directive_lines(&text); + let mut conditionals = Vec::new(); + let mut defines = HashSet::new(); + for (i, line) in lines.iter().enumerate() { + if let Some(define) = re!(r"^\s*#\s*define\s+(\w+)").captures(line) { + defines.insert(define[1].to_string()); + continue; + } + let Some(m) = re!(r"^\s*#\s*(ifdef|ifndef|if|elifdef|elifndef|elif|else|endif)\b(.*)$").captures(line) else { continue }; + let op = match &m[1] { + "ifdef" => Op::Ifdef, "ifndef" => Op::Ifndef, "if" => Op::If, "elifdef" => Op::Elifdef, + "elifndef" => Op::Elifndef, "elif" => Op::Elif, "else" => Op::Else, _ => Op::Endif, + }; + // A test continued with a backslash goes on over the next lines. + let mut expression = m[2].to_string(); + let mut j = i; + while re!(r"\\\s*$").is_match(&lines[j]) && j + 1 < lines.len() { + expression = format!("{} {}", re!(r"\\\s*$").replace(&expression, ""), lines[j + 1]); + j += 1; + } + conditionals.push(Directive { op, expression, line: i as i64 + 1 }); + } + let directives = Rc::new(FileDirectives { conditionals, defines }); + self.c_variants.files.insert(file.to_string(), directives.clone()); + directives + } + + fn variant_build_text(&mut self, dir: &str) -> Rc { + if let Some(hit) = self.c_variants.build_text.get(dir) { return hit.clone(); } + let text: Vec = BUILD_FILES.iter().map(|name| { + let path = if dir.is_empty() { name.to_string() } else { format!("{dir}/{name}") }; + self.read_file_uncached(&path).map(|s| s.text().to_string()).unwrap_or_default() + }).collect(); + let text: Rc = text.join("\n").into(); + self.c_variants.build_text.insert(dir.to_string(), text.clone()); + text + } + + /// The nearest directory at or above `dir` with a build file, else the project root. + fn variant_component(&mut self, dir: &str) -> String { + if let Some(hit) = self.c_variants.components.get(dir) { return hit.clone(); } + let component = if dir.is_empty() || !self.variant_build_text(dir).trim().is_empty() { dir.to_string() } else { self.variant_component(dir_of(dir)) }; + self.c_variants.components.insert(dir.to_string(), component.clone()); + component + } + + /// Does a C or C++ file under `component` `#define` `name`? Read only from files whose bytes hold the name. + fn variant_defined_in(&mut self, name: &str, component: &str) -> bool { + let key = (name.to_string(), component.to_string()); + if let Some(&hit) = self.c_variants.defines.get(&key) { return hit; } + let files = match self.c_variants.c_files.clone() { + Some(files) => files, + None => { + let all: Vec = match self.sorted_files() { + Some(files) => files.to_vec(), + None => self.table().map(|t| t.files.iter().cloned().collect()).unwrap_or_default(), + }; + let files = Rc::new(all.into_iter().filter(|f| re!(r"(?i)\.(c|h|cc|cpp|cxx|c\+\+|hpp|hh|hxx|h\+\+|inl|ipp|tcc|inc|m|mm)$").is_match(f)).collect::>()); + self.c_variants.c_files = Some(files.clone()); + files + } + }; + let prefix = if component.is_empty() { String::new() } else { format!("{component}/") }; + let defined = shared_regex(&format!(r"(?m)^[ \t]*#[ \t]*define[ \t]+{}\b", regex::escape(name))).is_ok_and(|define| { + files.iter().any(|file| file.starts_with(&prefix) + && self.read_file_uncached(file).is_some_and(|s| s.text().contains(name) && define.is_match(s.text()))) + }); + self.c_variants.defines.insert(key, defined); + defined + } +} + +/// The branches around `line`, innermost last: for each, the tests of its group's branches up to +/// and including the one `line` is in (`#else` has no test), as indexes into `conditionals`. +fn enclosing_tests(conditionals: &[Directive], line: i64) -> Vec> { + let mut groups: Vec> = Vec::new(); + for (i, d) in conditionals.iter().enumerate() { + if d.line >= line { break; } + match d.op { + Op::If | Op::Ifdef | Op::Ifndef => groups.push(vec![i]), + Op::Endif => { groups.pop(); } + _ => { if let Some(group) = groups.last_mut() { group.push(i); } } + } + } + groups +} + +/// Whether the last branch of `tests` is taken: no earlier test held, and its own does. +fn branch_truth(tests: &[&Directive], flag: &mut dyn FnMut(&str) -> Truth) -> Truth { + let mut earlier: Truth = Some(false); + for d in &tests[..tests.len() - 1] { + earlier = or(earlier, test_truth(d, flag)); + if earlier == Some(true) { return Some(false); } + } + and(not(earlier), test_truth(tests[tests.len() - 1], flag)) +} + +fn test_truth(d: &Directive, mut flag: &mut dyn FnMut(&str) -> Truth) -> Truth { + let trimmed = d.expression.trim(); + let name = (!trimmed.is_empty() && trimmed.bytes().all(|b| b.is_ascii_alphanumeric() || b == b'_')).then_some(trimmed); + match d.op { + Op::Ifdef | Op::Elifdef => name.and_then(&mut flag), + Op::Ifndef | Op::Elifndef => not(name.and_then(&mut flag)), + Op::Else => Some(true), + _ => evaluate(&d.expression, flag), + } +} + +/// A `#if` expression of `defined`, flags, integers, `!`, `&&`, `||` and parentheses; anything +/// else is unknown. A flag that is set nowhere reads as `0`, as the preprocessor reads a name no +/// `#define` gave a value. +fn evaluate(expression: &str, flag: &mut dyn FnMut(&str) -> Truth) -> Truth { + let tokens: Vec<&str> = re!(r"(?i)0x[\da-f]+\w*|\d\w*|[A-Za-z_]\w*|&&|\|\||\S").find_iter(expression).map(|m| m.as_str()).collect(); + // Check the shape first, so an unsupported test reads no flag. + let mut i = 0; + if tokens.is_empty() || !shape(&tokens, &mut i) || i != tokens.len() { return None; } + let mut i = 0; + disjunction(&tokens, &mut i, flag) +} + +fn is_identifier(t: &str) -> bool { + re!(r"^[A-Za-z_]\w*$").is_match(t) +} + +fn shape(tokens: &[&str], i: &mut usize) -> bool { + if !shape_term(tokens, i) { return false; } + while matches!(tokens.get(*i), Some(&"&&") | Some(&"||")) { + *i += 1; + if !shape_term(tokens, i) { return false; } + } + true +} + +fn shape_term(tokens: &[&str], i: &mut usize) -> bool { + let t = tokens.get(*i).copied(); + *i += 1; + match t { + Some("!") => shape_term(tokens, i), + Some("(") => { + if !shape(tokens, i) { return false; } + let close = tokens.get(*i) == Some(&")"); + *i += 1; + close + } + Some("defined") => { + let paren = tokens.get(*i) == Some(&"("); + if paren { *i += 1; } + let named = tokens.get(*i).is_some_and(|t| is_identifier(t)); + *i += 1; + named && (!paren || { let close = tokens.get(*i) == Some(&")"); *i += 1; close }) + } + Some(t) => re!(r"(?i)^(0x[\da-f]+|\d+)[ul]*$").is_match(t) || is_identifier(t), + None => false, + } +} + +fn disjunction(tokens: &[&str], i: &mut usize, flag: &mut dyn FnMut(&str) -> Truth) -> Truth { + let mut value = conjunction(tokens, i, flag); + while tokens.get(*i) == Some(&"||") { + *i += 1; + value = or(value, conjunction(tokens, i, flag)); + } + value +} + +fn conjunction(tokens: &[&str], i: &mut usize, flag: &mut dyn FnMut(&str) -> Truth) -> Truth { + let mut value = term(tokens, i, flag); + while tokens.get(*i) == Some(&"&&") { + *i += 1; + value = and(value, term(tokens, i, flag)); + } + value +} + +fn term(tokens: &[&str], i: &mut usize, flag: &mut dyn FnMut(&str) -> Truth) -> Truth { + let t = tokens[*i]; + *i += 1; + match t { + "!" => not(term(tokens, i, flag)), + "(" => { + let value = disjunction(tokens, i, flag); + *i += 1; + value + } + "defined" => { + if tokens.get(*i) == Some(&"(") { + let value = flag(tokens[*i + 1]); + *i += 3; + value + } else { + let value = flag(tokens[*i]); + *i += 1; + value + } + } + _ if t.as_bytes()[0].is_ascii_digit() => { + // (Nonzero when any digit is: `0x10`, `1UL`.) + let digits = t.strip_prefix("0x").or_else(|| t.strip_prefix("0X")).unwrap_or(t).trim_end_matches(['u', 'l', 'U', 'L']); + Some(digits.bytes().any(|b| b != b'0')) + } + _ => if flag(t) == Some(false) { Some(false) } else { None }, + } +} diff --git a/codegraph-kernel/src/resolve/cpp.rs b/codegraph-kernel/src/resolve/cpp.rs index 481cfa0e19..61e419a5bf 100644 --- a/codegraph-kernel/src/resolve/cpp.rs +++ b/codegraph-kernel/src/resolve/cpp.rs @@ -14,9 +14,9 @@ use super::*; /// Three-valued: `None` = depends on an unknown build flag. -type Truth = Option; +pub(super) type Truth = Option; -fn and(a: Truth, b: Truth) -> Truth { +pub(super) fn and(a: Truth, b: Truth) -> Truth { match (a, b) { (Some(false), _) | (_, Some(false)) => Some(false), (Some(true), Some(true)) => Some(true), @@ -24,7 +24,7 @@ fn and(a: Truth, b: Truth) -> Truth { } } -fn or(a: Truth, b: Truth) -> Truth { +pub(super) fn or(a: Truth, b: Truth) -> Truth { match (a, b) { (Some(true), _) | (_, Some(true)) => Some(true), (Some(false), Some(false)) => Some(false), @@ -32,7 +32,7 @@ fn or(a: Truth, b: Truth) -> Truth { } } -fn not(a: Truth) -> Truth { +pub(super) fn not(a: Truth) -> Truth { a.map(|v| !v) } @@ -293,7 +293,9 @@ impl KernelResolver { } else { self.resolve_import_path(spec, file, language)? }; - if target.is_none() { + // By the end of its path when nothing else places it (jemalloc's `jemalloc/internal/*.h`), + // but a standard header is never the project's: `` is not jemalloc's test/math.h. + if target.is_none() && !self.is_external_import(spec, language, file) { if self.cpp_macros.by_basename.is_none() { let files: Vec = match self.sorted_files() { Some(files) => files.to_vec(), @@ -845,7 +847,7 @@ fn wraps_file(lines: &[String], index: usize, define: usize) -> bool { /// The file's lines with comments removed only as far as the preprocessor /// needs: a line inside a block comment is blank, a directive line loses its /// trailing `//` / `/* … */`, and every other line is kept verbatim. -fn directive_lines(source: &str) -> Vec { +pub(super) fn directive_lines(source: &str) -> Vec { let masked = mask_cpp_raw_strings(source); let mut out = Vec::new(); let mut in_block = false; diff --git a/codegraph-kernel/src/resolve/cpp_implicit_this.rs b/codegraph-kernel/src/resolve/cpp_implicit_this.rs index 337faf5947..d14dda5bab 100644 --- a/codegraph-kernel/src/resolve/cpp_implicit_this.rs +++ b/codegraph-kernel/src/resolve/cpp_implicit_this.rs @@ -283,6 +283,7 @@ impl KernelResolver { } if belongs { own.push(n); } } + let own = self.cpp_specialization_pool(own, r); let holder = if scope.holds { scope.decls.first() } else { None }; let (inline, rest): (Vec<_>, Vec<_>) = match holder { Some(h) => own.into_iter().partition(|n| n.file_path == h.file_path && range_within(n, h)), @@ -313,17 +314,26 @@ impl KernelResolver { .iter().filter(|n| n.id != inherited.id && n.kind == "method" && c_family(&n.language)).cloned().collect(); let mut overloads = vec![inherited]; overloads.extend(prefer_call_site_file(siblings, &r.file_path)); + let overloads = self.cpp_specialization_pool(overloads, r); + if overloads.is_empty() { return Ok(Lookup::Missing); } return Ok(Lookup::Found(self.cpp_overload_for(overloads, name, r)?)); } } Ok(Lookup::Missing) } - /// C++ class `cls`'s method `name`: its own, else the nearest one of a class it derives from, - /// through the base edges every declaration of `cls` has. + /// C++ class `cls`'s method `name`: its own (of a member template's explicit specializations, + /// the one its template arguments name, cpp_specialization_pool; None when it names none of + /// them and they are all `cls` has), else the nearest one of a class it derives from, through + /// the base edges every declaration of `cls` has. pub(super) fn cpp_method_of(&mut self, cls: &KNode, name: &str, r: &ResolveRefIn, depth: usize, seen: &mut HashSet) -> Res>> { - let own: Vec<_> = self.nodes_by_qualified_name(&format!("{}::{name}", cls.qualified_name))? + let family: Vec<_> = self.nodes_by_qualified_name(&format!("{}::{name}", cls.qualified_name))? .iter().filter(|n| n.kind == "method" && c_family(&n.language)).cloned().collect(); + let had = !family.is_empty(); + let own = self.cpp_specialization_pool(family, r); + // Specializations of the class's own member template, none the call's: what C++ finds is + // the template, which has no body here. + if own.is_empty() && had { return Ok(None); } if !own.is_empty() { // Of its overloads, the one the call's arguments fit; a chained call `inner(…).name(…)` // keeps the first, since the arguments at its position are the inner call's. diff --git a/codegraph-kernel/src/resolve/cpp_namespace.rs b/codegraph-kernel/src/resolve/cpp_namespace.rs index 7c76363130..ebe65d06cd 100644 --- a/codegraph-kernel/src/resolve/cpp_namespace.rs +++ b/codegraph-kernel/src/resolve/cpp_namespace.rs @@ -22,7 +22,9 @@ impl KernelResolver { if !re!(r"(?i)\.(?:h|hh|hpp|hxx|inl|ipp|tcc|c|cc|cpp|cxx)$").is_match(file) {continue;} let Some(source)=self.read_file(file) else {continue}; let code=super::awaited::strip_ts_comments(&source.text().replace("\\\r\n"," ").replace("\\\n"," ")); - for m in re!(r"(?m)^\s*namespace\s+(\w+)\s*=\s*(?:::)?([\w:]+)\s*;").captures_iter(&code) {cache.aliases.entry(file.clone()).or_default().insert(m[1].to_string(),m[2].to_string());} + // (Of a file's alternatives for one build or another, `#if … namespace backend = …upb; + // #elif … namespace backend = …cpp;`, the first.) + for m in re!(r"(?m)^\s*namespace\s+(\w+)\s*=\s*(?:::)?([\w:]+)\s*;").captures_iter(&code) {cache.aliases.entry(file.clone()).or_default().entry(m[1].to_string()).or_insert_with(|| m[2].to_string());} for m in re!(r"(?m)^[ \t]*#[ \t]*define[ \t]+([A-Za-z_]\w*)(\(\s*([A-Za-z_]\w*)?\s*\))?[ \t]+([^\n]*)$").captures_iter(&code) { let body=m[4].trim(); if re!(r"^(?:[A-Za-z_]\w*\s+)*\}(?:\s*\})*\s*;?$").is_match(body) {cache.closers.insert((file.clone(),m[1].to_string()));} @@ -106,15 +108,39 @@ impl KernelResolver { } self.cpp_namespaces.includes.insert(file.to_string(),seen.clone());Ok(seen) } + /// The namespace `name` is an alias of where `file` writes it: `pybind11` for `py` under + /// `namespace py = pybind11;`, declared in the file itself, else alike in the files it includes. + pub(super) fn cpp_namespace_alias(&mut self,file:&str,name:&str)->Res> { + self.prepare_namespace_macros()?; + if let Some(local)=self.cpp_namespaces.aliases.get(file).and_then(|a|a.get(name)) {return Ok(Some(local.clone()));} + let visible=self.namespace_visible_files(file,"cpp")?; + let aliases:HashSet<_>=visible.iter().filter_map(|file|self.cpp_namespaces.aliases.get(file).and_then(|a|a.get(name))).cloned().collect(); + Ok((aliases.len()==1).then(||aliases.into_iter().next().unwrap())) + } + /// `qualified` spelled through the name macros `file` sees, when that differs: rocksdb's types + /// are `ROCKSDB_NAMESPACE::DB` in the index and `rocksdb::DB` to code that names them, under + /// `#define ROCKSDB_NAMESPACE rocksdb`; a tree declaring `namespace rocksdb` itself is named + /// `ROCKSDB_NAMESPACE::Cache` the other way round. + pub(super) fn cpp_name_macro_spelling(&mut self,file:&str,qualified:&str)->Res> { + self.prepare_namespace_macros()?; + if self.cpp_namespaces.replacements.is_empty() {return Ok(None);} + let visible=self.namespace_visible_files(file,"cpp")?; + let spelled:Vec=qualified.split("::").map(|segment| { + let forward:HashSet<_>=self.cpp_namespaces.replacements.iter().filter(|((f,name),_)|visible.contains(f) && name==segment).map(|(_,body)|body.clone()).collect(); + if forward.len()==1 {return forward.into_iter().next().unwrap();} + let reverse:HashSet<_>=self.cpp_namespaces.replacements.iter().filter(|((f,_),body)|visible.contains(f) && *body==segment).map(|((_,name),_)|name.clone()).collect(); + if reverse.len()==1 {reverse.into_iter().next().unwrap()} else {segment.to_string()} + }).collect(); + let spelled=spelled.join("::"); + Ok((spelled!=qualified).then_some(spelled)) + } pub(super) fn match_cpp_macro_namespaced(&mut self,r:&ResolveRefIn)->Res> { if !matches!(r.language.as_str(),"c"|"cpp") || !r.reference_name.trim_start_matches("::").contains("::") {return Ok(None);} self.prepare_namespace_macros()?; let visible=self.namespace_visible_files(&r.file_path,&r.language)?; let mut parts:Vec=r.reference_name.split("::").filter(|p|!p.is_empty()).map(str::to_string).collect(); - if let Some(first)=parts.first() { - let local=self.cpp_namespaces.aliases.get(&r.file_path).and_then(|a|a.get(first)).cloned(); - let aliases:HashSet<_>=visible.iter().filter_map(|file|self.cpp_namespaces.aliases.get(file).and_then(|a|a.get(first))).cloned().collect(); - if let Some(alias)=local.or_else(||(aliases.len()==1).then(||aliases.into_iter().next().unwrap())) {parts.splice(0..1,alias.split("::").map(str::to_string));} + if let Some(first)=parts.first().cloned() { + if let Some(alias)=self.cpp_namespace_alias(&r.file_path,&first)? {parts.splice(0..1,alias.split("::").map(str::to_string));} } let Some(name)=parts.last() else {return Ok(None)}; let candidates=self.nodes_by_name(name)?; @@ -128,14 +154,22 @@ impl KernelResolver { path.extend(n.qualified_name.split("::").map(str::to_string)); if path==parts {matches.push(n.clone());} } + let matches=self.cpp_specialization_pool(matches,r); let args = if r.reference_kind=="calls" { self.call_arguments(r,name) } else { None }; - let mut best=None; let mut best_score=i64::MIN;let mut tied=false; + let mut best:Vec>=Vec::new(); let mut best_score=i64::MIN; for n in matches { if args.as_ref().is_some_and(|args| !self.cpp_overload_compatible(&n,args,r)) {continue;} let score=if is_test_path(&n.file_path){-10}else{0} + args.as_ref().map(|args|self.cpp_overload_fit(&n,args,r)).unwrap_or(0); - if score>best_score {best_score=score;best=Some(n);tied=false;}else if score==best_score{tied=true;} + if score>best_score {best_score=score;best=vec![n];}else if score==best_score{best.push(n);} } - if tied{return Ok(None);} + // Overloads the arguments fit alike by count are told apart by the types they have (fmt's + // `write(out, count)` on an `int count`), else the call is left alone. + let best=match best.len() { + 0=>None, + 1=>best.pop(), + _ if args.is_some()=>self.cpp_argument_type_choice(best,name,r)?, + _=>None, + }; Ok(best.map(|node|KCand{node,confidence:0.8,resolved_by:"qualified-name"})) } } diff --git a/codegraph-kernel/src/resolve/cpp_overloads.rs b/codegraph-kernel/src/resolve/cpp_overloads.rs index dd86759763..b242d75c93 100644 --- a/codegraph-kernel/src/resolve/cpp_overloads.rs +++ b/codegraph-kernel/src/resolve/cpp_overloads.rs @@ -9,7 +9,6 @@ //! The arguments are read from the syntax tree: a literal's kind, or the type a name, `&name`, //! `name[i]` or another expression has as written where it is declared. use super::cpp_implicit_this::strip_template_arguments; -use super::cpp_range_for::CppWritten; use super::iteration::{descendant_for_position, named_children}; use super::receivers::cpp_last_segment; use super::*; @@ -21,7 +20,10 @@ pub(super) enum CppArgument { Zero, Bool, Null, - Type(String), + /// `lvalue`: whether the argument names an object (a name, a dereference, a member) rather than + /// a value a call returns, where that can be told. `dependent`: its type is a template + /// parameter of the calling function or its class, which could be a pointer as well as an object. + Type { text: String, lvalue: Option, dependent: bool }, } /// What a C++ type is, for an argument to fit a parameter of it. @@ -65,6 +67,14 @@ pub(super) fn cpp_type_category(ty: &str) -> CppTypeCategory { } /// A C++ parameter declaration's type: without its default argument and its name. +/// An array type (`Char[]`) as the pointer to its element an argument of it decays to (`Char*`). +fn cpp_array_as_pointer(text: String) -> String { + match super::cpp_range_for::cpp_array_split(&text) { + Some((base, dimensions)) if dimensions > 0 => format!("{base}*"), + _ => text, + } +} + pub(super) fn cpp_parameter_type(param: &str) -> String { let mut text = param; let mut depth = 0i32; @@ -129,24 +139,79 @@ impl KernelResolver { let top = fits.iter().copied().max().unwrap_or(0); let tied: Vec<_> = overloads.iter().zip(&fits).filter(|(_, f)| **f == top).map(|(n, _)| n.clone()).collect(); if tied.len() == 1 || args.is_empty() { return Ok(tied[0].clone()); } - let Some(types) = self.cpp_call_argument_types(name, r)? else { return Ok(tied[0].clone()) }; - if types.iter().all(Option::is_none) { return Ok(tied[0].clone()); } - let mut best = tied[0].clone(); - let mut best_fit = self.cpp_argument_type_fit(&best, &types, r)?; - for n in tied.into_iter().skip(1) { - let fit = self.cpp_argument_type_fit(&n, &types, r)?; - if fit > best_fit { - best = n; - best_fit = fit; + let first = tied[0].clone(); + Ok(self.cpp_argument_type_choice(tied, name, r)?.unwrap_or(first)) + } + + /// Of overloads a call's arguments fit alike by count (in preference order), the one the types + /// the arguments have where the call reads them fit best; the first of those that fit best + /// alike. None when no argument's type can be read, or every overload fits the types alike. + pub(super) fn cpp_argument_type_choice(&mut self, tied: Vec>, name: &str, r: &ResolveRefIn) -> Res>> { + let Some(types) = self.cpp_call_argument_types(name, r)? else { return Ok(None) }; + if types.iter().all(Option::is_none) { return Ok(None); } + let mut fits = Vec::with_capacity(tied.len()); + for n in &tied { fits.push(self.cpp_argument_type_fit(n, &types, r)?); } + let top = fits.iter().copied().max().unwrap_or(0); + let at = fits.iter().position(|f| *f == top).unwrap_or(0); + let best = tied[at].clone(); + let others: Vec<_> = tied.iter().zip(&fits).enumerate().filter(|(i, (_, f))| **f == top && *i != at).map(|(_, (n, _))| n.clone()).collect(); + // Fit alike by type: decided only when a const twin is. + let alike = others.len() + 1 == tied.len(); + let choice = self.cpp_const_twin_choice(best.clone(), others, &types, r); + Ok((!alike || choice.id != best.id || at != 0).then_some(choice)) + } + + /// The C++ function a call of `chosen`'s name reaches among `candidates` (cppFunctionOverload, + /// name-matcher.ts at 4a591aee): of its overloads, the functions of its qualified name the + /// call's file can see, `chosen` first, the one the call's arguments fit. The name strategies + /// pick a function by where it is; protobuf's `ParseWithLimit(limit, data, result)` on an + /// `int32_t result` is the overload taking an `int32_t&`, not the `int64_t&` one nearer the + /// call. A function hides a class of its name, so a call is to the function when `chosen` is + /// that class. `chosen` itself for anything but a C or C++ call, or when no function is in sight. + pub(super) fn cpp_function_overload(&mut self, chosen: Arc, candidates: &[Arc], r: &ResolveRefIn) -> Res> { + if !matches!(r.language.as_str(), "c" | "cpp") || r.reference_kind != "calls" { return Ok(chosen); } + let mut overloads = Vec::new(); + for n in candidates { + if n.kind == "function" && n.qualified_name == chosen.qualified_name && self.is_visible_across_files(n, r)? { overloads.push(n.clone()); } + } + if overloads.is_empty() { return Ok(chosen); } + // C has no overloads: a C call reaches the function of the name its file sees, its own or + // one a header it includes defines, else one with external linkage, the best placed of + // them. jemalloc's tests call the `static inline do_arena_create` of the arena_util.h they + // include, not a neighbouring test's own `static do_arena_create`. What the file sees only + // when the build sets a flag nothing in the project sets (is_c_feature_variant) yields to + // a definition with external linkage that the build without it calls: lua-cjson's fpconv.h + // defines `fpconv_init` under `#ifdef USE_INTERNAL_FPCONV`, and redis's build calls + // fpconv.c's. A call compiled only with such a flag itself is left to the file's own: + // redis's quicklist.c tests call its `static mstime` inside `#ifdef REDIS_TEST`. + if r.language == "c" { + let visible = self.namespace_visible_files(&r.file_path, "c")?; + let (own, others): (Vec<_>, Vec<_>) = overloads.iter().cloned().partition(|n| n.file_path == r.file_path || visible.contains(&n.file_path)); + let mut external = Vec::new(); + if !own.is_empty() { + for n in others { if !self.is_static_c_function(&n)? { external.push(n); } } } + let mut outside = Vec::new(); + if !external.is_empty() && own.iter().all(|n| self.is_c_feature_variant(n, r)) && !self.is_off_by_default(&r.file_path, r.line) { + outside = external.into_iter().filter(|n| !self.is_c_feature_variant(n, r)).collect(); + } + let pool = if !outside.is_empty() { &outside } else if !own.is_empty() { &own } else { &overloads }; + return Ok(self.find_best_match(r, pool).unwrap_or(chosen)); + } + if let Some(at) = overloads.iter().position(|n| n.id == chosen.id) { + let own = overloads.remove(at); + overloads.insert(0, own); } - Ok(best) + let name = chosen.name.clone(); + self.cpp_overload_for(overloads, &name, r) } /// Whether the C++ function or method `n`, named `name`, is declared deleted: from its name to /// the declaration's `;` or body, it says `= delete` (protobuf's `void reset(T* - /// PROTOBUF_NULLABLE) = delete;`). + /// PROTOBUF_NULLABLE) = delete;`). A method a macro defines never is. fn cpp_deleted_function(&mut self, n: &KNode, name: &str) -> bool { + // A method a macro defines has a body. + if cpp_macro_method_head(n).is_some() { return false; } let Some(source) = self.read_file(&n.file_path) else { return false }; let from = (n.start_line - 1).max(0) as usize; let Ok(word) = shared_regex(&format!(r"\b{}\s*\(", regex::escape(name))) else { return false }; @@ -169,6 +234,7 @@ impl KernelResolver { fn cpp_argument_type_fit(&mut self, n: &KNode, types: &[Option], r: &ResolveRefIn) -> Res { let Some(params) = self.declaration_parameters(n, r) else { return Ok(0) }; let params: Vec<_> = params.into_iter().filter(|p| p != "void").collect(); + let own = if types.iter().any(|t| matches!(t, Some(CppArgument::Type { .. }))) { self.cpp_own_template_parameters(&Arc::new(n.clone()), r)? } else { Vec::new() }; let mut score = 0; for (argument, param) in types.iter().zip(¶ms) { let Some(argument) = argument else { continue }; @@ -176,15 +242,80 @@ impl KernelResolver { if param.contains("...") { continue; } let category = cpp_type_category(¶m); score += match argument { - CppArgument::Type(text) => self.cpp_written_type_fit(text, ¶m, category)?, + CppArgument::Type { text, lvalue, dependent } => { + // (A parameter of the function's own template parameter's type, `T* x`, takes the argument's `const` into `T`.) + let deduced = !own.is_empty() && self.normalize_cpp_type_name(¶m, false)?.is_some_and(|p| own.contains(&p)); + self.cpp_written_type_fit(text, ¶m, category, *lvalue, *dependent, deduced)? + } literal => literal_fit(literal, category), }; } Ok(score) } - /// How an argument written as the type `argument` fits a parameter of the type `param`. - fn cpp_written_type_fit(&mut self, argument: &str, param: &str, category: CppTypeCategory) -> Res { + /// An overload's parameter types without their `const`s: what its const twins share. + fn cpp_constless_parameters(&mut self, n: &KNode, r: &ResolveRefIn) -> Option> { + let params = self.declaration_parameters(n, r)?; + Some(params.into_iter().filter(|p| p != "void").map(|p| re!(r"\bconst\b").replace_all(&cpp_parameter_type(&p), " ").split_whitespace().collect::()).collect()) + } + + /// How many of the arguments `types` an overload's pointer or reference parameters add `const` to. + fn cpp_const_added(&mut self, n: &KNode, types: &[Option], r: &ResolveRefIn) -> usize { + let Some(params) = self.declaration_parameters(n, r) else { return 0 }; + let params: Vec<_> = params.into_iter().filter(|p| p != "void").collect(); + types.iter().zip(¶ms).filter(|(argument, param)| { + let param = cpp_parameter_type(param); + matches!(argument, Some(CppArgument::Type { text, .. }) if cpp_refers_to_const(¶m) == Some(true) && cpp_argument_const(text, ¶m) == Some(false)) + }).count() + } + + /// `best`, or of its twins among `others` (overloads alike but for the `const` of a pointer or + /// reference parameter) the one that adds `const` to the fewest of the arguments `types`, as + /// C++ ranks them: protobuf's `DynamicCastMessage(ptr.get())` on a + /// `std::shared_ptr ptr` is `T* DynamicCastMessage(MessageLite* from)`, not the `const + /// T* DynamicCastMessage(const MessageLite* from)` declared before it. (A const object fits only + /// the const twin: `cpp_written_type_fit`.) + fn cpp_const_twin_choice(&mut self, best: Arc, others: Vec>, types: &[Option], r: &ResolveRefIn) -> Arc { + if others.is_empty() { return best; } + let Some(shape) = self.cpp_constless_parameters(&best, r) else { return best }; + let mut added = self.cpp_const_added(&best, types, r); + let mut choice = best; + for n in others { + if self.cpp_constless_parameters(&n, r).as_ref() != Some(&shape) { continue; } + let count = self.cpp_const_added(&n, types, r); + if count < added { + choice = n; + added = count; + } + } + choice + } + + /// How an argument written as the type `argument` fits a parameter of the type `param` (of + /// category `category`). An object is no pointer, nor a pointer to characters a pointer to + /// anything but characters or `void`, or the other way round: protobuf's + /// `DynamicCastMessage(*msg)` takes a reference, and rocksdb's `db_->Put(opts, key, ts, + /// value)` on a `char key[]` no `ColumnFamilyHandle*`. A pointer or reference to a non-const + /// type never takes a const object, nor a reference a value a call returns (`lvalue` false). + /// None of that holds for an argument of a template parameter's type (`dependent`), and the + /// pointer and `const` rules not for a parameter whose type the function's own template + /// parameter is (`deduced`). (Of two overloads alike but for a `const`, which one a non-const + /// object takes is `cpp_const_twin_choice`'s.) + fn cpp_written_type_fit(&mut self, argument: &str, param: &str, category: CppTypeCategory, lvalue: Option, dependent: bool, deduced: bool) -> Res { + use CppTypeCategory::*; + let fit = self.cpp_unqualified_type_fit(argument, param, category)?; + if fit < 0 || dependent { return Ok(fit); } + let argument_category = cpp_type_category(argument); + if category == Pointer && argument_category == Class { return Ok(-3); } + if !deduced && category == Pointer && argument_category == CString && !re!(r"\bvoid\b").is_match(param) { return Ok(-3); } + if category == CString && argument_category == Pointer { return Ok(-3); } + if cpp_refers_to_const(param) != Some(false) { return Ok(fit); } + if !deduced && cpp_argument_const(argument, param) == Some(true) { return Ok(-3); } + Ok(if lvalue == Some(false) && !strip_template_arguments(param).contains('*') { -3 } else { fit }) + } + + /// How an argument written as the type `argument` fits a parameter of the type `param`, `const` aside. + fn cpp_unqualified_type_fit(&mut self, argument: &str, param: &str, category: CppTypeCategory) -> Res { use CppTypeCategory::*; let argument_category = cpp_type_category(argument); let pointer = |c: CppTypeCategory| c == Pointer || c == CString; @@ -235,18 +366,122 @@ impl KernelResolver { "pointer_expression" => { let operator = node.child_by_field_name("operator").map(|o| o.kind()); match node.child_by_field_name("argument").filter(|a| a.kind() == "identifier" && operator == Some("&")) { - Some(name) => self.cpp_written_name(name, source, r)?.map(|w| CppArgument::Type(format!("{}*", w.text))), - None => self.cpp_written_expression(node, source, r)?.map(|w| CppArgument::Type(w.text)), + // (`&ref` is a pointer to the referee.) + Some(name) => match self.cpp_written_name(name, source, r)? { + Some(w) => { + let text = format!("{}*", cpp_declared_constness(&w.text, name, source).trim_end_matches('&').trim_end()); + let dependent = self.cpp_dependent_argument(&text, r)?; + Some(CppArgument::Type { text, lvalue: Some(false), dependent }) + } + None => None, + }, + None => match self.cpp_written_expression(node, source, r)? { + Some(w) => { + let dependent = self.cpp_dependent_argument(&w.text, r)?; + Some(CppArgument::Type { lvalue: cpp_names_object(text, &w.text), text: w.text, dependent }) + } + None => None, + }, } } - "identifier" => self.cpp_written_name(node, source, r)?.map(|w| CppArgument::Type(w.text)), - // `keys[i]`: an element of the container, or what the pointer points at. - "subscript_expression" => { - let Some(container) = node.child_by_field_name("argument").filter(|a| a.kind() == "identifier") else { return Ok(None) }; - let Some(written) = self.cpp_written_name(container, source, r)? else { return Ok(None) }; - self.cpp_subscript_element(written, r)?.map(|w| CppArgument::Type(w.text)) - } - _ => self.cpp_written_expression(node, source, r)?.map(|w: CppWritten| CppArgument::Type(w.text)), + // (An array is a pointer to its element: fmt's `format_base2e(4, buf, …)` on a `Char + // buf[8]` is the `Char* out` overload.) + "identifier" => match self.cpp_written_name(node, source, r)? { + Some(w) => { + let text = cpp_array_as_pointer(cpp_declared_constness(&w.text, node, source)); + let dependent = self.cpp_dependent_argument(&text, r)?; + Some(CppArgument::Type { text, lvalue: Some(true), dependent }) + } + None => None, + }, + // `keys[i]`: the element a subscript reads (`cpp_subscript_element`), through the expression typer. + _ => match self.cpp_written_expression(node, source, r)? { + Some(w) => { + let text = cpp_array_as_pointer(w.text); + let dependent = self.cpp_dependent_argument(&text, r)?; + Some(CppArgument::Type { lvalue: cpp_names_object(source.text()[node.byte_range()].trim(), &text), text, dependent }) + } + None => None, + }, }) } + + /// Whether an argument of the written type `ty` has the type of a template parameter of the + /// calling function or its class (fmt's `const S& fmt`), which could be a pointer as well as an + /// object. + fn cpp_dependent_argument(&mut self, ty: &str, r: &ResolveRefIn) -> Res { + let Some(name) = self.normalize_cpp_type_name(ty, false)? else { return Ok(false) }; + let Some(caller) = self.node_by_id(&r.from_node_id)? else { return Ok(false) }; + self.cpp_template_parameter(&caller, &name, r) + } +} + +/// `written`, with a `const` in front when the declaration of the name `node` writes one before +/// its type (`const Buffer* b`, `constexpr char key[]`), which the declared type leaves out. +fn cpp_declared_constness(written: &str, node: tree_sitter::Node, source: &SourceFile) -> String { + if !re!(r"^\s*const\b").is_match(written) && super::cpp_receivers::cpp_declared_const(node, source) { format!("const {written}") } else { written.to_string() } +} + +/// Whether the C++ argument `expr`, of the written type `ty`, names an object, which a reference to +/// a non-const type can bind: a dereference, a subscript, a member, or a call that returns an +/// lvalue reference. Some(false) for a call that returns a value; None when that can't be told. +fn cpp_names_object(expr: &str, ty: &str) -> Option { + let mut text = expr.trim(); + while text.starts_with('(') && text.ends_with(')') && opening_paren(text, text.len() - 1) == Some(0) { text = text[1..text.len() - 1].trim(); } + if text.starts_with('*') || text.ends_with(']') { return Some(true); } + if !text.ends_with(')') { return re!(r"^[A-Za-z_][\w\s.:>-]*$").is_match(text).then_some(true); } + if re!(r"&&\s*$").is_match(ty) { return None; } + Some(re!(r"&\s*$").is_match(ty)) +} + +/// Whether the C++ type `ty`, a pointer or an lvalue reference of one level, refers to a const +/// object: `const T*`, `T const&`. (A pointer's own `const`, `T* const`, is not the object's.) +/// None for any other type. +fn cpp_refers_to_const(ty: &str) -> Option { + let text = strip_template_arguments(ty); + let text = text.trim(); + let stars = text.matches('*').count(); + if stars > 1 || (stars == 0 && !re!(r"[^&]&\s*$").is_match(text)) { return None; } + let object = if stars == 1 { &text[..text.find('*').unwrap_or(text.len())] } else { text }; + Some(re!(r"\bconst\b").is_match(object)) +} + +/// Whether the object an argument of the written type `argument` gives a pointer or reference +/// parameter `param` is const: what the pointer points at, or the object itself. None when the +/// argument is no such thing. +fn cpp_argument_const(argument: &str, param: &str) -> Option { + let argument_pointer = strip_template_arguments(argument).contains('*'); + if strip_template_arguments(param).contains('*') { return if argument_pointer { cpp_refers_to_const(argument) } else { None }; } + if argument_pointer { None } else { Some(re!(r"\bconst\b").is_match(&strip_template_arguments(argument))) } +} + +/// Where the `(` the `)` at byte `close` of `text` closes is. +fn opening_paren(text: &str, close: usize) -> Option { + let mut depth = 0usize; + for (i, ch) in text[..=close].char_indices().rev() { + match ch { + ')' => depth += 1, + '(' => { depth -= 1; if depth == 0 { return Some(i); } } + _ => (), + } + } + None +} + +/// The expanded definition a C++ method that a macro defines carries as its signature +/// (cpp-macro-methods.ts), `inline const FileDescriptor* Descriptor::file() const` for protobuf's +/// `PROTOBUF_DEFINE_ACCESSOR(Descriptor, file, const FileDescriptor*)`: its source is the +/// invocation, which spells neither its return type nor its parameters. None for any other node. +pub(super) fn cpp_macro_method_head(n: &KNode) -> Option<&str> { + if n.kind != "method" { return None; } + let signature = n.signature.as_deref()?; + shared_regex(&format!(r"::\s*{}\s*\(", regex::escape(&n.name))).ok()?.is_match(signature).then_some(signature) +} + +/// The expansion `head` of a method a macro defines, parsed as a definition of its own. +pub(super) fn cpp_macro_method_tree(head: &str) -> Option<(Rc, tree_sitter::Tree)> { + let source = Rc::new(SourceFile::new(vec![format!("{head} {{}}")])); + let text = super::cpp_annotations::cpp_without_annotations(source.text()).into_owned(); + let tree = crate::tree::parse_with_cached_parser(&text, "cpp").ok()?; + Some((source, tree)) } diff --git a/codegraph-kernel/src/resolve/cpp_range_for.rs b/codegraph-kernel/src/resolve/cpp_range_for.rs index 7354d3ed16..e371f3e20d 100644 --- a/codegraph-kernel/src/resolve/cpp_range_for.rs +++ b/codegraph-kernel/src/resolve/cpp_range_for.rs @@ -39,9 +39,49 @@ const LIBRARY_SEQUENCES: &[&str] = &[ /// Widely used project containers of the same kind: protobuf's repeated fields, rocksdb's `autovector`. const PROJECT_SEQUENCES: &[&str] = &["RepeatedField", "RepeatedPtrField", "autovector"]; /// The standard library's and abseil's maps, whose elements are `std::pair`s of a key and a value. -const LIBRARY_MAPS: &[&str] = &["map", "multimap", "unordered_map", "unordered_multimap", "flat_hash_map", "node_hash_map", "btree_map", "btree_multimap"]; +pub(super) const LIBRARY_MAPS: &[&str] = &["map", "multimap", "unordered_map", "unordered_multimap", "flat_hash_map", "node_hash_map", "btree_map", "btree_multimap"]; +/// The standard library's container adaptors, whose element is their first template argument. +const LIBRARY_ADAPTORS: &[&str] = &["stack", "queue", "priority_queue"]; +/// The members of a library container that return an iterator into it, with the iterator's kind. +const ITERATOR_MEMBERS: &[(&str, &str)] = &[ + ("begin", "iterator"), ("end", "iterator"), ("find", "iterator"), ("lower_bound", "iterator"), ("upper_bound", "iterator"), + ("cbegin", "const_iterator"), ("cend", "const_iterator"), ("rbegin", "reverse_iterator"), ("rend", "reverse_iterator"), + ("crbegin", "const_reverse_iterator"), ("crend", "const_reverse_iterator"), +]; const MAX_DEPTH: u32 = 4; +/// A written type and the array dimensions `cpp_ast_declared_type` gives it (`Status[]` for +/// `Status statuses[3]`, `int[][]` for two): the element type, and how many. None when a `[` is +/// anywhere else. +pub(super) fn cpp_array_split(text: &str) -> Option<(&str, usize)> { + let mut base = text.trim_end(); + let mut dimensions = 0; + while let Some(inner) = base.strip_suffix("[]") { + base = inner.trim_end(); + dimensions += 1; + } + (!base.contains('[')).then_some((base, dimensions)) +} + +/// An element of the array `text` (`Status[]`): its type with one dimension less. None for a +/// type that is no array. +fn cpp_array_element(text: &str) -> Option { + let (base, dimensions) = cpp_array_split(text)?; + (dimensions > 0).then(|| format!("{base}{}", "[]".repeat(dimensions - 1))) +} + +/// A written type without `typename`, `const` and `&`, which name no part of it. +fn without_type_keywords(text: &str) -> String { + re!(r"\b(?:typename|const)\b").replace_all(text, " ").replace('&', " ").trim().to_string() +} + +/// The container of a container's iterator type (`std::vector` of +/// `std::vector::const_iterator`). +fn iterator_container(text: &str) -> Option { + let bare = without_type_keywords(text); + re!(r"^(.*>)\s*::\s*(?:const_)?(?:reverse_)?iterator$").captures(&bare).map(|m| m[1].trim().to_string()) +} + /// A written type without nullability annotations (`const FieldDescriptor* absl_nonnull`) or /// cv-qualifiers, which are no part of the name it is looked up by. pub(super) fn cpp_plain_type(text: &str) -> String { @@ -49,9 +89,9 @@ pub(super) fn cpp_plain_type(text: &str) -> String { text.split_whitespace().collect::>().join(" ") } -/// The class a member expression reads from, the member's name, and its explicit template +/// The object a member expression reads from, the member's name, and its explicit template /// arguments (`<…>` as written). -type CppMemberRead = (Arc, String, Option); +type CppMemberRead = (CppWritten, String, Option); /// A type written `auto`, `auto&`, `const auto*`. pub(super) fn is_cpp_auto(raw: &str) -> bool { @@ -59,7 +99,7 @@ pub(super) fn is_cpp_auto(raw: &str) -> bool { } /// Whether `text` is a pointer: a `*` outside its template arguments. -fn top_level_pointer(text: &str) -> bool { +pub(super) fn top_level_pointer(text: &str) -> bool { let mut depth = 0i32; text.chars().any(|c| { match c { '<' => depth += 1, '>' => depth -= 1, _ => {} } @@ -246,7 +286,7 @@ impl KernelResolver { } /// Whether the type `written` names is a template parameter's, which a template argument decides. - fn cpp_written_template_parameter(&mut self, written: &CppWritten) -> Res { + pub(super) fn cpp_written_template_parameter(&mut self, written: &CppWritten) -> Res { let Some(parts) = type_segments(&written.text) else { return Ok(false) }; let Some(node) = self.node_by_id(&written.site.from_node_id)? else { return Ok(false) }; self.cpp_template_parameter(&node, &parts[0], &written.site) @@ -256,6 +296,11 @@ impl KernelResolver { /// rocksdb's `const BlobFiles&` is `std::vector>`. None when an /// alias on the way can't be read: a member of an aliased type, `typename`, `decltype`. pub(super) fn cpp_follow_written_aliases(&mut self, written: CppWritten, r: &ResolveRefIn) -> Res> { + // An array of an alias (`Handle handles[2]`) is an array of what the alias names. + if let Some((base, dimensions)) = cpp_array_split(&written.text).filter(|(_, d)| *d > 0) { + let base = CppWritten { text: base.to_string(), site: written.site }; + return Ok(self.cpp_follow_written_aliases(base, r)?.map(|f| CppWritten { text: format!("{}{}", f.text, "[]".repeat(dimensions)), site: f.site })); + } let included = self.namespace_visible_files(&r.file_path, "cpp")?; let mut current = written; for _ in 0..4 { @@ -317,6 +362,12 @@ impl KernelResolver { /// FieldDescriptor*>` (the index keeps only `vector`), and its parameter list without spaces. /// None for a constructor, `void`, `auto` or `decltype(…)`. fn cpp_written_return_type(&mut self, function: &Arc, r: &ResolveRefIn) -> Res> { + // A method a macro defines is read from its expansion. + if let Some(head) = super::cpp_overloads::cpp_macro_method_head(function) { + let Some((source, tree)) = super::cpp_overloads::cpp_macro_method_tree(head) else { return Ok(None) }; + let Some(declaration) = named_children(tree.root_node()).into_iter().find(|n| n.kind() == "function_definition") else { return Ok(None) }; + return Ok(cpp_declared_return(declaration, &source)); + } let Some(source) = self.read_file(&function.file_path) else { return Ok(None) }; let site = site_at(r, function); let Some(tree) = self.parsed_tree(&source, &site) else { return Ok(None) }; @@ -333,12 +384,7 @@ impl KernelResolver { let Some(parent) = node.parent() else { return Ok(None) }; node = parent; }; - let (Some(ty), Some(declarator)) = (declaration.child_by_field_name("type"), declaration.child_by_field_name("declarator")) else { return Ok(None) }; - let Some(function_declarator) = declared_function(declarator) else { return Ok(None) }; - let raw = cpp_plain_type(&cpp_ast_declared_type(ty, declarator, &source)); - if raw.is_empty() || raw == "void" || is_cpp_auto(&raw) || raw.contains("decltype") { return Ok(None); } - let parameters = function_declarator.child_by_field_name("parameters").map(|p| without_spaces(&source.text()[p.byte_range()])).unwrap_or_default(); - Ok(Some((raw, parameters))) + Ok(cpp_declared_return(declaration, &source)) } /// The type the C++ class `cls` declares its data member `field` with, as written in the class, @@ -490,9 +536,14 @@ impl KernelResolver { let Some(cls) = self.cpp_written_class(&object, r)? else { return Ok(if self.cpp_outside_qualified_type(&object.text, &object.site)? { Some(None) } else { None }); }; - if let Some(method) = self.cpp_methods_named(&cls, &chain.1, r)?.into_iter().next() { + let methods = self.cpp_methods_named(&cls, &chain.1, r)?; + let named = !methods.is_empty(); + // Of a member template's explicit specializations, the one the call's template arguments + // name; none of them when it names another (cpp_specialization_pool). + if let Some(method) = self.cpp_specialization_pool(methods, r).into_iter().next() { return Ok(Some(Some(KCand { node: method, confidence: 0.85, resolved_by: "instance-method" }))); } + if named { return Ok(Some(None)); } if self.supertypes_complete && !self.cpp_alias_template_owner(&cls, r)? && self.cpp_bases_known(&cls, 0, &mut HashSet::new())? { return Ok(Some(None)); } Ok(None) } @@ -518,11 +569,20 @@ impl KernelResolver { let Some(argument) = node.child_by_field_name("argument") else { return Ok(None) }; let Some(pointer) = self.cpp_expression_written(argument, source, r, depth + 1)? else { return Ok(None) }; let Some(pointer) = self.cpp_follow_written_aliases(pointer, r)? else { return Ok(None) }; - Ok(cpp_pointee(&pointer.text).map(|text| CppWritten { text, site: pointer.site })) + self.cpp_dereferenced(pointer, r) + } + // `statuses[i]`, `(*m)[key]`: the element a subscript reads. + "subscript_expression" => { + let Some(argument) = node.child_by_field_name("argument") else { return Ok(None) }; + let Some(container) = self.cpp_expression_written(argument, source, r, depth + 1)? else { return Ok(None) }; + self.cpp_subscript_element(container, r) } - // `compact_->sub_compact_states`: a data member of the object's class. + // `compact_->sub_compact_states`: a data member of the object's class; `it->second` + // through a map's iterator: the member of the `std::pair`. "field_expression" => { - let Some((cls, field, None)) = self.cpp_member_of_expression(node, source, r, depth)? else { return Ok(None) }; + let Some((object, field, None)) = self.cpp_member_object_of(node, source, r, depth)? else { return Ok(None) }; + if let Some(element) = self.cpp_pair_member(&object, &field, r)? { return Ok(Some(element)); } + let Some(cls) = self.cpp_written_class(&object, r)? else { return Ok(None) }; self.cpp_data_member_written(&cls.qualified_name, &field, r) } // `new Foo(…)`: a pointer to what it constructs. @@ -556,7 +616,14 @@ impl KernelResolver { // `recv.method(…)`, `recv->method(…)`, `recv->method(…)`: the method of the // receiver's class. "field_expression" => { - let Some((cls, method, template_arguments)) = self.cpp_member_of_expression(function, source, r, depth)? else { return Ok(None) }; + let Some((object, method, template_arguments)) = self.cpp_member_object_of(function, source, r, depth)? else { return Ok(None) }; + if template_arguments.is_none() { + // `logs_.begin()`, `map.find(key)`: the iterator a library container hands out. + if let Some(iterator) = cpp_library_iterator_of(&object, &method) { return Ok(Some(iterator)); } + // `ws.front()`, `m.at(key)`, `p.get()`, `s.value()`: what it holds. + if let Some(element) = self.cpp_library_element_of(&object, &method, r)? { return Ok(Some(element)); } + } + let Some(cls) = self.cpp_written_class(&object, r)? else { return Ok(None) }; let methods = self.cpp_methods_named(&cls, &method, r)?; let Some(arguments) = template_arguments else { return self.cpp_return_type_of(&methods, r) }; // Explicit template arguments call a template, never an overload that is none. @@ -581,6 +648,11 @@ impl KernelResolver { } } let spelled: String = strip_template_arguments(callee).split_whitespace().collect(); + // `std::lower_bound(v.begin(), v.end(), key)`: the iterator it is given. + if re!(r"^(?:::)?std::(?:find|find_if|find_if_not|lower_bound|upper_bound|min_element|max_element|partition_point|adjacent_find|next|prev)$").is_match(&spelled) { + let first = node.child_by_field_name("arguments").and_then(|a| named_children(a).into_iter().find(|n| !n.kind().contains("comment"))); + return match first { Some(first) => self.cpp_expression_written(first, source, r, depth + 1), None => Ok(None) }; + } if !re!(r"^(?:[A-Za-z_]\w*::)*[A-Za-z_]\w*$").is_match(&spelled) { return Ok(None); } let Some(caller) = self.node_by_id(&r.from_node_id)? else { return Ok(None) }; // `NewIterator(…)` in a method, or in a gtest body: what the method of its @@ -616,18 +688,112 @@ impl KernelResolver { self.cpp_expression_written(node, source, r, 0) } - /// What `x[i]` is when `x` has the type `container`: what a pointer points at, or a - /// container's element. + /// What `x[i]` is when `x` has the type `container`: an array's element, what a pointer points + /// at, a library map's mapped value, a sequence's element (a library container's, or one of + /// PROJECT_SEQUENCES'), or what a class's `operator[]` returns. None for anything else: a + /// `std::string`'s `char`, or a class whose `operator[]` the index doesn't hold. pub(super) fn cpp_subscript_element(&mut self, container: CppWritten, r: &ResolveRefIn) -> Res> { let Some(followed) = self.cpp_follow_written_aliases(container, r)? else { return Ok(None) }; + if let Some(text) = cpp_array_element(&followed.text) { return Ok(Some(CppWritten { text, site: followed.site })); } if top_level_pointer(&followed.text) { return Ok(cpp_pointee(&followed.text).map(|text| CppWritten { text, site: followed.site })); } - self.cpp_range_element_of(followed, r, 0) + let plain = cpp_plain_type(&followed.text); + let Some(parts) = type_segments(&plain) else { return Ok(None) }; + let last = parts.last().map(String::as_str).unwrap_or(""); + let library = parts[0] == "std" || parts[0] == "absl"; + if library && LIBRARY_MAPS.contains(&last) { + return Ok(template_arguments(&plain).and_then(|a| a.into_iter().nth(1)).map(|value| CppWritten { text: cpp_plain_type(&value), site: followed.site })); + } + if (library && LIBRARY_SEQUENCES.contains(&last)) || PROJECT_SEQUENCES.contains(&last) { return self.cpp_range_element_of(followed, r, 0); } + if library { return Ok(None); } + let Some(cls) = self.cpp_written_class(&followed, r)? else { return Ok(None) }; + let subscript = self.cpp_methods_named(&cls, "operator[]", r)?; + Ok(self.cpp_return_type_of(&subscript, r)?.map(|t| CppWritten { text: t.text.trim_end_matches('&').trim().to_string(), site: t.site })) + } + + /// What dereferencing a value of the type `written` gives, for `*x` and `x->`: an array's first + /// element, what a pointer points at or a holder holds (`cpp_pointee`), or the element a + /// container's iterator points at (`cpp_iterator_target`): the `Node*` of `*nodes.begin()` + /// over a `std::vector`. + pub(super) fn cpp_dereferenced(&mut self, written: CppWritten, r: &ResolveRefIn) -> Res> { + if let Some(text) = cpp_array_element(&written.text) { return Ok(Some(CppWritten { text, site: written.site })); } + if let Some(text) = cpp_pointee(&written.text) { return Ok(Some(CppWritten { text, site: written.site })); } + self.cpp_iterator_target(written, r) } - /// The class of the object a member expression `x.m` or `x->m` names a member of, and `m`. - /// The class of the object a member expression `recv.name`, `recv->name` or `recv->name<…>` - /// reads from, with the member's name and its explicit template arguments (`<…>` as written). - fn cpp_member_of_expression(&mut self, member: TsNode, source: &Rc, r: &ResolveRefIn, depth: u32) -> Res> { + /// The element a container's iterator of the type `iterator` points at, read from the + /// container's template arguments before the `iterator` alias would lose them: `Slice` for + /// `std::deque::const_iterator`, protobuf's `SourceCodeInfo_Location` for + /// `RepeatedPtrField::iterator`. None for any other type, and for an + /// element a class template's own `begin()` or `operator*` writes with its template parameter + /// (googletest's `ParamGenerator::iterator`), which the specialization decides. + pub(super) fn cpp_iterator_target(&mut self, iterator: CppWritten, r: &ResolveRefIn) -> Res> { + let Some(container) = iterator_container(&iterator.text) else { return Ok(None) }; + let Some(element) = self.cpp_range_element_of(CppWritten { text: container, site: iterator.site.clone() }, r, 1)? else { return Ok(None) }; + if element.site.from_node_id != iterator.site.from_node_id && self.cpp_written_template_parameter(&element)? { return Ok(None); } + Ok(Some(element)) + } + + /// The member `first` or `second` of a `std::pair` of the type `pair`, as a map's iterator + /// points at one: its first or second element. None for any other member or type. + fn cpp_pair_member(&mut self, pair: &CppWritten, member: &str, r: &ResolveRefIn) -> Res> { + let index = match member { "first" => 0, "second" => 1, _ => return Ok(None) }; + let Some(followed) = self.cpp_follow_written_aliases(pair.clone(), r)? else { return Ok(None) }; + let plain = cpp_plain_type(&followed.text); + if top_level_pointer(&plain) || !type_segments(&plain).is_some_and(|parts| parts.len() == 2 && parts[0] == "std" && parts[1] == "pair") { return Ok(None); } + Ok(template_arguments(&plain).and_then(|a| a.into_iter().nth(index)).map(|text| CppWritten { text: cpp_plain_type(&text), site: followed.site })) + } + + /// What the member `member` of a library container or holder of the type `object` hands out, + /// where the container's type is written: an element of a sequence for `front()`, `back()` and + /// `at(i)` (of one of PROJECT_SEQUENCES' too), the value a map's `at(key)` maps to, the element + /// of a `std::stack`'s `top()` or a `std::queue`'s `front()` and `back()`, the pointer a smart + /// pointer's `get()` returns and the value an optional's or an `absl::StatusOr`'s `value()` + /// returns. rocksdb's `cfds.front()->GetName()` over an `autovector` calls + /// ColumnFamilyData's. None for any other member or type. + fn cpp_library_element_of(&mut self, object: &CppWritten, member: &str, r: &ResolveRefIn) -> Res> { + if top_level_pointer(&object.text) { return Ok(None); } + let Some(parts) = type_segments(&without_type_keywords(&object.text)) else { return Ok(None) }; + let last = parts.last().map(String::as_str).unwrap_or(""); + // (Its own `const` and `&` aside: a `const` in a template argument, `std::shared_ptr`, stays.) + let text = re!(r"(?:\s*\bconst\b)?\s*&+\s*$").replace(re!(r"^\s*(?:typename\s+)?const\s+").replace(&object.text, "").as_ref(), "").trim().to_string(); + let library = parts[0] == "std" || parts[0] == "absl"; + let element = |text: Option| text.map(|t| CppWritten { text: cpp_plain_type(&t), site: object.site.clone() }); + if matches!(member, "front" | "back" | "at") { + if (library && LIBRARY_SEQUENCES.contains(&last)) || PROJECT_SEQUENCES.contains(&last) { return self.cpp_range_element_of(object.clone(), r, 0); } + if member == "at" && library && LIBRARY_MAPS.contains(&last) { return Ok(element(template_arguments(&text).and_then(|a| a.into_iter().nth(1)))); } + } + if parts[0] == "std" && LIBRARY_ADAPTORS.contains(&last) && (member == "top" || (matches!(member, "front" | "back") && last == "queue")) { + return Ok(element(template_arguments(&text).and_then(|a| a.into_iter().next()))); + } + let spaced = text.split_whitespace().collect::>().join(" "); + let arguments = cpp_holder_arguments(&spaced); + let Some(held) = arguments.and_then(cpp_held_type) else { return Ok(None) }; + if held.contains('[') { return Ok(None); } + // (What it holds keeps its `const`: `get()` of a `std::shared_ptr` is a `const Message*`.) + let held = if arguments.is_some_and(|a| re!(r"^\s*const\s").is_match(a)) { format!("const {held}") } else { held }; + let holder = parts.join("::"); + if member == "get" && matches!(holder.as_str(), "std::unique_ptr" | "std::shared_ptr" | "absl::NoDestructor") { return Ok(Some(CppWritten { text: format!("{held}*"), site: object.site.clone() })); } + if matches!(member, "value" | "value_or") && matches!(holder.as_str(), "std::optional" | "absl::optional" | "absl::StatusOr") { return Ok(Some(CppWritten { text: held, site: object.site.clone() })); } + Ok(None) + } + + /// The object a C++ call through the operator `operator` (`.` or `->`) on the receiver + /// expression `receiver` is made on, as `cpp_member_object_of` reads it. + pub(super) fn cpp_call_object(&mut self, receiver: TsNode, operator: &str, source: &Rc, r: &ResolveRefIn) -> Res> { + let Some(object) = self.cpp_expression_written(receiver, source, r, 1)? else { return Ok(None) }; + let Some(object) = self.cpp_follow_written_aliases(object, r)? else { return Ok(None) }; + match operator { + "->" => self.cpp_dereferenced(object, r), + "." if !top_level_pointer(&object.text) && cpp_array_element(&object.text).is_none() => Ok(Some(object)), + _ => Ok(None), + } + } + + /// The type of the object a member expression `recv.name`, `recv->name` or `recv->name<…>` + /// reads from: `recv`'s own type for `.`, what it points at, holds or iterates over for `->` + /// (`cpp_dereferenced`), with the member's name and its explicit template arguments (`<…>` as + /// written). + fn cpp_member_object_of(&mut self, member: TsNode, source: &Rc, r: &ResolveRefIn, depth: u32) -> Res> { let (Some(argument), Some(mut field)) = (member.child_by_field_name("argument"), member.child_by_field_name("field")) else { return Ok(None) }; let mut template_arguments = None; if field.kind() == "template_method" { @@ -641,20 +807,20 @@ impl KernelResolver { let Some(object) = self.cpp_expression_written(argument, source, r, depth + 1)? else { return Ok(None) }; let Some(object) = self.cpp_follow_written_aliases(object, r)? else { return Ok(None) }; let object = match operator.trim() { - "->" => match cpp_pointee(&object.text) { - Some(text) => CppWritten { text, site: object.site }, + "->" => match self.cpp_dereferenced(object, r)? { + Some(object) => object, None => return Ok(None), }, - "." if !top_level_pointer(&object.text) => object, + "." if !top_level_pointer(&object.text) && cpp_array_element(&object.text).is_none() => object, _ => return Ok(None), }; - Ok(self.cpp_written_class(&object, r)?.map(|cls| (cls, name, template_arguments))) + Ok(Some((object, name, template_arguments))) } /// The names of the template parameters a C++ function template's own `template <…>` header /// declares, in order, without their defaults: `["Type"]` for protobuf's `template const Type& GetRaw(…)`. Empty for a function that is no template. - fn cpp_own_template_parameters(&mut self, function: &Arc, r: &ResolveRefIn) -> Res> { + pub(super) fn cpp_own_template_parameters(&mut self, function: &Arc, r: &ResolveRefIn) -> Res> { let Some(source) = self.read_file(&function.file_path) else { return Ok(Vec::new()) }; let site = site_at(r, function); let Some(tree) = self.parsed_tree(&source, &site) else { return Ok(Vec::new()) }; @@ -706,7 +872,8 @@ impl KernelResolver { while let Some(parent) = at.parent() { if let Some(bound) = self.cpp_structured_binding_written(parent, at, name, source, r)? { return Ok(bound); } if let Some(raw) = cpp_binding_type(parent, at, name, source) { - if raw.is_empty() || raw.contains('[') { return Ok(None); } + // (An array, `Status statuses[3]`, is written `Status[]`.) + if raw.is_empty() || cpp_array_split(&raw).is_none_or(|(base, dimensions)| dimensions > 0 && is_cpp_auto(base)) { return Ok(None); } if !is_cpp_auto(&raw) { return Ok(Some(CppWritten { text: raw, site: r.clone() })); } // `const auto& blob_files = vstorage->GetBlobFiles();`: what its initializer is. // (Read with a depth budget of its own: a chain of calls on the local, whose own @@ -781,6 +948,7 @@ impl KernelResolver { fn cpp_range_element_of(&mut self, range: CppWritten, r: &ResolveRefIn, hops: u32) -> Res> { if hops > 2 { return Ok(None); } let Some(range) = self.cpp_follow_written_aliases(range, r)? else { return Ok(None) }; + if let Some(text) = cpp_array_element(&range.text) { return Ok(Some(CppWritten { text, site: range.site })); } if top_level_pointer(&range.text) { return Ok(None); } let Some(parts) = type_segments(&range.text) else { return Ok(None) }; let last = parts.last().map(String::as_str).unwrap_or(""); @@ -809,9 +977,8 @@ impl KernelResolver { let Some(iterator) = self.cpp_follow_written_aliases(iterator, r)? else { return Ok(None) }; if top_level_pointer(&iterator.text) { return Ok(cpp_pointee(&iterator.text).map(|text| CppWritten { text, site: iterator.site })); } // `std::vector::const_iterator`: the container's element. - let bare = re!(r"\b(?:typename|const)\b").replace_all(&iterator.text, " ").replace('&', " "); - if let Some(nested) = re!(r"^(.*>)\s*::\s*(?:const_)?(?:reverse_)?iterator$").captures(bare.trim()) { - return self.cpp_range_element_of(CppWritten { text: nested[1].trim().to_string(), site: iterator.site.clone() }, r, hops + 1); + if let Some(container) = iterator_container(&iterator.text) { + return self.cpp_range_element_of(CppWritten { text: container, site: iterator.site.clone() }, r, hops + 1); } let Some(cls) = self.cpp_written_class(&iterator, r)? else { return Ok(None) }; let star = self.cpp_methods_named(&cls, "operator*", r)?; @@ -838,6 +1005,20 @@ impl KernelResolver { } } +/// The iterator the member `member` of a standard library or abseil container of the type `object` +/// returns, where the container's type is written: `std::deque::iterator` for +/// rocksdb's `logs_.begin()`, whose `->` reaches the container's element (`cpp_iterator_target`). +/// None for any other member or type. +fn cpp_library_iterator_of(object: &CppWritten, member: &str) -> Option { + let kind = ITERATOR_MEMBERS.iter().find(|(m, _)| *m == member)?.1; + if top_level_pointer(&object.text) { return None; } + let container = without_type_keywords(&object.text); + let parts = type_segments(&container)?; + let last = parts.last()?.as_str(); + if parts[0] != "std" && parts[0] != "absl" { return None; } + (LIBRARY_SEQUENCES.contains(&last) || LIBRARY_MAPS.contains(&last)).then(|| CppWritten { text: format!("{container}::{kind}"), site: object.site.clone() }) +} + /// Where `name` is in the structured binding `declarator` writes (`[cfh, iter]` in `auto& [cfh, /// iter]`, `auto [a, b] = …`), when it writes one. fn binding_place(mut declarator: TsNode, name: &str, source: &SourceFile) -> Option { @@ -885,3 +1066,14 @@ mod tests { assert_eq!(cpp_pointee("std::vector"), None); } } + +/// The return type a C or C++ function `declaration` writes, and its parameter list without +/// spaces (cpp_written_return_type). +fn cpp_declared_return(declaration: tree_sitter::Node, source: &SourceFile) -> Option<(String, String)> { + let (Some(ty), Some(declarator)) = (declaration.child_by_field_name("type"), declaration.child_by_field_name("declarator")) else { return None }; + let function_declarator = declared_function(declarator)?; + let raw = cpp_plain_type(&cpp_ast_declared_type(ty, declarator, source)); + if raw.is_empty() || raw == "void" || is_cpp_auto(&raw) || raw.contains("decltype") { return None; } + let parameters = function_declarator.child_by_field_name("parameters").map(|p| without_spaces(&source.text()[p.byte_range()])).unwrap_or_default(); + Some((raw, parameters)) +} diff --git a/codegraph-kernel/src/resolve/cpp_receivers.rs b/codegraph-kernel/src/resolve/cpp_receivers.rs index 81f3f00feb..f7ef7e23d5 100644 --- a/codegraph-kernel/src/resolve/cpp_receivers.rs +++ b/codegraph-kernel/src/resolve/cpp_receivers.rs @@ -92,9 +92,21 @@ impl KernelResolver { site = written.site; element = true; } - let Some(raw) = raw else { return Ok(None) }; + let Some(mut raw) = raw else { return Ok(None) }; if super::cpp_range_for::is_cpp_auto(&raw) { return Ok(None); } if raw.contains('[') { return Ok(Some(None)); } + // `it->data()` on a container's iterator (`std::deque::const_iterator it`, an `auto` + // local holding `logs_.begin()`): a member of the container's element, looked up where the + // container's type is written. One a class template's own `operator*` writes with its + // template parameter goes on below. + if operator.trim() == "->" { + if let Some(target) = self.cpp_iterator_target(super::cpp_range_for::CppWritten { text: raw.clone(), site: site.clone() }, r)? { + if super::cpp_range_for::top_level_pointer(&target.text) { return Ok(None); } + raw = format!("{}*", super::cpp_range_for::cpp_plain_type(&target.text)); + site = target.site; + element = true; + } + } let included = self.namespace_visible_files(&r.file_path, "cpp")?; if self.cpp_alias_expansion(&raw, &site, 0, &included)?.is_some_and(|alias| alias.pointer || (operator.trim() == "->" && alias.raw.as_deref().is_some_and(|ty| ty.ends_with("::iterator") || ty.ends_with("::const_iterator")))) { return Ok(None); } let mut depth = 0usize; @@ -123,7 +135,16 @@ impl KernelResolver { } _ => return Ok(Some(None)), }; - let primary = ty.split('<').next().unwrap_or(&ty).trim(); + // A library template's member type: its iterator is the library's, and its `value_type` + // (`std::vector::value_type key`) is its element. + let mut ty = ty; + if let Some(member) = cpp_library_member_type(&ty) { + let Some(held) = member else { return Ok(Some(None)) }; + ty = held; + element = true; + } + let primary = cpp_primary_type(&ty); + let primary = primary.as_str(); // A standard type's own member (`iter.reset()` on a `std::unique_ptr` element) is no project method. if element && re!(r"^(?:::)?(?:std|absl)::").is_match(primary) { return Ok(Some(None)); } if let Some(caller) = self.node_by_id(&site.from_node_id)? { if self.cpp_template_parameter(&caller, primary, &site)? { return Ok(Some(None)); } } @@ -141,14 +162,23 @@ impl KernelResolver { if owner.is_none() { owner = self.cpp_type_via_using(primary, &parts, &site, &included)?; } } } + // A class whose methods alone are indexed: protobuf's `descriptor.pb.h`, too large to index, + // declares `DescriptorProto`, and `descriptor.pb.cc` defines `DescriptorProto::Clear`. + if owner.is_none() { + if let Some(method) = self.cpp_unindexed_class_method(primary, method, &site)? { + return Ok(Some(Some(KCand { node: method, confidence: 0.85, resolved_by: "instance-method" }))); + } + } let Some(owner) = owner else { // A held type that is an alias of a library type holds no project class. let aliased_library = held && self.cpp_alias_expansion(primary, &site, 0, &included)? .is_some_and(|alias| alias.raw.as_deref().is_some_and(|ty| ty.trim_start_matches("::").starts_with("std::"))); return Ok(aliased_library.then_some(None)); }; - // Preserve the existing guarded alias and specialization fallback on a miss. - let member = self.match_bound_type_member(&owner.qualified_name, method, r)?; + // Preserve the existing guarded alias and specialization fallback on a miss. (The owner's + // qualified name is looked up from the global scope: inside `namespace hpb`, `upb::Arena` + // would find `hpb::upb::Arena`.) + let member = self.match_bound_type_member(&format!("::{}", owner.qualified_name), method, r)?; // A class that lacks the method, and whose bases lack it too, is called nothing (but a // class template's specialization may declare it, and so may a base the index doesn't link, // held in a smart pointer or not). @@ -158,6 +188,31 @@ impl KernelResolver { Ok(member.map(Some)) } + /// The method `method` of the class `ty` names where only the class's methods are indexed: + /// looked up as `ty` is, from the scopes around `site` innermost first, and only where no + /// class, struct, union or alias of that name is indexed at all. + fn cpp_unindexed_class_method(&mut self, ty: &str, method: &str, site: &ResolveRefIn) -> Res>> { + let Some(parts) = super::cpp_aliases::type_segments(ty) else { return Ok(None) }; + let global = ty.trim_start().starts_with("::"); + let ty = parts.join("::"); + let scopes: Vec = if global { Vec::new() } else { + self.node_by_id(&site.from_node_id)?.map(|n| n.qualified_name.split("::").map(str::to_string).collect()).unwrap_or_default() + }; + let mut names: Vec = (1..scopes.len()).rev().map(|i| format!("{}::{ty}", scopes[..i].join("::"))).collect(); + names.push(ty.clone()); + for owner in names { + if self.nodes_by_qualified_name(&owner)?.iter().any(|n| matches!(n.language.as_str(), "c" | "cpp")) { return Ok(None); } + let methods = self.nodes_by_qualified_name(&format!("{owner}::{method}"))?; + let methods: Vec<_> = methods.iter().filter(|n| n.language == "cpp" && n.kind == "method").cloned().collect(); + match methods.as_slice() { + [] => continue, + [only] => return Ok(Some(only.clone())), + _ => return Ok(None), + } + } + Ok(None) + } + /// A class `ty` names from inside a class that declares or inherits it, which /// C++ looks up before the namespaces around: the scopes of the caller, innermost /// first, each with the classes it derives from. @@ -204,7 +259,14 @@ impl KernelResolver { let children = named_children(shape); let [only] = children.as_slice() else { return Ok(None) }; shape = *only; } - if !matches!(shape.kind(), "subscript_expression" | "field_expression") { return Ok(None); } + // `(*it)->Visit()`, `(*node).Weight()`; not `(*this)`, which is implicit this's. + let dereference = shape.kind() == "pointer_expression" && shape.child_by_field_name("operator").is_some_and(|o| o.kind() == "*") + && shape.child_by_field_name("argument").is_some_and(|a| a.kind() != "this"); + if !dereference && !matches!(shape.kind(), "subscript_expression" | "field_expression") { return Ok(None); } + let operator = super::awaited::strip_ts_comments(&source.text()[receiver.end_byte()..field.map_or(receiver.end_byte(), |f| f.start_byte())]); + if let Some(found) = self.cpp_unspelled_receiver_call(receiver, operator.trim(), &source, r)? { return Ok(Some(found)); } + // A dereference whose type can't be read goes on to the name strategies, as before. + if dereference { return Ok(None); } let Some(owner) = self.cpp_expression_owner(receiver, &source, r, 0)? else { return Ok(Some(None)) }; let methods = self.nodes_by_qualified_name(&format!("{}::{}", owner.qualified_name, r.reference_name))?; let methods: Vec<_> = methods.iter().filter(|n| n.kind == "method" && n.file_path == owner.file_path).collect(); @@ -219,6 +281,36 @@ impl KernelResolver { Ok(None) } + /// A C++ call the extractor records by its bare name on a receiver it drops: `(*it)->Visit()` + /// over a `std::vector`, `(*node).Weight()` on a `Node*`, `it->second.Flush()` through + /// a map's iterator, `statuses[i].ok()` on an array. It calls a member of the class that + /// receiver has, read as written (`cpp_call_object`) and looked up as C++ looks it up there; + /// by its bare name the call went to whichever class had a method of that name. + /// - `Some(Some(_))`: the member of that class, or of a class it derives from. + /// - `Some(None)`: no project method is called: a library type's (`(*it).size()` over a + /// `std::vector`), the calling function's template parameter's, or a class that + /// surely lacks the method. + /// - `None`: the type can't be read or placed. + fn cpp_unspelled_receiver_call(&mut self, receiver: tree_sitter::Node, operator: &str, source: &Rc, r: &ResolveRefIn) -> Res>> { + let Some(object) = self.cpp_call_object(receiver, operator, source, r)? else { return Ok(None) }; + if super::cpp_range_for::top_level_pointer(&object.text) { return Ok(None); } + let plain = super::cpp_range_for::cpp_plain_type(&object.text); + if let Some(parts) = super::cpp_aliases::type_segments(&plain) { + let last = parts.last().map(String::as_str).unwrap_or(""); + if parts[0] == "std" || (parts[0] == "absl" && last.starts_with(|c: char| c.is_ascii_lowercase())) { return Ok(Some(None)); } + } + let Some(cls) = self.cpp_written_class(&object, r)? else { + // A template parameter's type names no particular class when the calling function + // writes it; one another template writes is what the call's specialization makes it. + if self.cpp_written_template_parameter(&object)? { return Ok((object.site.from_node_id == r.from_node_id).then_some(None)); } + return Ok(self.cpp_outside_qualified_type(&object.text, &object.site)?.then_some(None)); + }; + if let Some(member) = self.match_bound_type_member(&format!("::{}", cls.qualified_name), &r.reference_name, r)? { return Ok(Some(Some(member))); } + if self.supertypes_complete && !self.cpp_alias_template_owner(&cls, r)? && self.cpp_methods_named(&cls, &r.reference_name, r)?.is_empty() + && self.cpp_bases_known(&cls, 0, &mut HashSet::new())? { return Ok(Some(None)); } + Ok(None) + } + fn cpp_expression_owner(&mut self, node: tree_sitter::Node, source: &SourceFile, r: &ResolveRefIn, depth: u32) -> Res>> { if depth > 4 { return Ok(None); } let text = |n: tree_sitter::Node| &source.text()[n.start_byte()..n.end_byte()]; @@ -230,7 +322,8 @@ impl KernelResolver { if let Some(raw) = cpp_binding_type(parent, at, text(node), source) { let ty = if raw == "auto" && parent.kind() == "compound_statement" { self.infer_cpp_receiver_type(text(node), r, 0, true)? } else { Some(raw) }; let Some(ty) = ty else { return Ok(None) }; - let primary = ty.split('<').next().unwrap_or(&ty).trim(); + let primary = cpp_primary_type(&ty); + let primary = primary.as_str(); if let Some(caller) = self.node_by_id(&r.from_node_id)? { if self.cpp_template_parameter(&caller, primary, r)? { return Ok(None); } } return self.cpp_type_owner(primary, r, 0, false); } @@ -312,7 +405,8 @@ impl KernelResolver { fn cpp_field_owner(&mut self, owner: &Arc, field: &str, r: &ResolveRefIn) -> Res>> { let Some(ty) = self.cpp_field_type(owner, field, r)? else { return Ok(None) }; - let primary = ty.split('<').next().unwrap_or(&ty).trim(); + let primary = cpp_primary_type(&ty); + let primary = primary.as_str(); let mut site = r.clone().at(owner); site.from_node_id = owner.id.clone(); site.column = owner.start_column; if self.cpp_template_parameter(owner, primary, &site)? { return Ok(None); } let included = self.namespace_visible_files(&r.file_path, "cpp")?; @@ -415,6 +509,52 @@ fn cpp_element_type(raw: &str) -> Option { /// The first template argument of a standard smart pointer or optional, or None when it /// is a pointer: `Foo`, `ns::Foo`, `const Foo` for `Foo, Deleter`. +/// The type `ty` names as its owner is looked up: `SkipList` for `SkipList`, +/// and `SkipList::Iterator` for the member type `SkipList::Iterator`, which the +/// class template declares whatever its arguments. +pub(super) fn cpp_primary_type(ty: &str) -> String { + let mut depth = 0usize; + let mut member = false; + for (i, ch) in ty.char_indices() { + match ch { + '<' => depth += 1, + '>' => { depth = depth.saturating_sub(1); if depth == 0 && ty[i + 1..].trim_start().starts_with("::") { member = true; } } + _ => (), + } + } + if member { + if let Some(parts) = super::cpp_aliases::type_segments(ty) { + let global = re!(r"^(?:(?:typename|const)\s+)*::").is_match(ty.trim()); + return format!("{}{}", if global { "::" } else { "" }, parts.join("::")); + } + } + ty.split('<').next().unwrap_or(ty).trim().to_string() +} + +/// What a library template's member type is: Some(its element) for a container's `value_type`, +/// `reference` or a smart pointer's `element_type` (the first template argument), a map's +/// `mapped_type` (the second); Some(None) for any other member, its `iterator` included, which is +/// the library's, and for a map's `value_type`, a `std::pair` of the key and the value. None for a +/// type that is no library template's member. +fn cpp_library_member_type(ty: &str) -> Option> { + let hit = re!(r"^(?:(?:typename|const)\s+)*(?:::)?(?:std|absl)::(\w+)\s*<(.*)>\s*::\s*([A-Za-z_]\w*)\s*&?$").captures(ty.trim())?; + let arguments = &hit[2]; + let map = super::cpp_range_for::LIBRARY_MAPS.contains(&&hit[1]); + Some(match &hit[3] { + "value_type" | "reference" | "const_reference" if map => None, + "value_type" | "reference" | "const_reference" | "element_type" => cpp_held_type(arguments), + "mapped_type" => { + let mut depth = 0usize; + let comma = arguments.char_indices().find(|&(_, ch)| { + match ch { '<' | '(' => depth += 1, '>' | ')' => depth = depth.saturating_sub(1), _ => () } + ch == ',' && depth == 0 + }); + comma.and_then(|(at, _)| cpp_held_type(&arguments[at + 1..])) + } + _ => None, + }) +} + pub(super) fn cpp_held_type(arguments: &str) -> Option { let mut depth = 0usize; let mut end = arguments.len(); @@ -497,6 +637,40 @@ pub(super) fn cpp_binding_type(parent: tree_sitter::Node, at: tree_sitter::Node, None } +/// Whether the declaration of the name `node` in scope writes `const` (or `constexpr`) in front of +/// its type: `const Buffer* b`, `constexpr char name[] = "name"`, which the declared type +/// (`cpp_ast_declared_type`) leaves out. +pub(super) fn cpp_declared_const(node: tree_sitter::Node, source: &SourceFile) -> bool { + let name = &source.text()[node.byte_range()]; + let leading = |declaration: tree_sitter::Node| declaration.child_by_field_name("type") + .is_some_and(|ty| re!(r"\b(?:const|constexpr)\b").is_match(&source.text()[declaration.start_byte()..ty.start_byte()])); + let mut at = node; + while let Some(parent) = at.parent() { + let block = matches!(parent.kind(), "compound_statement" | "declaration_list" | "translation_unit"); + let initializer = super::cpp_range_for::statement_initializer(parent); + if block || initializer.is_some() { + let children = if block { named_children(parent) } else { initializer.into_iter().collect() }; + for declaration in children.into_iter().rev().filter(|n| n.kind() == "declaration" && n.end_byte() <= at.start_byte()) { + let mut cursor = declaration.walk(); + if declaration.children(&mut cursor).enumerate().any(|(i, child)| declaration.field_name_for_child(i as u32) == Some("declarator") && cpp_declarator_binds(child, name, source)) { + return leading(declaration); + } + } + } + if parent.kind() == "for_range_loop" && parent.child_by_field_name("body").is_some_and(|body| body.id() == at.id()) + && parent.child_by_field_name("declarator").is_some_and(|d| cpp_declarator_binds(d, name, source)) { return leading(parent); } + if matches!(parent.kind(), "lambda_expression" | "function_definition") { + let mut queue = named_children(parent).into_iter().filter(|n| n.kind() != "compound_statement").collect::>(); + while let Some(parameter) = queue.pop() { + if parameter.kind() == "parameter_declaration" && parameter.child_by_field_name("declarator").is_some_and(|n| cpp_declarator_binds(n, name, source)) { return leading(parameter); } + if parameter.kind() != "compound_statement" { queue.extend(named_children(parameter)); } + } + } + at = parent; + } + false +} + pub(super) fn cpp_ast_declared_type(ty: tree_sitter::Node, mut declarator: tree_sitter::Node, source: &SourceFile) -> String { let type_node = if matches!(ty.kind(), "class_specifier" | "struct_specifier" | "union_specifier" | "enum_specifier") { ty.child_by_field_name("name").unwrap_or_else(|| { diff --git a/codegraph-kernel/src/resolve/cpp_specializations.rs b/codegraph-kernel/src/resolve/cpp_specializations.rs new file mode 100644 index 0000000000..e079ed72e7 --- /dev/null +++ b/codegraph-kernel/src/resolve/cpp_specializations.rs @@ -0,0 +1,171 @@ +//! C++ explicit specializations (name-matcher.ts cppSpecializationPool, upstream #2609). An +//! explicit specialization is named like the template it specializes, so a call meets both among +//! that name's candidates; the template arguments the call writes choose between them. The +//! TypeScript synthesizers read the same arguments (src/resolution/cpp-specializations.ts). + +use super::*; + +impl KernelResolver { + /// Of the C++ functions `candidates` a call may reach, the ones the template arguments it + /// writes allow: + /// - a call that writes template arguments is to the specialization written with the same + /// ones (rocksdb's `Deserialize(src, offset)` is `template <> int64_t + /// Deserialize(…)`), else to the template itself or an ordinary overload: fmt's + /// `num_bits()` is `num_bits()`, not the `num_bits()` beside the + /// call. None is left when specializations are all there is: protobuf's template + /// `ReadPrimitive(…)` calls whichever one its arguments become. + /// - a call that writes none, or another reference, reaches a specialization only when there + /// is nothing else. + /// + /// The candidates as they are when none is a specialization. + pub(super) fn cpp_specialization_pool(&mut self, candidates: Vec>, r: &ResolveRefIn) -> Vec> { + if r.language != "cpp" { return candidates; } + let written: Vec>>> = candidates.iter().map(|n| self.cpp_specialization_arguments(n)).collect(); + if written.iter().all(Option::is_none) { return candidates; } + let others: Vec> = candidates.iter().zip(&written).filter(|(_, w)| w.is_none()).map(|(n, _)| n.clone()).collect(); + let Some(args) = self.cpp_call_template_arguments(r) else { return if others.is_empty() { candidates } else { others } }; + let named: Vec> = candidates.iter().zip(&written) + .filter(|(_, w)| w.as_ref().is_some_and(|w| cpp_template_arguments_match(w, &args))) + .map(|(n, _)| n.clone()).collect(); + if named.is_empty() { others } else { named } + } + + /// The template arguments the C++ function or method `n` is an explicit specialization for, + /// as its definition writes them after its name: `int32_t` and `WireFormatLite::TYPE_INT32` for + /// protobuf's `template <> inline bool WireFormatLite::ReadPrimitive(…)`. A method a macro defines is read from its expansion. None + /// for a template's own definition, any other function, and a specialization that leaves its + /// arguments to be deduced. + pub(super) fn cpp_specialization_arguments(&mut self, n: &KNode) -> Option>> { + // A constructor, whose signature is its parameter list, is never specialized by name: + // gmock's `template Action(const Action& action)` writes + // `Action` for a parameter. + if n.language != "cpp" || (n.kind != "function" && n.kind != "method") || n.signature.as_deref().is_some_and(|s| s.starts_with('(')) { return None; } + if let Some(hit) = self.cpp_specializations.get(&n.id) { return hit.clone(); } + // (A method a macro defines carries its expansion as its signature.) + let head = match n.signature.as_deref().filter(|s| !s.is_empty()) { + Some(signature) => signature.to_string(), + None => self.read_file(&n.file_path).map(|source| { + let from = (n.start_line - 1).max(0) as usize; + let text = source.iter().skip(from).take(12).cloned().collect::>().join("\n"); + let column = source.get(from).map_or(0, |first| super::names::js_unit_to_byte(first, n.start_column.max(0) as usize)); + text.get(column..).unwrap_or("").to_string() + }).unwrap_or_default(), + }; + let args = cpp_template_arguments_after(&head, &n.name, 0).map(Rc::new); + self.cpp_specializations.insert(n.id.clone(), args.clone()); + args + } + + /// The template arguments the C++ call `r` writes after the name it calls (`unsigned short` + /// for fmt's `num_bits()`), or None when it writes none, leaving them to be + /// deduced. + fn cpp_call_template_arguments(&mut self, r: &ResolveRefIn) -> Option> { + let source = self.read_file(&r.file_path)?; + let from = (r.line - 1).max(0) as usize; + let text = source.iter().skip(from).take(4).cloned().collect::>().join("\n"); + let column = source.get(from).map_or(0, |first| super::names::js_unit_to_byte(first, r.column.max(0) as usize)); + let name = re!(r"::|\.|->").split(&r.reference_name).last().unwrap_or(""); + cpp_template_arguments_after(&text, name, column) + } +} + +/// The template arguments `text` writes right after `name`, the first time from `from` on it is +/// written as a word of its own, as they compare (cpp_template_argument_key). None when it writes +/// none there, or `<>`. +fn cpp_template_arguments_after(text: &str, name: &str, from: usize) -> Option> { + if name.is_empty() { return None; } + let bytes = text.as_bytes(); + let word = |at: Option<&u8>| at.is_some_and(|&b| b.is_ascii_alphanumeric() || b == b'_' || b == b'$'); + let mut at = text.get(from..).and_then(|rest| rest.find(name)).map(|i| i + from); + while let Some(start) = at { + let open = start + name.len(); + if !word(start.checked_sub(1).and_then(|i| bytes.get(i))) && !word(bytes.get(open)) { + let end = if bytes.get(open) == Some(&b'<') { cpp_template_arguments_end(bytes, open) } else { None }; + let args: Vec = end.map(|end| split_cpp_top_level(&text[open + 1..end]).iter().map(|a| cpp_template_argument_key(a)).collect()).unwrap_or_default(); + return (!args.is_empty()).then_some(args); + } + at = text.get(start + 1..).and_then(|rest| rest.find(name)).map(|i| i + start + 1); + } + None +} + +/// A C++ template argument as it compares: spaced only between words. +fn cpp_template_argument_key(arg: &str) -> String { + let tight = re!(r"\s*([^\w\s])\s*").replace_all(arg, "$1"); + re!(r"\s+").replace_all(&tight, " ").into_owned() +} + +/// Whether a call's template arguments `call` are the ones a specialization writes, `written`: +/// each the same, or the same name qualified further on one side (`TYPE_INT64` inside the class, +/// `WireFormatLite::TYPE_INT64` outside it). +fn cpp_template_arguments_match(written: &[String], call: &[String]) -> bool { + written.len() == call.len() && written.iter().zip(call).all(|(w, c)| w == c || w.ends_with(&format!("::{c}")) || c.ends_with(&format!("::{w}"))) +} + +/// Split `a, f(b, c), d` at its top-level commas. +fn split_cpp_top_level(text: &str) -> Vec { + let bytes = text.as_bytes(); + let mut out = Vec::new(); + let mut depth = 0usize; + // Where each template argument list open around the current character closes. + let mut closes: Vec = Vec::new(); + let mut start = 0; + let mut i = 0; + while i < bytes.len() { + let ch = bytes[i]; + if ch == b'"' || ch == b'\'' { + let mut j = i + 1; + while j < bytes.len() && bytes[j] != ch { j += if bytes[j] == b'\\' { 2 } else { 1 }; } + i = j + 1; + continue; + } + match ch { + b'(' | b'[' | b'{' => depth += 1, + b')' | b']' | b'}' => depth = depth.saturating_sub(1), + b'<' => { if let Some(end) = cpp_template_arguments_end(bytes, i).filter(|&end| end > i) { closes.push(end); } } + b'>' if closes.last() == Some(&i) => { closes.pop(); } + b',' if depth == 0 && closes.is_empty() => { + out.push(text[start..i].trim().to_string()); + start = i + 1; + } + _ => {} + } + i += 1; + } + let rest = text.get(start..).unwrap_or("").trim(); + if !rest.is_empty() { out.push(rest.to_string()); } + out +} + +/// Where the template argument list the `<` at `at` opens closes: the index of its `>`, or None +/// when the `<` compares or shifts. +fn cpp_template_arguments_end(bytes: &[u8], at: usize) -> Option { + let before = at.checked_sub(1).and_then(|i| bytes.get(i)); + if !before.is_some_and(|b| b.is_ascii_alphanumeric() || *b == b'_') || matches!(bytes.get(at + 1), Some(b'<') | Some(b'=')) { return None; } + let mut angles = 0usize; + let mut others = 0usize; + let mut i = at; + while i < bytes.len() { + let ch = bytes[i]; + if ch == b'"' || ch == b'\'' { + let mut j = i + 1; + while j < bytes.len() && bytes[j] != ch { j += if bytes[j] == b'\\' { 2 } else { 1 }; } + i = j; + } else if matches!(ch, b'(' | b'[' | b'{') { + others += 1; + } else if matches!(ch, b')' | b']' | b'}') { + if others == 0 { return None; } + others -= 1; + } else if others == 0 { + if ch == b'<' { + angles += 1; + } else if ch == b'>' && (i == 0 || bytes[i - 1] != b'-') { + angles = angles.saturating_sub(1); + if angles == 0 { return Some(i); } + } + } + i += 1; + } + None +} diff --git a/codegraph-kernel/src/resolve/cpp_types.rs b/codegraph-kernel/src/resolve/cpp_types.rs index 30e6f856fa..0f5f9be8ed 100644 --- a/codegraph-kernel/src/resolve/cpp_types.rs +++ b/codegraph-kernel/src/resolve/cpp_types.rs @@ -134,6 +134,20 @@ impl KernelResolver { if let Some(found) = self.cpp_type_through_bases(raw, r, included)? { return self.cpp_expand_owner(&found, r, depth, constructor, included); } + // A name written through a macro that stands for a namespace, or the other way round. + if ty.contains("::") { + if let Some(spelled) = self.cpp_name_macro_spelling(&r.file_path, ty)? { + let raw = if raw.trim_start().starts_with("::") { format!("::{spelled}") } else { spelled.clone() }; + if let [owner] = self.cpp_scoped_types(&raw, &spelled, r, included)?.as_slice() { return self.cpp_expand_owner(owner, r, depth, constructor, included); } + } + } + // A name written through a namespace alias another file declares: protobuf hpb's + // `backend::Arena` under `namespace backend = hpb::internal::backend::upb;`. + if let Some((head, rest)) = ty.split_once("::").filter(|_| !raw.trim_start().starts_with("::")) { + if let Some(target) = self.cpp_namespace_alias(&r.file_path, head)? { + if let Some(found) = self.cpp_type_owner_visible(&format!("{target}::{rest}"), r, depth + 1, constructor, included)? { return Ok(Some(found)); } + } + } // Namespace-opening macros can leave a header's indexed owner unqualified. if ty.contains("::") { let named = self.nodes_by_name(simple)?; diff --git a/codegraph-kernel/src/resolve/declarations_upstream.rs b/codegraph-kernel/src/resolve/declarations_upstream.rs index 41a69e7f4d..69c2181fc1 100644 --- a/codegraph-kernel/src/resolve/declarations_upstream.rs +++ b/codegraph-kernel/src/resolve/declarations_upstream.rs @@ -432,6 +432,23 @@ impl KernelResolver { } fn cpp_parameter_facts(&mut self, n: &KNode) -> Option { + // A method a macro defines writes its list in its expansion. + if let Some(head) = super::cpp_overloads::cpp_macro_method_head(n) { + let (source, tree) = super::cpp_overloads::cpp_macro_method_tree(head)?; + let mut queue = vec![tree.root_node()]; + while let Some(node) = queue.pop() { + if node.kind() == "function_declarator" && cpp_function_identity(node, &source).is_some_and(|(name, _)| name == n.name) { + let params = node.child_by_field_name("parameters").filter(|p| cpp_parameter_shape_complete(*p))?; + let mut facts = cpp_parameters(node, params, &source, &n.file_path, &[], &[]); + // The expansion is parsed alone, so it knows only the qualifiers it writes; the + // extractor composed the enclosing namespaces into the qualified name. + facts.owner = n.qualified_name.rsplit_once("::").map(|(owner, _)| owner.split("::").map(str::to_string).collect()).unwrap_or_default(); + return Some(facts); + } + queue.extend(super::iteration::named_children(node)); + } + return None; + } let source = self.read_file(&n.file_path)?; if let Some(cached) = self.cpp_declaration_memo.get(&n.id) { if Rc::ptr_eq(&cached.source, &source) { diff --git a/codegraph-kernel/src/resolve/mod.rs b/codegraph-kernel/src/resolve/mod.rs index 42b9c478fa..d467890d13 100644 --- a/codegraph-kernel/src/resolve/mod.rs +++ b/codegraph-kernel/src/resolve/mod.rs @@ -586,6 +586,8 @@ mod cpp_namespace; mod cpp_annotations; mod cpp_construction; mod cpp_overloads; +mod c_feature_variants; +mod cpp_specializations; mod cpp_translation_unit; mod inherited_method; mod store; @@ -755,6 +757,10 @@ pub struct KernelResolver { /// C/C++ directive summaries and per-root macro timelines (#1838). cpp_macros: cpp::MacroCache, cpp_namespaces: cpp_namespace::NamespaceCache, + /// C definitions only an unset build flag compiles (c_feature_variants). + c_variants: c_feature_variants::VariantCache, + /// A C++ function's id → the template arguments it is an explicit specialization for. + cpp_specializations: HashMap>>>, inherited_class_methods: HashMap<(String,String), Option>>, /// A Python class and its in-repo ancestors, by class id. python_hierarchies: HashMap>>>, @@ -938,6 +944,8 @@ impl KernelResolver { factory_init_memo: HashMap::new(), cpp_macros: cpp::MacroCache::default(), cpp_namespaces: cpp_namespace::NamespaceCache::default(), + c_variants: c_feature_variants::VariantCache::default(), + cpp_specializations: HashMap::new(), inherited_class_methods: HashMap::new(), python_hierarchies: HashMap::new(), python_rebinds: HashMap::new(), diff --git a/codegraph-kernel/src/resolve/names.rs b/codegraph-kernel/src/resolve/names.rs index c62d80002c..8974232172 100644 --- a/codegraph-kernel/src/resolve/names.rs +++ b/codegraph-kernel/src/resolve/names.rs @@ -7,6 +7,12 @@ use super::name_scope::{can_name_in_type_position, is_bare_rust_name}; /// no repository class uses may claim: a guess, not a proven link. const PHP_UNSEEN_ANCESTOR_CONFIDENCE: f64 = 0.5; +/// Whether `nodes` are one C++ function's overloads (and declarations): functions all of one +/// qualified name, which a call's arguments choose among (cpp_function_overload). +fn cpp_overload_set(nodes: &[Arc]) -> bool { + nodes.first().is_some_and(|first| nodes.iter().all(|n| n.kind == "function" && n.language == "cpp" && n.qualified_name == first.qualified_name)) +} + impl KernelResolver { // ----------------------------------------------------------------------- // Name machinery (name-matcher.ts) @@ -316,7 +322,10 @@ impl KernelResolver { || !self.is_static_c_function(candidate)? || (matches!(r.language.as_str(), "c" | "cpp") && (self.c_includes(&r.file_path, &candidate.file_path)? - || self.c_sole_includer(candidate, &r.file_path)?)); + || self.c_sole_includer(candidate, &r.file_path)? + // Or by any path its include writes: hiredis's async.c includes + // `deps/hiredis/dict.c`. + || self.namespace_visible_files(&r.file_path, &r.language)?.contains(&candidate.file_path))); // So is what a source file's unnamed namespace declares, and a type only source files // declare, with its members (cpp_translation_unit). return Ok(static_seen && !self.cpp_translation_unit_local(candidate, r)?); @@ -822,6 +831,7 @@ impl KernelResolver { // A bare Dart call means the nearest member of the hierarchy around it. let candidates = self.nearest_dart_members(candidates, r)?; let candidates = self.nearest_swift_members(candidates, r)?; + let candidates = self.cpp_specialization_pool(candidates, r); // C/C++ call-site form. let Some(cpp_form) = self.apply_cpp_call_site_form(r, candidates)? else { @@ -858,6 +868,7 @@ impl KernelResolver { let Some(best) = self.find_best_match(r, &candidates) else { return Ok(None); }; + let best = self.cpp_function_overload(best, &candidates, r)?; if bare_js && !is_bare_call_target_kind(&best.kind) { return Ok(None); } @@ -970,6 +981,7 @@ impl KernelResolver { } else { same_language }; + let final_candidates = self.cpp_specialization_pool(final_candidates, r); if final_candidates.len() == 1 { let only = final_candidates[0].clone(); let reachable = self.is_lexically_reachable(&only, r)? @@ -1370,9 +1382,11 @@ impl KernelResolver { *start <= node.start_line && node.start_line <= *end) { continue; } exact.push(node); } + let exact = self.cpp_specialization_pool(exact, r); let exact = prefer_call_site_file(exact, &r.file_path); - return Ok(exact.first().filter(|node| exact.len() == 1 || node.file_path == r.file_path) - .map(|node| KCand { node: node.clone(), confidence: 0.95, resolved_by: "qualified-name" })); + let Some(first) = exact.first().filter(|node| exact.len() == 1 || node.file_path == r.file_path || cpp_overload_set(&exact)).cloned() else { return Ok(None) }; + let node = self.cpp_function_overload(first, &exact, r)?; + return Ok(Some(KCand { node, confidence: 0.95, resolved_by: "qualified-name" })); } // Relative C++ qualification searches the caller's enclosing scopes @@ -1386,9 +1400,11 @@ impl KernelResolver { let named = keep_for_ref(&self.nodes_by_qualified_name(&qualified)?); let candidates: Vec<_> = named.into_iter().filter(|n| matches!(n.language.as_str(), "c" | "cpp") && visible.contains(&n.file_path)).collect(); if candidates.is_empty() { continue; } + let candidates = self.cpp_specialization_pool(candidates, r); let candidates = prefer_call_site_file(candidates, &r.file_path); - return Ok(candidates.first().filter(|n| candidates.len() == 1 || n.file_path == r.file_path) - .map(|node| KCand { node: node.clone(), confidence: 0.95, resolved_by: "qualified-name" })); + let Some(first) = candidates.first().filter(|n| candidates.len() == 1 || n.file_path == r.file_path || cpp_overload_set(&candidates)).cloned() else { return Ok(None) }; + let node = self.cpp_function_overload(first, &candidates, r)?; + return Ok(Some(KCand { node, confidence: 0.95, resolved_by: "qualified-name" })); } } @@ -1412,6 +1428,7 @@ impl KernelResolver { } found } else { keep_for_ref(&self.nodes_by_qualified_name(&r.reference_name)?) }; + let candidates = self.cpp_specialization_pool(candidates, r); if candidates.len() == 1 { return Ok(Some(KCand { node: candidates[0].clone(), @@ -1423,7 +1440,7 @@ impl KernelResolver { let ordered = prefer_call_site_file(candidates, &r.file_path); if ordered[0].file_path == r.file_path { return Ok(Some(KCand { - node: ordered[0].clone(), + node: self.cpp_function_overload(ordered[0].clone(), &ordered, r)?, confidence: 0.95, resolved_by: "qualified-name", })); @@ -1467,10 +1484,11 @@ impl KernelResolver { .into_iter() .filter(|n| n.qualified_name.ends_with(&r.reference_name)) .collect(); + let partial = self.cpp_specialization_pool(partial, r); let chosen = prefer_call_site_file(partial, &r.file_path); - if let Some(first) = chosen.into_iter().next() { + if let Some(first) = chosen.first().cloned() { return Ok(Some(KCand { - node: first, + node: self.cpp_function_overload(first, &chosen, r)?, confidence: 0.85, resolved_by: "qualified-name", })); diff --git a/src/extraction/languages/c-cpp.ts b/src/extraction/languages/c-cpp.ts index 45d6018152..33a7d75d93 100644 --- a/src/extraction/languages/c-cpp.ts +++ b/src/extraction/languages/c-cpp.ts @@ -218,6 +218,29 @@ function cppWrittenClassParts(node: SyntaxNode, source: string): string[] | unde return parts.length > 0 && parts.every((p) => /^[A-Za-z_]\w*$/.test(p)) ? parts : undefined; } +/** + * The name a C++ `qualified_identifier` ends in, past its scopes: the + * `template_function` `ReadPrimitive` for + * `WireFormatLite::ReadPrimitive`. Null + * when the tree lacks one. + */ +function cppQualifiedIdName(qid: SyntaxNode): SyntaxNode | null { + let name: SyntaxNode | null = qid; + while (name?.type === 'qualified_identifier') name = getChildByField(name, 'name'); + return name; +} + +/** + * A C++ definition's name node, as named: an explicit specialization's + * (`template_function`) without its template arguments, so + * `ReadPrimitive` is `ReadPrimitive`, + * the name every call to it writes. Mirrored in the kernel (ccpp/mod.rs). + */ +function cppDefinedName(name: SyntaxNode, source: string): string { + const own = name.type === 'template_function' ? getChildByField(name, 'name') : null; + return getNodeText(own ?? name, source).trim(); +} + function extractCppQualifiedMethodName(node: SyntaxNode, source: string): string | undefined { const macroDefined = recoverSingleArgMacroDefinedName(node, source) ?? recoverCppMacroDefinedName(node, source); if (macroDefined) return macroDefined; @@ -225,24 +248,53 @@ function extractCppQualifiedMethodName(node: SyntaxNode, source: string): string if (!declarator) return undefined; const qid = findDeclaratorQualifiedId(declarator); if (!qid) return undefined; + const name = cppQualifiedIdName(qid); + if (name) return cppDefinedName(name, source); const parts = getNodeText(qid, source).trim().split('::').filter(Boolean); return parts[parts.length - 1]; } +/** + * The name of a C++ function an explicit specialization defines, without its + * template arguments: `Swap` for `template <> void Swap(Foo& a, Foo& b)`. + * Undefined for any other definition. Mirrored in the kernel (ccpp/mod.rs). + */ +function cppSpecializedFunctionName(node: SyntaxNode, source: string): string | undefined { + if (node.type !== 'function_definition') return undefined; + let declarator = getChildByField(node, 'declarator'); + while (declarator && (declarator.type === 'pointer_declarator' || declarator.type === 'reference_declarator')) { + declarator = getChildByField(declarator, 'declarator') || declarator.namedChild(0); + } + if (declarator?.type !== 'function_declarator') return undefined; + const inner = getChildByField(declarator, 'declarator'); + return inner?.type === 'template_function' ? cppDefinedName(inner, source) : undefined; +} + function extractCppReceiverType(node: SyntaxNode, source: string): string | undefined { const declarator = getChildByField(node, 'declarator'); if (!declarator) return undefined; const qid = findDeclaratorQualifiedId(declarator); if (!qid) return undefined; - const parts = getNodeText(qid, source).trim().split('::').filter(Boolean); - if (parts.length <= 1) return undefined; + // The scopes are what the qualified id writes before its name: a + // specialization's arguments (`ReadPrimitive`) are no scope, though they may write `::`. + const name = cppQualifiedIdName(qid); + let scopes = getNodeText(qid, source).trim().split('::').filter(Boolean).slice(0, -1).join('::'); + if (name) { + scopes = source.slice(qid.startIndex, name.startIndex).trimEnd(); + // A `::` the parser supplied in recovery writes nothing: Metal's `vertex + // VertexOut vertexShader(…)` reads as `VertexOut::vertexShader`. + if (!scopes.endsWith('::')) return undefined; + } + const parts = scopes.split('::').filter(Boolean); + if (parts.length === 0) return undefined; // An out-of-line template method definition carries the class's template // parameter list in the qualifier (`template T Box::get()`), // but the class node is indexed as bare `Box` — strip `<…>` so the receiver // matches it, the same normalization #1043 applies to base-class refs. // Multi-line parameter lists otherwise leak whole `<…>` blocks (newlines // included) into qualified_name, which can exceed NAME_MAX (#1286). - const receiver = stripCppTemplateArgs(parts.slice(0, -1).join('::')); + const receiver = stripCppTemplateArgs(parts.join('::')); return receiver || undefined; } @@ -2431,7 +2483,8 @@ export const cppExtractor: LanguageExtractor = { bodyField: 'body', paramsField: 'parameters', resolveName: (node, source) => - extractCppQualifiedMethodName(node, source) ?? unnamedTypeVariableName(node, source) ?? cppWrittenClassParts(node, source)?.at(-1), + extractCppQualifiedMethodName(node, source) ?? unnamedTypeVariableName(node, source) ?? cppWrittenClassParts(node, source)?.at(-1) ?? + cppSpecializedFunctionName(node, source), getDeclarationWrapper: (node) => definingDeclaration(node) ?? undefined, getReceiverType: extractCppReceiverType, getClassScope: (node, source) => cppWrittenClassParts(node, source)?.slice(0, -1).join('::') || undefined, diff --git a/src/extraction/languages/cpp-brace-scopes.ts b/src/extraction/languages/cpp-brace-scopes.ts index f83968117e..27e3ff6bc5 100644 --- a/src/extraction/languages/cpp-brace-scopes.ts +++ b/src/extraction/languages/cpp-brace-scopes.ts @@ -81,6 +81,11 @@ export interface CppBraceScopes { closeOf(open: number): number | undefined; /** The named namespaces whose braces hold `offset`, outermost first, as written (`a::b`). */ namespacesAt(offset: number): string[]; + /** + * Whether `offset` is at namespace scope: every brace around it opens a + * namespace, named or not, and none a class, a function or a linkage block. + */ + atNamespaceScope(offset: number): boolean; /** The 1-based line and the column of `offset`. */ positionOf(offset: number): { line: number; column: number }; } @@ -204,6 +209,7 @@ export function scanCppBraceScopes(source: string): CppBraceScopes | null { // first branch (null until a second branch starts). const branches: Array<{ atIf: number[]; afterFirst: number[] | null }> = []; const namespaceAt = new Map(); // `{` offset → name, for named namespaces + const namespaceBraces = new Set(); // `{` offsets of every namespace, named or not let pendingNamespace: { name: string; brace: number } | null = null; let balanced = true; let lineStart = true; @@ -322,6 +328,7 @@ export function scanCppBraceScopes(source: string): CppBraceScopes | null { open.push(i); if (pendingNamespace && pendingNamespace.brace === i) { if (pendingNamespace.name) namespaceAt.set(i, pendingNamespace.name); + namespaceBraces.add(i); pendingNamespace = null; } } else if (c === 125) { @@ -339,9 +346,17 @@ export function scanCppBraceScopes(source: string): CppBraceScopes | null { if (close !== undefined) namespaces.add(brace, close, name); } let lineStarts: number[] | null = null; + let scopes: NestedIntervals | null = null; return { closeOf: (at) => closes.get(at), namespacesAt: (offset) => namespaces.at(offset), + atNamespaceScope: (offset) => { + if (!scopes) { + scopes = new NestedIntervals(); + for (const [brace, close] of [...closes].sort((a, b) => a[0] - b[0])) scopes.add(brace, close, namespaceBraces.has(brace)); + } + return scopes.at(offset).every((isNamespace) => isNamespace); + }, positionOf: (offset) => { if (!lineStarts) { lineStarts = [0]; diff --git a/src/extraction/languages/cpp-macro-methods.ts b/src/extraction/languages/cpp-macro-methods.ts new file mode 100644 index 0000000000..fd03766aa8 --- /dev/null +++ b/src/extraction/languages/cpp-macro-methods.ts @@ -0,0 +1,571 @@ +/** + * C++ methods a function-like macro defines, minted where the file invokes + * the macro. + * + * protocolbuffers/protobuf defines most of its descriptor accessors this way. + * `FieldDescriptor` declares `int number() const;`, and descriptor.h defines + * it further down with `PROTOBUF_DEFINE_ACCESSOR(FieldDescriptor, number, + * int)`, the macro being `inline TYPE CLASS::FIELD() const { return FIELD##_; }`. + * The parser never sees that definition (the preParse blanks the invocation, + * an unknown statement without a semicolon), and a declaration without a body + * mints no node, so `FieldDescriptor::number` had no node at all and none of + * its callers linked. + * + * This reads the file's own `#define`s whose body defines `Class::method(…) … {` + * at its top level, expands each invocation at namespace scope (its + * parameters, and `##` pasting), and mints a method node per definition, as the + * walker mints an out-of-line definition: named and qualified like one (the + * enclosing namespaces composed with the written receiver), and contained by + * the file and by the same-file class of the receiver's name. The node spans + * the invocation. Its signature is the expanded head (`inline const + * FileDescriptor* Descriptor::file() const`), which the resolver reads where it + * would read a definition's source: the return type a chained call goes on + * through, and the parameters an overload is picked by. The expanded body is + * not read, so the method has no calls of its own. + * + * It runs on the original source after either extractor (the kernel or the + * wasm walker), so both mint the same nodes. Only macros the file defines + * itself are expanded: an extraction sees one file. Constructors and + * destructors are left out: constructor resolution reads a constructor's + * parameter list as its signature, and a destructor is not called by name. + */ +import { Edge, ExtractionResult, Node } from '../../types'; +import { generateNodeId } from '../tree-sitter-helpers'; +import { normalizeCppReturnType, stripCppTemplateArgs } from './c-cpp'; +import { scanCppBraceScopes } from './cpp-brace-scopes'; + +/** A function-like macro a file defines (not a variadic one: none defines a method in practice). */ +interface CppMacro { + name: string; + params: string[]; + /** Its replacement text: continuations spliced, comments blanked. */ + body: string; + /** Where its `#` is. */ + at: number; +} + +/** A method definition at the top level of a macro's expansion. */ +interface CppDefinitionHead { + /** The receiver as written, template arguments dropped (`Descriptor`, `internal::ExtensionSet`). */ + receiver: string; + name: string; + /** The text before the receiver: specifiers and the return type (`inline const FileDescriptor*`). */ + returnText: string; + /** Everything up to the body's `{`, whitespace collapsed. */ + head: string; +} + +const isWordStart = (c: number): boolean => (c >= 97 && c <= 122) || (c >= 65 && c <= 90) || c === 95; +const isWordChar = (c: number): boolean => isWordStart(c) || (c >= 48 && c <= 57); + +/** A function-like `#define` whose replacement, on any of its lines, writes `::` and a `{`. */ +const METHOD_MACRO_CANDIDATE = /^[ \t]*#[ \t]*define[ \t]+[A-Za-z_]\w*\([^)\n]*\)((?:[^\n]*\\\r?\n)*[^\n]*)/gm; + +/** + * Whether the C++ `source` may define a method with a macro of its own: a + * cheap test of the raw text, so the bulk-index fast path, which never sees + * this pass, is left only for a file that can't have one. + */ +export function mayDefineCppMacroMethods(source: string): boolean { + if (!source.includes('define')) return false; + METHOD_MACRO_CANDIDATE.lastIndex = 0; + for (let m = METHOD_MACRO_CANDIDATE.exec(source); m; m = METHOD_MACRO_CANDIDATE.exec(source)) { + if (m[1]!.includes('::') && m[1]!.includes('{')) return true; + } + return false; +} + +/** Add the methods the file's own macros define (see the module comment) to an extraction result. */ +export function appendCppMacroMethods(result: ExtractionResult, filePath: string, source: string, preParse: (source: string) => string): void { + if (!mayDefineCppMacroMethods(source)) return; + const minted = cppMacroMethods(filePath, source, preParse, result.nodes); + result.nodes.push(...minted.nodes); + result.edges.push(...minted.edges); +} + +/** + * The method nodes, and their `contains` edges, for every definition the + * file's own method-defining macros expand to where the file invokes them at + * namespace scope. `nodes` are the file's extracted nodes: the class a + * method's receiver names, and the ids already taken. + */ +function cppMacroMethods( + filePath: string, + source: string, + preParse: (source: string) => string, + nodes: readonly Node[], +): { nodes: Node[]; edges: Edge[] } { + const out = { nodes: [] as Node[], edges: [] as Edge[] }; + const { macros, undefs } = cppMacroDirectives(source); + const defining = macros.filter((m) => cppDefinitionHeads(m.body.replace(/\s*##\s*/g, '')).length > 0); + if (defining.length === 0) return out; + // Scopes as the parser saw them (after preParse, offsets unchanged), like the walker's. + const scopes = scanCppBraceScopes(preParse(source)); + if (!scopes) return out; + + const names = new Set(defining.map((m) => m.name)); + const taken = new Set(nodes.map((n) => n.id)); + const fileId = `file:${filePath}`; + for (const call of cppMacroInvocations(source, names)) { + if (!scopes.atNamespaceScope(call.at)) continue; + const macro = liveMacro(defining, undefs, call.name, call.at); + const expansion = macro && expandCppMacro(macro, call.args); + if (!expansion) continue; + const start = scopes.positionOf(call.at); + const end = scopes.positionOf(call.end); + const prefix = scopes.namespacesAt(call.at); + for (const def of cppDefinitionHeads(expansion)) { + const receiverLast = def.receiver.split('::').pop()!; + // A constructor's or destructor's. + if (def.name === receiverLast || def.name.startsWith('~')) continue; + let id = generateNodeId(filePath, 'method', def.name, start.line); + if (taken.has(id)) id = `${id}:${start.column}`; + for (let k = 1; taken.has(id); k++) id = `${generateNodeId(filePath, 'method', def.name, start.line)}:${start.column}:${k}`; + taken.add(id); + const returnType = normalizeCppReturnType(cppTypeText(def.returnText)); + out.nodes.push({ + id, + kind: 'method', + name: def.name, + qualifiedName: composeQualifiedName(prefix, def.receiver, def.name), + filePath, + language: 'cpp', + startLine: start.line, + endLine: end.line, + startColumn: start.column, + endColumn: end.column, + signature: def.head, + ...(returnType ? { returnType } : {}), + updatedAt: Date.now(), + }); + out.edges.push({ source: fileId, target: id, kind: 'contains' }); + // As for an out-of-line definition, the class of its receiver's name in the same file holds it. + const owner = nodes.find((n) => n.name === def.receiver && n.filePath === filePath && + (n.kind === 'struct' || n.kind === 'union' || n.kind === 'class' || n.kind === 'enum' || n.kind === 'trait')); + if (owner) out.edges.push({ source: owner.id, target: id, kind: 'contains' }); + } + } + return out; +} + +/** + * The qualified name of a method written `receiver::name` inside the + * namespaces `prefix`, as the walker composes an out-of-line definition's + * (TreeSitterExtractor.composeReceiverQualifiedName): the receiver is anchored + * at the first namespace it re-spells. + */ +function composeQualifiedName(prefix: readonly string[], receiver: string, name: string): string { + const base = `${receiver}::${name}`; + if (prefix.length === 0) return base; + const anchor = prefix.indexOf(receiver.split('::')[0]!); + const kept = anchor === -1 ? prefix : prefix.slice(0, anchor); + return kept.length > 0 ? `${kept.join('::')}::${base}` : base; +} + +/** The type a definition's leading text writes, without its specifiers, template header and attributes. */ +function cppTypeText(returnText: string): string { + let text = returnText.trim(); + for (;;) { + const before = text; + if (text.startsWith('template')) { + const open = text.indexOf('<'); + const close = open < 0 ? -1 : matching(text, open, '<', '>'); + if (close < 0) return ''; + text = text.slice(close + 1).trim(); + } + text = text.replace(/^\[\[[^\]]*\]\]\s*/, '') + .replace(/^(?:inline|static|virtual|constexpr|consteval|explicit|extern|friend|__forceinline)\b\s*/, '') + // An attribute macro in front (`PROTOBUF_NOINLINE`, `ABSL_MUST_USE_RESULT`). + .replace(/^[A-Z][A-Z0-9]*_[A-Z0-9_]*(?:\s*\([^()]*\))?\s+/, ''); + if (text === before) return text; + } +} + +/** Where the `close` matching the `open` at `text[from]` is, or -1. */ +function matching(text: string, from: number, open: string, close: string): number { + let depth = 0; + for (let i = from; i < text.length; i++) { + const c = text[i]!; + if (c === '"' || c === "'") i = quotedEnd(text, i) - 1; + else if (c === open) depth++; + else if (c === close && --depth === 0) return i; + } + return -1; +} + +/** Past the string or character literal whose quote is at `i`. */ +function quotedEnd(text: string, i: number): number { + const quote = text[i]; + let j = i + 1; + while (j < text.length && text[j] !== quote && text[j] !== '\n') j += text[j] === '\\' ? 2 : 1; + return Math.min(text.length, j + 1); +} + +/** Past the blanks at `i`. */ +function skipBlank(text: string, i: number): number { + while (i < text.length && /\s/.test(text[i]!)) i++; + return i; +} + +/** The end of the identifier at `i`, or -1. */ +function wordEnd(text: string, i: number): number { + if (!isWordStart(text.charCodeAt(i))) return -1; + let j = i + 1; + while (j < text.length && isWordChar(text.charCodeAt(j))) j++; + return j; +} + +/** + * The qualified name starting at `i` that an argument list follows + * (`Descriptor::file(`, `internal::ExtensionSet::Has(`, `Box::get(`, + * `Matrix::operator()(`): its segments and the offset of the `(`. Null when no + * such name starts there, or it has no `::`. + */ +function qualifiedCallableAt(text: string, i: number): { segments: string[]; open: number } | null { + const segments: string[] = []; + let j = i; + if (text.startsWith('::', j)) j = skipBlank(text, j + 2); + for (;;) { + if (text.startsWith('operator', j) && !isWordChar(text.charCodeAt(j + 8))) { + const op = /^operator\s*(\(\s*\)|\[\s*\]|[^\s\w(]+|\w+)\s*\(/.exec(text.slice(j)); + if (!op || segments.length === 0) return null; + segments.push(`operator${op[1]!.replace(/\s+/g, '')}`); + return { segments, open: j + op[0].length - 1 }; + } + const tilde = text[j] === '~'; + const end = wordEnd(text, tilde ? skipBlank(text, j + 1) : j); + if (end < 0) return null; + const word = text.slice(tilde ? skipBlank(text, j + 1) : j, end); + segments.push(tilde ? `~${word}` : word); + let k = skipBlank(text, end); + if (text[k] === '<' && !tilde) { + const close = matching(text, k, '<', '>'); + if (close < 0) return null; + segments[segments.length - 1] += text.slice(k, close + 1); + k = skipBlank(text, close + 1); + } + if (text.startsWith('::', k)) { + if (tilde) return null; + j = skipBlank(text, k + 2); + continue; + } + return text[k] === '(' && segments.length > 1 ? { segments, open: k } : null; + } +} + +/** + * After a definition's parameter list (from `i`): the offset of its body's + * `{`, past the qualifiers it may write (`const`, `noexcept(…)`, `override`, + * `&&`, an attribute, an attribute macro, a trailing return type). -1 for + * anything else: a declaration (`;`), `= default`, a constructor's + * initializer list. + */ +function bodyOpenAfter(text: string, i: number): number { + let j = skipBlank(text, i); + for (;;) { + const c = text[j]; + if (c === '{') return j; + if (c === '&') { + j = skipBlank(text, j + (text[j + 1] === '&' ? 2 : 1)); + continue; + } + if (text.startsWith('[[', j)) { + const close = text.indexOf(']]', j + 2); + if (close < 0) return -1; + j = skipBlank(text, close + 2); + continue; + } + if (text.startsWith('->', j)) { + // A trailing return type runs to the body. + let depth = 0; + for (let k = j + 2; k < text.length; k++) { + const d = text[k]; + if (d === '(' || d === '<' || d === '[') depth++; + else if (d === ')' || d === '>' || d === ']') depth--; + else if (d === '{' && depth <= 0) return k; + else if (d === ';' && depth <= 0) return -1; + } + return -1; + } + const end = wordEnd(text, j); + if (end < 0) return -1; + const word = text.slice(j, end); + if (!/^(?:const|volatile|override|final|noexcept|throw|[A-Z][A-Z0-9]*_[A-Z0-9_]*)$/.test(word)) return -1; + j = skipBlank(text, end); + if (text[j] === '(' && word !== 'const' && word !== 'volatile' && word !== 'override' && word !== 'final') { + const close = matching(text, j, '(', ')'); + if (close < 0) return -1; + j = skipBlank(text, close + 1); + } + } +} + +/** The method definitions at the top level of a macro's expansion `text`, in order. */ +function cppDefinitionHeads(text: string): CppDefinitionHead[] { + const heads: CppDefinitionHead[] = []; + let segment = 0; // where the current top-level declaration began + for (let i = 0; i < text.length; ) { + const c = text[i]!; + if (c === '"' || c === "'") { + i = quotedEnd(text, i); + continue; + } + if (c === ';' || c === '}') { + segment = ++i; + continue; + } + if (c === '{' || c === '(') { + // A body or group that is no definition read here (a namespace, a class, a statement). + const close = matching(text, i, c, c === '{' ? '}' : ')'); + if (close < 0) return heads; + if (c === '{') segment = close + 1; + i = close + 1; + continue; + } + const callable = isWordStart(text.charCodeAt(i)) || (c === ':' && text[i + 1] === ':') ? qualifiedCallableAt(text, i) : null; + if (!callable) { + const end = wordEnd(text, i); + i = end < 0 ? i + 1 : end; + continue; + } + const close = matching(text, callable.open, '(', ')'); + if (close < 0) return heads; + const open = bodyOpenAfter(text, close + 1); + if (open < 0) { + i = close + 1; + continue; + } + // An explicit specialization's arguments (`Read`) are no part of its name. + const last = callable.segments[callable.segments.length - 1]!; + const name = last.startsWith('operator') ? last : stripCppTemplateArgs(last); + const receiver = stripCppTemplateArgs(callable.segments.slice(0, -1).join('::')); + if (receiver) { + heads.push({ + receiver, + name, + returnText: text.slice(segment, i).trim(), + head: text.slice(segment, open).replace(/\s+/g, ' ').trim(), + }); + } + const bodyClose = matching(text, open, '{', '}'); + if (bodyClose < 0) return heads; + segment = i = bodyClose + 1; + } + return heads; +} + +/** + * The macro of `name` in effect at `offset`: its last `#define` before it, + * unless an `#undef` came between. + */ +function liveMacro(macros: readonly CppMacro[], undefs: ReadonlyArray<{ name: string; at: number }>, name: string, offset: number): CppMacro | null { + let found: CppMacro | null = null; + for (const m of macros) if (m.name === name && m.at < offset && (!found || m.at > found.at)) found = m; + if (!found) return null; + return undefs.some((u) => u.name === name && u.at > found!.at && u.at < offset) ? null : found; +} + +/** A macro's expansion for the arguments `args` (as written), or null when they don't fit its parameters. */ +function expandCppMacro(macro: CppMacro, args: string[]): string | null { + const values = macro.params.length === 0 && args.length === 1 && args[0]!.trim() === '' ? [] : args.map((a) => a.trim()); + if (values.length !== macro.params.length) return null; + const valueOf = new Map(macro.params.map((p, k) => [p, values[k]!])); + return macro.body + .replace(/[A-Za-z_]\w*/g, (word) => valueOf.get(word) ?? word) + // `a ## b`: one token. + .replace(/\s*##\s*/g, ''); +} + +/** + * The function-like `#define`s of a C++ source and its `#undef`s, in order, + * read past comments and literals: a `#define` in a comment or a raw string + * defines nothing. + */ +function cppMacroDirectives(source: string): { macros: CppMacro[]; undefs: Array<{ name: string; at: number }> } { + const macros: CppMacro[] = []; + const undefs: Array<{ name: string; at: number }> = []; + for (const d of cppCodeScan(source, null).directives) { + const text = withoutLineComment(d.text.replace(/\\\r?\n/g, ' ').replace(/\/\*[\s\S]*?\*\//g, ' ')); + const define = /^#[ \t]*define[ \t]+([A-Za-z_]\w*)\(([^)]*)\)/.exec(text); + if (define) { + const params = define[2]!.split(',').map((p) => p.trim()).filter((p) => p !== ''); + if (!params.some((p) => p.endsWith('...'))) macros.push({ name: define[1]!, params, body: text.slice(define[0].length), at: d.at }); + continue; + } + const undef = /^#[ \t]*undef[ \t]+([A-Za-z_]\w*)/.exec(text); + if (undef) undefs.push({ name: undef[1]!, at: d.at }); + } + return { macros, undefs }; +} + +/** The invocations of the macros `names` in code (not in a directive, comment or literal), with their arguments. */ +function cppMacroInvocations(source: string, names: ReadonlySet): Array<{ name: string; at: number; end: number; args: string[] }> { + const calls: Array<{ name: string; at: number; end: number; args: string[] }> = []; + for (const word of cppCodeScan(source, names).words) { + const open = skipBlankAndComments(source, word.end); + if (source[open] !== '(') continue; + const args: string[] = []; + let depth = 0; + let from = open + 1; + let close = -1; + for (let i = open; i < source.length; i++) { + const c = source[i]!; + if (c === '"' || c === "'") i = quotedEnd(source, i) - 1; + else if (c === '/' && source[i + 1] === '*') { + const end = source.indexOf('*/', i + 2); + i = end < 0 ? source.length : end + 1; + } else if (c === '/' && source[i + 1] === '/') { + const end = source.indexOf('\n', i); + i = end < 0 ? source.length : end - 1; + } else if (c === '(') depth++; + else if (c === ')' && --depth === 0) { + close = i; + break; + } else if (c === ',' && depth === 1) { + args.push(source.slice(from, i)); + from = i + 1; + } + } + if (close < 0) continue; + args.push(source.slice(from, close)); + calls.push({ name: word.name, at: word.at, end: close + 1, args: args.map(withoutComments) }); + } + return calls; +} + +function withoutComments(text: string): string { + return text.replace(/\/\*[\s\S]*?\*\//g, ' ').replace(/\/\/[^\n]*/g, ' '); +} + +/** A spliced directive without the `//` comment that ends it, one in a string literal aside. */ +function withoutLineComment(text: string): string { + for (let i = 0; i < text.length; i++) { + const c = text[i]; + if (c === '"' || c === "'") i = quotedEnd(text, i) - 1; + else if (c === '/' && text[i + 1] === '/') return text.slice(0, i); + } + return text; +} + +/** Past blanks and comments from `i`. */ +function skipBlankAndComments(source: string, i: number): number { + for (;;) { + i = skipBlank(source, i); + if (source.startsWith('//', i)) { + const end = source.indexOf('\n', i); + i = end < 0 ? source.length : end; + } else if (source.startsWith('/*', i)) { + const end = source.indexOf('*/', i + 2); + i = end < 0 ? source.length : end + 2; + } else { + return i; + } + } +} + +const RAW_STRING_PREFIX = /^(?:u8|[uUL])?R$/; + +/** + * One pass over C/C++ source, skipping comments and string, character and raw + * string literals: the preprocessor directives (each with its `\` + * continuations), and, when `names` is given, the code's identifiers among + * them. + */ +function cppCodeScan(source: string, names: ReadonlySet | null): { + directives: Array<{ at: number; text: string }>; + words: Array<{ name: string; at: number; end: number }>; +} { + const directives: Array<{ at: number; text: string }> = []; + const words: Array<{ name: string; at: number; end: number }> = []; + const n = source.length; + let lineStart = true; + let lastWordStart = -1; + let lastWordEnd = -1; + for (let i = 0; i < n; ) { + const c = source.charCodeAt(i); + if (c === 10) { + lineStart = true; + i++; + continue; + } + if (c === 32 || c === 9 || c === 13 || c === 11 || c === 12) { + i++; + continue; + } + if (c === 47 && source.charCodeAt(i + 1) === 47) { + // A `\` at the end continues a line comment. + let end = i; + for (;;) { + end = source.indexOf('\n', end); + if (end < 0) { + end = n; + break; + } + let k = end - 1; + if (source.charCodeAt(k) === 13) k--; + if (source.charCodeAt(k) !== 92) break; + end++; + } + i = end; + continue; + } + if (c === 47 && source.charCodeAt(i + 1) === 42) { + const end = source.indexOf('*/', i + 2); + i = end < 0 ? n : end + 2; + continue; + } + if (c === 35 && lineStart) { + let j = i + 1; + for (; j < n; j++) { + const d = source.charCodeAt(j); + if (d === 47 && source.charCodeAt(j + 1) === 42) { + const end = source.indexOf('*/', j + 2); + j = end < 0 ? n - 1 : end + 1; + } else if (d === 10) { + let k = j - 1; + if (source.charCodeAt(k) === 13) k--; + if (source.charCodeAt(k) !== 92) break; + } + } + directives.push({ at: i, text: source.slice(i, j) }); + i = j; + continue; + } + lineStart = false; + if (c === 34) { + if (lastWordEnd === i && RAW_STRING_PREFIX.test(source.slice(lastWordStart, i))) { + const paren = source.indexOf('(', i + 1); + const delimiter = paren < 0 ? '' : source.slice(i + 1, paren); + if (paren >= 0 && /^[^\s()\\"]{0,16}$/.test(delimiter)) { + const end = source.indexOf(`)${delimiter}"`, paren + 1); + i = end < 0 ? n : end + delimiter.length + 2; + continue; + } + } + i = quotedEnd(source, i); + continue; + } + if (c === 39) { + // A digit separator (`1'000`) is no character literal. + if (i > 0 && isWordChar(source.charCodeAt(i - 1)) && lastWordEnd !== i) { + i++; + continue; + } + i = quotedEnd(source, i); + continue; + } + if (isWordChar(c)) { + let j = i + 1; + while (j < n && isWordChar(source.charCodeAt(j))) j++; + if (isWordStart(c)) { + lastWordStart = i; + lastWordEnd = j; + if (names?.has(source.slice(i, j))) words.push({ name: source.slice(i, j), at: i, end: j }); + } + i = j; + continue; + } + i++; + } + return { directives, words }; +} diff --git a/src/extraction/parse-worker.ts b/src/extraction/parse-worker.ts index 974d556f7e..e1a2c5ade6 100644 --- a/src/extraction/parse-worker.ts +++ b/src/extraction/parse-worker.ts @@ -19,6 +19,7 @@ import { detectLanguage } from './grammars'; import { tryKernelExtractRaw } from './kernel'; import { commonJsRequireRefs } from './commonjs-requires'; import { getAllFrameworkResolvers, getApplicableFrameworks } from '../resolution/frameworks'; +import { mayDefineCppMacroMethods } from './languages/cpp-macro-methods'; import type { Language, ExtractionResult } from '../types'; parentPort!.on('message', async (msg: { type: string; id?: number; filePath?: string; content?: string; languages?: Language[]; frameworkNames?: string[]; language?: Language }) => { @@ -45,6 +46,8 @@ parentPort!.on('message', async (msg: { type: string; id?: number; filePath?: st // buffer clone is a flat memcpy). Only when no applicable framework has // an extract() hook: those merge extra nodes/refs into the DECODED // result inside extractFromSource, so such files keep the decoded path. + // So does a C++ file that may define methods with its own macros: they + // are minted after the kernel, in extractFromSource. let result: ExtractionResult | undefined; const frameworksNeedDecode = frameworkNames && frameworkNames.length > 0 @@ -53,7 +56,7 @@ parentPort!.on('message', async (msg: { type: string; id?: number; filePath?: st language ).some((fw) => !!fw.extract) : false; - if (!frameworksNeedDecode) { + if (!frameworksNeedDecode && !(language === 'cpp' && mayDefineCppMacroMethods(content!))) { const raw = tryKernelExtractRaw(filePath!, content!, language); if (raw) { result = { diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index eb255da79e..dbe03192f9 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -26,6 +26,7 @@ import type { LanguageExtractor, ExtractorContext } from './tree-sitter-types'; import { EXTRACTORS } from './languages'; import { stripCppTemplateArgs, isCppConstructorDeclaration, cDeclaratorIdentifier, isClassLikeDefinition } from './languages/c-cpp'; import { NestedIntervals, scanCppBraceScopes, type CppBraceScopes } from './languages/cpp-brace-scopes'; +import { appendCppMacroMethods } from './languages/cpp-macro-methods'; import { rustImplTypeName } from './languages/rust'; import { goAliasTypeNames, goEmbeddedTypeName } from './languages/go'; import { dartMisparsedGenericCall, dartReceiverOf, isDartTypeName, pushDartTypeRefs } from './languages/dart'; @@ -8635,6 +8636,12 @@ export function extractFromSource( // generic extractor over the kernel's serialized tree. result = new TreeSitterExtractor(filePath, source, detectedLanguage).extract(); } + // C++ methods the file's own macros define, read from the original source + // after either extractor so both mint them (cpp-macro-methods.ts). + if (detectedLanguage === 'cpp') { + const preParse = EXTRACTORS.cpp?.preParse; + appendCppMacroMethods(result, filePath, source, (text) => (preParse ? preParse(text, filePath) : text)); + } } // Framework-specific extraction (routes, middleware, etc.) diff --git a/src/resolution/callback-synthesizer.ts b/src/resolution/callback-synthesizer.ts index 121872acb2..ec0e4132d6 100644 --- a/src/resolution/callback-synthesizer.ts +++ b/src/resolution/callback-synthesizer.ts @@ -44,6 +44,7 @@ import { rnModuleMethods } from './frameworks/react-native'; import { resolveImportPath } from './import-resolver'; import { isDistinctiveIdentifier } from '../search/query-utils'; import { crossesCodeBoundary, innermostBinding } from './gates'; +import { cppSpecializationArguments } from './cpp-specializations'; import { declaredComponent, exportedComponent } from './frameworks/react'; import { jsxCode } from './jsx-code'; @@ -1037,8 +1038,10 @@ async function arkuiRouterEdges(ctx: ResolutionContext, onYield: MaybeYield): Pr * (the override), so trace/callees from the interface method reach the * implementation(s). Over-approximation accepted (reachability-correct); capped * per class and gated to C++ to avoid touching other languages' dispatch. + * An explicit specialization of a member template, which shares the + * template's name, overrides nothing and is overridden by nothing. */ -async function cppOverrideEdges(queries: QueryBuilder, onYield: MaybeYield): Promise { +async function cppOverrideEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise { let scanned255 = 0; const edges: Edge[] = []; const seen = new Set(); @@ -1047,6 +1050,7 @@ async function cppOverrideEdges(queries: QueryBuilder, onYield: MaybeYield): Pro .getOutgoingEdges(classId, ['contains']) .map((e) => queries.getNodeById(e.target)) .filter((n): n is Node => !!n && n.kind === 'method'); + const overridable = (m: Node): boolean => cppSpecializationArguments(m, ctx) === null; for (const cls of queries.iterateNodesByKind('class')) { if ((++scanned255 & 63) === 0) await onYield(); const subMethods = methodsOf(cls.id).filter((n) => n.language === 'cpp'); @@ -1054,12 +1058,12 @@ async function cppOverrideEdges(queries: QueryBuilder, onYield: MaybeYield): Pro for (const ext of queries.getOutgoingEdges(cls.id, ['extends'])) { const base = queries.getNodeById(ext.target); if (!base || base.language !== 'cpp' || base.id === cls.id) continue; - const baseMethods = new Map(methodsOf(base.id).map((m) => [m.name, m])); + const baseMethods = new Map(methodsOf(base.id).filter(overridable).map((m) => [m.name, m])); let added = 0; for (const m of subMethods) { if (added >= MAX_CALLBACKS_PER_CHANNEL) break; const bm = baseMethods.get(m.name); - if (!bm || bm.id === m.id) continue; + if (!bm || bm.id === m.id || !overridable(m)) continue; const key = `${bm.id}>${m.id}`; if (seen.has(key)) continue; seen.add(key); @@ -6561,7 +6565,7 @@ export const SYNTH_PASSES: SynthPassDef[] = [ { name: 'arkuiStateEdges', gate: (has) => has('arkts'), run: (q, c, y) => arkuiStateBuildEdges(q, c, y) }, { name: 'arkuiEmitter', gate: (has) => has('arkts'), run: (_q, c, y) => arkuiEmitterEdges(c, y) }, { name: 'arkuiRoutes', gate: (has) => has('arkts'), run: (_q, c, y) => arkuiRouterEdges(c, y) }, - { name: 'cppEdges', gate: (has) => has('cpp'), run: (q, _c, y) => cppOverrideEdges(q, y) }, + { name: 'cppEdges', gate: (has) => has('cpp'), run: (q, c, y) => cppOverrideEdges(q, c, y) }, { name: 'ifaceEdges', gate: (has) => has('java', 'kotlin', 'csharp', 'swift', 'scala', 'go', 'rust', 'arkts', ...JS_FAMILY), diff --git a/src/resolution/cpp-specializations.ts b/src/resolution/cpp-specializations.ts new file mode 100644 index 0000000000..39e11374c1 --- /dev/null +++ b/src/resolution/cpp-specializations.ts @@ -0,0 +1,110 @@ +import type { Node } from '../types'; +import type { ResolutionContext } from './types'; + +/** + * C++ explicit specializations for the TypeScript synthesizers: the kernel + * resolver reads the same arguments natively (resolve/cpp_specializations.rs). + */ + +/** Per context: a C++ function's id → the template arguments it is an explicit specialization for (null: none). */ +const CPP_SPECIALIZATION_ARGUMENTS = new WeakMap>(); + +/** + * The template arguments the C++ function or method `n` is an explicit + * specialization for, as its definition writes them after its name: + * `int32_t` and `WireFormatLite::TYPE_INT32` for protobuf's `template <> + * inline bool WireFormatLite::ReadPrimitive(…)`. + * A method a macro defines is read from its expansion. Null for a template's + * own definition, any other function, and a specialization that leaves its + * arguments to be deduced. + */ +export function cppSpecializationArguments(n: Node, context: ResolutionContext): string[] | null { + // A constructor, whose signature is its parameter list, is never specialized + // by name: gmock's `template Action(const Action& action)` + // writes `Action` for a parameter. + if (n.language !== 'cpp' || (n.kind !== 'function' && n.kind !== 'method') || n.signature?.startsWith('(')) return null; + let memo = CPP_SPECIALIZATION_ARGUMENTS.get(context); + if (!memo) CPP_SPECIALIZATION_ARGUMENTS.set(context, (memo = new Map())); + const hit = memo.get(n.id); + if (hit !== undefined) return hit; + // (A method a macro defines carries its expansion as its signature.) + const lines = n.signature ? [] : context.getFileLines?.(n.filePath) ?? context.readFile(n.filePath)?.split(/\r?\n/) ?? []; + const head = n.signature || lines.slice(n.startLine - 1, n.startLine + 11).join('\n').slice(n.startColumn); + const args = cppTemplateArgumentsAfter(head, n.name, 0); + memo.set(n.id, args); + return args; +} + +/** + * The template arguments `text` writes right after `name`, the first time + * from `from` on it is written as a word of its own, as they compare + * (cppTemplateArgumentKey). Null when it writes none there, or `<>`. + */ +function cppTemplateArgumentsAfter(text: string, name: string, from: number): string[] | null { + for (let at = text.indexOf(name, from); at >= 0; at = text.indexOf(name, at + 1)) { + const open = at + name.length; + if (/[\w$]/.test(text[at - 1] ?? '') || /[\w$]/.test(text[open] ?? '')) continue; + const end = text[open] === '<' ? cppTemplateArgumentsEnd(text, open) : -1; + const args = end < 0 ? [] : splitCppTopLevel(text.slice(open + 1, end)).map(cppTemplateArgumentKey); + return args.length > 0 ? args : null; + } + return null; +} + +/** A C++ template argument as it compares: spaced only between words. */ +function cppTemplateArgumentKey(arg: string): string { + return arg.replace(/\s*([^\w\s])\s*/g, '$1').replace(/\s+/g, ' '); +} + +/** Split `a, f(b, c), d` at its top-level commas. */ +function splitCppTopLevel(text: string): string[] { + const out: string[] = []; + let depth = 0; + // Where each template argument list open around the current character closes. + const closes: number[] = []; + let cur = ''; + for (let i = 0; i < text.length; i++) { + const ch = text[i]!; + if (ch === '"' || ch === "'") { + let j = i + 1; + while (j < text.length && text[j] !== ch) j += text[j] === '\\' ? 2 : 1; + cur += text.slice(i, j + 1); + i = j; + continue; + } + if (ch === '(' || ch === '[' || ch === '{') depth++; + else if (ch === ')' || ch === ']' || ch === '}') depth = Math.max(0, depth - 1); + else if (ch === '<') { + const end = cppTemplateArgumentsEnd(text, i); + if (end > i) closes.push(end); + } else if (ch === '>' && closes[closes.length - 1] === i) closes.pop(); + if (ch === ',' && depth === 0 && closes.length === 0) { out.push(cur.trim()); cur = ''; continue; } + cur += ch; + } + if (cur.trim()) out.push(cur.trim()); + return out; +} + +/** + * Where the template argument list the `<` at `at` opens closes: the index + * of its `>`, or -1 when the `<` compares or shifts. + */ +function cppTemplateArgumentsEnd(text: string, at: number): number { + if (!/\w/.test(text[at - 1] ?? '') || text[at + 1] === '<' || text[at + 1] === '=') return -1; + let angles = 0; + let others = 0; + for (let i = at; i < text.length; i++) { + const ch = text[i]!; + if (ch === '"' || ch === "'") { + let j = i + 1; + while (j < text.length && text[j] !== ch) j += text[j] === '\\' ? 2 : 1; + i = j; + } else if (ch === '(' || ch === '[' || ch === '{') others++; + else if (ch === ')' || ch === ']' || ch === '}') { + if (--others < 0) return -1; + } else if (others > 0) continue; + else if (ch === '<') angles++; + else if (ch === '>' && text[i - 1] !== '-' && --angles === 0) return i; + } + return -1; +}