diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d4a5efb1..2e504c5ad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -77,6 +77,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In C and C++, a header or source file named like a test, such as protocolbuffers/protobuf's `conformance_test.h` and `test_runner.h`, now counts as part of the code that `#include`s it, directly or through other headers. Before, calls from that code into it were dropped, so protobuf's conformance suites' `suite_.ReportFailure(…)` and `RunValidInputTest(…)` calls, and its unit tests' calls to the shared `TestUtil` helpers, linked to nothing. A test file that only tests include is still kept out of reach of the rest of your code. Re-index C and C++ projects after upgrading. - In C++, a class, struct or method now keeps the namespaces and the class it is declared in when the parser misreads something earlier in the file, such as an unknown macro in front of a member or in a class header. Before, everything after the misread code could lose them, or land inside a class it isn't in: in protocolbuffers/protobuf, `FieldDescriptor` was indexed without `google::protobuf::`, and in RocksDB, the `Opts` struct declared inside a cache table class was indexed outside it, so finding a class's base classes and the methods called on its objects fell back to guessing by name. Methods that had come loose from their class are its members again, and classes the misread code had hidden, like leveldb's POSIX file and environment classes, are now indexed. Re-index C++ projects after upgrading. - In C++, a call on a variable, parameter or member declared as a standard-library or other outside type with a lowercase name, like `std::string`, `std::vector`, `std::set` or `absl::flat_hash_set`, no longer links to a project method that only shares the method's name. In google/leveldb, `saved_key_.clear()` on a `std::string saved_key_;` reached `Slice::clear`, and in protocolbuffers/protobuf, `proto.append(…)` on a `std::string` reached an unrelated `LeftoverBuffer::append`. The same goes for `->` through a raw pointer like `std::string* out`, while `->` through a smart pointer, iterator or optional still links to the type it holds. Comments also no longer pass for a variable's declaration, so more calls now link to the method of the type the variable is really declared with. Re-index C++ projects after upgrading. +- In C++, a class written with attribute macros the parser can't read now keeps its members. That covers several macros between `class` and the class name, like protocolbuffers/protobuf's generated `class PROTOBUF_EXPORT PROTOBUF_FUTURE_ADD_EARLY_WARN_UNUSED Any final`, a macro with arguments there, like RocksDB's `struct ALIGN_AS(64U) HandleImpl`, or one before a template specialization, and macros on a member: in front of it (`PROTOBUF_FUTURE_ADD_EARLY_NODISCARD`), between a pointer and its name (`PROTOBUF_NONNULL`), after its parameters or its name (`ABSL_ATTRIBUTE_LIFETIME_BOUND`, google/leveldb's `LOCKS_EXCLUDED(mu_)` and `GUARDED_BY(mu_)`), or alone on a line above a comment ({fmt}'s `FMT_BEGIN_EXPORT`). Before, protobuf's generated message classes were missing and their members, like `GetDescriptor`, `GetReflection` and `unknown_fields`, were indexed as functions outside any class, and leveldb's annotated methods were indexed under the annotation's name, so calls to them linked to nothing. Re-index C++ projects after upgrading. - Framework support now switches on when an app's `package.json` sits three or more folders deep, as with an ASP.NET solution's Angular or React app in `src/Web/ClientApp/`, or a React app in a workspace folder like `web/ui/mantine-ui/`. Before, only the project root and the first two folder levels were read, so frameworks such as Angular, React Router, Vue Router, SvelteKit and TanStack Router could go undetected there: an Angular app's routes, its `routerLink` and `router.navigate(…)` navigation and its template's event bindings were missing, so a handler used only from a `(click)` binding had no callers, and a React app's `navigate('/login')` calls didn't reach their route. Re-index projects laid out this way after upgrading. - In a repository that mixes languages, a web framework's naming conventions now link only code written in that framework's own languages. Before, Express's also linked Go and Python code: in etcd a Go method's `*zap.Logger` result type linked to the method itself, and in Harbor a Python `self.jaeger.validate()` call linked to another class's `validate`. Links that cross languages on purpose, like React Native and Expo native modules, Swift and Objective-C, Drupal's `routing.yml`, SvelteKit's `$lib/…` imports in `.ts` files and a Razor page's `@model`, are unchanged. - In Go, an interface that embeds another, like gin's `IRouter` embedding `IRoutes`, now links to it, so `codegraph_explore` and impact show the embedded interface as its base. Before, every embedded interface was dropped. An embedded field written through another package of your project, like `*refresh.Discovery` or `storage.Appender`, or with type arguments, like `List[int]`, now links too. One from the standard library or another module, like `io.Closer` or `sync.Mutex`, links to nothing rather than to a project type that shares its name, and a type-set constraint like `~int | ~float64` is not an embedding. An embedded type written without a package now always links to its own package's type: before, etcd's `Client`, which embeds its own `Lease` interface, was linked to a `Lease` struct in another package. Re-index Go projects after upgrading. diff --git a/__tests__/cpp-attribute-macros.test.ts b/__tests__/cpp-attribute-macros.test.ts new file mode 100644 index 000000000..af90c4a29 --- /dev/null +++ b/__tests__/cpp-attribute-macros.test.ts @@ -0,0 +1,536 @@ +/** + * Attribute macros tree-sitter-cpp can't read, in the places the C++ preParse + * did not yet blank them: + * + * - several macros, or one with arguments, between `class` and the name + * (protobuf's generated `class PROTOBUF_EXPORT + * PROTOBUF_FUTURE_ADD_EARLY_WARN_UNUSED Any final : public Message`, + * rocksdb's `struct ALIGN_AS(64U) HandleImpl`), and one before a partial + * specialization's name; + * - a macro between a pointer and the declared name (`const Descriptor* + * PROTOBUF_NONNULL descriptor()`), after a parameter list + * (`unknown_fields() const ABSL_ATTRIBUTE_LIFETIME_BOUND {`, leveldb's + * `LOCKS_EXCLUDED(mu_) {`), after a declared name (`int count_ + * GUARDED_BY(mu_);`), or opening a declaration + * (`PROTOBUF_FUTURE_ADD_EARLY_NODISCARD absl::string_view name() const`); + * - a lone macro line with a comment under it ({fmt}'s `FMT_BEGIN_EXPORT`). + * + * Each misparsed the class around it: protobuf's generated messages had their + * members indexed as namespace-level functions, and leveldb's annotated + * methods became phantoms named after the annotation. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { + blankCppExportMacros, + blankCppPointerAnnotationMacros, + blankCppTrailingAttributeMacros, + blankCppDeclaratorAttributeMacros, + blankCppLeadingAttributeMacros, + blankLoneMacroLines, + cppExtractor, +} from '../src/extraction/languages/c-cpp'; + +/** Assert `blank` removes exactly `macros` from `input` and nothing else, offsets kept. */ +function expectBlanked(blank: (s: string) => string, input: string, macros: string[]): void { + const out = blank(input); + expect(out.length).toBe(input.length); + let expected = input; + for (const macro of macros) { + const at = expected.indexOf(macro); + expect(at, `${macro} in ${input}`).toBeGreaterThanOrEqual(0); + expected = expected.slice(0, at) + macro.replace(/[^\r\n]/g, ' ') + expected.slice(at + macro.length); + } + expect(out).toBe(expected); +} + +describe('C++ attribute-macro blanks', () => { + it('blanks every macro between class and the name, and before a specialization', () => { + expectBlanked( + blankCppExportMacros, + 'class PROTOBUF_EXPORT PROTOBUF_FUTURE_ADD_EARLY_WARN_UNUSED Any final : public ::google::protobuf::Message {};', + ['PROTOBUF_EXPORT PROTOBUF_FUTURE_ADD_EARLY_WARN_UNUSED'] + ); + expectBlanked( + blankCppExportMacros, + 'template \nclass ABSL_ATTRIBUTE_WARN_UNUSED PROTOBUF_DECLSPEC_EMPTY_BASES\n RepeatedField final\n : private internal::RepeatedFieldBase {};', + ['ABSL_ATTRIBUTE_WARN_UNUSED', 'PROTOBUF_DECLSPEC_EMPTY_BASES'] + ); + expectBlanked( + blankCppExportMacros, + 'template \nclass PROTOBUF_DECLSPEC_EMPTY_BASES RepeatedFieldProxyWithSet<\n T, kOrProxy,\n std::enable_if_t>> {};', + ['PROTOBUF_DECLSPEC_EMPTY_BASES'] + ); + expectBlanked(blankCppExportMacros, 'struct ALIGN_AS(64U) HandleImpl : public ClockHandle {};', ['ALIGN_AS(64U)']); + // An all-caps class name after a macro is the name, `final` included. + expectBlanked(blankCppExportMacros, 'class FOO_API BAR_IMPL final : public X {};', ['FOO_API']); + }); + + it('leaves class heads alone when nothing is defined there', () => { + for (const c of [ + 'class FOO_API Name;', + 'class FOO_API Name var;', + 'struct STAT_T st = {0};', + 'class GTEST_1_TUPLE_(T) {', + '// class PROTOBUF_EXPORT Any final : public Message {', + 'const char* s = "class FOO Bar {";', + ]) { + expect(blankCppExportMacros(c)).toBe(c); + } + }); + + it('blanks annotations between a pointer or reference and the declared name', () => { + expectBlanked( + blankCppPointerAnnotationMacros, + 'static const Descriptor* PROTOBUF_NONNULL descriptor() { return d; }', + ['PROTOBUF_NONNULL'] + ); + expectBlanked( + blankCppPointerAnnotationMacros, + 'void Any::InternalSwap(Any* PROTOBUF_RESTRICT PROTOBUF_NONNULL other) {}', + ['PROTOBUF_RESTRICT PROTOBUF_NONNULL'] + ); + expectBlanked( + blankCppPointerAnnotationMacros, + 'void f(const FieldDescriptor * PROTOBUF_NULLABLE *\n PROTOBUF_NONNULL field, const T& ABSL_ATTRIBUTE_LIFETIME_BOUND v);', + ['PROTOBUF_NULLABLE', 'PROTOBUF_NONNULL', 'ABSL_ATTRIBUTE_LIFETIME_BOUND'] + ); + expectBlanked( + blankCppPointerAnnotationMacros, + 'static constexpr const EnumDescriptor* PROTOBUF_NONNULL* PROTOBUF_NULLABLE\n file_level_enum_descriptors = nullptr;', + ['PROTOBUF_NONNULL', 'PROTOBUF_NULLABLE'] + ); + }); + + it('leaves products, bit tests, unnamed parameters and comments alone', () => { + for (const c of [ + 'int n = count * MAX_LEN + 1;', + 'bool b = (flags & FOO_BIT) != 0;', + 'bool b = flags & FOO_BIT\n && enabled;', + 'bool b = flags & FOO_BIT and enabled;', + 'void f(Foo* PROTOBUF_NONNULL);', + '/**\n * NOTE_THIS applies\n * TODO_LATER fix it\n */', + 'const char* s = "a * PROTOBUF_NONNULL b";', + 'int digits10 = int(sizeof(int) * CHAR_BIT * 3 / 10);', + 'int n = 2 * PAGE_SIZE * pages;', + '#define PTR_T int *\nFOO_BAR name;', + ]) { + expect(blankCppPointerAnnotationMacros(c)).toBe(c); + } + }); + + it('blanks attribute macros after a parameter list', () => { + expectBlanked( + blankCppTrailingAttributeMacros, + 'const UnknownFieldSet& unknown_fields() const\n ABSL_ATTRIBUTE_LIFETIME_BOUND {\n return f;\n}', + ['ABSL_ATTRIBUTE_LIFETIME_BOUND'] + ); + expectBlanked(blankCppTrailingAttributeMacros, '~Any() PROTOBUF_FINAL;', ['PROTOBUF_FINAL']); + expectBlanked( + blankCppTrailingAttributeMacros, + 'void Lock() EXCLUSIVE_LOCK_FUNCTION() { mu_.lock(); }\nvoid Wait() LOCKS_EXCLUDED(mu_);', + ['EXCLUSIVE_LOCK_FUNCTION()', 'LOCKS_EXCLUDED(mu_)'] + ); + expectBlanked( + blankCppTrailingAttributeMacros, + 'explicit MutexLock(port::Mutex* mu) EXCLUSIVE_LOCK_FUNCTION(mu) : mu_(mu) {}', + ['EXCLUSIVE_LOCK_FUNCTION(mu)'] + ); + expectBlanked( + blankCppTrailingAttributeMacros, + 'bool operator==(const A& a) const ABSL_MUST_USE_RESULT;', + ['ABSL_MUST_USE_RESULT'] + ); + }); + + it('leaves statements, casts and macro calls followed by macros alone', () => { + for (const c of [ + 'if (x) RETURN_FALSE;', + 'while (busy) SPIN_PAUSE;', + 'int v = (int) MAX_VALUE;', + 'FMT_PRAGMA_CLANG(diagnostic ignored "-Wbit-int-extension")\n\nTEST(std_test, bitint) {\n}', + 'DECLARE_FOO(a) DECLARE_BAR(b);', + 'void f()\n\nBAR_BAZ;', + 'bool b = f(x) == FOO_BAR;', + // A directive's `)` closes no parameter list. + '# if FMT_USE_FCNTL && !defined(__MINGW32__)\nTEST(file_test, open_windows_file) {\n}', + 'if constexpr (POS + 1 == str.size())\n FMT_THROW(format_error("unmatched brace"));', + ]) { + expect(blankCppTrailingAttributeMacros(c)).toBe(c); + } + }); + + it('blanks attribute macros after a declared name or its brackets', () => { + expectBlanked(blankCppDeclaratorAttributeMacros, 'port::CondVar cv GUARDED_BY(mu);', ['GUARDED_BY(mu)']); + expectBlanked(blankCppDeclaratorAttributeMacros, 'int refs_ GUARDED_BY(refs_mutex_);', ['GUARDED_BY(refs_mutex_)']); + expectBlanked(blankCppDeclaratorAttributeMacros, 'Version* const version GUARDED_BY(mu);', ['GUARDED_BY(mu)']); + expectBlanked( + blankCppDeclaratorAttributeMacros, + 'std::map cache_ ABSL_GUARDED_BY(mutex_) = {};', + ['ABSL_GUARDED_BY(mutex_)'] + ); + expectBlanked( + blankCppDeclaratorAttributeMacros, + 'const AnyGlobalsTypeInternal Any_globals_\n PROTOBUF_MESSAGE_GLOBALS_SECTION(.data.rel.ro);', + ['PROTOBUF_MESSAGE_GLOBALS_SECTION(.data.rel.ro)'] + ); + expectBlanked( + blankCppDeclaratorAttributeMacros, + 'const ::uint32_t Table::offsets[] ABSL_ATTRIBUTE_SECTION_VARIABLE(\n protodesc_cold) = {1, 2};', + ['ABSL_ATTRIBUTE_SECTION_VARIABLE(\n protodesc_cold)'] + ); + expectBlanked( + blankCppDeclaratorAttributeMacros, + 'explicit CordInputStream(const absl::Cord* cord ABSL_ATTRIBUTE_LIFETIME_BOUND);', + ['ABSL_ATTRIBUTE_LIFETIME_BOUND'] + ); + }); + + it('leaves variables named in capitals and names without a type before them alone', () => { + for (const c of [ + 'Foo DEFAULT_OPTIONS;', + 'static Foo DEFAULT_OPTIONS;', + 'const Foo DEFAULT_OPTIONS = Make();', + 'ns::Foo DEFAULT_OPTIONS;', + 'typedef Foo BAR_T;', + 'template T MAX_VALUE;', + 'return x FOO_SUFFIX;', + 'f(a, b FOO_FLAG);', + 'x = rate FOO_UNITS;', + 'EXPECT_EQ(2 SECS_X, d);', + ]) { + expect(blankCppDeclaratorAttributeMacros(c)).toBe(c); + } + }); + + it('blanks attribute macros that open a declaration', () => { + expectBlanked( + blankCppLeadingAttributeMacros, + ' PROTOBUF_FUTURE_ADD_EARLY_NODISCARD absl::string_view name() const { return n; }', + ['PROTOBUF_FUTURE_ADD_EARLY_NODISCARD'] + ); + expectBlanked( + blankCppLeadingAttributeMacros, + ' PROTOBUF_FUTURE_ADD_EARLY_NODISCARD static constexpr size_t\n kMax = 3;', + ['PROTOBUF_FUTURE_ADD_EARLY_NODISCARD'] + ); + expectBlanked( + blankCppLeadingAttributeMacros, + ' ABSL_ATTRIBUTE_REINITIALIZES void Clear() final;', + ['ABSL_ATTRIBUTE_REINITIALIZES'] + ); + expectBlanked( + blankCppLeadingAttributeMacros, + ' [[nodiscard]] PROTOBUF_NDEBUG_INLINE Ptr Make(Args&&... args) {}', + ['PROTOBUF_NDEBUG_INLINE'] + ); + expectBlanked( + blankCppLeadingAttributeMacros, + 'GTEST_API_ std::string JoinAsTuple(const Strings& fields);', + ['GTEST_API_'] + ); + expectBlanked(blankCppLeadingAttributeMacros, 'FMT_EXPORT struct as_identifiers_t {};', ['FMT_EXPORT']); + expectBlanked( + blankCppLeadingAttributeMacros, + 'PROTOBUF_NDEBUG_INLINE Any::Impl_::Impl_(Arena* arena) : x_(arena) {}', + ['PROTOBUF_NDEBUG_INLINE'] + ); + expectBlanked( + blankCppLeadingAttributeMacros, + 'PROTOBUF_ATTRIBUTE_NO_DESTROY PROTOBUF_CONSTINIT\n PROTOBUF_ATTRIBUTE_INIT_PRIORITY1 const AnyGlobals Any_globals_;', + ['PROTOBUF_ATTRIBUTE_NO_DESTROY PROTOBUF_CONSTINIT\n PROTOBUF_ATTRIBUTE_INIT_PRIORITY1'] + ); + }); + + it('leaves a type in capitals alone', () => { + for (const c of [ + 'DWORD_PTR value = 0;', + 'SIZE_T const n = 3;', + 'UINT_PTR const* p = q;', + 'HANDLE_T Open(int flags);', + 'UINT_PTR Foo::Bar() { return 0; }', + 'RESULT_T operator==(const A& a);', + 'RESULT_T\nGnuStyle(int x) {}', + 'int x = a |\n SOME_CONST\n | OTHER;', + '// PROTOBUF_FUTURE_ADD_EARLY_NODISCARD absl::string_view name() const;', + // A macro inside a string concatenation, with a declaration after it. + 'static const char kHelp[] =\n "using " GTEST_NAME_\n ". You can use the\\n"\n " --" GTEST_FLAG_PREFIX_\n "list_tests\\n";\nstatic bool g_help = false;', + ]) { + expect(blankCppLeadingAttributeMacros(c)).toBe(c); + } + }); + + it('blanks only the macros of a run, not a comment between them', () => { + expectBlanked( + blankCppLeadingAttributeMacros, + 'PROTOBUF_ATTRIBUTE_NO_DESTROY /* keep */ PROTOBUF_CONSTINIT const Foo kFoo;', + ['PROTOBUF_ATTRIBUTE_NO_DESTROY', 'PROTOBUF_CONSTINIT'] + ); + }); + + it('blanks a lone macro line with comments under it, but not an operand', () => { + const fmt = 'FMT_BEGIN_EXPORT\n\n// A generic formatting context with custom output iterator and character\n/* (code unit) support. */\ntemplate class generic_context {};\n'; + expectBlanked(blankLoneMacroLines, fmt, ['FMT_BEGIN_EXPORT']); + // A string literal continues the expression, comment or not. + const operand = 'const char* s =\n "x"\n FOO_PREFIX\n // comment\n "y";\n'; + expect(blankLoneMacroLines(operand)).toBe(operand); + }); + + it('keeps every offset through the whole C++ preParse', () => { + const src = 'class PROTOBUF_EXPORT X final : public Y {\n const D* PROTOBUF_NONNULL d() const ABSL_ATTRIBUTE_LIFETIME_BOUND;\n int n_ GUARDED_BY(mu_);\n};\r\n'; + const out = cppExtractor.preParse!(src, 'x.h'); + expect(out.length).toBe(src.length); + expect(out.split(/\r?\n/).map((l) => l.length)).toEqual(src.split(/\r?\n/).map((l) => l.length)); + expect(out).not.toMatch(/PROTOBUF_|ABSL_|GUARDED_BY/); + }); +}); + +const PB_H = `// Generated by the protocol buffer compiler. DO NOT EDIT! +#ifndef GOOGLE_PROTOBUF_INCLUDED_any_2eproto +#define GOOGLE_PROTOBUF_INCLUDED_any_2eproto + +#include "google/protobuf/message.h" + +namespace google { +namespace protobuf { + +class PROTOBUF_EXPORT PROTOBUF_FUTURE_ADD_EARLY_WARN_UNUSED Any final : public ::google::protobuf::Message +/* @@protoc_insertion_point(class_definition:google.protobuf.Any) */ { + public: + Any() : Any(nullptr) {} + ~Any() PROTOBUF_FINAL; + + [[nodiscard]] const ::google::protobuf::UnknownFieldSet& unknown_fields() const + ABSL_ATTRIBUTE_LIFETIME_BOUND { + return _internal_metadata_.unknown_fields(); + } + [[nodiscard]] ::google::protobuf::UnknownFieldSet* PROTOBUF_NONNULL mutable_unknown_fields() + ABSL_ATTRIBUTE_LIFETIME_BOUND { + return _internal_metadata_.mutable_unknown_fields(); + } + + [[nodiscard]] static const ::google::protobuf::Descriptor* PROTOBUF_NONNULL descriptor() { + return GetDescriptor(); + } + [[nodiscard]] static const ::google::protobuf::Descriptor* PROTOBUF_NONNULL + GetDescriptor() { + return default_instance().GetMetadata().descriptor; + } + [[nodiscard]] static const ::google::protobuf::Reflection* PROTOBUF_NONNULL GetReflection() { + return default_instance().GetMetadata().reflection; + } + void Swap(Any* PROTOBUF_NONNULL other) { + InternalSwap(other); + } + [[nodiscard]] ::std::string* PROTOBUF_NONNULL mutable_type_url(); + + private: + void InternalSwap(Any* PROTOBUF_NONNULL other); +}; + +inline ::std::string* PROTOBUF_NONNULL Any::mutable_type_url() { + return _impl_.type_url_.Mutable(); +} + +} // namespace protobuf +} // namespace google + +#endif // GOOGLE_PROTOBUF_INCLUDED_any_2eproto +`; + +const MESSAGE_H = `namespace google { +namespace protobuf { + +class Message { + public: + virtual ~Message(); +}; + +} // namespace protobuf +} // namespace google +`; + +const PB_CC = `#include "google/protobuf/any.pb.h" + +namespace google { +namespace protobuf { + +PROTOBUF_ATTRIBUTE_NO_DESTROY PROTOBUF_CONSTINIT PROTOBUF_EXPORT + PROTOBUF_ATTRIBUTE_INIT_PRIORITY1 const AnyGlobalsTypeInternal Any_globals_ + PROTOBUF_MESSAGE_GLOBALS_SECTION(.data.rel.ro); + +void Any::InternalSwap(Any* PROTOBUF_RESTRICT PROTOBUF_NONNULL other) { + _impl_.type_url_.InternalSwap(&other->_impl_.type_url_); +} + +} // namespace protobuf +} // namespace google +`; + +const REPEATED_H = `#include + +namespace google { +namespace protobuf { + +template +class ABSL_ATTRIBUTE_WARN_UNUSED PROTOBUF_DECLSPEC_EMPTY_BASES + RepeatedField final + : private internal::RepeatedFieldBase { + public: + int size() const { return size_; } + + private: + int size_; +}; + +} // namespace protobuf +} // namespace google +`; + +const DESCRIPTOR_H = `#include + +namespace google { +namespace protobuf { + +class PROTOBUF_EXPORT Descriptor : private internal::SymbolBase { + public: + PROTOBUF_FUTURE_ADD_EARLY_NODISCARD absl::string_view name() const { + return all_names_[0]; + } + PROTOBUF_FUTURE_ADD_EARLY_NODISCARD absl::string_view full_name() const { + return all_names_[1]; + } + PROTOBUF_FUTURE_ADD_EARLY_NODISCARD const FileDescriptor* file() const; +}; + +} // namespace protobuf +} // namespace google +`; + +const FMT_H = `#include + +#define FMT_BEGIN_EXPORT +FMT_BEGIN_NAMESPACE + +FMT_BEGIN_EXPORT + +// A generic formatting context with custom output iterator and character +// (code unit) support. +template class generic_context { + private: + OutputIt out_; + + public: + constexpr auto out() -> OutputIt { return out_; } +}; + +FMT_END_EXPORT +FMT_END_NAMESPACE +`; + +const LEVELDB_CC = `#include "port/port.h" + +namespace leveldb { + +class Counter { + public: + void IncrementBy(int count) LOCKS_EXCLUDED(mu_) { + MutexLock l(&mu_); + count_ += count; + } + void Lock() EXCLUSIVE_LOCK_FUNCTION() { mu_.Lock(); } + + private: + port::Mutex mu_; + int count_ GUARDED_BY(mu_); +}; + +} // namespace leveldb +`; + +describe('C++ classes written around attribute macros', () => { + let root = ''; + let cg: CodeGraph; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-cpp-attr-macros-')); + const files: Record = { + 'src/google/protobuf/message.h': MESSAGE_H, + 'src/google/protobuf/any.pb.h': PB_H, + 'src/google/protobuf/any.pb.cc': PB_CC, + 'src/google/protobuf/repeated_field.h': REPEATED_H, + 'src/google/protobuf/descriptor.h': DESCRIPTOR_H, + 'include/fmt/format.h': FMT_H, + 'db/counter.cc': LEVELDB_CC, + }; + for (const [rel, text] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), text); + } + cg = await CodeGraph.init(root, { index: true }); + }); + + afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); + }); + + const nodes = (file: string) => cg.getNodesInFile(file); + const qualified = (file: string, kind: string) => + nodes(file).filter((n) => n.kind === kind).map((n) => n.qualifiedName).sort(); + + it('indexes a generated message class with its members as methods', () => { + const file = 'src/google/protobuf/any.pb.h'; + expect(qualified(file, 'class')).toEqual(['google::protobuf::Any']); + expect(qualified(file, 'method')).toEqual( + expect.arrayContaining([ + 'google::protobuf::Any::Any', + 'google::protobuf::Any::GetDescriptor', + 'google::protobuf::Any::GetReflection', + 'google::protobuf::Any::Swap', + 'google::protobuf::Any::descriptor', + 'google::protobuf::Any::mutable_type_url', + 'google::protobuf::Any::mutable_unknown_fields', + 'google::protobuf::Any::unknown_fields', + ]) + ); + // No member is left at namespace level, and no node is named after a macro. + expect(qualified(file, 'function')).toEqual([]); + expect(nodes(file).filter((n) => /PROTOBUF_|ABSL_/.test(n.qualifiedName))).toEqual([]); + const any = nodes(file).find((n) => n.kind === 'class')!; + const bases = cg.getOutgoingEdges(any.id).filter((e) => e.kind === 'extends'); + expect(bases.map((e) => cg.getNode(e.target)?.qualifiedName)).toEqual(['google::protobuf::Message']); + }); + + it('indexes an out-of-line definition whose parameter carries an annotation', () => { + expect(qualified('src/google/protobuf/any.pb.cc', 'method')).toEqual(['google::protobuf::Any::InternalSwap']); + }); + + it('indexes a class whose head stacks macros over several lines', () => { + const file = 'src/google/protobuf/repeated_field.h'; + expect(qualified(file, 'class')).toEqual(['google::protobuf::RepeatedField']); + expect(qualified(file, 'method')).toEqual(['google::protobuf::RepeatedField::size']); + }); + + it('indexes members that open with an attribute macro', () => { + const file = 'src/google/protobuf/descriptor.h'; + expect(qualified(file, 'method')).toEqual([ + 'google::protobuf::Descriptor::full_name', + 'google::protobuf::Descriptor::name', + ]); + }); + + it('indexes a class under a lone macro line and a comment', () => { + expect(qualified('include/fmt/format.h', 'method')).toEqual(['generic_context::out']); + expect(qualified('include/fmt/format.h', 'function')).toEqual([]); + }); + + it('keeps annotated methods instead of phantoms named after the annotation', () => { + const file = 'db/counter.cc'; + expect(qualified(file, 'method')).toEqual(['leveldb::Counter::IncrementBy', 'leveldb::Counter::Lock']); + expect(nodes(file).filter((n) => /LOCKS|GUARDED/.test(n.name))).toEqual([]); + }); +}); diff --git a/__tests__/cpp-brace-scopes.test.ts b/__tests__/cpp-brace-scopes.test.ts index 2740f6007..f29c76b09 100644 --- a/__tests__/cpp-brace-scopes.test.ts +++ b/__tests__/cpp-brace-scopes.test.ts @@ -19,6 +19,10 @@ * Lookups that start from a declaration's scope (base classes, alias * receivers, constructors) then had only a unique-name guess left. * + * The C++ preParse now blanks both of those macros, so the fixtures below + * keep the shapes with macros it doesn't know: a one-word attribute macro + * (`NODISCARD`) and a lowercase one in a class head (`cacheline_aligned(64U)`). + * * For a file whose tree has errors, the walker now takes each declaration's * namespaces and enclosing classes from the source's braces, which the * parser rarely misreads; a class the tree glued into a declaration's type is @@ -54,7 +58,7 @@ namespace protobuf { class Names { public: - PROTOBUF_FUTURE_ADD_EARLY_NODISCARD absl::string_view name() const { + NODISCARD absl::string_view name() const { return absl::string_view(payload_); } @@ -77,7 +81,7 @@ namespace clock_cache { class FixedTable : public BaseTable { public: - struct ALIGN_AS(64U) HandleImpl : public ClockHandle { + struct cacheline_aligned(64U) HandleImpl : public ClockHandle { RelaxedAtomic displacements{}; bool standalone = false; }; @@ -224,7 +228,7 @@ describe('C++ scopes from the source braces of a file whose tree has errors', () }); it('a file that parses cleanly keeps the tree scopes', () => { - const clean = PROTOBUF_SHAPE.replace('PROTOBUF_FUTURE_ADD_EARLY_NODISCARD ', ''); + const clean = PROTOBUF_SHAPE.replace('NODISCARD ', ''); const tree = getParser('cpp')!.parse(cppExtractor.preParse!(clean, 'descriptor.h'))!; expect(tree.rootNode.hasError).toBe(false); tree.delete(); diff --git a/__tests__/kernel-ccpp-parity.test.ts b/__tests__/kernel-ccpp-parity.test.ts index 720ffb623..33baf912e 100644 --- a/__tests__/kernel-ccpp-parity.test.ts +++ b/__tests__/kernel-ccpp-parity.test.ts @@ -232,6 +232,52 @@ describe.skipIf(!kernelBuilt)('kernel C/C++ extraction parity', () => { assertParity('fixtures/kern.cu', cuda, 'cpp'); }); + // protobuf's generated headers and leveldb's thread annotations put + // attribute macros where tree-sitter can't read them: stacked in a class + // head, between a pointer and its name, after a parameter list or a field, + // opening a member. The hoisted preParse blanks them all, so the file parses + // clean and goes through the kernel instead of deferring — at parity. + it.each(['\n', '\r\n'])('attribute-macro shapes parse clean through the hoisted preParse (%j)', (eol) => { + const source = [ + 'namespace google {', + 'namespace protobuf {', + 'class PROTOBUF_EXPORT PROTOBUF_FUTURE_ADD_EARLY_WARN_UNUSED Any final : public ::google::protobuf::Message', + '/* @@protoc_insertion_point(class_definition:google.protobuf.Any) */ {', + ' public:', + ' ~Any() PROTOBUF_FINAL;', + ' [[nodiscard]] const ::google::protobuf::UnknownFieldSet& unknown_fields() const', + ' ABSL_ATTRIBUTE_LIFETIME_BOUND {', + ' return _internal_metadata_.unknown_fields();', + ' }', + ' [[nodiscard]] static const ::google::protobuf::Descriptor* PROTOBUF_NONNULL', + ' GetDescriptor() {', + ' return default_instance().GetMetadata().descriptor;', + ' }', + ' PROTOBUF_FUTURE_ADD_EARLY_NODISCARD absl::string_view name() const { return name_; }', + ' void Swap(Any* PROTOBUF_RESTRICT PROTOBUF_NONNULL other) { InternalSwap(other); }', + '', + ' private:', + ' void InternalSwap(Any* PROTOBUF_NONNULL other);', + ' int count_ GUARDED_BY(mu_);', + '};', + 'void Any::InternalSwap(Any* PROTOBUF_NONNULL other) { swap(*other); }', + '} // namespace protobuf', + '} // namespace google', + '', + ].join(eol); + assertParity('fixtures/any.pb.h', source, 'cpp', 6); + process.env.CODEGRAPH_KERNEL_LANGS = 'all'; + const result = tryKernelExtract('fixtures/any.pb.h', source, 'cpp')!; + const methods = result.nodes.filter((n) => n.kind === 'method').map((n) => n.qualifiedName); + expect(methods).toEqual(expect.arrayContaining([ + 'google::protobuf::Any::GetDescriptor', + 'google::protobuf::Any::InternalSwap', + 'google::protobuf::Any::Swap', + 'google::protobuf::Any::name', + 'google::protobuf::Any::unknown_fields', + ])); + }); + // Every torture fixture again with CRLF line endings — the shape every // Windows autocrlf checkout has. Derived in memory (not a checked-in CRLF // file) so no platform or editor can silently normalize it away. Pins the diff --git a/src/extraction/languages/c-cpp.ts b/src/extraction/languages/c-cpp.ts index ac238978d..c2e1f59be 100644 --- a/src/extraction/languages/c-cpp.ts +++ b/src/extraction/languages/c-cpp.ts @@ -378,13 +378,389 @@ function isMacroMisparsedTypeDecl(node: SyntaxNode): boolean { * declarations (`struct FOO var;`, `class FOO obj = …`) untouched, since those * end in `;` / `=` / `[`, never `:` / `{`. C++-only (wired into cppExtractor), * so C's heavier use of `struct TAG var;` never reaches it. + * + * The head may carry several macros, and a macro may take arguments: protobuf's + * generated `class PROTOBUF_EXPORT PROTOBUF_FUTURE_ADD_EARLY_WARN_UNUSED Any + * final : public Message`, `class ABSL_ATTRIBUTE_WARN_UNUSED + * PROTOBUF_DECLSPEC_EMPTY_BASES RepeatedField final`, rocksdb's `struct + * ALIGN_AS(64U) HandleImpl : public ClockHandle`. The whole run is blanked; the + * last name before the guard is the class. The name may be followed by template + * arguments, for a specialization (`class PROTOBUF_DECLSPEC_EMPTY_BASES + * RepeatedFieldProxyWithSet {`). Matched on the code alone, so + * a comment or string never takes part. */ +const CPP_CLASS_HEAD_MACROS_RE = + /\b(?:class|struct)\s+((?:[A-Z][A-Z0-9_]+(?:\s*\((?:[^()]|\([^()]*\))*\))?\s+)+)(?!final\b)[A-Za-z_]\w*(?:\s*::\s*[A-Za-z_]\w*)*/g; export function blankCppExportMacros(source: string): string { if (source.indexOf('class') === -1 && source.indexOf('struct') === -1) return source; - return source.replace( - /\b(class|struct)(\s+)([A-Z][A-Z0-9_]+)(?=\s+[A-Za-z_]\w*(?:\s+final)?\s*[:{])/g, - (_m, kw, ws, macro) => kw + ws + ' '.repeat(macro.length) - ); + if (!/\b(?:class|struct)\s+[A-Z][A-Z0-9_]/.test(source)) return source; + const code = maskCppCode(source); + const spans: Array<[number, number]> = []; + const re = new RegExp(CPP_CLASS_HEAD_MACROS_RE.source, 'g'); + let m: RegExpExecArray | null; + while ((m = re.exec(code)) !== null) { + let i = skipCppSpace(code, m.index + m[0].length); + if (code[i] === '<') { + i = skipCppAngleArgs(code, i); + if (i < 0) continue; + i = skipCppSpace(code, i); + } + if (/^final\b/.test(code.slice(i, i + 6))) i = skipCppSpace(code, i + 5); + if (code[i] !== '{' && !(code[i] === ':' && code[i + 1] !== ':')) continue; + const run = m[1] as string; + const start = m.index + m[0].indexOf(run); + pushCppRunTokens(spans, run, start); + } + return blankCppSpans(source, spans, code); +} + +// Comments and string/char literals (raw strings are masked beforehand by +// maskCppRawStrings). The char-literal boundary leaves digit separators alone. +const CPP_COMMENT_OR_LITERAL_RE = + /\/\/(?:\\\r?\n|[^\r\n])*|\/\*[\s\S]*?(?:\*\/|$)|"(?:\\[\s\S]|[^"\\])*(?:"|$)|(? m.replace(/[^\r\n]/g, ' ')); +} + +/** + * The code the macro scans below match on, at the source's offsets: comments + * are spaces, a string or char literal keeps its place as `\x01`s (it is a + * token, not space: in `"… using " GTEST_NAME_` the macro does not start a + * line, and a run of macros never reaches across a literal), and + * preprocessor directives with their `\` continuation lines are spaces — the + * `)` ending `#if !defined(__MINGW32__)` closes no parameter list, so the + * `TEST(file_test, open_windows_file) {` under it is not a trailing macro. + */ +function maskCppCode(source: string): string { + if (cppMaskMemo !== null && cppMaskMemo.source === source) return cppMaskMemo.code; + const code = source + .replace(CPP_COMMENT_OR_LITERAL_RE, (m) => m.replace(/[^\r\n]/g, m.startsWith('/') ? ' ' : '\x01')) + .replace(/^[ \t]*#(?:\\\r?\n|[^\r\n])*/gm, (m) => m.replace(/[^\r\n]/g, ' ')); + cppMaskMemo = { source, code }; + return code; +} + +// The last source maskCppCode masked. The passes run one after another on +// the same text: one that blanks nothing hands the next the same string, and +// one that blanks hands it a mask blanked the same way (blankCppSpans), so a +// file is masked once rather than once per pass. +let cppMaskMemo: { source: string; code: string } | null = null; + +// One macro, with its arguments, inside a run matched on the masked code. +const CPP_RUN_TOKEN_RE = /[A-Z][A-Z0-9_]*(?:\s*\((?:[^()]|\((?:[^()]|\([^()]*\))*\))*\))?/g; + +/** Add a span for each macro of `run` (found at `start`), leaving what lies between them. */ +function pushCppRunTokens(spans: Array<[number, number]>, run: string, start: number): void { + for (const t of run.matchAll(CPP_RUN_TOKEN_RE)) spans.push([start + t.index, start + t.index + t[0].length]); +} + +/** + * Replace each [start, end) span of `source` with spaces, keeping line + * endings. Given the mask the spans were found on, blank it the same way and + * keep it for the next pass: a span holds whole macros (with their arguments), + * so the result is what masking the new source would give. + */ +function blankCppSpans(source: string, spans: ReadonlyArray, code?: string): string { + if (spans.length === 0) return source; + const blank = (text: string): string => { + const chars = text.split(''); + for (const [start, end] of spans) { + for (let k = start; k < end; k++) { + if (chars[k] !== '\n' && chars[k] !== '\r') chars[k] = ' '; + } + } + return chars.join(''); + }; + const out = blank(source); + if (code !== undefined) cppMaskMemo = { source: out, code: blank(code) }; + return out; +} + +function skipCppSpace(code: string, i: number): number { + while (i < code.length && /\s/.test(code[i] as string)) i++; + return i; +} + +/** + * The index just past the template argument list opening at `code[open]` + * (`<`), or -1 when it doesn't close before a `;`, `{` or `}` — then it was a + * comparison, not template arguments. Angle brackets inside parentheses don't + * count (`Foo`). + */ +function skipCppAngleArgs(code: string, open: number): number { + let angles = 0; + let parens = 0; + const limit = Math.min(code.length, open + 4000); + for (let i = open; i < limit; i++) { + const c = code[i]; + if (c === '(') parens++; + else if (c === ')') { + if (--parens < 0) return -1; + } else if (c === ';' || c === '{' || c === '}') return -1; + else if (parens === 0 && c === '<') angles++; + else if (parens === 0 && c === '>' && code[i - 1] !== '-') { + if (--angles === 0) return i + 1; + } + } + return -1; +} + +/** + * Blank attribute macros between a pointer or reference and the name it + * declares: protobuf's nullability annotations (`const Descriptor* + * PROTOBUF_NONNULL descriptor()`, `Any* PROTOBUF_RESTRICT PROTOBUF_NONNULL + * other`, `const FieldDescriptor* PROTOBUF_NULLABLE* PROTOBUF_NONNULL field`). + * tree-sitter takes the macro for the declared name and the real name for an + * error: a member keeps a broken declarator, an out-of-line definition gets the + * macro glued into its qualified name (`PROTOBUF_NONNULL Any::mutable_type_url`), + * and a definition whose parameter carries one (`void Any::Swap(Any* + * PROTOBUF_NONNULL other) { … }`) parses as a variable and drops out of the + * index. Generated `.pb.h`/`.pb.cc` files carry thousands of them. + * + * Matched on the code alone: the macro is ALL-CAPS with an underscore, right + * after `*` or `&`, and followed by the name (or, after `*`, by another `*`). + * C++ never puts two names in a row there, and a product (`a * MAX_LEN + 1`) + * or a bit test (`flags & FOO_BIT)`) is followed by an operator or a closer + * instead; a `*` after `)`, `]` or a number is a product too (`sizeof(int) * + * CHAR_BIT * 3`). A trailing macro with nothing after it (`Foo* + * PROTOBUF_NONNULL)`) parses as the parameter name and is left alone. + * C++-only. + */ +const CPP_POINTER_ANNOTATION_RE = + /([*&])\s*((?:[A-Z][A-Z0-9]*_[A-Z0-9_]*(?:\s+|(?=\*)))+)(?=\*|::|(?!(?:and|and_eq|bitand|bitor|compl|not|not_eq|or|or_eq|xor|xor_eq)\b)[A-Za-z_~])/g; +export function blankCppPointerAnnotationMacros(source: string): string { + if (!/[*&]\s*[A-Z][A-Z0-9]*_/.test(source)) return source; + const code = maskCppCode(source); + const spans: Array<[number, number]> = []; + const re = new RegExp(CPP_POINTER_ANNOTATION_RE.source, 'g'); + let m: RegExpExecArray | null; + while ((m = re.exec(code)) !== null) { + const run = m[2] as string; + const start = m.index + m[0].length - run.length; + // After `&`, only a name: `flags & FOO_BIT\n && ok` is an expression. + if (m[1] === '&' && code[start + run.length] === '*') continue; + // A `*` after a closing bracket or a number multiplies: `sizeof(int) * CHAR_BIT * 3`. + if (/(?:[)\]]|\b\d[\w.]*)\s*$/.test(code.slice(Math.max(0, m.index - 40), m.index))) continue; + pushCppRunTokens(spans, run, start); + } + return blankCppSpans(source, spans, code); +} + +// Keywords that take a parenthesized operand: a `)` closing one of these is +// not a parameter list, so a token after it is not a trailing attribute. +// `constexpr` / `consteval` stand before the condition of `if constexpr (…)`. +const CPP_PAREN_KEYWORDS = new Set([ + 'if', 'while', 'for', 'switch', 'catch', 'return', 'sizeof', 'alignof', 'alignas', 'decltype', + 'typeid', 'noexcept', 'throw', 'static_assert', 'new', 'delete', 'case', 'requires', 'co_await', + 'co_yield', 'co_return', 'asm', '__asm__', '__attribute__', '__attribute', '__declspec', '_Pragma', + 'else', 'do', 'constexpr', 'consteval', 'explicit', 'typeof', '__typeof__', '__typeof', '_Alignof', + '__alignof__', 'volatile', '__volatile__', +]); + +/** + * Blank attribute macros after a function's parameter list: protobuf's + * `unknown_fields() const\n ABSL_ATTRIBUTE_LIFETIME_BOUND {` and `~Any() + * PROTOBUF_FINAL;`, and the thread-safety annotations of leveldb, rocksdb and + * abseil (`void Wait() LOCKS_EXCLUDED(mu_) {`, `void Lock() + * EXCLUSIVE_LOCK_FUNCTION();`). tree-sitter reads the macro as the start of a + * new declaration, so the method loses its `;` or its body: an inline body + * becomes a stray block, and in a long run of members the class can end early. + * + * Matched on the code alone, so it can't touch an expression: + * - the `)` closes a parameter list: the `(` it matches follows a function + * name (`name(`, `~Name(`, `operator==(`, `operator()(`), not a keyword + * (`if (x) FOO;`), a cast (`= (int) MAX_VALUE;`) or a macro call + * (`FMT_PRAGMA_CLANG(…)` above `TEST(std_test, bitint) {`); + * - between `)` and the macro only `const`, `volatile`, `&`, `&&`, + * `noexcept`, `override` or `final` may stand, and no blank line; + * - the macro (ALL-CAPS, optionally with balanced arguments; several may + * stack) is followed by what may end a declarator: `{`, `;`, `=`, a + * constructor's `:`, `->`, or `override`/`final`/`const`/`noexcept`. + * C++-only. + */ +const CPP_TRAILING_ATTR_CANDIDATE_RE = + /\)((?:\s*(?:const|volatile|noexcept|override|final|&&|&)(?![\w]))*)\s*(?=[A-Z][A-Z0-9_]{2,}\b)/g; +const CPP_TRAILING_ATTR_RUN_RE = + /(?:[A-Z][A-Z0-9_]{2,}\b(?:\s*\((?:[^()]|\((?:[^()]|\([^()]*\))*\))*\))?\s*)+/y; +const CPP_TRAILING_ATTR_FOLLOW_RE = /(?:\{|;|=(?!=)|:(?!:)|->|(?:override|final|const|noexcept)\b)/y; +export function blankCppTrailingAttributeMacros(source: string): string { + if (!/\)[\s\w&]*?\b[A-Z][A-Z0-9_]{2,}\b/.test(source)) return source; + const code = maskCppCode(source); + const spans: Array<[number, number]> = []; + const re = new RegExp(CPP_TRAILING_ATTR_CANDIDATE_RE.source, 'g'); + const run = new RegExp(CPP_TRAILING_ATTR_RUN_RE.source, 'y'); + const follow = new RegExp(CPP_TRAILING_ATTR_FOLLOW_RE.source, 'y'); + let m: RegExpExecArray | null; + while ((m = re.exec(code)) !== null) { + const macroStart = m.index + m[0].length; + if (/\n[ \t]*\r?\n/.test(m[0])) continue; + run.lastIndex = macroStart; + const r = run.exec(code); + if (!r) continue; + follow.lastIndex = macroStart + r[0].length; + if (!follow.test(code)) continue; + if (!closesParameterList(code, m.index)) continue; + pushCppRunTokens(spans, r[0], macroStart); + re.lastIndex = macroStart + r[0].length; + } + return blankCppSpans(source, spans, code); +} + +// Type keywords that may stand right before a declared name (`int count_ …`). +const CPP_TYPE_KEYWORDS = new Set([ + 'void', 'bool', 'char', 'char8_t', 'char16_t', 'char32_t', 'wchar_t', 'short', 'int', 'long', 'float', + 'double', 'signed', 'unsigned', 'auto', +]); +const CPP_KEYWORDS = new Set([ + ...CPP_TYPE_KEYWORDS, ...CPP_PAREN_KEYWORDS, 'const', 'volatile', 'static', 'extern', 'inline', 'virtual', + 'explicit', 'constexpr', 'consteval', 'constinit', 'friend', 'mutable', 'thread_local', 'register', + 'typedef', 'using', 'template', 'typename', 'class', 'struct', 'union', 'enum', 'namespace', 'operator', + 'goto', 'break', 'continue', 'default', 'public', 'private', 'protected', 'this', 'true', 'false', + 'nullptr', 'and', 'or', 'not', 'xor', 'bitand', 'bitor', 'compl', 'and_eq', 'or_eq', 'xor_eq', 'not_eq', +]); + +/** + * Blank attribute macros after the name a declaration declares, or after its + * array brackets: thread-safety annotations on fields (leveldb's `port::CondVar + * cv GUARDED_BY(mu);`, abseil's `ABSL_GUARDED_BY`), annotated parameters + * (`const absl::Cord* cord ABSL_ATTRIBUTE_LIFETIME_BOUND)`), and the section + * attributes of protobuf's generated globals (`Any_globals_ + * PROTOBUF_MESSAGE_GLOBALS_SECTION(.data.rel.ro);`, `offsets[] + * ABSL_ATTRIBUTE_SECTION_VARIABLE(protodesc_cold) = {`). tree-sitter reads + * the macro as a second declarator and errors. + * + * Matched on the code alone, so it can't touch a variable named in capitals: + * - the macro is ALL-CAPS with an underscore, optionally with balanced + * arguments, and is followed by `;`, `=`, `,` or `)`; + * - before it stands `]`, or a name with a lowercase letter that itself + * follows a type: a type name, a type keyword, `*`, `&`, `>`, or `const` + * after a pointer. `Foo DEFAULT_OPTIONS;` and `const Foo kMAX_SIZE;` + * declare those capitals (nothing names a type before `Foo`), `typedef Foo + * BAR_T;` names a type, and `return x FOO;` or `f(a, b FOO)` have no type + * before the name. + * C++-only. + */ +const CPP_DECLARATOR_ATTR_CANDIDATE_RE = /(? = []; + const re = new RegExp(CPP_DECLARATOR_ATTR_CANDIDATE_RE.source, 'g'); + const run = new RegExp(CPP_DECLARATOR_ATTR_RUN_RE.source, 'y'); + let m: RegExpExecArray | null; + while ((m = re.exec(code)) !== null) { + const macroStart = m.index; + // Whitespace before the macro, and before that a name or `]`. + let last = macroStart - 1; + while (last >= 0 && /\s/.test(code[last] as string)) last--; + if (last === macroStart - 1 || last < 0 || !/[\w\]]/.test(code[last] as string)) continue; + run.lastIndex = macroStart; + const r = run.exec(code); + if (!r || !/^[;,)]|^=(?!=)/.test(code.slice(macroStart + r[0].length, macroStart + r[0].length + 2))) continue; + if (code[last] !== ']' && !followsDeclaredName(code, last)) continue; + pushCppRunTokens(spans, r[0], macroStart); + re.lastIndex = macroStart + r[0].length; + } + return blankCppSpans(source, spans, code); +} + +/** True when the identifier ending at `code[last]` is a name a declaration declares. */ +function followsDeclaredName(code: string, last: number): boolean { + let start = last; + while (start > 0 && /\w/.test(code[start - 1] as string)) start--; + const name = code.slice(start, last + 1); + if (!/^[A-Za-z_]\w*$/.test(name) || !/[a-z]/.test(name) || CPP_KEYWORDS.has(name)) return false; + let i = start - 1; + while (i >= 0 && /\s/.test(code[i] as string)) i--; + const c = code[i]; + if (c === '*' || c === '&') return true; + if (c === '>') return code[i - 1] !== '-'; + if (!c || !/\w/.test(c)) return false; + let j = i; + while (j > 0 && /\w/.test(code[j - 1] as string)) j--; + const before = code.slice(j, i + 1); + if (before === 'const' || before === 'volatile') { + let k = j - 1; + while (k >= 0 && /\s/.test(code[k] as string)) k--; + return code[k] === '*' || code[k] === '&'; + } + if (CPP_TYPE_KEYWORDS.has(before)) return true; + return /^[A-Za-z_]\w*$/.test(before) && !CPP_KEYWORDS.has(before); +} + +/** True when the `)` at `code[close]` closes a function declarator's parameter list. */ +function closesParameterList(code: string, close: number): boolean { + let depth = 0; + let open = -1; + for (let i = close; i >= 0 && close - i < 4000; i--) { + const c = code[i]; + if (c === ')') depth++; + else if (c === '(') { + if (--depth === 0) { open = i; break; } + } else if (c === ';' || c === '{' || c === '}') return false; + } + if (open < 0) return false; + const before = code.slice(Math.max(0, open - 200), open); + // `operator()(…)`, `operator==(…)`, `operator bool(…)` + if (/\boperator\s*(?:\(\s*\)|\[\s*\]|[^\s\w()[\]]{1,3}|\s+[A-Za-z_][\w\s:<>*&]*)\s*$/.test(before)) return true; + const name = /([A-Za-z_]\w*)\s*$/.exec(before)?.[1]; + return !!name && !CPP_PAREN_KEYWORDS.has(name) && !/^[A-Z][A-Z0-9_]*$/.test(name); +} + +/** + * Blank attribute macros that open a declaration, at the start of a line or + * after a `[[…]]` attribute: protobuf's `PROTOBUF_FUTURE_ADD_EARLY_NODISCARD + * absl::string_view name() const {` (250 members of descriptor.h alone), + * `ABSL_ATTRIBUTE_REINITIALIZES void Clear()`, `[[nodiscard]] + * PROTOBUF_NDEBUG_INLINE Ptr Make(…)`, googletest's `GTEST_API_ + * std::string JoinAsTuple(…)`, {fmt}'s `FMT_EXPORT struct …`. tree-sitter + * takes the macro for the type and everything after it for an error, which + * swallows the members that follow. `blankCLeadingAttrMacros` only knows a + * plain `MACRO Ret name(`; C++ adds qualified and template types, specifiers + * and keywords. + * + * The macro (ALL-CAPS with an underscore; several may stack) must be followed + * by what can't follow a type name: a declaration keyword (`static`, + * `inline`, `virtual`, `void`, `struct`, `template`, `return`, `[[`, …) or a + * whole declaration of its own, a type and then the declared name (`Type + * name(`, `ns::Type* name;`). A type is followed by the name alone + * (`DWORD_PTR value = 0;`, `HANDLE_T Open(…)`), so it never matches. `const` + * after the token could be either (`SIZE_T const n`), so it only counts with a + * type and name after it. Matched on the code alone. C++-only. + */ +const CPP_TYPE_ARGS = '<(?:[^<>;{}]|<(?:[^<>;{}]|<[^<>;{}]*>)*>)*>'; +const CPP_TYPE_NAME = `(?:::\\s*)?(?!(?:const|volatile|operator|return)\\b)[A-Za-z_]\\w*(?:\\s*${CPP_TYPE_ARGS})?(?:\\s*::\\s*[A-Za-z_]\\w*(?:\\s*${CPP_TYPE_ARGS})?)*`; +const CPP_DECLARED_NAME = '(?:(?:::\\s*)?~?[A-Za-z_]\\w*(?:\\s*::\\s*~?[A-Za-z_]\\w*)*\\s*[(;=[{]|operator\\b)'; +const CPP_LEADING_ATTR_RE = new RegExp( + '(^[ \\t]*|\\]\\][ \\t]*)((?:[A-Z][A-Z0-9]*_[A-Z0-9_]*\\s+)+)(?=' + + '(?:static|inline|virtual|explicit|constexpr|consteval|constinit|friend|extern|thread_local|mutable|' + + 'typedef|using|template|typename|class|struct|union|enum|void|bool|char|char8_t|char16_t|char32_t|' + + 'wchar_t|short|int|long|float|double|signed|unsigned|auto|decltype|return)\\b|\\[\\[|' + + // An out-of-line constructor or destructor: `Any::Impl_::Impl_(`, `Any::~Any(`. + '(?:[A-Za-z_]\\w*\\s*::\\s*)*([A-Za-z_]\\w*)\\s*::\\s*~?\\3\\s*\\(|' + + `(?:(?:const|volatile)\\s+)*${CPP_TYPE_NAME}(?:\\s*(?:const|volatile)\\b)?(?:\\s*[*&]+\\s*|\\s+)(?:(?:const|volatile)\\s+)?${CPP_DECLARED_NAME})`, + 'gm' +); +export function blankCppLeadingAttributeMacros(source: string): string { + if (!/(?:^|\]\])[ \t]*[A-Z][A-Z0-9]*_[A-Z0-9_]*\s/m.test(source)) return source; + const code = maskCppCode(source); + const spans: Array<[number, number]> = []; + const re = new RegExp(CPP_LEADING_ATTR_RE.source, 'gm'); + let m: RegExpExecArray | null; + while ((m = re.exec(code)) !== null) { + const start = m.index + (m[1] as string).length; + pushCppRunTokens(spans, m[2] as string, start); + } + return blankCppSpans(source, spans, code); } /** @@ -680,9 +1056,11 @@ export function blankCppAnnotationMacroCalls(source: string): string { * continuation) — so an ALL-CAPS operand split onto its own line inside a * multi-line expression (`int x =\n SOME_CONST\n | OTHER;`) is left * alone; and - * - the NEXT non-blank line starts like a declaration/scope token + * - the NEXT line of code starts like a declaration/scope token * (letter, `_`, `#`, `{`, `}`, or `~`) or the file ends — an operator, - * string literal, or `;` continuation rejects the match. + * string literal, or `;` continuation rejects the match. Comments in + * between are skipped: {fmt}'s `FMT_BEGIN_EXPORT` sits above `// A + * generic formatting context …` and then the class it opens. * Shared by C and C++ (the idiom is identical in both). */ const LONE_MACRO_LINE_RE = /^[ \t]*([A-Z][A-Z0-9_]{3,})[ \t]*(?:\/\/[^\n\r]*|\/\*[^\n\r]*\*\/[ \t]*)?\r?$/; @@ -709,12 +1087,8 @@ export function blankLoneMacroLines(source: string): string { const prevText = prev >= 0 ? content(lines[prev] as string) : ''; const prevCode = /^(?:\/\/|\/\*|\*)/.test(prevText) ? '' : prevText.replace(/\/\*.*?\*\/|\/\/.*$/g, '').trim(); if (prevCode !== '' && !/^template\s* m.replace(/[^\r\n]/g, ' ') - ); + let code = maskCppCommentsAndLiterals(source); const hasAlias = /^[ \t]*#[ \t]*define[ \t]+interface[ \t]+struct\b/m.test(code); code = code.replace(/^[ \t]*#(?:\\\r?\n|[^\r\n])*/gm, (m) => m.replace(/[^\r\n]/g, '\0')); const declaration = /(^|[;{}])\s*\binterface\s+[A-Za-z_]\w*\s*(:[^;{}]+)?\{/gm; @@ -1065,12 +1458,23 @@ function preParseCppSource(source: string, filePath?: string): string { source = flattenMidStatementConditionals(rewriteCppAccessMacros(rawStrings.source)); // blankCLeadingAttrMacros runs AFTER the api-prefix blank so a stacked // `FMT_NORETURN FMT_API void f(…)` reduces to the `MACRO Ret name(` shape - // it matches (the _API token is already spaces by then). + // it matches (the _API token is already spaces by then). The C++ leading + // blank runs after the pointer, trailing and declarator ones, so the + // declaration it looks for (`Type* name(`, `Type name;`) is already free of + // their macros. let blanked = blankLoneMacroLines( - blankCLeadingAttrMacros( - blankCppAnnotationMacroCalls( - blankCppInlineAnnotationMacros( - blankCppApiPrefixMacros(blankCppInlineMacros(blankCppExportMacros(normalizeCppComInterfaces(source)))) + blankCppLeadingAttributeMacros( + blankCLeadingAttrMacros( + blankCppAnnotationMacroCalls( + blankCppDeclaratorAttributeMacros( + blankCppTrailingAttributeMacros( + blankCppPointerAnnotationMacros( + blankCppInlineAnnotationMacros( + blankCppApiPrefixMacros(blankCppInlineMacros(blankCppExportMacros(normalizeCppComInterfaces(source)))) + ) + ) + ) + ) ) ) )