Standard Library documentation includes unavailable enumerations

[benrimmington]

The new documentation for the Standard Library includes enumerations such as Unicode.Block and Unicode.Script.

https://docs.swift.org/latest/documentation/swift/unicode#Enumerations

These are public enumerations from the _RegexParser module.

https://github.com/swiftlang/swift-experimental-string-processing/blob/main/Sources/_RegexParser/Utility/MissingUnicode.swift

Should they have the package access modifier, or should their module be excluded from the documentation?

[heckj]

@benrimmington I need a smidge of help understanding the report to work through how to solve/resolve it - it's basically that the docs are including some references that shouldn't be exposed - and aren't available in Swift 6.4?

This isn't a space I'm super familiar with - poking around, I'm not sure there's a Unicode.Script that I can get access to - is that what we're talking about here? (https://docs.swift.org/latest/documentation/swift/unicode/script)? Is it all of the enumerations under https://docs.swift.org/latest/documentation/swift/unicode (all the ones listed in that MissingUnicode.swift: BinaryProperty, Block, ExtendedGeneralCategory, POSIXProperty, Script)?

/cc @invalidname

[benrimmington]

… it's basically that the docs are including some references that shouldn't be exposed - and aren't available in Swift 6.4?

Is it all of the enumerations under https://docs.swift.org/latest/documentation/swift/unicode (all the ones listed in that MissingUnicode.swift: BinaryProperty, Block, ExtendedGeneralCategory, POSIXProperty, Script)?

@heckj Yes and yes. I think those enumerations, and perhaps other string-processing APIs, should be excluded from documentation. Thanks for taking a look.

[invalidname]

If they're here it's because they're visible in the symbol graph when CI builds swift and the stdlib. The fact that they're in a default bucket ("Enumerations") means we didn't expect them to be there. Maybe someone like @d-ronnqvist could speak to why they're being picked up and if there's a way to not do so.

[heckj]

I did a bit more digging, and it looks like these are appearing because the _RegexParser module extends the Unicode with those additional types, even though it's imported as @_implementationOnly - so those symbols aren't projected out to consumers, if I'm understanding this correctly.

I'm looking to see if we can strip out the symbols from _RegexParser, and if that's a correct solution here (and the extension types that build over it). We currently suppress those symbols in the navigation for the standard library, but if they're not accessible, then we need to suppress them further. We might be able to use @_documentation(visibility:) controls here as well - although if it's the whole module that wouldn't make sense as a solution.

I suspect we may be wanting to strip out all the symbols for _RegexParser prior to when we assemble the standard library docs, but I need to dig further to verify.

[heckj]

Worth a quick note - while it's not available in the Swift with ships with Xcode, the open source toolchain does include the module _RegexParser, so you can import _RegexParser and access those types with that toolchain:

import _RegexParser
print(Unicode.Script.allCases.count)

[heckj]

digging into the low-level build processes for the std lib - CMake uses IS_STDLIB to mark a number of things part of the standard library, including internal/private testing modules. It looks like IS_SDK_OVERLAY also enables IS_STDLIB - and where most of the models are directly annotated, Observation gets into the collection from the IS_SDK_OVERLAY setup (AddSwiftStdlib.cmake:2266-2270)

Depending on the build host, it looks like other modules could get included in there as well through this same path: Glibc, Musl, WASILibc, EmscriptenLibc, CRT, _math, and Android.

Right now the processing just grabs all the symbolgraphs in one big sweep, recursing through the build directories for them (under something akin to build/Ninja-Release/swift-macosx-arm64/lib/symbol-graph/) - so it's picking up quite a few more than we'd ideally like to have included.

I think we want to filter - but I'm not clear if if we should tag an explicit list somewhere - in CMake modules, in a list external, or something else, if we should make a list of symbolgraphs to explicit exclude and do some file-system filtering after they're all built, but before we hand off to docc to process them all together.

There is an explicit gap in there for swiftDarwin and swiftDifferentiationUnittest (in stdlib/cmake/modules/AddSwiftStdlib.cmake:1043-1051.) that excludes a few specific files by module name, with that list baked into the CMake

[heckj]

Note:

utils/build-script --build-stdlib-docs --reconfigure --release \
    --no-assertions --enable-llvm-assertions=0 \
    --swift-enable-assertions=0 --swift-stdlib-enable-assertions=0 \
    --swift-enable-ast-verifier=0

Does a run through of the build, assembling all the standard library pieces into symbolgraphs - so some of this could be handled at the python layer for generating symbolgraphs, but maybe it would be better to move and assemble things at the CMake layer beneath it.

There's also a CMake filter list in FILTERED_UNITTESTS (AddSwiftStdlib.cmake:3084), which may be a pattern we could follow. Potentially start with IS_STDLIB for the broad set of everything potentially STDLIB, and then exclude the bits/symbolgraphs we don't want with a list of those modules. We could also potentially exclude the symbolgraph generation up front, like happens for swiftDarwin and the differentiation unit test, with a broader set of hardcoded modules. That way, if something got added later - it would "show up" by default, not be excluded by default - which I think may be preferable.

[heckj]

posted into the forums with a potential solution: https://forums.swift.org/t/cmake-guidance-tweaking-the-process-to-build-the-docs-for-the-standard-library/89845

[heckj]

The list of all the symbolgraphs generated when I build on macOS (w/ Swift from Xcode beta 27.2)

./swiftRuntime-macosx-x86_64/Runtime@Swift.symbols.json
./swiftRuntime-macosx-x86_64/Runtime.symbols.json
./swiftOSLogTestHelper-macosx-arm64/OSLogTestHelper.symbols.json
./swiftRegexBuilder-macosx-x86_64/RegexBuilder.symbols.json
./swiftRegexBuilder-macosx-x86_64/RegexBuilder@Swift.symbols.json
./swiftRegexBuilder-macosx-x86_64/RegexBuilder@_StringProcessing.symbols.json
./swiftRegexBuilder-macosx-arm64/RegexBuilder.symbols.json
./swiftRegexBuilder-macosx-arm64/RegexBuilder@Swift.symbols.json
./swiftRegexBuilder-macosx-arm64/RegexBuilder@_StringProcessing.symbols.json
./swiftSwiftPrivateThreadExtras-macosx-arm64/SwiftPrivateThreadExtras.symbols.json
./swiftCxx-macosx-arm64/Cxx.symbols.json
./swiftCxx-macosx-arm64/Cxx@Swift.symbols.json
./swiftCompatibilitySpan-macosx-arm64/CompatibilitySpan.symbols.json
./swift_RegexParser-macosx-x86_64/_RegexParser@Swift.symbols.json
./swift_RegexParser-macosx-x86_64/_RegexParser.symbols.json
./swiftRuntime-macosx-arm64/Runtime@Swift.symbols.json
./swiftRuntime-macosx-arm64/Runtime.symbols.json
./swift_Volatile-macosx-x86_64/_Volatile.symbols.json
./swiftSwiftOnoneSupport-macosx-x86_64/SwiftOnoneSupport.symbols.json
./swiftSynchronization-macosx-arm64/Synchronization@Swift.symbols.json
./swiftSynchronization-macosx-arm64/Synchronization.symbols.json
./swiftStdlibUnittest-macosx-arm64/StdlibUnittest.symbols.json
./swiftStdlibUnittest-macosx-arm64/StdlibUnittest@Swift.symbols.json
./swiftDistributed-macosx-arm64/Distributed.symbols.json
./swiftDistributed-macosx-arm64/Distributed@Swift.symbols.json
./swiftSwiftPrivate-macosx-x86_64/SwiftPrivate.symbols.json
./swiftCore-macosx-arm64/Swift.symbols.json
./swift_Builtin_float-macosx-x86_64/_Builtin_float.symbols.json
./swift_StringProcessing-macosx-x86_64/_StringProcessing@Swift.symbols.json
./swift_Differentiation-macosx-x86_64/_Differentiation@Swift.symbols.json
./swift_Differentiation-macosx-x86_64/_Differentiation.symbols.json
./swiftCompatibilitySpan-macosx-x86_64/CompatibilitySpan.symbols.json
./swift_Volatile-macosx-arm64/_Volatile.symbols.json
./swift_Concurrency-macosx-x86_64/_Concurrency@Swift.symbols.json
./swiftSwiftPrivateThreadExtras-macosx-x86_64/SwiftPrivateThreadExtras.symbols.json
./swiftSwiftOnoneSupport-macosx-arm64/SwiftOnoneSupport.symbols.json
./swiftCxxStdlib-macosx-arm64/CxxStdlib@Swift.symbols.json
./swiftCxxStdlib-macosx-arm64/CxxStdlib.symbols.json
./swift_Builtin_float-macosx-arm64/_Builtin_float.symbols.json
./swiftStdlibUnittest-macosx-x86_64/StdlibUnittest.symbols.json
./swiftStdlibUnittest-macosx-x86_64/StdlibUnittest@Swift.symbols.json
./swiftStdlibUnittestFoundationExtras-macosx-x86_64/StdlibUnittestFoundationExtras.symbols.json
./swiftStdlibUnittestFoundationExtras-macosx-x86_64/StdlibUnittestFoundationExtras@Foundation.symbols.json
./swiftSwiftPrivate-macosx-arm64/SwiftPrivate.symbols.json
./swiftCxx-macosx-x86_64/Cxx.symbols.json
./swiftCxx-macosx-x86_64/Cxx@Swift.symbols.json
./swiftOSLogTestHelper-macosx-x86_64/OSLogTestHelper.symbols.json
./swiftObservation-macosx-x86_64/Observation.symbols.json
./swift_StringProcessing-macosx-arm64/_StringProcessing@Swift.symbols.json
./swift_Differentiation-macosx-arm64/_Differentiation@Swift.symbols.json
./swift_Differentiation-macosx-arm64/_Differentiation.symbols.json
./swiftRuntimeUnittest-macosx-arm64/RuntimeUnittest.symbols.json
./swift_Concurrency-macosx-arm64/_Concurrency@Swift.symbols.json
./swiftCore-macosx-x86_64/Swift.symbols.json
./swiftDistributed-macosx-x86_64/Distributed.symbols.json
./swiftDistributed-macosx-x86_64/Distributed@Swift.symbols.json
./swiftRuntimeUnittest-macosx-x86_64/RuntimeUnittest.symbols.json
./swift_RegexParser-macosx-arm64/_RegexParser@Swift.symbols.json
./swift_RegexParser-macosx-arm64/_RegexParser.symbols.json
./swiftObservation-macosx-arm64/Observation.symbols.json
./swiftSwiftReflectionTest-macosx-arm64/SwiftReflectionTest.symbols.json
./swiftCxxStdlib-macosx-x86_64/CxxStdlib@Swift.symbols.json
./swiftCxxStdlib-macosx-x86_64/CxxStdlib.symbols.json
./swiftStdlibUnittestFoundationExtras-macosx-arm64/StdlibUnittestFoundationExtras.symbols.json
./swiftStdlibUnittestFoundationExtras-macosx-arm64/StdlibUnittestFoundationExtras@Foundation.symbols.json
./swiftSynchronization-macosx-x86_64/Synchronization@Swift.symbols.json
./swiftSynchronization-macosx-x86_64/Synchronization.symbols.json
./swiftSwiftPrivateLibcExtras-macosx-x86_64/SwiftPrivateLibcExtras.symbols.json
./swiftSwiftReflectionTest-macosx-x86_64/SwiftReflectionTest.symbols.json
./swiftSwiftPrivateLibcExtras-macosx-arm64/SwiftPrivateLibcExtras.symbols.json

Quite a few of these I think we'll want to explicit exclude