Delete coreclr PAL's unicode wrapper layer by am11 · Pull Request #136973 · dotnet/runtime

am11 · 2026-11-04T19:31:05Z

port minipal UTF8/16 tests from PAL tests to src/tests
use minipal APIs directly at call site
delete wrapper from coreclr PAL.

Port everything the PAL MultiByteToWideChar and WideCharToMultiByte tests cover to direct minipal_* calls, so they can replace those tests: valid and ill-formed input with and without the terminator, embedded in ASCII runs, embedded NULs, size queries, exactly sized and too small destinations, errno, MINIPAL_MB_NO_REPLACE_INVALID_CHARS, and MINIPAL_TREAT_AS_LITTLE_ENDIAN parity on big- and little-endian platforms. Add long runs around the UTF-16 lane count flush, and random input checked against a simple reference model.

…WideCharToMultiByte Call minipal_get_length_* and minipal_convert_* directly for every conversion in CoreCLR, on Windows as well as Unix, including the PAL, the PAL test helpers and the generated LTTng provider. Native code now treats narrow strings as UTF-8 everywhere instead of using the ANSI or console code page on Windows: console output of ilasm, ildasm and PrintToStdErrW, metadata, ETW and config names, the ildasm input file name, the dbghelp symbol path, GC config strings, the mcs command line and the assembly name hashed into the typelib GUID. ilasm opens resource files by their UTF-16 name instead of converting it. Callers that pass a destination size of 0 now ask for the length explicitly, which fixes GetVersionString writing past the caller's buffer on Unix. Also pass EventStaticEntry::WriteEntry a size in characters rather than bytes, and keep the terminator inside the buffer in the GDB JIT file names.

Nothing calls them anymore, so remove unicode.cpp along with GetConsoleOutputCP, which only ever returned CP_UTF8 on Unix, and the UNICODE debug channel. Remove their declarations and the code page structures and flags that only they used from pal.h, the DAC exports, and the PAL locale tests, which the native minipal tests in src/tests replace.

azure-pipelines · 2026-11-04T19:32:16Z

Azure Pipelines: Successfully started running 8 pipeline(s). 7 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

🟡 Changes recommended

Unresolved null-pointer handling, Windows encoding mismatches, and event-buffer sizing regressions can cause crashes, corrupted paths and output, and lost event data.

8 open issues

What changed in this PR

This PR removes CoreCLR PAL’s Unicode conversion wrappers and moves callers to the shared minipal UTF-8/UTF-16 APIs.

Changes:

Replaces wrapper calls across the runtime, metadata code, diagnostics, and tools.
Removes obsolete PAL declarations, implementation, debug channel, and tests.
Adds minipal test-driver integration and clarifies conversion API documentation.

File	Description
src/tests/minipal/NativeMinipalTests.csproj	Adds the managed test project and native driver reference.
src/tests/minipal/NativeMinipalTests.cs	Runs the native UTF conversion suite.
src/tests/minipal/native-minipal-tests-driver.c	Registers the UTF conversion test group.
src/tests/minipal/Directory.Build.targets	Reuses native test-driver targets.
src/tests/minipal/Directory.Build.props	Reuses native test-driver properties.
src/tests/minipal/CMakeLists.txt	Builds and installs the minipal test driver.
src/tests/CMakeLists.txt	Avoids a minipal build-directory collision.
src/native/minipal/utf8.h	Clarifies sizing, destination, and error contracts.
src/coreclr/vm/util.cpp	Uses minipal for Unix stderr conversion.
src/coreclr/vm/object.cpp	Converts native UTF-8 strings through minipal.
src/coreclr/vm/interoputil.cpp	Migrates Unix assembly-name conversion, preserving Windows behavior.
src/coreclr/vm/gdbjit.cpp	Migrates debugger names and source paths.
src/coreclr/vm/gctoclreventsink.cpp	Converts dynamic event names through minipal.
src/coreclr/vm/gcenv.ee.cpp	Migrates configuration conversions and handles missing strings.
src/coreclr/vm/eventtracepriv.h	Migrates field-name conversion and corrects capacity units.
src/coreclr/vm/dwbucketmanager.hpp	Migrates crash-bucket assembly-name conversion.
src/coreclr/vm/dllimport.cpp	Migrates entry-point diagnostic conversion.
src/coreclr/vm/commtmemberinfomap.cpp	Migrates COM property-name conversion.
src/coreclr/vm/cdacstress.cpp	Migrates diagnostic method-name conversion.
src/coreclr/vm/assemblyspec.cpp	Migrates locale conversion and error handling.
src/coreclr/utilcode/util.cpp	Migrates quick-buffer UTF conversion.
src/coreclr/utilcode/stacktrace.cpp	Changes symbol-search-path conversion to UTF-8.
src/coreclr/utilcode/sstring.cpp	Migrates SString Unicode conversions.
src/coreclr/utilcode/fstring.cpp	Migrates string sizing and conversion helpers.
src/coreclr/tools/superpmi/superpmi/jithost.cpp	Migrates environment configuration conversion.
src/coreclr/tools/superpmi/superpmi-shared/spmiutil.cpp	Migrates shared UTF-8 conversion.
src/coreclr/tools/superpmi/mcs/verbmerge.h	Renames conversion helper declarations.
src/coreclr/tools/superpmi/mcs/verbmerge.cpp	Migrates merge-path conversion helpers and callers.
src/coreclr/tools/metainfo/mdobj.cpp	Migrates archive-member-name conversion.
src/coreclr/tools/metainfo/mdinfo.cpp	Migrates metadata display conversions.
src/coreclr/scripts/genLttngProvider.py	Generates direct minipal conversion calls.
src/coreclr/pal/tests/palsuite/paltestlist.txt	Removes obsolete wrapper test registrations.
src/coreclr/pal/tests/palsuite/paltestlist_to_be_reviewed.txt	Removes the legacy best-fit test listing.
src/coreclr/pal/tests/palsuite/locale_info/WideCharToMultiByte/test5/test5.cpp	Removes the PAL UTF-16-to-UTF-8 suite.
src/coreclr/pal/tests/palsuite/locale_info/WideCharToMultiByte/test4/test4.cpp	Removes legacy code-page best-fit tests.
src/coreclr/pal/tests/palsuite/locale_info/WideCharToMultiByte/test3/test3.cpp	Removes wrapper error-condition tests.
src/coreclr/pal/tests/palsuite/locale_info/WideCharToMultiByte/test2/test2.cpp	Removes wrapper explicit-length tests.
src/coreclr/pal/tests/palsuite/locale_info/WideCharToMultiByte/test1/test1.cpp	Removes wrapper ASCII and sizing tests.
src/coreclr/pal/tests/palsuite/locale_info/MultiByteToWideChar/test4/test4.cpp	Removes the PAL UTF-8-to-UTF-16 suite.
src/coreclr/pal/tests/palsuite/locale_info/MultiByteToWideChar/test3/test3.cpp	Removes wrapper error-condition tests.
src/coreclr/pal/tests/palsuite/locale_info/MultiByteToWideChar/test2/test2.cpp	Removes wrapper explicit-length tests.
src/coreclr/pal/tests/palsuite/locale_info/MultiByteToWideChar/test1/test1.cpp	Removes wrapper ASCII and sizing tests.
src/coreclr/pal/tests/palsuite/compilableTests.txt	Removes obsolete test build listings.
src/coreclr/pal/tests/palsuite/common/palsuite.cpp	Migrates shared PAL test conversion helpers.
src/coreclr/pal/tests/palsuite/CMakeLists.txt	Removes deleted wrapper test sources.
src/coreclr/pal/src/thread/thread.cpp	Migrates thread-description conversion.
src/coreclr/pal/src/misc/utils.cpp	Migrates allocating conversion helpers.
src/coreclr/pal/src/misc/fmtmessage.cpp	Migrates format-message numeric conversion.
src/coreclr/pal/src/misc/environ.cpp	Migrates environment name and value conversions.
src/coreclr/pal/src/misc/dbgmsg.cpp	Removes the Unicode debug-channel name.
src/coreclr/pal/src/locale/unicode.cpp	Deletes the Unicode wrapper implementation.
src/coreclr/pal/src/loader/module.cpp	Migrates module and library path conversions.
src/coreclr/pal/src/init/pal.cpp	Migrates executable-path conversion.
src/coreclr/pal/src/include/pal/utils.h	Updates conversion comments to specify UTF-8.
src/coreclr/pal/src/include/pal/dbgmsg.h	Removes the Unicode debug-channel declaration.
src/coreclr/pal/src/file/path.cpp	Migrates full-path and search-path conversions.
src/coreclr/pal/src/file/file.cpp	Migrates file-creation path conversion.
src/coreclr/pal/src/file/directory.cpp	Migrates directory path conversions.
src/coreclr/pal/src/debug/debug.cpp	Migrates debug-output conversion.
src/coreclr/pal/src/cruntime/wchar.cpp	Migrates wide-string numeric parsing conversions.
src/coreclr/pal/src/CMakeLists.txt	Removes the deleted wrapper source.
src/coreclr/pal/inc/pal.h	Removes wrapper APIs and supporting definitions.
src/coreclr/nativeresources/resourcestring.cpp	Migrates resource-string conversion.
src/coreclr/nativeaot/Runtime/windows/PalMinWin.cpp	Migrates NativeAOT Windows string conversions.
src/coreclr/nativeaot/Runtime/windows/PalCommon.cpp	Migrates PDB-path conversion.
src/coreclr/md/runtime/stgpool.cpp	Migrates metadata string-pool conversion.
src/coreclr/md/runtime/metamodelro.cpp	Separates metadata sizing from conversion.
src/coreclr/md/enc/stgtiggerstorage.cpp	Migrates metadata stream-name conversions.
src/coreclr/md/enc/rwutil.cpp	Migrates metadata UTF-8 conversion.
src/coreclr/md/enc/metamodelrw.cpp	Separates writable metadata sizing from conversion.
src/coreclr/md/compiler/import.cpp	Migrates version-string sizing and conversion.
src/coreclr/md/compiler/disp.cpp	Migrates runtime-version option conversion.
src/coreclr/md/compiler/custattr_import.cpp	Migrates custom-attribute-name conversion.
src/coreclr/jit/utils.cpp	Migrates the JIT UTF-16 string helper.
src/coreclr/inc/utilcode.h	Migrates shared conversion macros.
src/coreclr/inc/mdfileformat.h	Migrates stream-header name conversion.
src/coreclr/ildasm/windasm.cpp	Migrates arguments and changes encoding selection.
src/coreclr/ildasm/dman.cpp	Migrates manifest display conversions.
src/coreclr/ildasm/dis.cpp	Migrates disassembly and output conversion helpers.
src/coreclr/ildasm/dasm.cpp	Migrates resource, name, and filename conversions.
src/coreclr/ilasm/writer.cpp	Migrates conversions and opens resources by wide path.
src/coreclr/ilasm/typar.hpp	Migrates generic-parameter-name conversions.
src/coreclr/ilasm/method.hpp	Migrates method generic-parameter lookup conversion.
src/coreclr/ilasm/main.cpp	Migrates argument and path conversions.
src/coreclr/ilasm/grammar_before.cpp	Removes the console-code-page declaration.
src/coreclr/ilasm/grammar_after.cpp	Migrates token conversion and removes diagnostic transcoding.
src/coreclr/ilasm/class.hpp	Migrates class generic-parameter lookup conversion.
src/coreclr/ilasm/assembler.cpp	Migrates assembler names and string literals.
src/coreclr/ilasm/assem.cpp	Migrates metadata emission conversions.
src/coreclr/ilasm/asmparse.h	Migrates source-filename conversion.
src/coreclr/ilasm/asmman.cpp	Migrates assembly manifest conversions.
src/coreclr/hosts/corerun/corerun.hpp	Migrates Windows host conversion helpers.
src/coreclr/dlls/mscoree/exports.cpp	Migrates hosting string conversion.
src/coreclr/dlls/mscordac/mscordac_unixexports.src	Removes commented-out wrapper export entries.
src/coreclr/debug/ee/debugger.cpp	Migrates debug-event configuration conversion.
src/coreclr/debug/di/rsmain.cpp	Migrates application-group identifier conversion.
src/coreclr/debug/daccess/daccess.cpp	Migrates diagnostic conversion helpers and validation.

_{Review effort: Balanced}

- Report truncation for a null metadata string buffer with a nonzero size instead of writing through it. - Reject null paths in PAL CreateDirectoryW and CreateFileW before converting. - Keep the ANSI code page for the dbghelp SymInitialize search path. - Reserve room for a 255 character field name in bulk statics events. Co-authored-by: Cursor <cursoragent@cursor.com>

huoyaoyuan · 2026-11-05T03:40:50Z

This was blocked in my mind by the API design discussions of the minipal functions related to #100451, but seems that you have the context here.

am11 · 2026-11-05T03:50:51Z

This was blocked in my mind by the API design discussions of the minipal functions related to #100451, but seems that you have the context here.

This is a straightforward replacement of inlining existing minipal API and adapting caller code which was earlier adapting to win32 PAL shape anyway that for the call through the wrapper. On Windows, it is slightly faster: #136915 (comment)

jkotas · 2026-11-05T03:53:03Z

This was blocked in my mind by the API design discussions of the minipal functions related to #100451, but seems that you have the context here.

minipal utf8 converters are same or faster compared to the Win32 methods after #136915, so the concern about perf regressions discussed in #100451 no longer applies.

We should get rid of the FString wrapper too (follow up PR).

Copilot

🟡 Changes recommended

The change introduces Windows GC log-path encoding mismatches that can fail or misdirect logging, and directory-scan allocations that leak because the renamed member shadows the shared UTF-8 helper.

2 open issues

8 resolved since last review

_{Review effort: Balanced}

Drop palsuite coverage for APIs that no longer exist (IsBad*, Sleep, CompareString*, GetLocaleInfoW, LoadLibraryA, PAL_RegisterLibraryW) and for plain C-runtime functions that do not go through the PAL. Co-authored-by: Cursor <cursoragent@cursor.com>

Keep conversion lengths as size_t to match the minipal APIs. Co-authored-by: Cursor <cursoragent@cursor.com>

minipal_convert_utf16_to_utf8 cannot exceed the destination size; keep the assert instead of a redundant upper-bound check. Co-authored-by: Cursor <cursoragent@cursor.com>

jkotas · 2026-11-08T02:32:36Z

Could you please resolve the conflicts?

…place-pal Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # src/coreclr/vm/gcenv.ee.cpp # src/coreclr/vm/gdbjit.cpp

Copilot

🟡 Changes recommended

Unchecked size multiplication can wrap on 32-bit targets, allocating an undersized buffer that the converter can overrun before the later INT_MAX check.

1 open issue

Check buffer-size multiplication before allocation on 32-bit targets · New

_{Review effort: Balanced}

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

jkotas · 2026-11-08T19:20:59Z

/ba-g opened #137160 on the unrelated test failure

jkotas

Thank you!

jkotas · 2026-11-08T21:02:58Z

/ba-g opened #137160 on the unrelated test failure

jkotas · 2026-11-08T21:04:01Z

We should get rid of the FString wrapper too (follow up PR).

This should be unblocked now

am11 · 2026-11-08T21:32:31Z

This should be unblocked now

#137164

Follow-up to #136973. * Replace the FString wrapper with direct minipal UTF-8 conversions in SString, CQuickBytes and JitHost. * Delete pal/src/cruntime/wchar.cpp. Its users within the PAL are inlined; the u16_strtoul, u16_strtoui64 and u16_strtod implementations move to coreclr minipal (dn-u16). * Add u16_stricmp and u16_strnicmp to dn-u16 and use them in place of _wcsicmp/_wcsnicmp. * Delete the corresponding PAL tests. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

am11 added 3 commits November 4, 2026 21:27

am11 requested a review from MichalStrehovsky as a code owner November 4, 2026 19:31

github-actions Bot added the area-PAL-coreclr only for closed issues label Nov 4, 2026

am11 requested review from jkoritzinsky and jkotas November 4, 2026 19:31

dotnet-policy-service Bot added the community-contribution Indicates that the PR has been added by a community member label Nov 4, 2026

Revert two windows callsites affected by codepage

521d2a7

jkoritzinsky reviewed Nov 4, 2026

View reviewed changes

Comment thread src/coreclr/tools/metainfo/mdobj.cpp Resolved

Comment thread src/coreclr/vm/interoputil.cpp Resolved

Comment thread src/coreclr/vm/util.cpp Resolved

build-analysis Bot mentioned this pull request Nov 4, 2026

[ci-scan] Test failure: GC/API/GC/GetConfigurationVariables on GCStandAlone windows-arm64 #131318

Open

jkotas reviewed Nov 4, 2026

View reviewed changes

Comment thread src/tests/minipal/CMakeLists.txt Resolved

am11 added 3 commits November 5, 2026 01:43

Move tests under src/tests/minipal

85cdee2

Merge branch 'main' into chore/minipal-utf8-replace-pal

700b372

Don't crash on null config as before

fad6ae5

jkotas requested a balanced review from Copilot November 5, 2026 00:44

Copilot started reviewing on behalf of jkotas November 5, 2026 00:44

View session

Copilot AI reviewed Nov 5, 2026

View reviewed changes

build-analysis Bot mentioned this pull request Nov 5, 2026

"We stopped hearing from agent Azure Pipelines 32. Verify the agent machine is running and has a healthy network connection" dotnet/dnceng#1886

Open

3 tasks

Merge branch 'main' into chore/minipal-utf8-replace-pal

36ae0ce

jkotas requested a balanced review from Copilot November 6, 2026 21:45

Copilot started reviewing on behalf of jkotas November 6, 2026 21:45

View session

Copilot AI reviewed Nov 6, 2026

View reviewed changes

Comment thread src/coreclr/tools/superpmi/mcs/verbmerge.cpp Resolved

Comment thread src/coreclr/vm/gcenv.ee.cpp Resolved

This was referenced Nov 7, 2026

[ci-scan] Hang: dotnet-dump opening macOS x64 Mach-O dumps #137115

Open

[ci-scan] Build break: unable to pull Helix prerequisite image #134176

Open

am11 mentioned this pull request Nov 7, 2026

Remove PAL file, path and file-mapping APIs #136821

Merged

jkotas reviewed Nov 7, 2026

View reviewed changes

Comment thread src/coreclr/utilcode/util.cpp Resolved

jkotas reviewed Nov 7, 2026

View reviewed changes

Comment thread src/coreclr/vm/eventtracepriv.h Resolved

am11 force-pushed the chore/minipal-utf8-replace-pal branch from 471b139 to d6fe8c4 Compare November 7, 2026 20:39

Avoid size_t-to-int casts in Utf2Quick

9939ca6

Keep conversion lengths as size_t to match the minipal APIs. Co-authored-by: Cursor <cursoragent@cursor.com>

am11 force-pushed the chore/minipal-utf8-replace-pal branch from 1b09ec2 to 9939ca6 Compare November 7, 2026 21:31

jkotas reviewed Nov 7, 2026

View reviewed changes

Comment thread src/coreclr/jit/helperexpansion.cpp Resolved

Simplify ReadUtf8 length check after minipal convert

2da64d8

minipal_convert_utf16_to_utf8 cannot exceed the destination size; keep the assert instead of a redundant upper-bound check. Co-authored-by: Cursor <cursoragent@cursor.com>

build-analysis Bot mentioned this pull request Nov 8, 2026

[browser] Wasm.Build.Tests timeout - Timed out after 10s waiting for 'WASM EXIT' #116697

Open

Merge remote-tracking branch 'dotnet/main' into chore/minipal-utf8-re…

49e1a52

…place-pal Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # src/coreclr/vm/gcenv.ee.cpp # src/coreclr/vm/gdbjit.cpp

jkotas requested a balanced review from Copilot November 8, 2026 15:41

Copilot started reviewing on behalf of jkotas November 8, 2026 15:41

View session

Copilot AI reviewed Nov 8, 2026

View reviewed changes

Comment thread src/coreclr/vm/object.cpp Resolved

Fix potential overflow in WCHAR allocation

4dbd351

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

jkotas mentioned this pull request Nov 8, 2026

DacDbiComWrappersDumpTests failure #137160

Open

jkotas approved these changes Nov 8, 2026

View reviewed changes

jkotas merged commit a8caf1f into dotnet:main Nov 8, 2026
245 of 249 checks passed

am11 mentioned this pull request Nov 8, 2026

Delete FString and coreclr PAL's wchar.cpp #137164

Merged

am11 deleted the chore/minipal-utf8-replace-pal branch November 8, 2026 22:42

dotnet-maestro Bot mentioned this pull request Nov 9, 2026

[main] Source code updates from dotnet/runtime dotnet/dotnet#10811

Merged

dotnet-milestone-bot Bot added this to the 12.0-preview1 milestone Nov 9, 2026

Conversation

am11 commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azure-pipelines Bot commented Nov 4, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

huoyaoyuan commented Nov 5, 2026

Uh oh!

am11 commented Nov 5, 2026

Uh oh!

jkotas commented Nov 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jkotas commented Nov 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

jkotas commented Nov 8, 2026

Uh oh!

jkotas left a comment

Choose a reason for hiding this comment

Uh oh!

jkotas commented Nov 8, 2026

Uh oh!

Uh oh!

jkotas commented Nov 8, 2026

Uh oh!

am11 commented Nov 8, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

am11 commented Nov 4, 2026 •

edited

Loading