feat(icu4c): Add ICU4C support to UTF8 string operations - #3187
CryoTheRenegade wants to merge 10 commits into
Conversation
PR Summary by QodoAdd ICU4C-backed UTF-8 conversions and link ICU into the engine
AI Description
Diagram
High-Level Assessment
Files changed (13)
|
Code Review by Qodo
1.
|
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit 0240a44 |
|
@greptileai review this |
|
0240a44 to
c27b37a
Compare
|
To use Codex here, create a Codex account and connect to github. |
|
@codex review this |
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
OmniBlade
left a comment
There was a problem hiding this comment.
There is a lot of needless complexity here IMO, why are we worrying about using ICU on VC6 at all and dynamically loading it? Alternate codepaths for wchar_t as UTF-16 and UTF-32 also bothers me, its a more invasive change, but putting the "wide" types behind an ifdef and removing wchar_t use from the code base allows for simpler conversion functions that are easier to understand and maintain once the pain of getting rid of wchar_t has been taken care of.
I'd prefer to see this broken down into two PRs, one to move all use of wchar_t behind a typedef and then a second to use ICU exclusively for none windows platforms and make the typedef UChar on those platforms for consistent use of UTF-8 and UTF-16 only.
|
I would prefer to have icu supported on windows so that we can get proper bidirectional string support without needing another library like FriBidi in the future. I'd be happy to separate this into two prs. |
xezon
left a comment
There was a problem hiding this comment.
Does VC6 use ICU or not? It is not clear from the code. Implementation looks confusing.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (7)
🚧 Files skipped from review as they are similar to previous changes (1)
Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review. WalkthroughThe build adds a dedicated ChangesICU UTF-8 Integration
Priority: ➖ Normal Estimated code review effort: 4 (Complex) | ~45 minutes Change: Feature Sequence Diagram(s)sequenceDiagram
participant GameMain
participant UnicodeString_translate
participant ICU_utf8
participant IcuLoader
GameMain->>IcuLoader: create IcuScope
UnicodeString_translate->>ICU_utf8: request UTF-8 to wide-character conversion
ICU_utf8->>IcuLoader: query ICU functions when dynamic loading is enabled
IcuLoader-->>ICU_utf8: return resolved conversion functions
ICU_utf8-->>UnicodeString_translate: return converted length or conversion result
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: 0d0a0605-d996-4f8e-a869-b3b6770009eb
📒 Files selected for processing (18)
CMakeLists.txtCore/GameEngine/CMakeLists.txtCore/GameEngine/Source/Common/System/AsciiString.cppCore/GameEngine/Source/Common/System/UnicodeString.cppCore/GameEngine/Source/GameNetwork/GameInfo.cppCore/GameEngine/Source/GameNetwork/GameSpy/Thread/ThreadUtils.cppCore/Libraries/Source/WWVegas/WWLib/CMakeLists.txtCore/Libraries/Source/WWVegas/WWLib/utf8.cppCore/Libraries/Source/WWVegas/WWLib/utf8.hDependencies/ICU/CMakeLists.txtDependencies/ICU/ICU/IcuLoader.cppDependencies/ICU/ICU/IcuLoader.hDependencies/ICU/ICU/IcuSupport.hDependencies/ICU/ICU/utf8.cppDependencies/ICU/ICU/utf8.hDependencies/ICU/README.mdcmake/icu.cmakevcpkg.json
💤 Files with no reviewable changes (2)
- Core/Libraries/Source/WWVegas/WWLib/utf8.cpp
- Core/Libraries/Source/WWVegas/WWLib/utf8.h
Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.
VC6 does use ICU through a fallback icu.dll load |
Wide Char PR is up #3321 |
87e922d to
38453f5
Compare
Use vcpkg or the Windows SDK C API on modern builds, keep VC6 on runtime LoadLibrary, and only probe system icu.dll for the delay-loaded SDK path.
Preserve bobtista's original change comments and append the ICU conversion notes instead of replacing them.
Publish the availability result with InterlockedCompareExchange, search the normal DLL path instead of System32 only, and pass the CMake-found icu.lib into the link line.
Avoid InterlockedCompareExchange, whose VC6 and later SDK signatures disagree, so utf8.cpp compiles on both toolchains.
Main's LAN name truncation includes WWLib/utf8.h. After that header moves into the ICU library, GameInfo has to include ICU/utf8.h. Co-authored-by: Cursor <cursoragent@cursor.com>
cc8321a to
c1ee4e6
Compare
Replaces the hand-rolled WWLib UTF-8 converter with ICU4C, and links ICU into the engine so later code can use the rest of the suite.
AsciiString::translate/UnicodeString::translatenow convert through ICU instead of 7-bit ASCII. Invalid UTF-8 still falls back to the original one-byte-to-one-wide-unit behavior so legacy CP1252 data is preserved. LAN player names are truncated on a UTF-8 code-point boundary instead of chopping mid-sequence.ICU is selected in this order:
find_package(ICU)from vcpkg / system (C and C++ APIs)icu.lib+/DELAYLOAD:icu.dllon modern MSVCLoadLibraryof OSicu.dll, with Win32CP_UTF8if that DLL is missingWWLib/IcuSupport.his the engine include for linked ICU. vcpkg now depends onicuon all platforms.In the future we can also use ICU's baked in bidirectional string support for languages that require them