Increase decompiler test-suite CPU utilization via NUnit parallelism

[christophwille]

A full ICSharpCode.Decompiler.Tests run kept a 24-logical-CPU machine at only ~46% average CPU. This PR makes the suite use the machine it runs on: fixtures run in parallel by default, the NUnit worker pool is oversubscribed to 2x logical CPUs, and the multi-minute tests are scheduled first. A companion doc describes the Windows Defender exclusions that would remove the remaining scan overhead (deliberately not automated).

Measured causes of the idle CPU

1. ~30 fixture files had no Parallelizable attribute (TypeSystem, Output, Util, Metadata, ProjectDecompiler, ...) - NUnit runs unattributed fixtures one at a time on its non-parallel queue. Only the 12 matrix runners declared Parallelizable(ParallelScope.All).
2. Worker count = ProcessorCount, but matrix tests spend most of their time blocked on child processes (csc/vbc/ilasm/msbuild/nunit-agent/TestRunner). Blocked workers leave cores idle.
3. The critical path is a handful of 3-5 minute roundtrip tests (Random_TestCase_1, ExplicitConversions*, NRefactory_CSharp, ...) that started mid-run and straggled at the end.
4. Windows Defender (MsMpEng.exe) ran at 1-4 cores continuously, scanning every compiled fixture and spawned process (out of scope for code changes; see doc/WindowsDefenderExclusions.md).

Options considered

For raising the worker count:

Option	Verdict
Local .runsettings with NumberOfTestWorkers	Rejected - a fixed number in a file, and an extra flag every run
Hard-coded [assembly: LevelOfParallelism(48)]	Rejected - tuned to one machine; wrong everywhere else (e.g. 4-core CI runners)
Build-time computed attribute, always 2x logical CPUs	Chosen - machine-independent policy, no number checked in

For the Defender overhead: apply exclusions vs. document them vs. skip. Machine-level AV configuration does not belong in the repo, so this PR only documents the exclusions (doc/WindowsDefenderExclusions.md).

How the oversubscription works

[assembly: LevelOfParallelism] only accepts a compile-time constant, so the csproj generates it during build:

<AssemblyAttribute Include="NUnit.Framework.LevelOfParallelism">
  <_Parameter1>$([MSBuild]::Multiply($([System.Environment]::ProcessorCount), 2))</_Parameter1>
  <_Parameter1_TypeName>System.Int32</_Parameter1_TypeName>
</AssemblyAttribute>

• The AssemblyAttribute item is emitted into the SDK-generated AssemblyInfo.cs (obj/.../ICSharpCode.Decompiler.Tests.AssemblyInfo.cs); _Parameter1_TypeName makes MSBuild emit the value as an int rather than a string.
• The value is 2x the logical CPU count of the machine building the tests - which is the machine running them, both locally and in CI. On the 24-thread dev box this generates LevelOfParallelism(48); on a 4-core CI runner it generates 8.
• 2x is deliberate oversubscription: NUnit workers are dedicated threads, and most matrix tests block on child compiler/runner processes, so twice as many in-flight tests keeps cores busy without thrashing. The NUnit adapter honors the attribute unless a runsettings value overrides it (none does).
• [assembly: Parallelizable(ParallelScope.Fixtures)] (in Properties/AssemblyInfo.cs) makes the previously-serial fixtures run concurrently with each other while tests within such a fixture stay sequential. Fixtures sharing process-global state can opt out with [NonParallelizable]; none currently needs to (DecompilerEventSourceTests was audited - its assertions already filter by payload marker).
• [Order(1)]/[Order(2)] on RoundtripAssembly and CorrectnessTestRunner enqueue the multi-minute tests first.

Before / after (24 logical CPUs, ILSpy-tests checked out)

	Baseline	This PR
In-flight tests (from TRX start/end stamps)	24, flat	47-48, flat
Longest test start	~t+150s	t=0
Wall time	8m16s (3966 tests)	8m11s-8m37s (4170 tests, two runs)
Average total CPU	45.8%	46.2%
Failures	0	0 (two consecutive green runs)

The test-count difference is unrelated new tests picked up by the rebuild. Concurrency verifiably doubled and scheduling is now optimal (every giant starts at t=0), yet wall time and CPU stayed flat - which proves the worker pool is no longer the constraint:

• The suite is now bounded by its single longest test. Random_TestCase_1 runs 464s wall-to-wall (up from 314s mid-run at baseline - it slows under the doubled contention); total wall time is essentially that one test. More workers cannot help; they only add contention against the critical path.
• The remaining idle CPU is Defender scan latency on thousands of process spawns and file writes (48 in-flight tests averaged ~0.23 cores each).

Round 2: sequential awaits inside the test infra

A second pass hunted for places where the infra itself awaits independent work sequentially (await a; await b; where a Task.WhenAll or an earlier start genuinely overlaps):

• Tester.Initialize now runs everything concurrently. It gates every test via the [SetUpFixture], and serialized nine NuGet toolset fetches plus two self-contained TestRunner builds. The fetches extract into disjoint directories and the builds depend on no fetched toolset, so all of it is now started eagerly and awaited once (registration dictionaries got a lock). Only the two Windows RID builds stay sequential with each other - they share the TestRunner project's obj/, and their implicit restores would race on project.assets.json. Biggest effect on a cold machine/CI, where the downloads dominated. Verified with 3 consecutive cold-cache runs (toolset dirs deleted each time), all green.
• The original and decompiled executables run concurrently. RunAndCompareOutput awaited the two runs back to back; they are independent processes with separately buffered output. Plain Task.WhenAll (no error aggregation) keeps NUnit's Ignore semantics, and exit codes are still asserted in the original order, so failure output is unchanged (verified by sabotaging one path and checking the message).
• The original executable starts before the decompile. In the correctness runners (RunCS/RunVB/RunIL) and the RunWithOutput roundtrips, the original binary is complete after the first compile, and the decompile/recompile stages only read it - the new Tester.StartRun hands the in-flight run into the comparison. For mcs configurations the .exe.config write moved ahead of the run start (the runtime reads it at process launch); the mcs matrix stays green. In roundtrips the pristine exe now overlaps the multi-minute whole-project decompile; the submodule-missing guard runs before the early start so those tests still report Ignored.
• FindMSBuild is cached (Lazy<Task<string>>): the parallel roundtrip fixture spawned one vswhere.exe per test for a process-invariant answer.
• The roundtrip testAction became Func<string, Task> along the way, removing GetAwaiter().GetResult() blocking on NUnit worker threads.

Full suite after the changes: 4170 tests, 0 failures, 20 skipped (environment-gated), wall in the same band as before - expected, since Random_TestCase_1 spends 98.6% of its 480s inside a single WholeProjectDecompiler.DecompileProject call (measured from the harness's Decompiled X in N stamps), which none of this touches. The wins are cold-start setup, the correctness fixture (864 cases), and removed serial slack off the critical path.

Follow-up ideas (non-Defender)

• Split the generated monoliths in ILSpy-tests. Random_TestCase_1 and the ExplicitConversions* variants are single generated executables from Random Tests/TestCaseGenerator in the ILSpy-tests submodule. Splitting each into several smaller assemblies (or emitting the conversions matrix as N partitions) would turn one 460s pipeline into parallelizable chunks - the only way to push wall time meaningfully below ~8 minutes.
• The per-file decompile phase is already parallel (WholeProjectDecompiler.MaxDegreeOfParallelism = ProcessorCount), so the giants' serial cost is in their single-assembly csc rebuild and execute/compare phases, which only splitting addresses.
• Scheduling slack, ~66s. From the TRX timeline: Random_TestCase_1 starts ~27s into the run despite Order(1) (setup + fixture-construction order), and ~44s of unordered fast tests drain after it finishes. Recovering that head/tail is worth more than any remaining in-process await.
• The Linux/macOS CI job runs its four test assemblies strictly serially (build-ilspy.yml, separate dotnet test --project steps), while the Windows job's --solution form runs them concurrently. Running them in parallel cuts roughly 40% of that job's test time; needs a solution filter without ILSpy.Tests.Windows (which must not run off-Windows) or backgrounded steps.
• ILSpy.Tests extracts zero parallelism by construction (one Avalonia dispatcher via AvaloniaTestIsolationLevel.PerAssembly plus the static MEF container; PerTest was tried and reverted). Only process-level sharding of fixtures helps; measured fixture durations bin-pack to ~185s slowest shard at 4 shards. Separate effort.

🤖 Generated with Claude Code

[christophwille]

Verification: do the NUnit parallelism attributes actually work under Microsoft.Testing.Platform?

The attributes are consumed by NUnit itself, not by the host platform, so MTP vs. VSTest makes no difference - but here is the full verified chain rather than an appeal to documentation.

1. The attributes are physically in the compiled assembly

Reading the PE metadata of the built ICSharpCode.Decompiler.Tests.dll (System.Reflection.Metadata, no runtime load) shows both assembly-level custom attributes with the expected arguments:

Attribute	Blob	Decoded
NUnit.Framework.LevelOfParallelismAttribute	01 00 30 00 00 00	int32 0x30 = 48 (2x24 logical CPUs of the build machine)
NUnit.Framework.ParallelizableAttribute	01 00 00 02 00 00	0x200 = ParallelScope.Fixtures

2. NUnit 4.6.1 honors the attribute unless the adapter passes an override

Decompiling nunit.framework.dll (with this repo's ilspycmd), NUnitTestAssemblyRunner.GetLevelOfParallelism is:

private int GetLevelOfParallelism(ITest loadedTest)
{
    if (!Settings.TryGetValue("NumberOfTestWorkers", out object value))
        return loadedTest.Properties.TryGet("LevelOfParallelism", DefaultLevelOfParallelism);
    return ConvertSetting<int>(value);
}

The assembly attribute populates the LevelOfParallelism property on the loaded test assembly; the result feeds new ParallelWorkItemDispatcher(48). The parallelism engine is entirely NUnit's in-process dispatcher - MTP (via Microsoft.Testing.Extensions.VSTestBridge) just hosts the adapter and adds nothing to this path.

3. NUnit3TestAdapter 6.2.0 does not inject NumberOfTestWorkers in our invocation

Decompiling NUnit3.TestAdapter.dll:

• AdapterSettings: NumberOfTestWorkers = GetInnerTextAsInt(xmlNode, "NumberOfTestWorkers", -1) - defaults to -1 when no runsettings node exists.
• NUnitTestAdapter.CreateTestPackage: the setting is only written into the test package when >= 0, so with the default -1 the framework falls through to the assembly attribute.
• The only paths that force it to 0 (serial): debugger attached (without AllowParallelWithDebugger), DisableParallelization in runsettings, or CollectDataForEachTestSeparately / Live Unit Testing in-proc collectors. None applies to a plain --report-trx run with no --settings.

4. Empirical proof from the TRX timelines

Reconstructing concurrency from per-test startTime/endTime stamps:

• Overall in-flight tests: 24, flat (baseline) vs. 47-48, flat (this PR) for the entire run.
• The ~30 fixtures that previously had no Parallelizable attribute (718 tests in the comparison set): 0 cross-fixture overlapping executions in the baseline run - perfectly serial, as predicted - vs. 4596 overlapping execution pairs across 206 distinct fixture pairs with this PR.

Caveat

The attribute is a default, not a mandate: --settings with NumberOfTestWorkers, DisableParallelization, CollectDataForEachTestSeparately, or running under a debugger overrides or disables it. That matches the adapter's long-standing behavior under VSTest as well.

🤖 Generated with Claude Code