Conversation
fe653b5 to
65d17a8
Compare
|
fyi I'm working on this in #13215, but won't be mad if you beat me to it. Chasing what looks like some Windows hangs at the moment, and as a low-pri background thing. |
|
Oh, I didn't know you are already working on it. |
|
@rainersigwald I tried to pull in some changes you did in the other PR here to see if it gets me here in a better state. We can sync next Tuesday to align the efforts. Generally speaking, a big difference between the two PRs is that in this PR I'm running with MTP, while the other PR runs with VSTest. But I would think most test failures will be related to the xunit 2 to xunit 3 move and not VSTest/MTP differences. In addition, in this PR I'm keeping the tests wrapped in One thing to note is that xunit.v3 might be running the tests in a different order compared to xunit 2, which might (or might not) be the cause for some of the failures (i.e, some tests might have been modifying global state without resetting it - that was somehow unnoticed with xunit 2 but might become apparent now that the ordering changed) |
I believe this is not currently working--my interpretation of a bunch of failures in my attempt was "something is making this work now and failing a bunch of tests that should have been failing before but weren't". I'd love to have all those tests fixed of course . . . |
In the other PR, I commented on why the approach there won't work. But I think it might have been working on main already. |
ec93ec7 to
6f93f7e
Compare
| var exceptionFilesBefore = | ||
| Directory.Exists(DebugUtils.DebugDumpPath) ? Directory.GetFiles(DebugUtils.DebugDumpPath, "MSBuild_*failure.txt") : Array.Empty<string>(); |
There was a problem hiding this comment.
For some unknown reason, sometimes the DebugDumpPath directory isn't there on disk, and then GetFiles throws an exception if the directory doesn't exist.
Maybe there is some other previous test that deletes DebugDumpPath or something. Anyways, it should be safe to keep this change. It just means that future refactoring for how tests work is ideal to avoid this kind of cross-test dependency issues.
| private static string AssemblyLocation { get; } = | ||
| typeof(TaskHostFactory_Tests).Assembly.Location | ||
| ?? Path.Combine(AppContext.BaseDirectory, "Microsoft.Build.Engine.UnitTests.dll"); |
There was a problem hiding this comment.
- NOTE 1: Under .NET Framework, the assembly is now the
.exewhichAssembly.Locationwill provide correctly. - NOTE 2: In the case of
Assembly.Locationbeing null (unlikely to happen for .NET Framework), we still useAppContext.BaseDirectoryand then get the dll. If this ever happened for .NET Framework, we would need to add extra condition to either find.dllor.exefor .NET Framework. But so farAssembly.Locationsimply works just fine.
| ParallelConsoleLogger cl2 = new ParallelConsoleLogger(LoggerVerbosity.Minimal, sc.Write, null, null); | ||
| EventSourceSink es = new EventSourceSink(); | ||
| cl2.Initialize(es); | ||
| cl2.Parameters = "ShowEnvironment"; |
There was a problem hiding this comment.
NOTE: This line will lead to:
So, we ensure we get back the static state to its original value (should be false).
There is some other test that would fail if Traits.LogAllEnvironmentVariables is true (I don't recall which one now).
In current main, DisplayEnvironmentInMinimal is run after that test so everything kinda works fine.
With xunit.v3, the order changed and now DisplayEnvironmentInMinimal is run earlier, causing the other test to fail.
Pulling some changes from #13215
…auses some caches to be bad/unexpected for tests.
cdf2641 to
9b1339b
Compare
There was a problem hiding this comment.
Pull request overview
This PR migrates the repo’s unit test infrastructure from xUnit v2 patterns to xUnit v3, aligning execution with Microsoft.Testing.Platform and updating shared test bootstrapping so test environment initialization remains consistent across the suite.
Changes:
- Switched test dependencies to xUnit v3 (and Verify.XunitV3), removed xUnit v2 console runner usage, and updated test runner/coverage configuration.
- Replaced the custom xUnit v2 “AssemblyFixtureSupport” framework with xUnit v3 pipeline startup + a custom test framework wrapper to create
TestEnvironmentper test case. - Updated many test projects/files to remove
Xunit.Abstractionsusages and stabilize a few test-global/static state interactions.
Reviewed changes
Copilot reviewed 160 out of 160 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Xunit.NetCore.Extensions/Xunit.NetCore.Extensions.csproj | Switch extension package reference to xUnit v3 extensibility core and adjust TFMs/settings. |
| src/Xunit.NetCore.Extensions/UseInvariantCultureAttribute.cs | Update BeforeAfterTestAttribute overrides to xUnit v3 signatures. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestMethodRunnerWithAssemblyFixture.cs | Removed legacy xUnit v2 assembly fixture runner implementation. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestFrameworkWithAssemblyFixture.cs | Removed legacy xUnit v2 custom test framework. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestFrameworkExecutorWithAssemblyFixture.cs | Removed legacy xUnit v2 executor wrapper. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestCollectionRunnerWithAssemblyFixture.cs | Removed legacy xUnit v2 collection runner wrapper. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestClassRunnerWithAssemblyFixture.cs | Removed legacy xUnit v2 class runner wrapper. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestAssemblyRunnerWithAssemblyFixture.cs | Removed legacy xUnit v2 assembly runner wrapper. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/README.md | Removed docs for the deleted xUnit v2 assembly fixture support. |
| src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/AssemblyFixtureAttribute.cs | Removed legacy assembly-level fixture attribute. |
| src/Utilities.UnitTests/ToolTask_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Utilities.UnitTests/ToolLocationHelper_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Utilities.UnitTests/ProcessExtensions_Tests.cs | Remove xUnit v2 abstractions import. |
| src/UnitTests.Shared/TestEnvironment.cs | Update default output helper implementation for xUnit v3 interface surface. |
| src/UnitTests.Shared/RunnerUtilities.cs | Update xUnit output helper import namespace. |
| src/UnitTests.Shared/ObjectModelHelpers.cs | Remove xUnit v2 abstractions import. |
| src/UnitTests.Shared/MockLogger.cs | Remove xUnit v2 abstractions import. |
| src/UnitTests.Shared/MockEngine.cs | Update xUnit output helper import namespace. |
| src/UnitTests.Shared/Microsoft.Build.UnitTests.Shared.csproj | Replace xUnit v2 packages with xUnit v3 assert/extensibility packages. |
| src/Tasks.UnitTests/XamlDataDrivenToolTask_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/WriteLinesToFile_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/VerifyFileHash_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/Unzip_Tests.cs | Replace conditional attribute usage with runtime skip under xUnit v3. |
| src/Tasks.UnitTests/TestResources/TestBinary.cs | Update xUnit-related imports. |
| src/Tasks.UnitTests/RoslynCodeTaskFactory_Tests.cs | Remove Verify-specific attribute usage. |
| src/Tasks.UnitTests/ResourceHandling/ResGenDependencies_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/ResourceHandling/MSBuildResXReader_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/ResourceHandling/GenerateResourceOutOfProc_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/ResourceHandling/GenerateResource_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/ResolveNonMSBuildProjectOutput_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/ResolveAssemblyReference_CustomCultureTests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/RemoveDir_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/RegressionTests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/PortableTasks_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/OutputPathTests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/MSBuildInternalMessage_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/MSBuild_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/Microsoft.Build.Tasks.UnitTests.csproj | Switch Verify package to xUnit v3 variant. |
| src/Tasks.UnitTests/HintPathResolver_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/GetSDKReference_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/GetFileHash_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/GetCompatiblePlatform_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/GetAssembliesMetadata_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/GenerateBindingRedirects_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/FormatUrl_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/Exec_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/CreateVisualBasicManifestResourceName_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/CreateItem_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/CreateCSharpManifestResourceName_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/Copy_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssignProjectConfiguration_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/WinMDTests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/VerifyTargetFrameworkHigherThanRedist.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/VerifyTargetFrameworkAttribute.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/VerifyIgnoreVersionForFrameworkReference.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/SuggestedRedirects.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/StronglyNamedDependencyAutoUnify.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/StronglyNamedDependencyAppConfig.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/StronglyNamedDependency.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/SpecificVersionPrimary.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/ResolveAssemblyReferenceTestFixture.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/ReferenceTests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/Perf.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/NonSpecificVersionStrictPrimary.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/Node/OutOfProcRarNode_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/Miscellaneous.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/InstalledSDKResolverFixture.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/GlobalAssemblyCacheTests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/FilePrimary.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AssemblyDependency/AssemblyFoldersFromConfig_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/Al_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Tasks.UnitTests/AddToWin32Manifest_Tests.cs | Remove xUnit v2 abstractions import. |
| src/StringTools.UnitTests/StringTools.UnitTests.net35.csproj | Force OutputType Exe for net35 test project under new runner approach. |
| src/StringTools.UnitTests/InterningTestData.cs | Update xUnit-related imports. |
| src/Shared/UnitTests/xunit.runner.json | Remove xUnit v2 runner settings (shadow copy/app domain). |
| src/Shared/UnitTests/TypeLoader_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Shared/UnitTests/TestProgram.cs | Removed custom xUnit v2 console entrypoint program. |
| src/Shared/UnitTests/TestAssemblyInfo.cs | Introduce xUnit v3 pipeline startup + custom test framework wrapper for TestEnvironment. |
| src/MSBuild.UnitTests/XMake_Tests.cs | Remove unused test platform + xUnit v2 abstractions imports. |
| src/MSBuild.UnitTests/XMake_BinlogSwitch_Tests.cs | Remove xUnit v2 abstractions import. |
| src/MSBuild.UnitTests/PerfLog_Tests.cs | Remove xUnit v2 abstractions import. |
| src/MSBuild.UnitTests/MSBuildServer_Tests.cs | Remove xUnit v2 abstractions import. |
| src/MSBuild.UnitTests/MSBuildMultithreaded_Tests.cs | Remove xUnit v2 abstractions import. |
| src/MSBuild.UnitTests/Microsoft.Build.CommandLine.UnitTests.csproj | Switch Verify package to xUnit v3 variant. |
| src/MSBuild.UnitTests/CommandLineSwitches_Tests.cs | Stabilize change-wave state setup for a test. |
| src/Framework.UnitTests/FileUtilities_Tests.cs | Replace conditional fact usage with runtime skip under xUnit v3. |
| src/Framework.UnitTests/FileMatcher_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Directory.Build.targets | Update runner args/trait filtering, add xUnit v3 extensions + MTP coverage integration, remove xUnit console runner wiring. |
| src/BuildCheck.UnitTests/TaskInvocationAnalysisDataTests.cs | Remove xUnit v2 abstractions import. |
| src/BuildCheck.UnitTests/EndToEndTests.cs | Remove xUnit v2 abstractions import. |
| src/BuildCheck.UnitTests/BuildCheckManagerProviderTests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/WarningsAsMessagesAndErrors_Tests.cs | Adjust assembly location resolution logic for task loading. |
| src/Build.UnitTests/Utilities_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/TerminalLoggerConfiguration_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/TerminalLogger_Tests.cs | Remove Verify-specific attribute usage and xUnit v2 abstractions import. |
| src/Build.UnitTests/Telemetry/Telemetry_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/TaskHostFactoryLifecycle_E2E_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Resources_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/ProjectCache/ProjectCacheTests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/NodeStatus_Transition_Tests.cs | Remove Verify-specific attribute usage. |
| src/Build.UnitTests/NodeStatus_SizeChange_Tests.cs | Remove Verify-specific attribute usage. |
| src/Build.UnitTests/NetTaskHost_E2E_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/MSBuildTaskHostTests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Microsoft.Build.Engine.UnitTests.csproj | Switch Verify package to xUnit v3 variant. |
| src/Build.UnitTests/InvalidProjectFileException_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Instance/ProjectInstance_Internal_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Graph/ResultCacheBasedBuilds_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Graph/ProjectGraph_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Graph/IsolateProjects_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Graph/GraphLoadedFromSolution_tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/ExpressionTreeExpression_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/EvaluationProfiler_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Evaluation/SdkResultEvaluation_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Evaluation/ExpanderFunction_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Evaluation/Expander_Tests.cs | Remove xUnit v2 abstractions import; reset change-wave state for stability. |
| src/Build.UnitTests/EscapingInProjects_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/EndToEndCondition_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Construction/SolutionProjectGenerator_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Construction/SolutionFilter_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Construction/SolutionFile_OldParser_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Construction/SolutionFile_NewParser_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/Construction/ProjectRootElement_Tests.cs | Clear cached read-only flags to prevent cross-test pollution in same process. |
| src/Build.UnitTests/ConsoleLogger_Tests.cs | Ensure Traits.LogAllEnvironmentVariables is restored after tests. |
| src/Build.UnitTests/CLR2TaskHost_E2E_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/ChangeWaves_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BinaryLogger_Tests.cs | Stabilize test behavior around Traits.Instance state and environment variables. |
| src/Build.UnitTests/BackEnd/TaskRouter_IntegrationTests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/TaskRegistry_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/TaskHostFactory_Tests.cs | Adjust assembly location resolution logic for <UsingTask AssemblyFile=...>. |
| src/Build.UnitTests/BackEnd/TaskHostCallback_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/TaskHost_MultiThreadableTask_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/TaskBuilder_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/TargetUpToDateChecker_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/TargetEntry_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/SdkResultOutOfProc_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/SdkResolverLoader_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/MSBuild_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/LoggingContext_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/DebugUtils_tests.cs | Avoid Directory.GetFiles when debug dump dir doesn’t exist. |
| src/Build.UnitTests/BackEnd/BuildRequestConfiguration_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/BuildManager_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/BuildManager_Logging_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/AssemblyTaskFactory_E2E_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.UnitTests/BackEnd/AppHostSupport_Tests.cs | Ensure DOTNET_ROOT_X86 is tracked for restoration in tests. |
| src/Build.OM.UnitTests/NugetRestoreTests.cs | Remove xUnit v2 abstractions import. |
| src/Build.OM.UnitTests/Instance/ProjectInstance_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.OM.UnitTests/Definition/ProjectCollection_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.OM.UnitTests/Definition/Project_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.OM.UnitTests/Construction/ProjectMetadataElement_Tests.cs | Remove xUnit v2 abstractions import. |
| src/Build.OM.UnitTests/Construction/ProjectFormatting_Tests.cs | Remove xUnit v2 abstractions import. |
| global.json | Configure test runner as Microsoft.Testing.Platform. |
| eng/Version.Details.xml | Rename Arcade dependency to Microsoft.DotNet.XUnitV3Extensions. |
| eng/Version.Details.props | Update Arcade xUnit v3 extensions version properties. |
| eng/dependabot/Directory.Packages.props | Add MTP coverage package; switch Verify dependency to Verify.XunitV3. |
| Directory.Packages.props | Switch xUnit extensions package and remove xunit.console version pin. |
| Directory.Build.props | Add xUnit v3/MTP-related properties (TestRunnerName, versions, OutputType for test projects, analyzer suppression). |
| CoverageWindowsFull.runsettings | Removed legacy VS CodeCoverage runsettings. |
| CoverageWindowsFull.config | Added new coverage configuration format for MTP coverage integration. |
| Coverage.runsettings | Removed legacy non-Windows runsettings. |
| Coverage.config | Added new non-Windows coverage configuration format. |
No description provided.