Cherry-pick for release/2.0: pipeline & hot-reload test fixes

[aaronburtle]

Why make this change?

Closes #3517

Ports into release/2.0

• Reliable cleanup of MSSQL integration tests by properly disposing the config file watcher and FileSystemRuntimeConfigLoader, eliminating intermittent IOException failures on File.Delete.
• Faster, more balanced MSSQL integration test pipeline: tests are split into two parallel Windows jobs (combined and configuration), with a shared mssql-test-steps.yml template, and DB-independent unit tests are removed from the MSSQL category so they run in the unit-test pipeline instead.
• Hot-reload test stability: replaced naive timeout waits with polling helpers (WaitForRestEndpointAsync and retry logic to cover the race between hot-reload validation completing and metadata providers finishing re-initialization.
• Autoentity hot-reload test now uses the same polling pattern, removing the last known race condition in that area.Reliable cleanup of MSSQL integration tests by properly disposing the config file watcher and FileSystemRuntimeConfigLoader, eliminating intermittent IOException failures on File.Delete

What is this change?

Cherry-picked PRs (chronological order on main):

• Fix IOException in MSSQL pipeline by disposing config file watchers Fix IOException in MSSQL pipeline by disposing config file watchers #3243
• Split MSSQL integration tests into parallel pipeline jobs and remove MSSQL category from generic unit tests Split MSSQL integration tests into parallel pipeline jobs and remove MSSQL category from generic unit tests #3245
• Fix intermittent hot-reload test failures caused by metadata initialization race condition Fix intermittent hot-reload test failures caused by metadata initialization race condition #3302
• Fix hot-reload race condition in Autoentities test Fix hot-reload race condition in Autoentities test #3320

How was this tested?

Existing and newly added unit/integration tests from each source PR.

[Copilot]

Pull request overview

This cherry-pick ports several reliability and CI pipeline improvements into release/2.0, focusing on stabilizing MSSQL test execution (file watcher disposal + hot-reload race fixes) and reducing overall pipeline time by splitting MSSQL tests into parallel Windows jobs.

Changes:

• Ensure config hot-reload file watchers are disposable and can be cleaned up reliably to prevent intermittent IOException on config file deletion.
• Stabilize hot-reload tests by replacing fixed waits with REST/GraphQL polling helpers to tolerate metadata re-initialization races.
• Speed up MSSQL CI by splitting Windows MSSQL tests into two parallel jobs and moving DB-independent unit tests out of the MSSQL category; introduce a shared MSSQL pipeline steps template.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/Service/Startup.cs	Registers FileSystemRuntimeConfigLoader via factory so DI can dispose it at shutdown.
src/Config/FileSystemRuntimeConfigLoader.cs	Implements IDisposable to tear down the config file watcher and release file handles.
src/Config/ConfigFileWatcher.cs	Implements IDisposable to stop events/unsubscribe and dispose the underlying watcher.
src/Core/Services/MetadataProviders/SqlMetadataProvider.cs	Removes generated autoentities during validate-only initialization to avoid config mutation side effects.
src/Core/Services/MetadataProviders/MsSqlMetadataProvider.cs	Enhances autoentity generation with definition-aware querying/logging and marks generated entities as autoentities.
src/Core/Configurations/RuntimeConfigProvider.cs	Adds removal of generated autoentities from runtime config (used in validate-only flow).
src/Config/ObjectModel/RuntimeConfig.cs	Adds removal helper for entity→datasource mapping and clarifies autoentity-not-found messaging.
src/Config/ObjectModel/Entity.cs	Introduces IsAutoentity (JsonIgnore) to tag auto-generated entities.
src/Service.Tests/ModuleInitializer.cs	Updates snapshot verifier config to ignore Entity.IsAutoentity.
src/Cli.Tests/ModuleInitializer.cs	Updates snapshot verifier config to ignore Entity.IsAutoentity.
src/Service.Tests/Configuration/HotReload/ConfigurationHotReloadTests.cs	Adds polling helpers and a new autoentity hot-reload test to reduce flakiness.
src/Service.Tests/Configuration/ConfigurationTests.cs	Adds retry/backoff for config file deletion and improves TestServer disposal patterns; updates expected autoentity conflict message.
src/Service.Tests/UnitTests/SqlMetadataProviderUnitTests.cs	Updates unit test callsite for new QueryAutoentitiesAsync signature.
src/Service.Tests/UnitTests/RuntimeConfigLoaderJsonDeserializerTests.cs	Removes MSSQL test category so it runs with unit tests instead of MSSQL pipeline.
src/Service.Tests/UnitTests/RestServiceUnitTests.cs	Removes MSSQL test category so it runs with unit tests instead of MSSQL pipeline.
src/Service.Tests/UnitTests/RequestValidatorUnitTests.cs	Removes MSSQL test category so it runs with unit tests instead of MSSQL pipeline.
src/Service.Tests/UnitTests/RequestContextUnitTests.cs	Removes MSSQL test category so it runs with unit tests instead of MSSQL pipeline.
src/Service.Tests/UnitTests/EntitySourceNamesParserUnitTests.cs	Removes MSSQL test category so it runs with unit tests instead of MSSQL pipeline.
src/Cli/ConfigGenerator.cs	Adjusts dab start behavior so --LogLevel is only passed when explicitly provided.
src/Cli/Commands/AutoConfigOptions.cs	Renames auto-config option to template.mcp.dml-tools for consistency.
src/Cli.Tests/AutoConfigTests.cs	Updates tests for renamed auto-config option parameter.
src/Cli.Tests/EndToEndTests.cs	Adds a test asserting Startup.IsLogLevelOverriddenByCli behavior (but currently has a lifecycle issue; see PR comments).
schemas/dab.draft.schema.json	Updates autoentities template mcp schema to allow boolean shorthand or object form.
.pipelines/templates/mssql-test-steps.yml	New shared Azure Pipelines template for MSSQL job setup/test execution.
.pipelines/mssql-pipelines.yml	Splits Windows MSSQL tests into parallel “combined” and “configuration” jobs using the shared template.

Comments suppressed due to low confidence (1)

src/Service.Tests/Configuration/HotReload/ConfigurationHotReloadTests.cs:832

• HotReloadAutoentities creates multiple HttpResponseMessage instances (restResult, failRestResult) that are not disposed. Because this test performs retries/polling and additional requests, leaving responses undisposed can unnecessarily hold connections/resources and may contribute to flakiness. Prefer using (or explicit Dispose) for all HttpResponseMessage instances created in the test.

        // Act
        HttpResponseMessage restResult = await _testClient.GetAsync($"rest/autoentity_books");

        GenerateConfigFile(
            connectionString: $"{ConfigurationTests.GetConnectionStringFromEnvironmentConfig(TestCategory.MSSQL).Replace("\\", "\\\\")}",
            autoentityName: "HotReload_{object}");
        await WaitForConditionAsync(
          () => WriterContains(HOT_RELOAD_SUCCESS_MESSAGE),
          TimeSpan.FromSeconds(HOT_RELOAD_TIMEOUT_SECONDS),
          TimeSpan.FromMilliseconds(500));

        // After hot-reload, the engine may still be re-initializing metadata providers.
        // Poll the REST endpoint to allow time for the engine to become fully ready.
        using HttpResponseMessage hotReloadRestResult = await WaitForRestEndpointAsync("rest/HotReload_books", HttpStatusCode.OK);

        // Once the engine is fully ready, verify the old autoentity name is no longer recognized.
        HttpResponseMessage failRestResult = await _testClient.GetAsync($"rest/autoentity_books");

        // Assert
        Assert.AreEqual(HttpStatusCode.OK, restResult.StatusCode,
                    $"REST request before hot-reload failed when it was expected to succeed. Response: {await restResult.Content.ReadAsStringAsync()}");
        Assert.AreEqual(HttpStatusCode.NotFound, failRestResult.StatusCode,
                    $"REST request after hot-reload succeeded when it was expected to fail. Response: {await failRestResult.Content.ReadAsStringAsync()}");
        Assert.AreEqual(HttpStatusCode.OK, hotReloadRestResult.StatusCode,