Make the ClearlyDefined batch size configurable via -lbs

[ChristophHornung]

Make the ClearlyDefined batch size configurable via -lbs

Problem

When -li true is set, LicenseInformationService.FetchLicenseInformationFromAPI sends components to the ClearlyDefined batch API in slices of a hardcoded batchSize = 500. For a project with a few hundred components that is a single POST to https://api.clearlydefined.io/definitions?expand=-files asking the service to resolve every coordinate at once.

That request regularly exceeds the origin timeout in front of api.clearlydefined.io, which responds HTTP 524. Because there is no retry and no per-batch fallback, one failed response means the SBOM ends up with no license information at all:

##[debug]Retrieving license information for 292 components...
##[debug]Retrieving license information for 292 components took 125.1564491 seconds
##[warning]Error encountered while fetching license information from API, resulting SBOM may have
          incomplete license information. Request returned status code: 524
##[information]Found license information for 0 out of 296 unique components.

Raising -lto does not help — the client timeout was already honored here (the request ran a full 125 s); the failure is on the server side, so a longer client timeout just waits longer for a response that never arrives. Splitting the work into smaller requests does help, but the batch size is not reachable from the outside.

What this changes

Per the second maintainer takeaway in #944 (expose the batch size as a CLI argument):

• New LicenseInformationBatchSize (-lbs) generation argument, plumbed through ConfigurationMapper, IConfiguration / Configuration / InputConfiguration, and sanitized in ConfigSanitizer exactly like -lto: null → default, non-positive → default with a warning, above max → truncated with a warning, and a warning when -lbs is supplied without -li.
• LicenseInformationService takes the batch size as a parameter instead of hardcoding it.
• Constants.DefaultLicenseFetchBatchSize = 500, Constants.MaxLicenseFetchBatchSize = 1000.

The default of 500 is deliberately the previous hardcoded value, so behaviour is unchanged unless -lbs is passed. The #944 discussion also floated lowering the default to 100; that is a one-line change on top of this if you would prefer users to get smaller batches without opting in. I kept it opt-in so this PR carries no behavioural change of its own — happy to switch if you would rather have the new default.

Notes for reviewers

This touches version-pinned interfaces. ILicenseInformationService.FetchLicenseInformationFromAPI and ILicenseInformationFetcher.FetchLicenseInformationAsync each gain an int batchSize parameter, and IConfiguration gains a property; VersionSpecificPins/Version_4_0/InterfaceConcretionTests.cs is updated accordingly. The precedent I followed is #1110, which added a property to the pinned IConfiguration and updated the same pin class, shipping in v4.1.0.

If you would rather not change that surface at all, the alternative is to inject IConfiguration into LicenseInformationService and read the batch size there, leaving both executor interfaces untouched. Happy to switch — just say which you prefer.

Not included here: the third takeaway in #944 (a CLI argument to fail the run when license fetching fails). That is independent of this change and also covered by #733; glad to open a separate PR for it.

Two adjacent gaps noticed while doing this — both pre-existing and left alone to keep the diff focused:

1. src/Microsoft.Sbom.Targets/README.md documents <SbomGenerationLicenseInformationTimeoutInSeconds>, but no such property exists in GenerateSbom.cs, Microsoft.Sbom.Targets.targets, or SbomCLIToolTask.cs — the MSBuild task cannot actually set -lto. -lbs is therefore not added to that README either, so as not to document a second option that does nothing.
2. ConfigFile.cs has no LicenseInformationTimeoutInSeconds, so -lto cannot be set through a config file. -lbs mirrors that and is CLI-only.

Let me know if you want either folded into this PR.

Testing

• Added LicenseInformationBatchSize sanitizer tests mirroring the existing -lto coverage: valid values pass through unchanged (1, 500, 1000), out-of-range values are corrected (negative, zero, max+1, int.MaxValue), and null falls back to the default with SettingSource.Default.
• Existing test suite unchanged apart from the interface pin updates.

Addresses #944.

[ChristophHornung]

@microsoft-github-policy-service agree