Problem with mapping of licenseConcluded and licenseDeclared values when importing SPDX 2.3 SBOM

[DennisClark]

Describe the bug
Using the load_sbom pipeline, an import of an SPDX 2.3 SBOM does not interpret and map licenseConcluded and licenseDeclared values correctly when licenseDeclared has a valid SPDX license expression but licenseConcluded has a value of NOASSERTION.

System configuration

• SCIO v37.1.0
• input: an SPDX 2.3 SBOM
• pipeline: load_sbom

To Reproduce
Please see the excerpt below, which was taken from a proprietary source.

1. Create an SPDX 2.3 SBOM containing the excerpt.
2. Import using load_sbom.
3. Note that all of the packages show unknown as the Declared license expression.

Expected behavior
The SPDX 2.3 SBOM field for declaredLicense should be used to compute and populate the SCIO Declared license expression. The SPDX 2.3 SBOM concludedLicense is also useful information and should be stored as well; ideally, it should be available in the visible details in SCIO.

    "packages": [
        {
            "SPDXID": "SPDXRef-Package-gstreamer1.0-plugins-good-video4linux2",
            "name": "gstreamer1.0-plugins-good-video4linux2",
            "downloadLocation": "NOASSERTION",
            "packageVerificationCode": {
                "packageVerificationCodeValue": "1c6cda1dc680b9c3f88649dba0b9df5ee898c4ee"
            },
            "licenseInfoFromFiles": [
                "NOASSERTION"
            ],
            "licenseConcluded": "NOASSERTION",
            "licenseDeclared": "LGPL-2.1-or-later",
            "copyrightText": "NOASSERTION",
            "versionInfo": "1.20.7",
            "supplier": "Organization: OpenEmbedded",
            "hasFiles": [
                "SPDXRef-PackagedFile-gstreamer1.0-plugins-good-video4linux2-1"
            ]
        },
        {
            "SPDXID": "SPDXRef-Package-glibc-gconv-ibm866",
            "name": "glibc-gconv-ibm866",
            "downloadLocation": "NOASSERTION",
            "packageVerificationCode": {
                "packageVerificationCodeValue": "834cd72c9bdf9656fe946861d8b3648b49e02f26"
            },
            "licenseInfoFromFiles": [
                "NOASSERTION"
            ],
            "licenseConcluded": "NOASSERTION",
            "licenseDeclared": "(GPL-2.0-only AND LGPL-2.1-only)",
            "copyrightText": "NOASSERTION",
            "versionInfo": "2.35",
            "supplier": "Organization: OpenEmbedded",
            "hasFiles": [
                "SPDXRef-PackagedFile-glibc-gconv-ibm866-1"
            ]
        },
        {
            "SPDXID": "SPDXRef-Package-dnsmasq",
            "name": "dnsmasq",
            "downloadLocation": "NOASSERTION",
            "packageVerificationCode": {
                "packageVerificationCodeValue": "33a1a3f16ba0bef5f130549a3aa7f837a2e5c57c"
            },
            "licenseInfoFromFiles": [
                "NOASSERTION"
            ],
            "licenseConcluded": "NOASSERTION",
            "licenseDeclared": "(GPL-2.0-only OR GPL-3.0-only)",
            "copyrightText": "NOASSERTION",
            "versionInfo": "2.90",
            "supplier": "Organization: OpenEmbedded",
            "hasFiles": [
                "SPDXRef-PackagedFile-dnsmasq-1",
                "SPDXRef-PackagedFile-dnsmasq-2",
                "SPDXRef-PackagedFile-dnsmasq-3",
                "SPDXRef-PackagedFile-dnsmasq-4",
                "SPDXRef-PackagedFile-dnsmasq-5"
            ]
        },
        {
            "SPDXID": "SPDXRef-Package-perl-module-mro",
            "name": "perl-module-mro",
            "downloadLocation": "NOASSERTION",
            "packageVerificationCode": {
                "packageVerificationCodeValue": "f6f6e875c25588b7a69262dce3d2023cbe467c4e"
            },
            "licenseInfoFromFiles": [
                "NOASSERTION"
            ],
            "licenseConcluded": "NOASSERTION",
            "licenseDeclared": "(Artistic-1.0 OR GPL-1.0-or-later)",
            "copyrightText": "NOASSERTION",
            "versionInfo": "5.34.3",
            "supplier": "Organization: OpenEmbedded",
            "hasFiles": [
                "SPDXRef-PackagedFile-perl-module-mro-1",
                "SPDXRef-PackagedFile-perl-module-mro-2"
            ]
        },
        {
            "SPDXID": "SPDXRef-Package-gstreamer1.0-plugins-bad-closedcaption",
            "name": "gstreamer1.0-plugins-bad-closedcaption",
            "downloadLocation": "NOASSERTION",
            "packageVerificationCode": {
                "packageVerificationCodeValue": "8ebae79d2303808ee241060d59336fbf97a9bf78"
            },
            "licenseInfoFromFiles": [
                "NOASSERTION"
            ],
            "licenseConcluded": "NOASSERTION",
            "licenseDeclared": "LGPL-2.1-or-later",
            "copyrightText": "NOASSERTION",
            "versionInfo": "1.20.7",
            "supplier": "Organization: OpenEmbedded",
            "hasFiles": [
                "SPDXRef-PackagedFile-gstreamer1.0-plugins-bad-closedcaption-1"
            ]
        },
        {
            "SPDXID": "SPDXRef-Recipe-iptables",
            "name": "iptables",
            "downloadLocation": "http://netfilter.org/projects/iptables/files/iptables-1.8.7.tar.bz2",
            "licenseInfoFromFiles": [
                "NOASSERTION"
            ],
            "licenseConcluded": "NOASSERTION",
            "licenseDeclared": "GPL-2.0-or-later",
            "copyrightText": "NOASSERTION",
            "versionInfo": "1.8.7",
            "summary": "Tools for managing kernel packet filtering capabilities",
            "supplier": "Organization: OpenEmbedded",
            "description": "iptables is the userspace command line program used to configure and control network packet filtering code in Linux.",
            "homepage": "http://www.netfilter.org/",
            "externalRefs": [
                {
                    "referenceCategory": "SECURITY",
                    "referenceType": "http://spdx.org/rdf/references/cpe23Type",
                    "referenceLocator": "cpe:2.3:a:*:iptables:1.8.7:*:*:*:*:*:*:*"
                }
            ]

[tdruez]

Summary

The current implementation in SCIO is wrong as it takes the SPDX licenseConcluded and puts it in the declared_license_expression field.

Current state of license-related fields on Package models:

DejaCode:

• license_expression (rendered as "Concluded license expression")
• declared_license_expression
• other_license_expression

ScanCode.io

• declared_license_expression
• other_license_expression

SPDX (2.2/2.3):

• licenseConcluded
• licenseDeclared

CycloneDX (1.6/1.7):

• licenses.expression + optional acknowledgement (Must be one of: "declared" or "concluded")

Issue

There's a gap regarding "concluded" license data in SCIO. This is mainly because the license fields were taken and kept in sync with scancode-toolkit model, as most of the data in SCIO comes from toolkit scans.
This is not good enough anymore as we need to keep the "concluded" license data from SBOM when available.
We need to make sure we also propagate that data to DejaCode.

One aspect to discuss regarding the computation of the compliance_alert values.
On the SCIO side, it's currently based on "declared_license_expression" as it's the only license data point available, but with the addition of a concluded_license_expression entry, we have to reconsider this approach.
On the DejaCode side, the concluded_license_expression is used for this computation.

SBOM SPECS

• https://spdx.github.io/spdx-spec/v2.3/package-information/#715-declared-license-field
• https://cyclonedx.org/docs/1.7/json/#metadata_tools_oneOf_i0_components_items_licenses_items_oneOf_i1_acknowledgement

[tdruez]

@DennisClark

1. a workaround fix was implemented in fix: fallback to license_declared when loading SPDX SBOM #2144 so you can load your SBOM without too much data loss.
   The "licenseDeclared" value is now used when "licenseConcluded" is empty/"NOASSERTION"

2. I've also started a proper long term fix adding a concluded_license_expression on the SCIO side but this will require more time to implement.

[DennisClark]

Thanks @tdruez -- excellent approach to a solution.