can-i-deploy doesn't check that ONLY compatible providers are deployed to an environment.

[paulspencerwilliams]

The can-i-deploy tool should take note of all versions of all collaborators released to a target environment. However, during my short evaluation of Pact, I've observed an issue where in one situation, it returns a false positives - advising it's safe to deploy when it isn't.

The following screenshot (taken from PactFlow.io) presents the Matrix for a consumer and provider:

I deliberately introduced a breaking change within the consumer at 0.0.3-SNAPSHOT when both participants where 'deployed to production' at 0.0.2-SNAPSHOT. At this stage, both can-i-deploy and the Matrix both agreed that this version was not safe to deploy.

I resolved the breakage with a resulting update to the provider in its 0.0.3-SNAPSHOT. At this stage, the Matrix correctly stated that the 0.0.3-SNAPSHOTs are compatible. can-i-deploy correctly stated that the consumer 0.0.3-SNAPSHOT was unsafe to release as the incompatible provider 0.0.2-SNAPSHOT was still deployed.

The screenshot demonstrates the state after releasing provider 0.0.3-SNAPSHOT. The second line within the Matrix correctly shows that consumer 0.0.3 is incompatible with provider 0.0.2 which is still released to production and therefore is unsafe to deploy. However, the output of can-i-deploy disagrees with the matrix:

will@host broker % podman run --rm \
    -e PACT_BROKER_BASE_URL=https://xxx.pactflow.io \
    -e PACT_BROKER_TOKEN=<token> \
    pactfoundation/pact-cli:latest pact-broker can-i-deploy \
    --pacticipant consumer \
    --version 0.0.3-SNAPSHOT \
    --to-environment production
Computer says yes \o/ 

CONSUMER | C.VERSION      | PROVIDER | P.VERSION      | SUCCESS? | RESULT#
---------|----------------|----------|----------------|----------|--------
consumer | 0.0.3-SNAPSHOT | provider | 0.0.3-SNAPSHOT | true     | 1      

VERIFICATION RESULTS
--------------------
1. https://xxx.pactflow.io/pacts/provider/provider/consumer/consumer/pact-version/4a9234eb022c262a1a3d960f70b64cf0f7c7dc73/metadata/Y3ZuPTAuMC4zLVNOQVBTSE9U/verification-results/221 (success)

All required verification results are published and successful

Whilst it correctly reports true for the provider 0.0.3 line, it fails to list the provider 0.0.2 line which should evaluate to false.

I've now replicated this with two separate codebases using Pact JVM consumer, provider and maven artefacts at 4.6.20. The first replication used the latest Docker based pact broker (2.118.0) and this latest attempt used PactFlow.io. Both used the latest Docker based CLI:

will@host broker % podman run --rm pactfoundation/pact-cli:latest version
1.5.0

These both failed on my local machine.

Steps to replicate:

1. Create a compatible consumer and provider and publish verification of both to a broker
2. Release both to 'production' in the broker
3. Introduce a breaking change in the consumer. can-i-deploy will agree with the Matrix and state that the breaking consumer is not safe to release to production.
4. Introduce a fix to the breakage in the provider and verify again. can-i-deploy will agree with the Matrix and state that the breaking consumer is not safe to release to production as non compatible providers are still in production.
5. Release the provider fix to production within the broker. At this point, can-i-deploy for the latest consumer will diverge from the Matrix. The Matrix will correctly hint that the consumer is not safe to deploy. However, the can-i-deploy tool will incorrectly state that release is safe and only include a line for the released provider with fix. It will not include the incompatible previous version of the provider that is still released as it has not been record-support-ended.

[mefellows]

OK, so the can-i-deploy result suggests that only 0.0.3-SNAPSHOT of the provider has been deployed (or released) to the production environment. The matrix shows that 0.0.3-SNAPSHOT of the consumer is compatible with this version.

So the question is why is the matrix ignoring a released version of the provider (0.0.2-SNAPSHOT, presumably).

Can you please navigate to the released-versions API. In the HAL browser you can:

1. go to /pacticipants/provider
2. Click on the link next to pb:deployed-environments or pb:released-environments
3. Share the results here

That should help us get some idea as to what is going on.

[github-actions]

👋 Hi! The 'smartbear-supported' label has just been added to this issue, which will create an internal tracking ticket in PactFlow's Jira (PACT-5936). We will use this to prioritise and assign a team member to this task. All activity will be public on this ticket. For now, sit tight and we'll update this ticket once we have more information on the next steps.

See our documentation for more information.

[paulspencerwilliams]

In the HAL Browser, at /pacticipants/provider, there are no pb:deployed-environments or pb:released-environments.

[Saup21]

We investigated this and confirmed that everything is working as expected.

Following the replication steps provided, the results looked correct. The Matrix results were consistent with can_i_deploy — no divergence was observed.

Regarding the screenshot showing the state after releasing provider 0.0.3-SNAPSHOT: the second line in the Matrix correctly indicates that consumer 0.0.3 is incompatible with provider 0.0.2, which is still deployed to production(likely a UI glitch), and is therefore unsafe to deploy. This is expected behaviour.

As for the concern about multiple provider versions appearing in production — this was, as mentioned, likely a UI glitch. Only one version of an application can be deployed to production at a time. Version 0.0.3 of both consumer and provider passes verification successfully, and since provider 0.0.3 is in the production environment, the can-i-deploy check passes as expected. This is also consistent with the Matrix result.

See results from my tests 👇

[mefellows]

this was, as mentioned, likely a UI glitch. Only one version of an application can be deployed to production at a time

Just for clarity (or ICYMI), these applications aren't deployed to production, rather they have been released to production (see "releases" in https://docs.pact.io/pact_broker/recording_deployments_and_releases). This means that it's possible for more than one version of an application to be in the target environment.

In this case, 0.0.2-SNAPSHOT and 0.0.3-SNAPSHOT of provider are both in production simultaneously. So when asking if it's safe to release 0.0.3-SNAPSHOT of consumer to production, the answer should be:

• overall false
• true against the 0.0.3-SNAPSHOT
• false against the 0.0.2-SNAPSHOT

[paulspencerwilliams]

this was, as mentioned, likely a UI glitch. Only one version of an application can be deployed to production at a time

Just for clarity (or ICYMI), these applications aren't deployed to production, rather they have been released to production (see "releases" in https://docs.pact.io/pact_broker/recording_deployments_and_releases). This means that it's possible for more than one version of an application to be in the target environment.

In this case, 0.0.2-SNAPSHOT and 0.0.3-SNAPSHOT of provider are both in production simultaneously. So when asking if it's safe to release 0.0.3-SNAPSHOT of consumer to production, the answer should be:

• overall false
• true against the 0.0.3-SNAPSHOT
• false against the 0.0.2-SNAPSHOT

Yes, thank you for summarising the exact issue I observed.

[paulspencerwilliams]

So testing again demonstrates a different result to earlier, making me believe this issue has been resolved in pactflow.io.

will@Pauls-MacBook-Pro consumer % podman run --rm \
    -e PACT_BROKER_BASE_URL=https://xxxxx.pactflow.io/ \
    -e PACT_BROKER_TOKEN=<token> \
    pactfoundation/pact-cli:latest pact-broker can-i-deploy \
    --pacticipant consumer \
    --version 0.0.3-SNAPSHOT \
    --to-environment production
Computer says no ¯\_(ツ)_/¯

CONSUMER | C.VERSION      | PROVIDER | P.VERSION      | SUCCESS? | RESULT#
---------|----------------|----------|----------------|----------|--------
consumer | 0.0.3-SNAPSHOT | provider | 0.0.2-SNAPSHOT | false    | 1

VERIFICATION RESULTS
--------------------
1. https://xxxxxx.pactflow.io/pacts/provider/provider/consumer/consumer/pact-version/4a9234eb022c262a1a3d960f70b64cf0f7c7dc73/metadata/Y3ZuPTAuMC4zLVNOQVBTSE9U/verification-results/237 (failure)

I will, over the coming days replicate with a locally deployed pact broker.

[Saup21]

I see the problem when using record-release instead of record-deployment

[pact-bot]

🤖 Great news! We've labeled this issue as SmartBear Supported and created a tracking ticket in PactFlow's Jira (PACT-6874). We'll keep work public and post updates here. Meanwhile, feel free to check out our docs. Thanks for your patience!

[paulspencerwilliams]

Cheers for re-opening. Please don’t hesitate to reach out for further information.