🐛 [BUG] - Failed to release `v2.6.0`

[lvaylet]

SLO Generator Version

v2.6.0

Python Version

3.9

What happened?

The release/release-pipy and deploy/cloudrun jobs of the v2.6.0 release pipeline failed:

https://github.com/google/slo-generator/actions/runs/7960838943 (release/release-pipy)
https://github.com/google/slo-generator/actions/runs/7960838947 (deploy/cloudrun)

Note that the deploy/cloudrun failure might be a consequence of the release/release-pipy failure.

What did you expect?

Release pipeline completes successfully.

Screenshots

No response

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[lvaylet]

Regarding the release/release-pipy job, investigate why Twine uses a legacy/ URL in

slo-generator/.github/workflows/release.yml

Line 29 in b18fc09

TWINE_REPOSITORY_URL: https://upload.pypi.org/legacy/

Then run some tests locally with the latest version of Twine and default settings, and upload to Test PyPi first to avoid updating the real index.

[lvaylet]

The logs of the 2.5.2 release have expired so I am unable to compare them to this failed run.

https://github.com/google/slo-generator/actions/runs/6826860828/job/18567693525

[lvaylet]

For the record, here is the output for the 2.6.0 release that failed:

adding 'slo_generator-2.6.0.dist-info/METADATA'
adding 'slo_generator-2.6.0.dist-info/WHEEL'
adding 'slo_generator-2.6.0.dist-info/entry_points.txt'
adding 'slo_generator-2.6.0.dist-info/top_level.txt'
adding 'slo_generator-2.6.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Uploading slo_generator-2.6.0-py2.py3-none-any.whl
25l
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/116.5 kB • --:-- • ?
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/116.5 kB • --:-- • ?
 70% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━ 81.9/116.5 kB • 00:01 • 70.2 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.5/116.5 kB • 00:00 • 1.4 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.5/116.5 kB • 00:00 • 1.4 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.5/116.5 kB • 00:00 • 1.4 MB/s
25hWARNING  Error during upload. Retry with the --verbose option for more details. 
ERROR    HTTPError: 403 Forbidden from https://upload.pypi.org/legacy/          
         Invalid or non-existent authentication information. See                
         https://pypi.org/help/#invalid-auth for more information.              
make: *** [Makefile:52: deploy] Error 1
Error: Process completed with exit code 2.

[lvaylet]

I managed to reproduce the error locally on my dev machine with:

source venv3.9.18/bin/activate
make clean
make install_twine
python setup.py sdist bdist_wheel
export TWINE_USERNAME=slo-generator
export TWINE_PASSWORD=<REDACTED> (stored in go/valentine)
twine upload -r testpypi dist/*

The output:

Uploading distributions to https://test.pypi.org/legacy/
Uploading slo_generator-2.6.0-py2.py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.5/116.5 kB • 00:00 • 50.2 MB/s
WARNING  Error during upload. Retry with the --verbose option for more details.                                                       
ERROR    HTTPError: 403 Forbidden from https://test.pypi.org/legacy/                                                                  
         Invalid or non-existent authentication information. See https://test.pypi.org/help/#invalid-auth for more information.

The --verbose option gives slightly more details:

$ twine upload -r testpypi dist/* --verbose
Uploading distributions to https://test.pypi.org/legacy/
INFO     dist/slo_generator-2.6.0-py2.py3-none-any.whl (91.2 KB)                                                                      
INFO     dist/slo-generator-2.6.0.tar.gz (66.4 KB)                                                                                    
INFO     password set by command options                                                                                              
INFO     username: __token__                                                                                                          
INFO     password: <hidden>                                                                                                           
Uploading slo_generator-2.6.0-py2.py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.5/116.5 kB • 00:00 • 44.5 MB/s
INFO     Response from https://test.pypi.org/legacy/:                                                                                 
         403 Invalid or non-existent authentication information. See https://test.pypi.org/help/#invalid-auth for more information.   
INFO     <html>                                                                                                                       
          <head>                                                                                                                      
           <title>403 Invalid or non-existent authentication information. See https://test.pypi.org/help/#invalid-auth for more       
         information.</title>                                                                                                         
          </head>                                                                                                                     
          <body>                                                                                                                      
           <h1>403 Invalid or non-existent authentication information. See https://test.pypi.org/help/#invalid-auth for more          
         information.</h1>                                                                                                            
           Access was denied to this resource.<br/><br/>                                                                              
         Invalid or non-existent authentication information. See https://test.pypi.org/help/#invalid-auth for more information.       
                                                                                                                                      
                                                                                                                                      
          </body>                                                                                                                     
         </html>                                                                                                                      
ERROR    HTTPError: 403 Forbidden from https://test.pypi.org/legacy/                                                                  
         Invalid or non-existent authentication information. See https://test.pypi.org/help/#invalid-auth for more information.

Notice how the /legacy endpoint seems linked to the Test PyPI instance, even without the TWINE_REPOSITORY_URL environment variable specified in

slo-generator/.github/workflows/release.yml

Line 29 in b18fc09

TWINE_REPOSITORY_URL: https://upload.pypi.org/legacy/

I tried to log in to Test PyPI with the credentials above. I get the following error:

So there might an authentication issue indeed. But then I am not sure how the previous releases (up to 2.5.2) landed on the real instance of PyPI, while the upload URL explicitly points to the test instance.

[lvaylet]

Every blog article about Twine and releasing to PyPI recommends using an access token instead of the usual username/password pair.

For example: https://dev.to/arnu515/create-a-pypi-pip-package-test-it-and-publish-it-using-github-actions-part-1-3cp8

[lvaylet]

When this issue is solved, it would make sense to use OpenID Connect (OIDC) to create a passwordless connection between PyPI and GitHub Actions, as described in:

https://pypi.org/manage/project/slo-generator/settings/publishing/

[lvaylet]

Running the deployment code against release v2.6.0 gives the same result and the same error message, with the same /legacy url:

$ git checkout v2.6.0 
$ make clean install_twine build
$ twine upload dist/*
twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Uploading slo_generator-2.6.0-py2.py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.5/116.5 kB • 00:00 • 975.0 kB/s
WARNING  Error during upload. Retry with the --verbose option for more details.                                                       
ERROR    HTTPError: 403 Forbidden from https://upload.pypi.org/legacy/                                                                
         Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for more information.

Not sure how to interpret that. The username and password match the real instance of PyPI.

[lvaylet]

Got it. PyPI enforces 2FA since Jan 1, 2024: https://blog.pypi.org/posts/2023-12-13-2fa-enforcement/

As a result, an API Token or Trusted Publisher must be used to upload packages. A username/password pair can no longer be used.

Release v2.5.2 was uploaded in November 2023, while release v2.6.0 is from February 2024.

Actions: set up Trusted Publisher for a credential-free experience.

[lvaylet]

For the record, Trusted Published is configured like:

[lvaylet]

I needed to confirm whether 2FA was actually the source of the issue. As the fix for this bug did not trigger a new release workflow, I generated an API Token on PyPI, saved it to go/valentine and used it from my local machine to upload v2.6.0 with the same commands above. I just provided Twine with the API Token when requested, and the process completed successfully in a couple of seconds.