Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Repository	Step Name	Console log	Pull Request
1569528	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132632/merge to s	Log	dotnet/runtime#132632
1569068	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68829/merge to s	Log	dotnet/aspnetcore#68829
1569103	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132795/merge to s	Log	dotnet/runtime#132795
1568612	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55940/merge to s	Log	dotnet/sdk#55940
1568485	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132633/merge to s	Log	dotnet/runtime#132633
1568526	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55520/merge to s	Log	dotnet/sdk#55520
1568432	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132769/merge to s	Log	dotnet/runtime#132769
1568519	dotnet/maui	Checkout dotnet/maui@refs/pull/37802/merge to s	Log	dotnet/maui#37802
1567862	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132737/merge to s	Log	dotnet/runtime#132737
1567394	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68784/merge to s	Log	dotnet/aspnetcore#68784
1567253	dotnet/maui	Checkout dotnet/maui@release/11.0.1xx-rc1 to s	Log
1566335	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1565575	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68754/merge to s	Log	dotnet/aspnetcore#68754
1565264	dotnet/maui	Checkout dotnet/maui@inflight/current to s	Log
1562783	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132633/merge to s	Log	dotnet/runtime#132633
1562675	dotnet/maui	Checkout dotnet/maui@net11.0 to s	Log
1562372	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132438/merge to s	Log	dotnet/runtime#132438
1561613	dotnet/runtime	Checkout dotnet/runtime@release/10.0 to s	Log
1561525	dotnet/performance	Checkout dotnet/performance@refs/pull/5287/merge to s	Log	dotnet/performance#5287
1561578	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132593/merge to s	Log	dotnet/runtime#132593
1561116	dotnet/maui	Checkout dotnet/maui@refs/pull/37688/merge to s	Log	dotnet/maui#37688
1560580	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132556/merge to s		dotnet/runtime#132556
1560388	dotnet/maui	Checkout dotnet/maui@refs/pull/37642/merge to s	Log	dotnet/maui#37642
1559803	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132529/merge to s	Log	dotnet/runtime#132529
1558545	dotnet/runtime	Checkout dotnet/runtime@release/11.0-rc1 to s	Log
1557990	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132432/merge to s	Log	dotnet/runtime#132432
1557800	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132424/merge to s	Log	dotnet/runtime#132424
1557752	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55841/merge to s	Log	dotnet/sdk#55841
1557745	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55837/merge to s	Log	dotnet/sdk#55837
1557283	dotnet/maui	Checkout dotnet/maui@main to s	Log
1551774	dotnet/maui	Checkout dotnet/maui@refs/pull/37111/merge to s	Log	dotnet/maui#37111
1556156	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132138/merge to s	Log	dotnet/runtime#132138
1555842	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131552/merge to s	Log	dotnet/runtime#131552
1556026	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131686/merge to s	Log	dotnet/runtime#131686
1556037	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132424/merge to s	Log	dotnet/runtime#132424
1555997	dotnet/xharness	Checkout dotnet/xharness@refs/pull/1659/merge to s	Log	dotnet/xharness#1659
1555918	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132421/merge to s	Log	dotnet/runtime#132421
1555994	dotnet/aspnetcore	Checkout dotnet/aspnetcore@main to s	Log
1555984	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68598/merge to s	Log	dotnet/aspnetcore#68598
1555991	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68600/merge to s	Log	dotnet/aspnetcore#68600
1555992	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68600/merge to s	Log	dotnet/aspnetcore#68600
1555977	dotnet/sdk	Checkout dotnet/sdk@main to s	Log
1555938	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1555935	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132388/merge to s	Log	dotnet/runtime#132388
1555874	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68594/merge to s	Log	dotnet/aspnetcore#68594
1555927	dotnet/roslyn	Checkout dotnet/roslyn@refs/pull/84917/merge to s	Log	dotnet/roslyn#84917
1555888	dotnet/maui	Checkout dotnet/maui@refs/pull/37424/merge to s	Log	dotnet/maui#37424
1555887	dotnet/maui	Checkout dotnet/maui@refs/pull/37424/merge to s	Log	dotnet/maui#37424
1555925	dotnet/maui	Checkout dotnet/maui@refs/pull/37433/merge to s	Log	dotnet/maui#37433
1555914	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132420/merge to s	Log	dotnet/runtime#132420
1555900	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132416/merge to s	Log	dotnet/runtime#132416
1555891	dotnet/aspnetcore	Checkout dotnet/aspnetcore@main to s	Log
1555880	dotnet/xharness	Checkout dotnet/xharness@release/11.0 to s	Log
1555618	dotnet/arcade	Checkout dotnet/arcade@refs/pull/17356/merge to s	Log	#17356
1555421	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2751/merge to s	Log	maestro-auth-test/build-result-analysis-test#2751
1555422	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2751/merge to s	Log	maestro-auth-test/build-result-analysis-test#2751
1554876	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68563/merge to s	Log	dotnet/aspnetcore#68563
1553358	dotnet/maui	Checkout dotnet/maui@net11.0 to s	Log
1553274	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132163/merge to s	Log	dotnet/runtime#132163
1552999	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68505/merge to s	Log	dotnet/aspnetcore#68505
1552326	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68491/merge to s	Log	dotnet/aspnetcore#68491
1552517	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55776/merge to s	Log	dotnet/sdk#55776
1552470	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55774/merge to s	Log	dotnet/sdk#55774
1552427	dotnet/machinelearning	Checkout dotnet/machinelearning@refs/pull/7675/merge to s	Log
1552101	dotnet/maui	Checkout dotnet/maui@refs/pull/37364/merge to s	Log
1552102	dotnet/maui	Checkout dotnet/maui@refs/pull/37120/merge to s	Log
1552159	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132254/merge to s	Log	dotnet/runtime#132254
1551891	dotnet/maui	Checkout dotnet/maui@refs/pull/37393/merge to s	Log	dotnet/maui#37393
1551030	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55760/merge to s	Log	dotnet/sdk#55760
1550178	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6571/merge to s	Log	dotnet/arcade-services#6571
1549848	dotnet/xharness	Checkout dotnet/xharness@refs/pull/1666/merge to s	Log	dotnet/xharness#1666
1549677	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68269/merge to s	Log	dotnet/aspnetcore#68269
1549392	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55737/merge to s	Log	dotnet/sdk#55737
1549267	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132169/merge to s	Log	dotnet/runtime#132169
1548695	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55721/merge to s	Log	dotnet/sdk#55721
1548870	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55677/merge to s	Log	dotnet/sdk#55677
1548539	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55718/merge to s	Log	dotnet/sdk#55718
1548404	dotnet/arcade	Checkout dotnet/arcade@refs/pull/17304/merge to s	Log	#17304
1548265	dotnet/runtime	Checkout dotnet/runtime@release/10.0 to s	Log
1547353	dotnet/maui	Checkout dotnet/maui@main to s	Log
1547216	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131783/merge to s	Log	dotnet/runtime#131783
1547122	dotnet/maui	Checkout dotnet/maui@refs/pull/36212/merge to s	Log	dotnet/maui#36212
1546374	dotnet/runtime	Checkout dotnet/runtime@release/8.0-staging to s	Log
1544487	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132001/merge to s	Log	dotnet/runtime#132001
1543532	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55660/merge to s	Log	dotnet/sdk#55660
1542922	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55638/merge to s	Log	dotnet/sdk#55638
1542869	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131876/merge to s	Log	dotnet/runtime#131876
1542524	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131788/merge to s	Log	dotnet/runtime#131788
1541489	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55285/merge to s	Log	dotnet/sdk#55285
1541420	dotnet/maui	Checkout dotnet/maui@refs/pull/37119/merge to s	Log	dotnet/maui#37119
1540351	dotnet/maui	Checkout dotnet/maui@darc-net11.0-94daab64-3d05-4315-92d8-198252a8f3d2 to s	Log
1539964	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55595/merge to s	Log	dotnet/sdk#55595
1539833	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131827/merge to s	Log	dotnet/runtime#131827
1539464	dotnet/maui	Checkout dotnet/maui@refs/pull/37087/merge to s	Log	dotnet/maui#37087
1539427	dotnet/arcade	Checkout dotnet/arcade@refs/pull/17241/merge to s	Log	#17241
1539345	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131786/merge to s	Log	dotnet/runtime#131786
1538085	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131768/merge to s	Log	dotnet/runtime#131768
1537538	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131750/merge to s	Log	dotnet/runtime#131750
1536977	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55563/merge to s	Log	dotnet/sdk#55563
1536577	dotnet/diagnostics	Checkout dotnet/diagnostics@refs/pull/5937/merge to s	Log	dotnet/diagnostics#5937
Displaying 100 of 139 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
8	23	139

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?