Skip to content

Commit 604aeec

Browse files
tothszabitothszabi
authored
[CI-5827] Tweak download settings (#26)
* Tweak download settings * Update readme * Update e2e claim * Update env * Update e2e settings * Check for type
1 parent bc4de7e commit 604aeec

4 files changed

Lines changed: 265 additions & 118 deletions

File tree

‎README.md‎

Lines changed: 89 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -38,30 +38,103 @@ To configure the Step:
3838

3939
## 🧩 Get started
4040

41-
Add this step directly to your workflow in the [Bitrise Workflow Editor](https://devcenter.bitrise.io/steps-and-workflows/steps-and-workflows-index/).
41+
Add this step directly to your workflow in the [Bitrise Workflow Editor](https://docs.bitrise.io/en/bitrise-ci/workflows-and-pipelines/steps/adding-steps-to-a-workflow.html).
4242

4343
You can also run this step directly with [Bitrise CLI](https://github.com/bitrise-io/bitrise).
4444

4545
#### Examples
4646

47-
##### Basic step config
47+
##### Graph pipeline
48+
49+
###### Basic step config
50+
51+
```yaml
52+
steps:
53+
- pull-intermediate-files@1:
54+
inputs:
55+
- verbose: "true"
56+
- artifact_sources: build
57+
```
58+
59+
Use the `artifact_sources` input variable to limit the downloads to a set of workflows. Simply specify the Workflow name directly — no stage prefix is needed:
60+
61+
- `build` - Gets intermediate files from the 'build' Workflow.
62+
- `build,test` - Gets intermediate files from both the 'build' and 'test' Workflows.
63+
- `test.*` - Gets every intermediate file from all previous Workflows with names starting with 'test'.
64+
- `.*` - Gets every intermediate file from all previous Workflows.
65+
66+
###### Wildcard based artifact pull
67+
68+
During a pipeline, workflows receive the finished workflows object. Developers can find it on a build VM's environment variable: `BITRISEIO_FINISHED_WORKFLOWS`.
69+
70+
Let's suppose that we get the following JSON object about the previously finished workflows.
71+
72+
```json
73+
[
74+
{
75+
"external_id": "73d33fb5-35c6-495f-bd80-015ae681db33",
76+
"finished_at": "2021-12-07T14:04:45Z",
77+
"id": "b1c6f0a1-06e7-4f63-a172-ac541a467d71",
78+
"name": "build",
79+
"started_at": "2021-12-07T14:04:27Z",
80+
"status": "succeeded"
81+
},
82+
{
83+
"external_id": "39404bee-52ba-4ca2-8508-91489e7f6afa",
84+
"finished_at": "2021-12-07T14:05:07Z",
85+
"id": "f3bda7bb-37be-409f-9291-b377717cba60",
86+
"name": "test",
87+
"started_at": "2021-12-07T14:04:48Z",
88+
"status": "succeeded"
89+
},
90+
{
91+
"external_id": "ed0da0cf-66cc-4109-b23f-8a156d61b0c3",
92+
"finished_at": "2021-12-07T14:06:41Z",
93+
"id": "f572ca4e-2f06-40f1-a4cf-c208af15ff28",
94+
"name": "deploy",
95+
"started_at": "2021-12-07T14:06:13Z",
96+
"status": "succeeded"
97+
}
98+
]
99+
```
100+
101+
As the key names in the object are self-describing, we will not cover those names except the `external_id`. The `external_id` is the build's slug in the PipelineService context.
102+
103+
Let's see the following use-cases, the use cases first part is the demand, the second is the `artifact_sources` config:
104+
105+
- As a developer, I would like to get the build artifact(s) of the _build_ workflow: `build`.
106+
107+
- As a developer, I would like to get the build artifact(s) of both the _build_ and _test_ workflows: `build,test`. The two expressions are separated by a comma.
108+
109+
- As a developer, I would like to retrieve already generated artifacts from all previous workflows: `.*`. As the example shows, developers can use regex.
110+
111+
- As a developer, I would like to get artifacts from all workflows whose names start with _test_: `test.*`.
112+
113+
And so on. The syntax is: `{workflow-name}`.
114+
Do not forget to escape the special characters when using a regex pattern.
115+
116+
---
117+
118+
##### Staged pipeline
119+
120+
###### Basic step config
48121

49122
```yaml
50123
steps:
51124
- pull-intermediate-files@1:
52125
inputs:
53126
- verbose: "true"
54-
- artifact_sources: workflow1
127+
- artifact_sources: stage-1\..*
55128
```
56129

57130
Use the `artifact_sources` input variable to limit the downloads to a set of stages or workflows:
58131

59-
- `build` - Gets files from the 'build' Workflow.
60-
- `build,test` - Gets files from both the 'build' and 'test' Workflows.
61-
- `test.*` - Gets every intermediate files from all previous Workflows with names starting with 'test'.
62-
- `.*` - Gets every intermediate files from all previous Workflows.
132+
- `stage1.workflow1` - Gets the artifacts from the stage1's workflow1.
133+
- `stage1\..*` - Gets all artifacts from the stage1's workflows.
134+
- `.*\.workflow1` - Gets workflow1s' artifacts from the previous stages.
135+
- `.*` - Gets every generated artifacts from the previous stages.
63136

64-
##### Wildcard based intermediate file pull
137+
###### Wildcard based artifact pull
65138

66139
During a pipeline, workflows receive the finished stages and workflows object. Developers can find it on a build VM's environment variable: `BITRISEIO_FINISHED_STAGES`.
67140

@@ -120,15 +193,15 @@ As the key names in the object are self-describing, we will not cover those name
120193

121194
Let's see the following use-cases, the use cases first part is the demand, the second is the `artifact_sources` config:
122195

123-
- As a developer, I would like to get the intermediate files of the _stage-1_'s _placeholder_'s workflow: `stage-1.placeholder`.
196+
- As a developer, I would like to get the build artifact(s) of the _stage-1_'s _placeholder_'s workflow: `stage-1.placeholder`.
124197

125-
- As a developer, I would like to get the intermediate files of the _stage-2_'s _deployer_'s workflow and the _stage-1_'s _placeholder_'s workflow: `stage-1.placeholder,stage-2.deployer`. The two expressions are separated by a comma.
198+
- As a developer, I would like to get the build artifact(s) of the _stage-2_'s _deployer_'s workflow and the _stage-1_'s _placeholder_'s workflow: `stage-1.placeholder,stage-2.deployer`. The two expressions are separated by a comma.
126199

127-
- As a developer, I would like to retrieve all intermediate files: `.*`. As the example shows, developers can use regex.
200+
- As a developer, I would like to retrieve already generated artifacts: `.*`. As the example shows, developers can use regex.
128201

129-
- As a developer, I would like to retrieve the intermediate files from the _stage-2_ stage: `stage-2\..*`.
202+
- As a developer, I would like to retrieve the generated artifacts from the _stage-2_ stage: `stage-2\..*`.
130203

131-
- As a developer, I would like to get the _textfile_generator_ workflow intermediate files: `.*\.textfile_generator`
204+
- As a developer, I would like to get the _textfile_generator_ workflow artifacts: `.*\.textfile_generator`
132205

133206
And so on. The syntax is: `{stage-name}.{workflow-name}`.
134207
Do not forget to escape the special characters when using a regex pattern.
@@ -141,7 +214,7 @@ Do not forget to escape the special characters when using a regex pattern.
141214

142215
| Key | Description | Flags | Default |
143216
| --- | --- | --- | --- |
144-
| `artifact_sources` | A comma (`,`) separated list of Workflow names or Stage and Workflow paths, used to specify which workflows' intermediate files to download. **Graph Pipelines:** Simply specify the workflow name directly. For example, `build` or `test`. Examples: - `build` - Gets intermediate files from the 'build' workflow. - `build,test` - Gets intermediate files from both the 'build' and 'test' workflows. - `test.*` - Gets every intermediate files from all previous workflows with names starting with 'test'. - `.*` - Gets every intermediate files from all previous workflows. **Staged Pipelines:** The input uses a `{stage}.{workflow}` syntax. The dot character (`.`) is the delimiter between the Stage and the Workflow. Examples: - `stage1.workflow1` - Gets intermediate files from stage1's workflow1. - `stage1\..*` - Gets all intermediate files from stage1's workflows. - `.*\.workflow1` - Gets workflow1's intermediate files from all previous stages. - `.*` - Gets every intermediate files from all previous stages. You can use regular expressions for both pipeline types. Remember to escape special characters. | required | `.*` |
217+
| `artifact_sources` | A comma (`,`) separated list of Workflow names or Stage and Workflow paths, used to specify which Workflows' files to download. **Graph Pipelines:** Simply specify the Workflow name directly. For example, `build` or `test`. Examples: - `build` - Gets intermediate files from the 'build' Workflow. - `build,test` - Gets intermediate files from both the 'build' and 'test' Workflows. - `test.*` - Gets every intermediate files from all previous Workflows with names starting with 'test'. - `.*` - Gets every intermediate files from all previous Workflows. **Staged Pipelines:** The input uses a `{stage}.{workflow}` syntax. The dot character (`.`) is the delimiter between the Stage and the Workflow. Examples: - `stage1.workflow1` - Gets intermediate files from stage1's workflow1. - `stage1\..*` - Gets all intermediate files from stage1's workflows. - `.*\.workflow1` - Gets workflow1's intermediate files from all previous stages. - `.*` - Gets every intermediate files from all previous stages. You can use regular expressions for both pipeline types. Remember to escape special characters. | required | `.*` |
145218
| `verbose` | Enable logging additional information for debugging | required | `false` |
146219
| `app_slug` | The slug that uniquely identifies your app on bitrise.io. It’s part of the app URL, too. | required | `$BITRISE_APP_SLUG` |
147220
| `finished_stage` | This is a JSON representation of the finished staged pipeline stages for which the step can download intermediate files. | required | `$BITRISEIO_FINISHED_STAGES` |
@@ -159,11 +232,10 @@ There are no outputs defined in this step
159232

160233
We welcome [pull requests](https://github.com/bitrise-steplib/bitrise-step-pull-intermediate-files/pulls) and [issues](https://github.com/bitrise-steplib/bitrise-step-pull-intermediate-files/issues) against this repository.
161234

162-
For pull requests, work on your changes in a forked repository and use the Bitrise CLI to [run step tests locally](https://devcenter.bitrise.io/bitrise-cli/run-your-first-build/).
235+
For pull requests, work on your changes in a forked repository and use the Bitrise CLI to [run step tests locally](https://docs.bitrise.io/en/bitrise-ci/bitrise-cli/running-your-first-local-build-with-the-cli.html).
163236

164237
Note: this step's end-to-end tests (defined in e2e/bitrise.yml) are working with secrets which are intentionally not stored in this repo. External contributors won't be able to run those tests. Don't worry, if you open a PR with your contribution, we will help with running tests and make sure that they pass.
165238

166239
Learn more about developing steps:
167240

168-
- [Create your own step](https://devcenter.bitrise.io/contributors/create-your-own-step/)
169-
- [Testing your Step](https://devcenter.bitrise.io/contributors/testing-and-versioning-your-steps/)
241+
- [Create your own step](https://docs.bitrise.io/en/bitrise-ci/workflows-and-pipelines/developing-your-own-bitrise-step/developing-a-new-step.html)

‎docs/examples.md‎

Lines changed: 75 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,79 @@
11
#### Examples
22

3-
##### Basic step config
3+
##### Graph pipeline
4+
5+
###### Basic step config
6+
7+
```yaml
8+
steps:
9+
- pull-intermediate-files@1:
10+
inputs:
11+
- verbose: "true"
12+
- artifact_sources: build
13+
```
14+
15+
Use the `artifact_sources` input variable to limit the downloads to a set of workflows. Simply specify the Workflow name directly — no stage prefix is needed:
16+
17+
- `build` - Gets intermediate files from the 'build' Workflow.
18+
- `build,test` - Gets intermediate files from both the 'build' and 'test' Workflows.
19+
- `test.*` - Gets every intermediate file from all previous Workflows with names starting with 'test'.
20+
- `.*` - Gets every intermediate file from all previous Workflows.
21+
22+
###### Wildcard based artifact pull
23+
24+
During a pipeline, workflows receive the finished workflows object. Developers can find it on a build VM's environment variable: `BITRISEIO_FINISHED_WORKFLOWS`.
25+
26+
Let's suppose that we get the following JSON object about the previously finished workflows.
27+
28+
```json
29+
[
30+
{
31+
"external_id": "73d33fb5-35c6-495f-bd80-015ae681db33",
32+
"finished_at": "2021-12-07T14:04:45Z",
33+
"id": "b1c6f0a1-06e7-4f63-a172-ac541a467d71",
34+
"name": "build",
35+
"started_at": "2021-12-07T14:04:27Z",
36+
"status": "succeeded"
37+
},
38+
{
39+
"external_id": "39404bee-52ba-4ca2-8508-91489e7f6afa",
40+
"finished_at": "2021-12-07T14:05:07Z",
41+
"id": "f3bda7bb-37be-409f-9291-b377717cba60",
42+
"name": "test",
43+
"started_at": "2021-12-07T14:04:48Z",
44+
"status": "succeeded"
45+
},
46+
{
47+
"external_id": "ed0da0cf-66cc-4109-b23f-8a156d61b0c3",
48+
"finished_at": "2021-12-07T14:06:41Z",
49+
"id": "f572ca4e-2f06-40f1-a4cf-c208af15ff28",
50+
"name": "deploy",
51+
"started_at": "2021-12-07T14:06:13Z",
52+
"status": "succeeded"
53+
}
54+
]
55+
```
56+
57+
As the key names in the object are self-describing, we will not cover those names except the `external_id`. The `external_id` is the build's slug in the PipelineService context.
58+
59+
Let's see the following use-cases, the use cases first part is the demand, the second is the `artifact_sources` config:
60+
61+
- As a developer, I would like to get the build artifact(s) of the _build_ workflow: `build`.
62+
63+
- As a developer, I would like to get the build artifact(s) of both the _build_ and _test_ workflows: `build,test`. The two expressions are separated by a comma.
64+
65+
- As a developer, I would like to retrieve already generated artifacts from all previous workflows: `.*`. As the example shows, developers can use regex.
66+
67+
- As a developer, I would like to get artifacts from all workflows whose names start with _test_: `test.*`.
68+
69+
And so on. The syntax is: `{workflow-name}`.
70+
Do not forget to escape the special characters when using a regex pattern.
71+
72+
---
73+
74+
##### Staged pipeline
75+
76+
###### Basic step config
477

578
```yaml
679
steps:
@@ -17,7 +90,7 @@ Use the `artifact_sources` input variable to limit the downloads to a set of sta
1790
- `.*\.workflow1` - Gets workflow1s' artifacts from the previous stages.
1891
- `.*` - Gets every generated artifacts from the previous stages.
1992

20-
##### Wildcard based artifact pull
93+
###### Wildcard based artifact pull
2194

2295
During a pipeline, workflows receive the finished stages and workflows object. Developers can find it on a build VM's environment variable: `BITRISEIO_FINISHED_STAGES`.
2396

‎downloader/artifact_downloader.go‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -138,7 +138,6 @@ func (ad *ConcurrentArtifactDownloader) downloadFile(targetDir, fileName, downlo
138138
start := time.Now()
139139

140140
err := downloadWithRetry(ctx, ad.createClient(), downloadURL, fileFullPath, ad.Logger)
141-
142141
if err != nil {
143142
// fallback to single threaded download - the error with the 416 status code seems to happen for 0 size files with got
144143
errorMessage := err.Error()
@@ -316,7 +315,7 @@ func fileSize(path string) int64 {
316315
}
317316

318317
func downloadWithRetry(ctx context.Context, httpClient *retryablehttp.Client, url, dest string, logger log.Logger) error {
319-
return retry.Times(3).Wait(5 * time.Second).TryWithAbort(func(attempt uint) (error, bool) {
318+
return retry.Times(5).Wait(5 * time.Second).TryWithAbort(func(attempt uint) (error, bool) {
320319
if attempt != 0 {
321320
logger.Debugf("Retrying intermediate file download... (attempt %d)", attempt+1)
322321
}
@@ -333,12 +332,15 @@ func downloadWithRetry(ctx context.Context, httpClient *retryablehttp.Client, ur
333332
}
334333

335334
func download(ctx context.Context, httpClient *retryablehttp.Client, url string, dest string, logger log.Logger) error {
336-
httpClient.HTTPClient.Transport.(*http.Transport).ForceAttemptHTTP2 = false
337-
httpClient.HTTPClient.Transport.(*http.Transport).DialContext = (&net.Dialer{
338-
Timeout: 30 * time.Second,
339-
KeepAlive: 30 * time.Second,
340-
DualStack: false,
341-
}).DialContext
335+
if t, ok := httpClient.HTTPClient.Transport.(*http.Transport); ok {
336+
t.ForceAttemptHTTP2 = false
337+
t.DialContext = (&net.Dialer{
338+
Timeout: 30 * time.Second,
339+
KeepAlive: 30 * time.Second,
340+
DualStack: false,
341+
}).DialContext
342+
t.ResponseHeaderTimeout = 30 * time.Second
343+
}
342344

343345
downloader := got.New()
344346
downloader.Client = httpClient.StandardClient()

0 commit comments

Comments
 (0)