Skip to content

Commit cb4e6b8

Browse files
PauloLory-ionosjannotti-glaucioGlaucio Jannotti
authored
Migration to EDC 0.7.2 (#7)
* Migrating to EDC 0.7.2 (Digital-Ecosystems#77) * feat: migrating to edc 0.7.3 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * Fixing connector-persistente to EDC 0.7.2 (Digital-Ecosystems#78) * feat: migrating to edc 0.7.3 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * feat: migrating to edc 0.7.2 * S3 Endpoint Regions Validation (Digital-Ecosystems#79) * feat: s3 endpoint regions validation * chore: fixing assets doc * fix: launchers config * fix: regions endpoint signature changes (Digital-Ecosystems#81) Co-authored-by: Glaucio Jannotti <glaucio.jannotti@dengun.com> --------- Co-authored-by: Glaucio Jannotti <111659831+jannotti-glaucio@users.noreply.github.com> Co-authored-by: Glaucio Jannotti <glaucio.jannotti@dengun.com>
1 parent 0366e9c commit cb4e6b8

87 files changed

Lines changed: 970 additions & 1507 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ launchers/demo-e2e/edc-config.properties
5151
*.hprof
5252

5353
**/vault-keys.json
54+
**/vault-tokens.json
5455
runtime_settings.properties
5556
generated_backend.tf
5657

‎README.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -59,15 +59,15 @@ The credentials can be found/configured in one of the following:
5959

6060
It is required to configure those parameters:
6161

62-
| Parameter name | Description | Mandatory |
63-
|-------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------|
64-
| `edc.ionos.access.key` | IONOS Access Key Id to access S3 | Yes if the context is accessing file |
65-
| `edc.ionos.secret.access.key` | IONOS Secret Access Key to access S3 | Yes if the context is accessing file |
66-
| `edc.ionos.token` | IONOS token to allow S3 provisioning | Yes if the context is provisioning access for others |
67-
| `edc.ionos.endpoint.region` | IONOS S3 endpoint region. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/endpoints) for further information. | No, the default value is de |
68-
| `edc.ionos.max.files` | Maximum number of files retrieved by list files function. | No, the default value is 5,000 files |
69-
| `edc.ionos.key.validation.attempts` | Maximum number of attemps to validate a temporary key after its creation. | No, the default values is 10 attempts |
70-
| `edc.ionos.key.validation.delay` | Time to wait (in milisseconds) before each key validation attempt. In each new attempt the delay is multiplied by the attempt number. | No, the default value is 3,000 (3 seconds) |
62+
| Parameter name | Description | Mandatory |
63+
|-------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------|
64+
| `edc.ionos.access.key` | IONOS Access Key Id to access S3 | Yes if the context is accessing file |
65+
| `edc.ionos.secret.access.key` | IONOS Secret Access Key to access S3 | Yes if the context is accessing file |
66+
| `edc.ionos.token` | IONOS token to allow S3 provisioning | Yes if the context is provisioning access for others |
67+
| `edc.ionos.endpoint.region` | IONOS S3 endpoint region. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/endpoints) for further information. | No, the default value is "de" |
68+
| `edc.ionos.max.files` | Maximum number of files retrieved by list files function. | No, the default value is 5,000 files |
69+
| `edc.ionos.key.validation.attempts` | Maximum number of attemps to validate a temporary key after its creation. | No, the default values is 10 attempts |
70+
| `edc.ionos.key.validation.delay` | Time to wait (in milisseconds) before each key validation attempt. In each new attempt the delay is multiplied by the attempt number. | No, the default value is 3,000 (3 seconds) |
7171

7272
To create the token please take a look at the following [documentation](./ionos_token.md).
7373

‎assets.md‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -11,21 +11,21 @@ The asset registration aims to specify which file/folder we want to share. We ca
1111

1212
| Parameter | Description | Mandatory |
1313
|------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|
14-
| `region` | IONOS S3 endpoint region. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/s3-endpoints) for further information. | no, default value = de |
15-
| `bucketName` | IONOS S3 bucket name. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/concepts/buckets) for further information. | yes |
16-
| `blobName` | File name or path to folder | yes |
17-
| `filterIncludes` | `filterIncludes` use regular expression that will be used to select the file name pattern from the asset's blobName that will be copied during the transfer <br/> * do not consider the blobName in the expression, but the path from it. example: blobName = folder1, filterIncludes=file1.csv, the file foloder1/file1.csv will be copied | no |
18-
| `filterExcludes` | `filterExcludes` use regular expression that will be used to select the file name pattern from the asset's blobName that will NOT be copied during the transfer <br/> | no |
14+
| `region` | IONOS S3 endpoint region. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/s3-endpoints) for further information.| no, default value = de |
15+
| `bucketName` | IONOS S3 bucket name. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/concepts/buckets) for further information.| yes |
16+
| `blobName` | File name or path to folder| yes |
17+
| `filterIncludes` | `filterIncludes` use regular expression that will be used to select the file name pattern from the asset's blobName that will be copied during the transfer <br/> * do not consider the blobName in the expression, but the path from it. example: blobName = folder1, filterIncludes=file1.csv, the file foloder1/file1.csv will be copied| no |
18+
| `filterExcludes` | `filterExcludes` use regular expression that will be used to select the file name pattern from the asset's blobName that will NOT be copied during the transfer <br/>| no |
1919

2020
Note: if `filterIncludes` and `filterExcludes` parameters are satisfied, the files to be copied will be selected using the `filterIncludes` and after that selected list, the files that have the pattern defined in the `filterExcludes` will be ignored.
2121

2222

2323
## Example
2424

2525
```json
26-
"dataAddress": {
26+
"dataAddress":{
2727
"type": "IonosS3", //from EDC
28-
"region": "de,
28+
"region": "de",
2929
"bucketName": "mybucket",
3030
"blobName": "folder1/",
3131
"filterIncludes": "file1.csv",
@@ -40,17 +40,17 @@ The transfer of assets aims to transfer the files/folders from one connector to
4040
### Requirements
4141

4242

43-
| Parameter | Description | Mandatory |
44-
|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|
45-
| `region` | IONOS S3 endpoint region. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/s3-endpoints) for further information. | no, default value = de |
46-
| `bucketName` | IONOS S3 bucket name. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/concepts/buckets) for further information. | yes |
47-
| `path` | Path of destination where the file/folder will be placed. </br> *if the path not filled, the file will be placed in the root of the bucket. | no |
43+
| Parameter | Description | Mandatory |
44+
|---------------|----------------------------------------------------------------------------------------------------------------------------------------------|------------------------|
45+
| `region` | IONOS S3 endpoint region. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/s3-endpoints) for further information.| no, default value = de |
46+
| `bucketName` | IONOS S3 bucket name. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/concepts/buckets) for further information| yes |
47+
| `path` | Path of destination where the file/folder will be placed. </br> *if the path not filled, the file will be placed in the root of the bucket. | no |
4848

4949

5050
## Example
5151

5252
```json
53-
"dataDestination": {
53+
"dataDestination":{
5454
"type": "IonosS3", //from EDC
5555
"region": "de",
5656
"bucketName": "mybucket",

‎deployment/README.md‎

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -86,13 +86,13 @@ In case you want to configure this Connector without Hashicorp Vault, you need t
8686

8787
```yaml
8888
ionos:
89-
region: <IONOS-S3-ENDPOINT-REGION>
89+
region: <YOUR-S3-ENDPOINT-REGION>
9090
accessKey: <YOUR-KEY>
9191
secretKey: <YOUR-SECRET-KEY>
9292
token: <IONOS-TOKEN>
9393
```
9494
95-
They should be the same as the ones set in the environment variables. The **ionos.endpoint** is set to the default S3 location, but it can be changed to any other location.
95+
They should be the same as the ones set in the environment variables. The **ionos.region** is set to the default S3 endpoint region, but it can be changed to any other location.
9696
9797
9898
If you don't want the Connector to be externally accessible, you need to set the following parameters in the helm [values.yaml](deployment/helm/edc-ionos-s3/values.yaml):
@@ -110,21 +110,19 @@ This will allocate a public IP address to the Connector. You can then access it
110110
111111
All commands paths are relative to the current directory where this readme is located.
112112
113-
### 1. Install the EDC Ionos S3 services
113+
### 1. Deploy the services
114114
115-
To install the services run the script ```deploy-services.sh``` in ```terraform``` directory.
115+
To deploy the services run the script ```deploy-services.sh``` in ```terraform``` directory.
116116

117117
```sh
118118
cd terraform
119119
./deploy-services.sh
120120
```
121+
### 2. Undeploy the services
121122

122-
### 2. Vault keys
123-
After the services are installed you will have ```vault-keys.json``` file containing the vault keys in ```terraform``` directory.
124-
125-
### 3. Destroy the services
123+
To undeploy the services run the script ```undeploy-services.sh``` in ```terraform``` directory.
126124

127125
```sh
128126
cd terraform
129-
./destroy-services.sh
127+
./undeploy-services.sh
130128
```

‎deployment/helm/edc-ionos-s3/templates/configmap.yaml‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ metadata:
44
name: {{ include "edc-ionos-s3.fullname" . }}-config
55
data:
66
config.properties: |
7+
edc.participant.id={{ .Values.edc.participant.id }}
78
web.http.port={{ .Values.web.http.port }}
89
web.http.path={{ .Values.web.http.path }}
910
web.http.management.port={{ .Values.web.http.management.port }}
@@ -14,20 +15,19 @@ data:
1415
web.http.public.path={{ .Values.web.http.public.path }}
1516
web.http.control.port={{ .Values.web.http.control.port }}
1617
web.http.control.path={{ .Values.web.http.control.path }}
18+
edc.dsp.callback.address={{ .Values.edc.dsp.callback.address }}
19+
edc.dataplane.token.validation.endpoint={{ .Values.edc.dataplane.token.validation.endpoint }}
20+
edc.dataplane.api.public.baseurl={{ .Values.edc.dataplane.api.public.baseurl }}
1721
edc.api.auth.key={{ .Values.edc.api.auth.key }}
18-
edc.participant.id={{ .Values.edc.participant.id }}
19-
edc.ionos.access.key={{ .Values.edc.ionos.accessKey }}
20-
edc.ionos.secret.key={{ .Values.edc.ionos.secretKey }}
21-
edc.ionos.endpoint.region={{ .Values.edc.ionos.endpoint.region }}
22-
edc.ionos.token={{ .Values.edc.ionos.token }}
22+
edc.transfer.proxy.token.signer.privatekey.alias={{ .Values.edc.vault.certificates.privateKey.alias }}
23+
edc.transfer.proxy.token.verifier.publickey.alias={{ .Values.edc.vault.certificates.publicKey.alias }}
2324
edc.vault.hashicorp.url={{ .Values.edc.vault.hashicorp.url }}
2425
edc.vault.hashicorp.token={{ .Values.edc.vault.hashicorp.token }}
2526
edc.vault.hashicorp.timeout.seconds={{ .Values.edc.vault.hashicorp.timeout.seconds }}
26-
edc.ids.id={{ .Values.edc.ids.id }}
27-
edc.dsp.callback.address={{ .Values.edc.dsp.callback.address }}:{{ .Values.web.http.protocol.port }}{{ .Values.web.http.protocol.path }}
28-
edc.receiver.http.endpoint={{ .Values.edc.receiver.http.endpoint }}/receiver/{{ .Values.edc.ids.id }}/callback
29-
edc.public.key.alias={{ .Values.edc.public.key.alias }}
30-
edc.dataplane.token.validation.endpoint={{ .Values.edc.dataplane.token.validation.endpoint }}:{{ .Values.web.http.control.port }}{{ .Values.web.http.control.path }}/token
27+
edc.ionos.access.key={{ .Values.edc.ionos.accessKey }}
28+
edc.ionos.secret.key={{ .Values.edc.ionos.secretKey }}
29+
edc.ionos.endpoint.region={{ .Values.edc.ionos.region }}
30+
edc.ionos.token={{ .Values.edc.ionos.token }}
3131
3232
{{- if eq .Values.edc.persistenceType "PostgreSQLaaS" }}
3333
edc.datasource.asset.name=asset

‎deployment/helm/edc-ionos-s3/values.yaml‎

Lines changed: 11 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -113,34 +113,31 @@ edc:
113113
auth:
114114
key: password
115115
vault:
116-
clientid: company1
117-
tenantid: 1
118-
certificate: /resources/
116+
certificates:
117+
publicKey:
118+
alias: edc.connector.public.key
119+
privateKey:
120+
alias: edc.connector.private.key
119121
hashicorp:
120122
url: http://vault:8200
121123
token:
122124
timeout:
123125
seconds: 30
124-
ids:
125-
id: urn:connector:provider
126126
ionos:
127-
endpoint: s3-eu-central-1.ionoscloud.com
127+
region: de
128128
accessKey: notnull
129129
secretKey: notnull
130130
token: notnull
131131
dsp:
132132
callback:
133-
address: http://localhost
134-
receiver:
135-
http:
136-
endpoint: http://localhost:4000
137-
public:
138-
key:
139-
alias: alias
133+
address: http://localhost:8281/protocol
140134
dataplane:
135+
api:
136+
public:
137+
baseurl: http://localhost:8282/public
141138
token:
142139
validation:
143-
endpoint: http://localhost
140+
endpoint: http://localhost:8283/control/token
144141
persistenceType: PostgreSQLaaS # 'PostgreSQLaaS', 'PostgreSQL' or 'None'
145142
postgresql: # Only used if persistenceType is 'PostgreSQLaaS' or 'PostgreSQL'
146143
host: postgresql

‎deployment/terraform/clean-state.sh‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
#!/bin/bash
2+
3+
# remove terraform state
4+
rm -rf ./configure-public-address/.terraform
5+
rm -f ./configure-public-address/terraform.tfstate
6+
rm -f ./configure-public-address/.terraform.lock.hcl
7+
rm -f ./configure-public-address/terraform.tfstate.backup
8+
9+
rm -rf ./ionos-s3-deploy/.terraform
10+
rm -f ./ionos-s3-deploy/terraform.tfstate
11+
rm -f ./ionos-s3-deploy/.terraform.lock.hcl
12+
rm -f ./ionos-s3-deploy/terraform.tfstate.backup
13+
14+
rm -rf ./vault-init/.terraform
15+
rm -f ./vault-init/terraform.tfstate
16+
rm -f ./vault-init/.terraform.lock.hcl
17+
rm -f ./vault-init/terraform.tfstate.backup
18+
19+
rm -rf ./vault-deploy/.terraform
20+
rm -f ./vault-deploy/terraform.tfstate
21+
rm -f ./vault-deploy/.terraform.lock.hcl
22+
rm -f ./vault-deploy/terraform.tfstate.backup
23+
24+
rm -rf ./ionos-postgresqlaas/.terraform
25+
rm -f ./ionos-postgresqlaas/terraform.tfstate
26+
rm -f ./ionos-postgresqlaas/.terraform.lock.hcl
27+
rm -f ./ionos-postgresqlaas/terraform.tfstate.backup
28+
29+
rm -rf ./postgresql-deploy/.terraform
30+
rm -f ./postgresql-deploy/terraform.tfstate
31+
rm -f ./postgresql-deploy/.terraform.lock.hcl
32+
rm -f ./postgresql-deploy/terraform.tfstate.backup
33+
34+
rm -rf ./db-scripts/.terraform
35+
rm -f ./db-scripts/terraform.tfstate
36+
rm -f ./db-scripts/.terraform.lock.hcl
37+
rm -f ./db-scripts/terraform.tfstate.backup
38+
39+
rm -f vault-init/vault-keys.json
40+
rm -f vault-init/vault-tokens.json
41+
42+
echo "Terraform state cleanup complete"

‎deployment/terraform/configure-public-address/public-addresses.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,9 @@ fi
1212
# Change public address in the config.properties in the configmap
1313
kubectl --kubeconfig=$TF_VAR_kubeconfig -n $TF_VAR_namespace get configmap edc-ionos-s3-config -o yaml | sed "s/edc.dsp.callback.address=.*/edc.dsp.callback.address=http:\/\/$CONNECTOR_ADDRESS:8281\/protocol/g" | kubectl --kubeconfig=$TF_VAR_kubeconfig apply -f -
1414

15-
kubectl --kubeconfig=$TF_VAR_kubeconfig -n $TF_VAR_namespace get configmap edc-ionos-s3-config -o yaml | sed "s/edc.receiver.http.endpoint=.*/edc.receiver.http.endpoint=http:\/\/$CONNECTOR_ADDRESS:4000\/receiver\/urn:connector:provider\/callback/g" | kubectl --kubeconfig=$TF_VAR_kubeconfig apply -f -
16-
1715
kubectl --kubeconfig=$TF_VAR_kubeconfig -n $TF_VAR_namespace get configmap edc-ionos-s3-config -o yaml | sed "s/edc.dataplane.token.validation.endpoint=.*/edc.dataplane.token.validation.endpoint=http:\/\/$CONNECTOR_ADDRESS:8283\/control\/token/g" | kubectl --kubeconfig=$TF_VAR_kubeconfig apply -f -
1816

17+
kubectl --kubeconfig=$TF_VAR_kubeconfig -n $TF_VAR_namespace get configmap edc-ionos-s3-config -o yaml | sed "s/edc.dataplane.api.public.baseurl=.*/edc.dataplane.api.public.baseurl=http:\/\/$CONNECTOR_ADDRESS:8282\/public/g" | kubectl --kubeconfig=$TF_VAR_kubeconfig apply -f -
18+
1919
# Restart the pods
2020
kubectl --kubeconfig=$TF_VAR_kubeconfig -n $TF_VAR_namespace delete pod -l app.kubernetes.io/name=edc-ionos-s3

0 commit comments

Comments
 (0)