Skip to content

Commit 9a3cfc7

Browse files
trigganRadulescu Kobler
triggan
and
Radulescu Kobler
authored
the very frist try (#202)
Co-authored-by: Radulescu Kobler <kobler@38f9d3ea0c9c.ant.amazon.com>
1 parent 32c5e2f commit 9a3cfc7

26 files changed

Lines changed: 9440 additions & 0 deletions

‎neptune-REST-IAM/.gitignore‎

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
# compiled output
2+
**/dist
3+
**/global-s3-assets
4+
**/regional-s3-assets
5+
**/open-source
6+
**/.zip
7+
**/tmp
8+
**/out-tsc
9+
**/*.d.ts
10+
/lib/*.js
11+
12+
# dependencies
13+
**/node_modules
14+
15+
# e2e
16+
**/e2e/*.js
17+
**/e2e/*.map
18+
19+
# misc
20+
**/npm-debug.log
21+
**/testem.log
22+
**/package-lock.json
23+
**/yarn-lock.json
24+
**/.vscode/settings.json
25+
26+
# System Files
27+
**/.DS_Store
28+
**/.vscode
29+
30+
31+
32+
# Byte-compiled / optimized / DLL files
33+
__pycache__/
34+
*.py[cod]
35+
*$py.class
36+
37+
# C extensions
38+
*.so
39+
40+
*.avi
41+
42+
*no_commit/
43+
44+
# Distribution / packaging
45+
.Python
46+
build/
47+
develop-eggs/
48+
dist/
49+
downloads/
50+
eggs/
51+
.eggs/
52+
lib64/
53+
parts/
54+
sdist/
55+
var/
56+
wheels/
57+
pip-wheel-metadata/
58+
share/python-wheels/
59+
*.egg-info/
60+
.installed.cfg
61+
*.egg
62+
MANIFEST
63+
64+
# PyInstaller
65+
# Usually these files are written by a python script from a template
66+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
67+
*.manifest
68+
*.spec
69+
70+
# Installer logs
71+
pip-log.txt
72+
pip-delete-this-directory.txt
73+
74+
# Unit test / coverage reports
75+
htmlcov/
76+
.tox/
77+
.nox/
78+
.coverage
79+
.coverage.*
80+
.cache
81+
nosetests.xml
82+
coverage.xml
83+
*.cover
84+
*.py,cover
85+
.hypothesis/
86+
.pytest_cache/
87+
88+
# Translations
89+
*.mo
90+
*.pot
91+
92+
# Django stuff:
93+
*.log
94+
local_settings.py
95+
db.sqlite3
96+
db.sqlite3-journal
97+
98+
# Flask stuff:
99+
instance/
100+
.webassets-cache
101+
102+
# Scrapy stuff:
103+
.scrapy
104+
105+
# Sphinx documentation
106+
docs/_build/
107+
108+
# PyBuilder
109+
target/
110+
111+
# Jupyter Notebook
112+
.ipynb_checkpoints
113+
114+
# IPython
115+
profile_default/
116+
ipython_config.py
117+
118+
# pyenv
119+
.python-version
120+
121+
# pipenv
122+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
123+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
124+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
125+
# install all needed dependencies.
126+
#Pipfile.lock
127+
128+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
129+
__pypackages__/
130+
131+
# Celery stuff
132+
celerybeat-schedule
133+
celerybeat.pid
134+
135+
# SageMath parsed files
136+
*.sage.py
137+
138+
# Environments
139+
.env
140+
.venv
141+
.env/
142+
.venv/
143+
env/
144+
venv/
145+
ENV/
146+
env.bak/
147+
venv.bak/
148+
149+
# Spyder project settings
150+
.spyderproject
151+
.spyproject
152+
153+
# Rope project settings
154+
.ropeproject
155+
156+
# mkdocs documentation
157+
/site
158+
159+
# mypy
160+
.mypy_cache/
161+
.dmypy.json
162+
dmypy.json
163+
164+
# Pyre type checker
165+
.pyre/
166+
167+
# InteliJ
168+
.idea/
169+
*.iml
170+
171+
# MacOs
172+
.DS_Store
173+
174+
# pyenv directory
175+
/.pyenv/
176+
177+
ignore.me.*
178+
ignore.me*/
179+
180+
packaged.yaml
181+
182+
# all the local stored certificates
183+
cert/
184+
185+
# cdk
186+
cdk.out/
187+
.cdk.staging
188+

‎neptune-REST-IAM/.npmignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*.ts
2+
!*.d.ts
3+
4+
# CDK asset staging directory
5+
.cdk.staging
6+
cdk.out

‎neptune-REST-IAM/.prettierrc‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"tabWidth": 2,
3+
"useTabs": false,
4+
"singleQuote": true
5+
}

‎neptune-REST-IAM/README.md‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
# Interaction with Neptune Cluster with IAM db authentication enabled
2+
3+
## About
4+
5+
This sample project shows how you can interact with a Neptune cluster when the
6+
IAM db authentication is enabled.
7+
8+
According with the Neptune documentation,
9+
the AWS Identity and Access Management (IAM) is an AWS service
10+
that helps an administrator securely control access to AWS resources.
11+
IAM administrators control who can be authenticated (signed in) and authorized
12+
(have permissions) to use Neptune resources.
13+
IAM is an AWS service that you can use with no additional charge.
14+
15+
You can use AWS Identity and Access Management (IAM) to authenticate to your Neptune DB instance or DB cluster. When IAM database authentication is enabled, each request must be signed using AWS Signature Version 4.
16+
17+
## How to install it?
18+
19+
This is a typescript [CDK](https://docs.aws.amazon.com/cdk/latest/guide/home.html) based project.
20+
CDK is a [NodeJS](https://nodejs.org/en/) module, and this is the only perquisite.
21+
22+
This project was tested with NodeJs version 12.16.1.
23+
24+
After you clone this repository, ensure that all the node dependencies are
25+
installed with the following command:
26+
27+
```bash
28+
npm install
29+
```
30+
31+
After all the nodeJS dependencies are claimed you can install the CDK stack with
32+
the the following command:
33+
34+
```bash
35+
./deploy.bash
36+
```
37+
38+
This operation will may take around 10 mins. If the deployment is successful
39+
then you may be able to see an output as the next one:
40+
41+
```bash
42+
./deploy.bash
43+
......
44+
......
45+
Synth Complete, deploying Stack
46+
NeptuneRestIamStack: deploying...
47+
[0%] start: Publishing ff6804beac5d60d6ec143c1d10f08e10591a5d671f588baf1cabf18e27949177:current
48+
[100%] success: Published ff6804beac5d60d6ec143c1d10f08e10591a5d671f588baf1cabf18e27949177:current
49+
NeptuneRestIamStack: creating CloudFormation changeset...
50+
[██████████████████████████████████████████████████████████] (42/42)
51+
52+
53+
54+
✅ NeptuneRestIamStack
55+
56+
Outputs:
57+
NeptuneRestIamStack.BulkLoaderRoleArn = arn:aws:iam::495161600685:role/NeptuneRestIamStack-NeptuneLoadFromS30422660D-G5ZI9V5C50J7
58+
NeptuneRestIamStack.NeptuneClusterEndpoint = neptune-test-cluster.cluster-cl9mufznljov.eu-west-1.neptune.amazonaws.com
59+
NeptuneRestIamStack.NeptuneInstanceEndpoint = neptunedbinstance-wnxhffni670a.cl9mufznljov.eu-west-1.neptune.amazonaws.com
60+
NeptuneRestIamStack.SecurityGroupId = sg-0ead6abc144196f49
61+
NeptuneRestIamStack.Subnets = subnet-0f759cdb19d8b13b3,subnet-081f56f0c0c288565
62+
63+
```
64+
65+
This CDK stack install an Neptune Cluster with IAM db authentication enabled and adds four six lambda able to interact with the Neptune Cluster.
66+
67+
### Interaction with Neptune Cluster
68+
69+
The interaction with the Neptune cluster can be done in two ways:
70+
71+
1. over the Neptune REST API
72+
2. over the [Gremlin](https://tinkerpop.apache.org/gremlin.html) client.
73+
74+
In botch cases you need to interact with the cluster you need to sing all the requests using
75+
[aws sig4](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html).
76+
77+
#### REST Based interaction
78+
79+
Here is an example for the REST based interaction
80+
originated from the bulkUploadHandler.ts file:
81+
82+
```typescript
83+
try {
84+
const response = await neptunePost(
85+
process.env.NEPTUNE_ENDPOINT,
86+
process.env.NEPTUNE_PORT,
87+
process.env.AWS_DEFAULT_REGION,
88+
'/loader',
89+
body
90+
);
91+
console.log(
92+
`File ${key} was processed with response response ${JSON.stringify(
93+
response,
94+
null,
95+
2
96+
)}.`
97+
);
98+
} catch (e) {
99+
console.error(`File ${key} can not be processed.`);
100+
console.error(e);
101+
}
102+
```
103+
104+
The `neptunePost` method can be found in the utils.ts file.
105+
The `neptunePost` uses the [aws4-axios](https://www.npmjs.com/package/aws4-axios)
106+
to sign all the requests send to the Neptune Cluster using aws4.
107+
The aws4 library uses your secretAccessKey, accessKeyId and sessionToken in order to sing the request.
108+
109+
If you use this code in a AWS Lambda, then the secretAccessKey, accessKeyId and sessionToken are obtained from the environment variables.
110+
111+
Without a sign request all the attempts to communicate with Neptune cluster will end with HTTP 403 error.
112+
113+
Here is the result bulkUploadHandlerv output after the file `edge.csv` was uploaded.
114+
115+
``
116+
2021-03-09T16:41:08.692Z 17c6d7a7-c870-42eb-9b36-9585fbfac76b INFO File edge.csv was processed with response response {
117+
"data": {
118+
"status": "200 OK",
119+
"payload": {
120+
"loadId": "f53d2153-95d4-4b23-b0af-b54753e353a5"
121+
}
122+
},
123+
"status": 200,
124+
"statusText": "OK"
125+
}.
126+
127+
``
128+
129+
Same logic for the clusterStateHandler and getAllBulkJobsHandler lambdas.
130+
131+
#### Gremlin based interaction
132+
133+
For Gremlin based access you need to use use a library able to sign the request.
134+
The next example uses the [gremlin-aws-sigv4](https://www.npmjs.com/package/gremlin-aws-sigv4)
135+
136+
```typescript
137+
const countVerticesHandler: TaskHandler = async (event: any, context: any) => {
138+
const result: CountVerticesResult = await gremlinQuery(
139+
NEPTUNE_ENDPOINT,
140+
NEPTUNE_PORT,
141+
defaultGremlinOpts,
142+
context,
143+
countVertices,
144+
getVerticesCount
145+
);
146+
console.log('countVertices=', result);
147+
return result;
148+
};
149+
150+
async function countVertices(
151+
g: gremlin.driver.AwsSigV4DriverRemoteConnection
152+
): Promise<any> {
153+
return g.V().count().next();
154+
}
155+
```
156+
157+
The `gremlinQuery` method can be found in the utils.ts file.
158+
The `gremlinQuery` uses setup the gremlin client and cares about the signing
159+
the request.
160+
Same logic for: `addDataToNeptuneHandler`, `dropAllHandler` and `countVerticesHandler`.

‎neptune-REST-IAM/bin/neptune-rest-iam.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/usr/bin/env node
2+
import 'source-map-support/register';
3+
import * as cdk from '@aws-cdk/core';
4+
import { NeptuneRestIamStack } from '../lib/neptune-rest-iam-stack';
5+
6+
const app = new cdk.App();
7+
new NeptuneRestIamStack(app, 'NeptuneRestIamStack');

‎neptune-REST-IAM/cdk.json‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"app": "npx ts-node --prefer-ts-exts bin/neptune-rest-iam.ts",
3+
"context": {
4+
"@aws-cdk/core:enableStackNameDuplicates": "true",
5+
"aws-cdk:enableDiffNoFail": "true",
6+
"@aws-cdk/core:stackRelativeExports": "true",
7+
"@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true,
8+
"@aws-cdk/aws-secretsmanager:parseOwnedSecretName": true,
9+
"@aws-cdk/aws-kms:defaultKeyPolicies": true
10+
}
11+
}

0 commit comments

Comments
 (0)