Skip to content

Commit 9a461fd

Browse files
jordanpadamsjordanpadams
authored
Merge pull request #446 from NASA-PDS/tloubrieu-jpl-patch-1
Remove scalable harvest deployment guidelines from docker compose README.md
2 parents 150f330 + bd2c020 commit 9a461fd

1 file changed

Lines changed: 0 additions & 177 deletions

File tree

‎docker/README.md‎

Lines changed: 0 additions & 177 deletions
Original file line numberDiff line numberDiff line change
@@ -286,181 +286,4 @@ docker compose --profile=int-registry-batch-loader down
286286
Note: Ignore any `failed to remove network` errors, because the related docker network may
287287
have active endpoints of other services.
288288

289-
## 🏃 Steps to configure the Scalable Harvest components to be executed with docker compose
290289

291-
#### 1. Open the `.env` file (located in the same directory with the `docker.compose.yml` file).
292-
293-
#### 2. Check and update (if necessary) the following common environment variable.
294-
295-
| Environment Variable | Description |
296-
| ----------------------------- | ----------- |
297-
| HARVEST_DATA_DIR | Absolute path of the Harvest data directory in the host machine (E.g.: `/tmp/registry-harvest-data`). If the Registry Harvest CLI is executed with the option to download test data, then this directory will be cleaned-up and populated with test data |
298-
299-
```
300-
# --------------------------------------------------------------------
301-
# Common Configuartions
302-
# --------------------------------------------------------------------
303-
304-
# Absolute path of the Harvest data directory in the host machine (E.g.: `./test-data/registry-harvest-data`).
305-
# If the Registry Harvest CLI is executed with the option to download test data, then this directory will be
306-
# cleaned-up and populated with test data. Make sure to have the same `HARVEST_DATA_DIR` value set in the
307-
# environment variables of the Registry Harvest Service, Registry Crawler Service and Registry Harvest CLI.
308-
# Also, this `HARVEST_DATA_DIR` location should be accessible from the docker containers of the Registry Harvest Service,
309-
# Registry Crawler Service and Registry Harvest CLI.
310-
HARVEST_DATA_DIR=./test-data/registry-harvest-data
311-
```
312-
313-
#### 2. Update the Registry Harvest Service configuration file.
314-
315-
* Get a copy of the `harvest-server.cfg` file from https://github.com/NASA-PDS/registry-harvest-service/blob/main/src/main/resources/conf/harvest-server.cfg and keep it in a local file location such as `/tmp/cfg/harvest-server.cfg`.
316-
* Update the properties such as `rmq.host`, `rmq.user`, `rmq.password` and `es.url` to match with your deployment environment.
317-
* Make sure to specify the exact IP address of the host machine (E.g.: `192.168.0.1`), when configuring the `rmq.host` and
318-
`es.url`.
319-
320-
#### 3. Check and update (if necessary) the following environment variables related with the Registry Harvest Service in the `.env` file.
321-
322-
323-
| Environment Variable | Description |
324-
| ----------------------------- | ----------- |
325-
| BIG_DATA_HARVEST_SERVER_IMAGE | Docker image of the Registry Harvest Service. Make sure this docker image is available. |
326-
| HARVEST_SERVER_CONFIG_FILE | Absolute path of the Registry Harvest Service configuration file in the host machine (E.g.: `./default-config/harvest-server.cfg`) |
327-
328-
```
329-
# --------------------------------------------------------------------
330-
# Registry Harvest Service
331-
# --------------------------------------------------------------------
332-
333-
# Docker image of the Registry Harvest Service
334-
REGISTRY_HARVEST_SERVICE_IMAGE=nasapds/registry-harvest-service
335-
336-
# Absolute path of the Registry Harvest Service configuration file in the host machine (E.g.: /tmp/cfg/harvest-server.cfg)
337-
HARVEST_SERVER_CONFIG_FILE=./default-config/harvest-server.cfg
338-
```
339-
340-
#### 4. Update the Registry Crawler Service configuration file.
341-
342-
* Get a copy of the `harvest-client.cfg` file from https://github.com/NASA-PDS/big-data-crawler-server/blob/main/src/main/resources/conf/crawler-server.cfg and
343-
keep it in a local file location such as `/tmp/cfg/crawler-server.cfg`.
344-
* Update the properties such as `rmq.host`, `rmq.user` and `rmq.password` to match with your deployment environment.
345-
* Make sure to specify the exact IP address of the host machine (E.g.: `192.168.0.1`), when configuring the `rmq.host`.
346-
347-
#### 5. Check and update (if necessary) the following environment variables related with the Registry Crawler Service in the `.env` file.
348-
349-
350-
| Environment Variable | Description |
351-
| ------------------------------ | ----------- |
352-
| REGISTRY_CRAWLER_SERVICE_IMAGE | Docker image of the Registry Crawler Service. Make sure this docker image is available. |
353-
| CRAWLER_SERVER_CONFIG_FILE | Absolute path of the Registry Crawler Service configuration file in the host machine (`E.g.: ./default-config/crawler-server.cfg`) |
354-
355-
```
356-
# --------------------------------------------------------------------
357-
# Registry Crawler Service
358-
# --------------------------------------------------------------------
359-
360-
# Docker image of the Registry Crawler Service
361-
REGISTRY_CRAWLER_SERVICE_IMAGE=nasapds/registry-crawler-service
362-
363-
# Absolute path of the Registry Crawler Service configuration file in the host machine (E.g.: /tmp/cfg/crawler-server.cfg)
364-
CRAWLER_SERVER_CONFIG_FILE=./default-config/crawler-server.cfg
365-
```
366-
367-
#### 6. Update the Registry Harvest CLI configuration file.
368-
369-
* Get a copy of the `harvest-client.cfg` file from https://github.com/NASA-PDS/registry-harvest-cli/blob/main/src/main/resources/conf/harvest-client.cfg and
370-
keep it in a local file location such as `./default-config/harvest-client.cfg`.
371-
* Update the properties such as `rmq.host`, `rmq.user` and `rmq.password` to match with your deployment environment.
372-
* Make sure to specify the exact IP address of the host machine (E.g.: `192.168.0.1`), when configuring the `rmq.host`.
373-
374-
375-
#### 7. Update the Harvest job file.
376-
377-
* Create a Harvest job file in a local file location (E.g.: `./default-config/cfg/harvest-job-config.xml`).
378-
* An example for a Harvest job file can be found at https://github.com/NASA-PDS/registry-harvest-cli/blob/main/src/main/resources/examples/directories.xml.
379-
Make sure to update the `/path/to/archive` in the Harvest job file to point to a valid Harvest data directory.
380-
381-
#### 8. Check and update (if necessary) the following environment variables related with the Registry Harvest CLI in the `.env` file.
382-
383-
| Environment Variable | Description |
384-
| ----------------------------- | ----------- |
385-
| REGISTRY_HARVEST_CLI_IMAGE | Docker image of the Registry Harvest CLI. Make sure this docker image is available. |
386-
| HARVEST_CLIENT_CONFIG_FILE | Absolute path of the Registry Harvest CLI configuration file in the host machine (E.g.: `./default-config/harvest-client.cfg`) |
387-
| HARVEST_JOB_CONFIG_FILE | Absolute path of the Harvest job file in the host machine (E.g.: `./default-config/harvest-job-config.xml`) |
388-
389-
```
390-
# --------------------------------------------------------------------
391-
# Registry Harvest CLI
392-
# --------------------------------------------------------------------
393-
394-
# Docker image of the Registry Harvest CLI
395-
REGISTRY_HARVEST_CLI_IMAGE=nasapds/registry-harvest-cli
396-
397-
# Absolute path of the Registry Harvest CLI configuration file in the host machine (E.g.: /tmp/conf/harvest-client.cfg)
398-
HARVEST_CLIENT_CONFIG_FILE=./default-config/harvest-client.cfg
399-
400-
# --------------------------------------------------------------------
401-
# Common Configuartions
402-
# --------------------------------------------------------------------
403-
404-
# Absolute path of the Harvest job file in the host machine (E.g.: ./default-config/harvest-job-config.xml)
405-
HARVEST_JOB_CONFIG_FILE=./default-config/harvest-job-config.xml
406-
```
407-
408-
#### 9. Configure RabbitMQ
409-
410-
The RabbitMQ can be configured by using the `rabbitmq-definitions.json` file available at the `registry/docker/config` directory.
411-
412-
* Open the `rabbitmq-definitions.json` file.
413-
* Locate the definition for the user `harvest` under the `users`.
414-
* The password of the user `harvest` should be specified as a RabbitMQ `password_hash`. Generate a password hash using
415-
the `rabbit_password_hashing_sha256` algorithm (The Python script available at the https://stackoverflow.com/questions/41306350/how-to-generate-password-hash-for-rabbitmq-management-http-api/53016240#53016240
416-
can be used to generate a password hash for a new password hash).
417-
* Update the `password_hash` of the `harvest` user with the newly generated password hash.
418-
419-
#### 10. Check and update (if necessary) the following environment variables related with the Postman Collection Test
420-
421-
| Environment Variable | Description |
422-
| ----------------------------- | ----------- |
423-
| POSTMAN_NEWMAN_IMAGE | Docker image of Newman (a command-line collection runner for Postman) |
424-
| POSTMAN_COLLECTION_FILE | Absolute path of the Postman collection to be executed with the test data (E.g.: ./postman/postman_collection.json) |
425-
426-
```
427-
# --------------------------------------------------------------------
428-
# Registry Harvest CLI
429-
# --------------------------------------------------------------------
430-
431-
# Docker image of Newman (a command-line collection runner for Postman)
432-
POSTMAN_NEWMAN_IMAGE=postman/newman
433-
434-
# Absolute path of the Postman collection to be executed with the test data (E.g.: ./postman/postman_collection.json)
435-
POSTMAN_COLLECTION_FILE=./postman/postman_collection.json
436-
```
437-
438-
## 🏃 Steps to execute the Scalable Harvest components with docker compose
439-
440-
#### 1. Open a terminal and change the current working directory to `registry/docker`.
441-
442-
#### 2. Setup Elasticsearch and RabbitMQ as explained in the following document.
443-
TODO: Add the link to the Scalable Harvest documentation.
444-
445-
#### 3. Start Scalable Harvest components as follows.
446-
447-
To execute Scalable Harvest Integration Tests with downloaded test data
448-
```
449-
docker compose --profile=int-registry-service-loader up
450-
```
451-
452-
#### 4. Test the deployment.
453-
454-
Follow the instructions in the following section at the end of the [Test Your Deployment](https://nasa-pds.github.io/pds-registry-app/install/test.html).
455-
456-
* Query Elasticsearch
457-
458-
## 🏃 Cleaning up the Scalable Harvest deployment
459-
460-
#### * The Scalable Harvest deployment can be cleaned up as follows.
461-
```
462-
docker compose --profile=int-registry-service-loader down
463-
```
464-
465-
Note: Ignore any `failed to remove network` errors, because the related docker network may
466-
have active endpoints of other services.

0 commit comments

Comments
 (0)