Skip to content

Commit bd2c020

Browse files
tloubrieu-jpltloubrieu-jpl
authored
Update README.md
Remove obsolete scalable harvest documenttion.
1 parent aa1981e commit bd2c020

1 file changed

Lines changed: 0 additions & 177 deletions

File tree

‎docker/README.md‎

Lines changed: 0 additions & 177 deletions
Original file line numberDiff line numberDiff line change
@@ -284,181 +284,4 @@ docker compose --profile=int-registry-batch-loader down
284284
Note: Ignore any `failed to remove network` errors, because the related docker network may
285285
have active endpoints of other services.
286286

287-
## 🏃 Steps to configure the Scalable Harvest components to be executed with docker compose
288287

289-
#### 1. Open the `.env` file (located in the same directory with the `docker.compose.yml` file).
290-
291-
#### 2. Check and update (if necessary) the following common environment variable.
292-
293-
| Environment Variable | Description |
294-
| ----------------------------- | ----------- |
295-
| HARVEST_DATA_DIR | Absolute path of the Harvest data directory in the host machine (E.g.: `/tmp/registry-harvest-data`). If the Registry Harvest CLI is executed with the option to download test data, then this directory will be cleaned-up and populated with test data |
296-
297-
```
298-
# --------------------------------------------------------------------
299-
# Common Configuartions
300-
# --------------------------------------------------------------------
301-
302-
# Absolute path of the Harvest data directory in the host machine (E.g.: `./test-data/registry-harvest-data`).
303-
# If the Registry Harvest CLI is executed with the option to download test data, then this directory will be
304-
# cleaned-up and populated with test data. Make sure to have the same `HARVEST_DATA_DIR` value set in the
305-
# environment variables of the Registry Harvest Service, Registry Crawler Service and Registry Harvest CLI.
306-
# Also, this `HARVEST_DATA_DIR` location should be accessible from the docker containers of the Registry Harvest Service,
307-
# Registry Crawler Service and Registry Harvest CLI.
308-
HARVEST_DATA_DIR=./test-data/registry-harvest-data
309-
```
310-
311-
#### 2. Update the Registry Harvest Service configuration file.
312-
313-
* Get a copy of the `harvest-server.cfg` file from https://github.com/NASA-PDS/registry-harvest-service/blob/main/src/main/resources/conf/harvest-server.cfg and keep it in a local file location such as `/tmp/cfg/harvest-server.cfg`.
314-
* Update the properties such as `rmq.host`, `rmq.user`, `rmq.password` and `es.url` to match with your deployment environment.
315-
* Make sure to specify the exact IP address of the host machine (E.g.: `192.168.0.1`), when configuring the `rmq.host` and
316-
`es.url`.
317-
318-
#### 3. Check and update (if necessary) the following environment variables related with the Registry Harvest Service in the `.env` file.
319-
320-
321-
| Environment Variable | Description |
322-
| ----------------------------- | ----------- |
323-
| BIG_DATA_HARVEST_SERVER_IMAGE | Docker image of the Registry Harvest Service. Make sure this docker image is available. |
324-
| HARVEST_SERVER_CONFIG_FILE | Absolute path of the Registry Harvest Service configuration file in the host machine (E.g.: `./default-config/harvest-server.cfg`) |
325-
326-
```
327-
# --------------------------------------------------------------------
328-
# Registry Harvest Service
329-
# --------------------------------------------------------------------
330-
331-
# Docker image of the Registry Harvest Service
332-
REGISTRY_HARVEST_SERVICE_IMAGE=nasapds/registry-harvest-service
333-
334-
# Absolute path of the Registry Harvest Service configuration file in the host machine (E.g.: /tmp/cfg/harvest-server.cfg)
335-
HARVEST_SERVER_CONFIG_FILE=./default-config/harvest-server.cfg
336-
```
337-
338-
#### 4. Update the Registry Crawler Service configuration file.
339-
340-
* Get a copy of the `harvest-client.cfg` file from https://github.com/NASA-PDS/big-data-crawler-server/blob/main/src/main/resources/conf/crawler-server.cfg and
341-
keep it in a local file location such as `/tmp/cfg/crawler-server.cfg`.
342-
* Update the properties such as `rmq.host`, `rmq.user` and `rmq.password` to match with your deployment environment.
343-
* Make sure to specify the exact IP address of the host machine (E.g.: `192.168.0.1`), when configuring the `rmq.host`.
344-
345-
#### 5. Check and update (if necessary) the following environment variables related with the Registry Crawler Service in the `.env` file.
346-
347-
348-
| Environment Variable | Description |
349-
| ------------------------------ | ----------- |
350-
| REGISTRY_CRAWLER_SERVICE_IMAGE | Docker image of the Registry Crawler Service. Make sure this docker image is available. |
351-
| CRAWLER_SERVER_CONFIG_FILE | Absolute path of the Registry Crawler Service configuration file in the host machine (`E.g.: ./default-config/crawler-server.cfg`) |
352-
353-
```
354-
# --------------------------------------------------------------------
355-
# Registry Crawler Service
356-
# --------------------------------------------------------------------
357-
358-
# Docker image of the Registry Crawler Service
359-
REGISTRY_CRAWLER_SERVICE_IMAGE=nasapds/registry-crawler-service
360-
361-
# Absolute path of the Registry Crawler Service configuration file in the host machine (E.g.: /tmp/cfg/crawler-server.cfg)
362-
CRAWLER_SERVER_CONFIG_FILE=./default-config/crawler-server.cfg
363-
```
364-
365-
#### 6. Update the Registry Harvest CLI configuration file.
366-
367-
* Get a copy of the `harvest-client.cfg` file from https://github.com/NASA-PDS/registry-harvest-cli/blob/main/src/main/resources/conf/harvest-client.cfg and
368-
keep it in a local file location such as `./default-config/harvest-client.cfg`.
369-
* Update the properties such as `rmq.host`, `rmq.user` and `rmq.password` to match with your deployment environment.
370-
* Make sure to specify the exact IP address of the host machine (E.g.: `192.168.0.1`), when configuring the `rmq.host`.
371-
372-
373-
#### 7. Update the Harvest job file.
374-
375-
* Create a Harvest job file in a local file location (E.g.: `./default-config/cfg/harvest-job-config.xml`).
376-
* An example for a Harvest job file can be found at https://github.com/NASA-PDS/registry-harvest-cli/blob/main/src/main/resources/examples/directories.xml.
377-
Make sure to update the `/path/to/archive` in the Harvest job file to point to a valid Harvest data directory.
378-
379-
#### 8. Check and update (if necessary) the following environment variables related with the Registry Harvest CLI in the `.env` file.
380-
381-
| Environment Variable | Description |
382-
| ----------------------------- | ----------- |
383-
| REGISTRY_HARVEST_CLI_IMAGE | Docker image of the Registry Harvest CLI. Make sure this docker image is available. |
384-
| HARVEST_CLIENT_CONFIG_FILE | Absolute path of the Registry Harvest CLI configuration file in the host machine (E.g.: `./default-config/harvest-client.cfg`) |
385-
| HARVEST_JOB_CONFIG_FILE | Absolute path of the Harvest job file in the host machine (E.g.: `./default-config/harvest-job-config.xml`) |
386-
387-
```
388-
# --------------------------------------------------------------------
389-
# Registry Harvest CLI
390-
# --------------------------------------------------------------------
391-
392-
# Docker image of the Registry Harvest CLI
393-
REGISTRY_HARVEST_CLI_IMAGE=nasapds/registry-harvest-cli
394-
395-
# Absolute path of the Registry Harvest CLI configuration file in the host machine (E.g.: /tmp/conf/harvest-client.cfg)
396-
HARVEST_CLIENT_CONFIG_FILE=./default-config/harvest-client.cfg
397-
398-
# --------------------------------------------------------------------
399-
# Common Configuartions
400-
# --------------------------------------------------------------------
401-
402-
# Absolute path of the Harvest job file in the host machine (E.g.: ./default-config/harvest-job-config.xml)
403-
HARVEST_JOB_CONFIG_FILE=./default-config/harvest-job-config.xml
404-
```
405-
406-
#### 9. Configure RabbitMQ
407-
408-
The RabbitMQ can be configured by using the `rabbitmq-definitions.json` file available at the `registry/docker/config` directory.
409-
410-
* Open the `rabbitmq-definitions.json` file.
411-
* Locate the definition for the user `harvest` under the `users`.
412-
* The password of the user `harvest` should be specified as a RabbitMQ `password_hash`. Generate a password hash using
413-
the `rabbit_password_hashing_sha256` algorithm (The Python script available at the https://stackoverflow.com/questions/41306350/how-to-generate-password-hash-for-rabbitmq-management-http-api/53016240#53016240
414-
can be used to generate a password hash for a new password hash).
415-
* Update the `password_hash` of the `harvest` user with the newly generated password hash.
416-
417-
#### 10. Check and update (if necessary) the following environment variables related with the Postman Collection Test
418-
419-
| Environment Variable | Description |
420-
| ----------------------------- | ----------- |
421-
| POSTMAN_NEWMAN_IMAGE | Docker image of Newman (a command-line collection runner for Postman) |
422-
| POSTMAN_COLLECTION_FILE | Absolute path of the Postman collection to be executed with the test data (E.g.: ./postman/postman_collection.json) |
423-
424-
```
425-
# --------------------------------------------------------------------
426-
# Registry Harvest CLI
427-
# --------------------------------------------------------------------
428-
429-
# Docker image of Newman (a command-line collection runner for Postman)
430-
POSTMAN_NEWMAN_IMAGE=postman/newman
431-
432-
# Absolute path of the Postman collection to be executed with the test data (E.g.: ./postman/postman_collection.json)
433-
POSTMAN_COLLECTION_FILE=./postman/postman_collection.json
434-
```
435-
436-
## 🏃 Steps to execute the Scalable Harvest components with docker compose
437-
438-
#### 1. Open a terminal and change the current working directory to `registry/docker`.
439-
440-
#### 2. Setup Elasticsearch and RabbitMQ as explained in the following document.
441-
TODO: Add the link to the Scalable Harvest documentation.
442-
443-
#### 3. Start Scalable Harvest components as follows.
444-
445-
To execute Scalable Harvest Integration Tests with downloaded test data
446-
```
447-
docker compose --profile=int-registry-service-loader up
448-
```
449-
450-
#### 4. Test the deployment.
451-
452-
Follow the instructions in the following section at the end of the [Test Your Deployment](https://nasa-pds.github.io/pds-registry-app/install/test.html).
453-
454-
* Query Elasticsearch
455-
456-
## 🏃 Cleaning up the Scalable Harvest deployment
457-
458-
#### * The Scalable Harvest deployment can be cleaned up as follows.
459-
```
460-
docker compose --profile=int-registry-service-loader down
461-
```
462-
463-
Note: Ignore any `failed to remove network` errors, because the related docker network may
464-
have active endpoints of other services.

0 commit comments

Comments
 (0)