[Docs] Add documentation about storage indexers (#1374)

This PR adds documentation about the indexers used by the Elastic
Package Registry, focusing on the storage indexers (in technical preview).
In this PR, the information about the current endpoints has been updated,
including the available query parameters for `/search` and `/categories`
endpoints; and the Storage indexer based on Google Cloud Storage has
been set as GA.

Author: mrodm

CHANGELOG.md

@@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 * Skip adding to cache requests containing package query parameter. [#1378](https://github.com/elastic/package-registry/pull/1378)
 * Added cache for categories endpoint (SQL storage indexer). [#1379](https://github.com/elastic/package-registry/1379)
 * Update default values for cache search and TTL settings search (SQL storage indexer). [#1379](https://github.com/elastic/package-registry/1379)
+* Storage indexer based on Google Cloud Storage is now GA. [#1374](https://github.com/elastic/package-registry/pull/1374)
 
 ### Deprecated
 

README.md

@@ -5,6 +5,7 @@
 Endpoints:
 
 * `/`: Info about the registry
+* `/health`: Health of the service. Returns 200 if service is ready.
 * `/search`: Search for packages. By default returns all the most recent packages available.
 * `/categories`: List of the existing package categories and how many packages are in each category.
 * `/package/{name}/{version}`: Info about a package
@@ -20,6 +21,17 @@ The `/search` API endpoint has few additional query parameters. More might be ad
 * `category`: Filters the package by the given category. Available categories can be seen when going to `/categories` endpoint.
 * `package`: Filters by a specific package name, for example `mysql`. Returns the most recent version.
 * `all`: This can be set to `true` to list all package versions. This is set to `false` by default.
+* `type`: Filters by a specific package type, for example `input`.
+* `capabilities`: Filters the packages according to the given capabilities. This query parameter accepts a comma-separated list.
+* `spec.min` and `spec.max`: Filters the packages by their `format_version` field given the version (major and minor numbers) set in the query parameter. It is not required to set both parameters. Examples:
+    - `?spec.min=2.2&spec.max=3.3`
+    - `?spec.max=3.3`
+    - `?spec.min=3.0`
+* `discovery`: Returns the packages that define the `discovery` setting and fulfill the conditions in the query parameter. There are two different options:
+    - Based on fields: Packages must include discovery fields in their manifest and all of fields must be included in the list included in the request
+        - Example: Packages that contain both `process.pid` and `host.os.name` fields: `?discovery=fields:process.pid,host.os.name`
+    - Based on datasets: Packages must include discovery datasets in their manifest and at least one of the datasets must be included in the list included in the request:
+        - Example: Packages that contain at least one of `nginx.access` or `nginx.error` datasets: `?discovery=datasets:nginx.access,nginx.error`
 * `prerelease`: This can be set to `true` to list prerelease versions of packages. Versions are considered prereleases if they are not stable according to semantic versioning, that is, if they are 0.x versions, or if they contain a prerelease tag. This is set to `false` by default.
 * `experimental` (deprecated): This can be set to `true` to list packages considered to be experimental. This is set to `false` by default.
 
@@ -33,6 +45,12 @@ The `/categories` API endpoint has two additional query parameters.
 * `prerelease`: This can be set to `true` to list prerelease versions of packages. Versions are considered prereleases if they are not stable according to semantic versioning, that is, if they are 0.x versions, or if they contain a prerelease tag. This is set to `false` by default.
 * `experimental` (deprecated): This can be set to `true` to list categories from experimental packages. This is set to `false` by default.
 * `include_policy_templates`: This can be set to `true` to include categories from policy templates. This is set to `false` by default.
+* `capabilities`: List categories filtering the packages according to the given capabilities. This query parameter accepts a comma-separated list.
+* `spec.min` and `spec.max`: List categories filtering the packages by their `format_version` field given the version (major and minor numbers) set in the query parameter. It is not required to set both parameters. Examples:
+    - `?spec.min=2.2&spec.max=3.3`
+    - `?spec.max=3.3`
+    - `?spec.min=3.0`
+* `discovery`: List categories filtering the packages that define the `discovery` setting and fulfill the conditions in the query parameter. These query parameter follow the same syntax and behaviour to obtain the corresponding categories as in [`/search` endpoint](#search).
 
 ## Package structure
 
@@ -170,130 +188,6 @@ For that, you need to build a new Package Registry docker image from your requir
    elastic-package stack up -v -d
    ```
 
-### Testing Storage indexers
-
-By default, Package Registry uses the FileSystem indexers.
-
-#### Start services locally
-Storage indexers can be tested locally following in two different ways:
-1. Running Package Registry and a fake GCS server as independent services:
-    1. Launch the fake GCS server in one terminal:
-        - It creates a new folder with the expected contents for the bucket.
-        - It manages a docker-compose scenario with the fake GCS server.
-        - The search index JSON file can be downloaded from the [internal CI](https://buildkite.com/elastic/package-storage-infra-indexing/builds?branch=main) and set that file via `-i` parameter.
-       ```shell
-       cd /path/to/repo/package-registry/
-       cd dev
-       bash launch_fake_gcs_server.sh -i ../storage/testdata/search-index-all-full.json -b example -c 1
-       ```
-    2. Tune the configuration used by Package Registry as you require:
-        - By default, it uses the `config.yml` file at the root of the repository.
-    3. Launch EPR service in a different terminal:
-        - It builds package-registry with the contents of the working copy.
-        - It triggers the EPR service with the required environment variables to use storage indexers.
-       ```shell
-       cd /path/to/repo/package-registry/
-       cd dev
-       bash launch_epr_service_storage_indexer.sh -p ../config.yml
-       ```
-    4. To stop both services, you just need to press `CTRL+Z` on each terminal. The scripts also manage the cleanup process.
-2. Running just Package Registry (fake GCS server runs using the golang library within Package Registry service):
-    1. Tune the configuration used by Package Registry as you require:
-        - By default, it uses the `config.yml` file at the root of the repository.
-    2. Launch EPR service:
-        - It builds package-registry with the contents of the working copy.
-        - It triggers the EPR service with the required environment variables to use storage indexers.
-            - Fake GCS server will run as part of the same process via the Golang library.
-       ```shell
-       cd /path/to/repo/package-registry/
-       cd dev
-       bash launch_epr_service_storage_indexer.sh -i ../storage/testdata/search-index-all-full.json -p ../config.yml
-       ```
-    3. To stop these services, you just need to press `CTRL+Z` on this terminal.
-
-Following these steps, EPR service should be reading files from the storage indexer and there should be log messages like these ones:
-```json
-{"log.level":"info","@timestamp":"2024-05-27T20:03:35.489+0200","log.origin":{"function":"github.com/elastic/package-registry/storage.(*Indexer).updateIndex","file.name":"storage/indexer.go","file.line":181},"message":"cursor will be updated","cursor.current":"","cursor.next":"1","ecs.version":"1.6.0"}
-{"log.level":"info","@timestamp":"2024-05-27T20:03:35.827+0200","log.origin":{"function":"github.com/elastic/package-registry/storage.(*Indexer).updateIndex","file.name":"storage/indexer.go","file.line":192},"message":"Downloaded new search-index-all index","index.packages.size":"1133","ecs.version":"1.6.0"}
-```
-
-Package registry service is available at `http://localhost:8080`. Example of query using `curl`:
-```shell
-curl -s "http://localhost:8080/search"
-```
-
-#### Update indices locally
-
-Following the steps mentioned above, a fake GCS server with the contents bucket is going to be
-available at http://localhost:4443.
-
-You can query directly to check if the contents of the bucket are available:
-    - `http://localhost:4443/storage/v1/b/<bucket>/o`
-    - Example:
-```shell
- $ curl  -s http://localhost:4443/storage/v1/b/fake-package-storage-internal/o |jq -r .
-{
-  "kind": "storage#objects",
-  "items": [
-    {
-      "kind": "storage#object",
-      "name": "v2/metadata/1/search-index-all.json",
-      "id": "fake-package-storage-internal/v2/metadata/1/search-index-all.json",
-      "bucket": "fake-package-storage-internal",
-      "size": "22157319",
-      "crc32c": "s7tNlg==",
-      "md5Hash": "49AZWmusj2eLpjH98ePdCw==",
-      "etag": "49AZWmusj2eLpjH98ePdCw==",
-      "storageClass": "STANDARD",
-      "timeCreated": "2025-06-11T09:54:21.239848+02:00",
-      "timeStorageClassUpdated": "2025-06-11T09:54:21.239848+02:00",
-      "updated": "2025-06-11T09:54:21.239848+02:00",
-      "generation": "1749628461332535",
-      "selfLink": "/storage/v1/b/fake-package-storage-internal/o/v2%2Fmetadata%2F1%2Fsearch-index-all.json",
-      "mediaLink": "/download/storage/v1/b/fake-package-storage-internal/o/v2%2Fmetadata%2F1%2Fsearch-index-all.json?alt=media",
-      "metageneration": "1"
-    },
-    {
-      "kind": "storage#object",
-      "name": "v2/metadata/cursor.json",
-      "id": "fake-package-storage-internal/v2/metadata/cursor.json",
-      "bucket": "fake-package-storage-internal",
-      "size": "15",
-      "crc32c": "oP5v4Q==",
-      "md5Hash": "vxi+8LZ201P+UukYETdzvQ==",
-      "etag": "vxi+8LZ201P+UukYETdzvQ==",
-      "storageClass": "STANDARD",
-      "timeCreated": "2025-06-11T09:54:21.239843+02:00",
-      "timeStorageClassUpdated": "2025-06-11T09:54:21.239847+02:00",
-      "updated": "2025-06-11T09:54:21.239847+02:00",
-      "generation": "1749628461239867",
-      "selfLink": "/storage/v1/b/fake-package-storage-internal/o/v2%2Fmetadata%2Fcursor.json",
-      "mediaLink": "/download/storage/v1/b/fake-package-storage-internal/o/v2%2Fmetadata%2Fcursor.json?alt=media",
-      "metageneration": "1"
-    }
-  ]
-}
-```
-
-As this URL is available, it can also be updated the contents of the bucket to test the update index process
-in the Elastic Package Registry service locally. For that you need to follow these steps:
-1. Check the current cursor set. In EPR logs should be a log like this:
-   ```json
-   {"log.level":"info","@timestamp":"2025-06-11T10:02:17.680+0200","log.origin":{"function":"github.com/elastic/package-registry/storage.(*Indexer).updateIndex","file.name":"storage/indexer.go","file.line":178},"message":"cursor is up-to-date","cursor.current":"1","ecs.version":"1.6.0"}
-   ```
-2. Update the new index JSON file to the new path `v2/metadata/<cursor>/search-index-all.json` and also update `cursor.json` file in the fake GCS server using the script helper:
-   ```shell
-   # bash dev/updateIndexDatabase.sh -i <path_to_index> -c <cursor_string> -b <bucket>
-   # Example adding a new index using as cursor "2":
-   bash dev/updateIndexDatabase.sh -i ./search-index-all.json -c 2 -b fake-package-storage-internal
-   ```
-
-After following these steps, the next log messages must appear in the EPR service:
-```json
-{"log.level":"info","@timestamp":"2025-06-11T10:07:17.680+0200","log.origin":{"function":"github.com/elastic/package-registry/storage.(*Indexer).updateIndex","file.name":"storage/indexer.go","file.line":181},"message":"cursor will be updated","cursor.current":"1","cursor.next":"2","ecs.version":"1.6.0"}
-{"log.level":"info","@timestamp":"2025-06-11T10:07:25.749+0200","log.origin":{"function":"github.com/elastic/package-registry/storage.(*Indexer).updateIndex","file.name":"storage/indexer.go","file.line":192},"message":"Downloaded new search-index-all index","index.packages.size":"10775","ecs.version":"1.6.0"}
-```
-
 ### Healthcheck
 
 Availability of the service can be queried using the `/health` endpoint. As soon as `/health` returns a 200, the service is ready to handle requests.
@@ -393,6 +287,13 @@ For example:
 package-registry --feature-proxy-mode=true -proxy-to=https://epr.elastic.co
 ```
 
+### Storage indexers
+
+Elastic Package Registry (EPR) supports multiple ways to retrieve package information. By default, it uses the File system indexer to read packages (folders or zip files) from the paths defined in the `config.yml`.
+
+[Here](./docs/storage_indexers.md#storage-indexers-used-in-elastic-package-registry) you can read more about the different
+indexers available in EPR as well as how to test EPR locally using these storage indexers without configuring a remote bucket.
+
 
 ## Release
 

dev/launch_epr_service_storage_indexer.sh

@@ -118,7 +118,7 @@ export EPR_CONFIG="${CONFIG_PATH}"
 # export EPR_SQL_INDEXER_CATEGORIES_CACHE_SIZE=100
 # export EPR_SQL_INDEXER_CATEGORIES_CACHE_TTL=24h
 # export EPR_SQL_INDEXER_READ_PACKAGES_BATCH_SIZE=2000
-# export EPR_SQL_INDEXER_DB_INSERT_BATCH_SIZE=2000
+# export EPR_SQL_INDEXER_DB_INSERT_BATCH_SIZE=500
 
 ./package-registry