Performance Testing: Container cpu memory limits (#184)

* Added new Compose file for data insertion into postgres

* Fix command in Docker Compose to handle line endings before executing load script

* Refactor Docker Compose command to handle line endings and streamline load script execution

* Aligned files in folder structure

* Enhance schema validation in bulk data loader and remove unused schema generator script

* Remove unused schema volume from Docker Compose and clean up load.sql comments

* Add container observability features with CPU and memory limits exporter

- Introduced Docker Compose configuration for observability services
- Added Prometheus metrics exporter for container CPU limits
- Created Grafana dashboards for monitoring CPU and memory usage
- Configured OpenTelemetry collector for metrics collection
- Implemented necessary environment variables for resource limits

* Update Docker images for twinengine-dataengine and dpp-plugin to use 'develop' tag

* Update container observability dashboard to improve CPU and memory metrics with new expressions and intervals

* Remove usage instructions from Docker Compose file

* Refactor load scripts to improve error handling and enhance readability by using constants for repeated values

* feat: Add observability stack with Grafana, Prometheus, and custom CPU limits exporter

- Introduced a Docker Compose setup for observability, including services for Grafana, Prometheus, and a custom CPU limits exporter.
- Added SQL script for truncating the database with a dynamic approach.
- Created Grafana dashboard configuration for monitoring container metrics.
- Implemented a Python-based CPU limits exporter that reads container limits from Docker and environment variables.
- Configured OpenTelemetry collector for gathering metrics from Docker containers.
- Established health checks and dependencies for services in the Docker Compose file.

* refactor: Improve Dockerfile structure and enhance error logging in CPU limits exporter

* refactor: Consolidate user creation and dependency installation in Dockerfile

* Remove SQL scripts for schema checks, data loading, and database truncation

- Deleted `check-schema.sql` which checked for required tables in the database schema.
- Removed `load.sql` which contained the logic for bulk data loading into various tables.
- Eliminated `truncate-db.sql` that was used to truncate all tables in the public schema.

* refactor: Update Docker setup and metrics exporter for resource limits

* refactor: Update Docker images and user permissions in docker-compose.yml

* refactor: Change user for otel-docker-stats-collector to root for improved permissions

* refactor: Adjust CPU and memory limits for performance services in Docker setup

* refactor: Reduce scrape intervals for container CPU and memory metrics to improve observability

---------

Co-authored-by: Hardi Shah <hsh@mm-software.com>

Author: mm-omdi

performance/databaseBulkDataLoader/.env.example

@@ -0,0 +1,105 @@
+# DataEngine Performance Test
+
+This folder contains a Docker Compose setup for running a full DataEngine performance environment with:
+
+1. TwinEngine DataEngine
+2. DPP plugin
+3. PostgreSQL
+4. Registry and template services
+5. LGTM observability stack (Grafana, Prometheus, OTLP)
+
+## What Is Included
+
+The stack in `docker-compose.yml` starts these main services:
+
+1. `twinengine-dataengine`
+2. `dpp-plugin`
+3. `postgres`
+4. `pgadmin`
+5. `otel-lgtm`
+6. `otel-docker-stats-collector`
+7. `docker-resource-exporter`
+8. Supporting services (`nginx`, `mongo`, template repositories, registries, UI)
+
+## What Is Required
+
+1. Docker with Compose support.
+2. Access to local container ports listed below.
+3. Valid resource settings in `.env`.
+
+## Environment Configuration
+
+This performance stack reads CPU and memory settings from `.env`:
+
+```env
+TWINENGINE_CPU=0.5
+TWINENGINE_MEMORY=1g
+
+DPP_PLUGIN_CPU=0.5
+DPP_PLUGIN_MEMORY=1g
+
+POSTGRES_CPU=4
+POSTGRES_MEMORY=8g
+```
+
+These values are applied through `cpus` and `mem_limit` in `docker-compose.yml`.
+
+## CPU and Memory Sizing Examples
+
+You can tune container resource usage in `docker-compose.yml`.
+
+Example service settings:
+
+```yaml
+services:
+  twinengine-dataengine:
+    cpus: ${TWINENGINE_CPU:-1}
+    mem_limit: ${TWINENGINE_MEMORY:-1g}
+```
+
+CPU examples:
+
+```yaml
+cpus: 1      # 1 core
+cpus: 2      # 2 cores
+cpus: 4      # 4 cores
+cpus: 0.5    # Half a core
+cpus: 1.25   # 1¼ cores
+```
+
+Memory examples:
+
+```yaml
+mem_limit: 512m   # 512 MB
+mem_limit: 1g     # 1 GB
+mem_limit: 2g     # 2 GB
+mem_limit: 4g     # 4 GB
+```
+
+## Run
+
+From this folder:
+
+```bash
+docker compose up -d
+```
+
+To stop and remove containers/volumes:
+
+```bash
+docker compose down -v
+```
+
+## Common Endpoints
+
+1. DataEngine entry via nginx: `http://localhost:8080`
+2. pgAdmin: `http://localhost:8081`
+3. Grafana (LGTM): `http://localhost:3000`
+4. Prometheus (LGTM): `http://localhost:9090`
+5. PostgreSQL: `localhost:9999`
+
+## Notes
+
+1. This folder runs the complete local performance stack, not just the bulk loader.
+2. For database-only loading, use the `databaseBulkDataLoader` subfolder.
+3. If you change CPU/memory values, restart the stack to apply updates.

test/performance/README.md

@@ -13,6 +13,14 @@ The runner uses the official `postgres:16-alpine` image only as a client contain
 
 The loader does not generate schema. It validates required tables first and exits with an error if any are missing.
 
+## Example .env file
+
+```env
+PG_CONN_STRING=postgresql://postgres:admin@localhost:9999/twinengine
+ASSET_COUNT=1000
+BATCH_SIZE=100
+```
+
 ## Configuration
 
 The loader reads only environment variables:
@@ -21,14 +29,6 @@ The loader reads only environment variables:
 2. `ASSET_COUNT` (optional, defaults to `1000`)
 3. `BATCH_SIZE` (optional, defaults to `100`)
 
-Create your runtime file:
-
-```bash
-cp .env.example .env
-```
-
-Then update at least `PG_CONN_STRING`.
-
 ## Run
 
 From this folder: