Enhance observability with Alloy service and telemetry routing

[ExtraToast]

This pull request migrates the observability stack from Promtail to Grafana Alloy, standardizes OpenTelemetry (OTEL) resource attributes and environment variables across services, upgrades Tempo to a newer version, and updates documentation to reflect the new observability architecture. The changes impact both Docker Compose and Nomad deployments, ensuring a unified, modernized observability pipeline for logs, traces, and metrics.

Observability Stack Migration and Service Updates:

Migration from Promtail to Alloy:

• Replaces Promtail with Grafana Alloy for log collection in both Docker Compose and Nomad, including new Alloy service definitions, configuration files, and removal of Promtail jobs/configs. [1] [2] [3] [4] [5] [6] [7]

Tempo and Grafana Configuration:

• Upgrades Tempo to version 2.10.3 and adjusts its configuration to keep metrics generation enabled for service graphs and span metrics. Updates Grafana datasource volume mounts for both Compose and Nomad. [1] [2] [3] [4] [5]

Standardization of OTEL and Service Metadata:

Environment Variable Consistency:

• Adds and standardizes DEPLOYMENT_ENVIRONMENT, SERVICE_VERSION, and OTEL_RESOURCE_ATTRIBUTES across all core application and infrastructure services in both Docker Compose and Nomad. Ensures all services export traces to Alloy and use consistent OTEL exporter endpoints. [1] [2] [3] [4] [5] [6]

Documentation and Architecture Updates:

Documentation Overhaul:

• Updates architecture docs to reflect the new Alloy-based logging pipeline, clarifies the metrics and tracing model, and adds a detailed OBSERVABILITY.md outlining the stack's default architecture, signal ownership, metrics policy, and platform defaults. [1] [2] [3]

Configuration and Wiring Improvements:

Datasource and Volume Mount Adjustments:

• Refines Grafana provisioning by mounting specific datasource files instead of entire directories, and adds new persistent volumes for Alloy data. [1] [2] [3]

Endpoint and Port Changes:

Collector Endpoint Updates:

• Changes OTEL exporter endpoints in all services to point to Alloy (port 4318 for Compose, 4319 for Nomad), and updates Traefik tracing configuration to use the new Alloy endpoint. [1] [2] [3] [4] [5]

These changes collectively modernize the observability stack, improve maintainability, and ensure consistent telemetry across all environments.