Skip to content

SLA_SLO.md

Latest commit

 

History

History
55 lines (40 loc) · 2.73 KB

SLA_SLO.md

File metadata and controls

55 lines (40 loc) · 2.73 KB
Raw

Service Level Agreements (SLA) & Service Level Objectives (SLO)

This document outlines the Service Objectives and Alert Thresholds implemented for the TalentTrust Backend operations. Setting these operational metrics allows us to proactively track performance, ensure high availability, and configure reliable alerts when our service degrades.

Core Concepts

  • Service Level Agreement (SLA): A contractual commitment to maintain specific performance benchmarks. This backend focuses on providing the tools and metrics to enforce internal SLAs.
  • Service Level Objective (SLO): Our internal target for a given operation. For example, maintaining a 99.9% success rate on the API.
  • Alert Thresholds: The actionable limits where error rates or response latencies degrade and require developer or operator intervention.

Defined Objectives

Health Check (/health)

  • Target Success Rate: 99.99%
  • Target Latency (P95): 50ms
  • Alert Trigger: Error Rate $\ge$ 0.1% OR Average Latency $\ge$ 150ms over a 5-minute rolling window.

Contracts API (/api/v1/contracts)

  • Target Success Rate: 99.9%
  • Target Latency (P95): 200ms
  • Alert Trigger: Error Rate $\ge$ 1.0% OR Average Latency $\ge$ 400ms over a 5-minute rolling window.

Usage in Codebase

The definitions are maintained within src/operations/service-objectives.ts. We use NatSpec-style comments to document types, thresholds, and objectives to align with secure systems and standard architectures.

import { isThresholdBreached, DefaultAlertThresholds } from './operations/service-objectives';

// Example: Evaluating if a breach occurred
const hasBreached = isThresholdBreached(
  DefaultAlertThresholds.contractsApi,
  currentErrorRate,      // e.g., fetched from metrics store like Prometheus
  currentAverageLatency  // e.g., measured via APM
);

if (hasBreached) {
  // Trigger PagerDuty, Slack alert, or perform fallbacks
}

Expanding Metrics

To define SLOs for new API routes or operations:

  1. Open src/operations/service-objectives.ts
  2. Define a new entry inside DefaultServiceObjectives.
  3. Define the corresponding alerting limits in DefaultAlertThresholds.
  4. Ensure relevant automated tests run cleanly.

Security and Threat Assumptions

Since SLO and SLA data can be sensitive and used for operational integrity:

  • Metric Spoofing: Ensure that metrics collected for the isThresholdBreached function are generated by trusted internal observability tools, preventing malicious external sources from triggering false positive alerts (DDoS on monitoring).
  • Efficiency: Evaluation logic is kept completely synchronous $O(1)$ and lightweight to prevent evaluating thresholds from becoming a bottleneck during high-load periods.