Skip to content

design-documents

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

design-documents

README.md

GitLab Orbit

Current State

This repository implements both Orbit Remote and Orbit Local. They share the Rust workspace, Code Graph pipeline, and ontology, but use different storage and query surfaces.

Today, the repository includes the following major components:

  • Orbit Remote's gkg-server binary, which runs in four modes: Webserver, Indexer, DispatchIndexing, and HealthCheck.
  • A ClickHouse-backed remote graph runtime with ontology-driven schema, ETL, authorization metadata, and Query DSL validation. The authoritative ontology lives in config/ontology/.
  • Remote HTTP, gRPC, REST, and MCP query surfaces that compile the JSON Query DSL into parameterized ClickHouse SQL. The Orbit query frontend provides a separate compiler-level text API.
  • A distributed remote indexing pipeline that consumes Siphon CDC through NATS JetStream, dispatches indexing work, and writes SDLC and code graph data into ClickHouse.
  • Orbit Local's standalone orbit CLI, which indexes a repository into DuckDB and supports direct SQL, schema inspection, repository maps, and a stateless stdio MCP server.
  • Shared crates for indexing, query compilation, formatting, ontology loading, database access, GitLab API access, health checks, and integration testing.

The Kuzu-backed local desktop architecture from the earlier gitlab-org/rust/knowledge-graph project remains historical context. It is distinct from the current DuckDB-backed Orbit Local product implemented in this repository.

Orbit Architecture High-Level Overview

Orbit Remote is primarily a data engineering system. It builds on the Data Insights Platform to index code and SDLC metadata as a distributed system. For an end-to-end overview of the Data Insights Platform (logical replication, NATS JetStream, ClickHouse), see the Data Insights Platform design doc.

A secure query layer on top of the graph lets developers and AI agents query that data. The current runtime components are:

  • Siphon, the CDC bridge that streams PostgreSQL logical replication events into NATS.
  • NATS, the durable message broker for CDC and event-driven work such as consuming p_knowledge_graph_code_indexing_tasks for code indexing (see ADR 005).
  • ClickHouse, the remote datalake and property-graph store. Orbit does not connect directly to the GitLab OLTP database.
  • Orbit Remote's service binary, with four runtime modes:
    • Webserver (gkg-server --mode Webserver): Serves HTTP, gRPC, REST, and MCP traffic; validates Query DSL requests against the JSON schema and ontology; compiles them to ClickHouse SQL; and applies authorization and formatting before returning results.
    • Indexer (gkg-server --mode Indexer): Runs the shared indexing engine, consumes SDLC and code indexing requests from NATS JetStream, and writes graph data into ClickHouse.
    • DispatchIndexing (gkg-server --mode DispatchIndexing): On a schedule, detects enabled root namespaces with recent Siphon changes and publishes deduplicated per-namespace indexing requests to the internal GKG_INDEXER stream. It also runs scheduled dispatchers for code indexing tasks, namespace deletion, stale-edge reconciliation, and schema-migration lifecycle, including ontology archive publication.
    • HealthCheck (gkg-server --mode HealthCheck): Aggregates cluster health by probing Kubernetes deployments and ClickHouse instances, and exposes the result on a single /health endpoint.
  • The orbit CLI, which exposes one flat command tree. Local commands parse repositories, store code graphs in DuckDB, accept read-only DuckDB SQL, and serve MCP tools over stdio; hosted commands query Orbit Remote. glab orbit installs and runs this binary.
  • UI and product experiences, which consume Orbit Remote through GitLab APIs and the GitLab Duo Agent Platform.
graph TD
    RailsAPI["GitLab Rails API Server"]
    Postgres["PostgreSQL"]
    DataInsights["Data Insights Platform"]
    KGService["Orbit"]

    subgraph Omnibus["Omnibus VM"]
        RailsAPI
        Postgres
    end

    subgraph KGS["Omnibus Adjacent - K8s"]
        DataInsights
        KGService
    end

    RailsAPI -->|Transactions| Postgres
    Postgres -->|Logical replication stream| DataInsights
    DataInsights <-->|Snapshots & change feeds| KGService
    RailsAPI -.->|queries| KGService
    KGService <-.->|Authorization checks, Redaction| RailsAPI

    class RailsAPI rails;
    class Postgres postgres;
    class DataInsights,KGService services;
    class Omnibus,KGS clusters;

    classDef rails fill:#FCA32,stroke:#E24329,stroke-width:3px,rx:15,ry:15;
    classDef postgres fill:#6E49CB,stroke:#5C3CAB,stroke-width:3px,rx:15,ry:15;
    classDef services fill:#FC6D26,stroke:#E24329,stroke-width:2px,rx:15,ry:15;
    classDef clusters fill:none,stroke:#6E49CB,stroke-width:2px,rx:10,ry:10;
Loading

Design Documents

Please see the following design documents for more details on the Orbit architecture:

Runtime Breakdown

Orbit Remote deploys the four gkg-server modes shown below. Orbit Local runs separately as the orbit CLI on a user's machine and stores its graph in ~/.orbit/graph.duckdb.

flowchart TD
  %% ====== Styles ======
  classDef bin fill:#eef2ff,stroke:#4338ca,color:#111,rx:12,ry:12;
  classDef queue fill:#fff7ed,stroke:#fb923c,color:#7c2d12,rx:12,ry:12;
  classDef db fill:#ecfeff,stroke:#06b6d4,color:#0e7490,rx:12,ry:12;
  classDef ext fill:#f5f5f5,stroke:#9ca3af,color:#111,rx:12,ry:12;

  %% ====== Environments ======
  subgraph GitLab_VM[GitLab VM]
    PG[(PostgreSQL)]
    RailsAPI[GitLab Rails / Duo / UI]:::ext
  end

  subgraph GKE_Cluster[GKE Cluster]
    subgraph SiphonGrp[Siphon - Go]
      SiphonMain[siphon-main]
      SiphonCI[siphon-ci]
    end

    subgraph NATSGrp[NATS JetStream]
      JS[(Streams & Consumers)]
    end

    subgraph KGGrp[Orbit - Rust]
      IDX[gkg-server --mode Indexer]
      WEB[gkg-server --mode Webserver]
      DSP[gkg-server --mode DispatchIndexing]
      HC[gkg-server --mode HealthCheck]
    end

    subgraph CHGrp[ClickHouse]
      CH[(ClickHouse Server / Cluster)]
    end
  end


  %% ====== Data Flow ======
  PG -- Logical replication --> SiphonMain
  PG -- CI-related replication --> SiphonCI
  SiphonMain --> JS
  SiphonCI --> JS

  JS -->|CDC events| DSP
  DSP -->|internal indexing requests| JS
  JS -->|indexing requests| IDX
  IDX -- archive download --> RailsAPI

  %% Durable store: ClickHouse only
  IDX -- INSERT/UPSERT graph tables --> CH
  WEB -- SELECT / Analytics --> CH

  %% Control/coordination
  RailsAPI -. HTTP/gRPC .-> WEB
  WEB --> |kv / subscriptions| JS
  JS --> |namespace cache, node registration| WEB

  %% ====== Classes ======
  class SiphonMain,SiphonCI,IDX,WEB,DSP,HC bin
  class JS queue
  class CH db
  class RailsAPI ext
Loading

Database & Database Ops

Orbit Remote's Graph Query Engine validates the JSON Query DSL against the ontology and compiles traversal, aggregation, neighbors, and path-finding requests into parameterized ClickHouse SQL. Cypher was evaluated during the storage and query-engine design, but it is not a current query surface. The JSON Query DSL and the compiler-level Orbit query frontend share the same compiler pipeline.

The current implementation uses ClickHouse for remote graph storage and query execution. In the current repository state:

  • Graph nodes live in typed gl_* ClickHouse tables such as gl_group, gl_project, gl_merge_request, gl_pipeline, gl_job, gl_vulnerability, gl_branch, gl_file, gl_definition, and gl_imported_symbol.
  • Relationships are stored in ontology-configured edge tables (defaulting to gl_edge) with adjacency-optimized ordering and projections. Each edge YAML can specify a table: field to route relationship types to dedicated tables; settings.edge_tables in schema.yaml defines available tables.
  • Code indexing progress is tracked in code_indexing_checkpoint.
  • The ontology in config/ontology/ defines the mapping between entity names, properties, redaction metadata, ETL sources, and relationship kinds.

Orbit Local generates its DuckDB tables from the same ontology, then writes Code Graph nodes and relationships into a workspace database. Local queries use read-only DuckDB SQL directly rather than the remote Query DSL and authorization pipeline. Release binaries statically link DuckDB's full-text search extension from a pinned source archive; development builds load the pinned extension artifact at runtime. Regenerate the source archive with scripts/duckdb/vendor-duckdb-fts-sources.sh.

ClickHouse was chosen over dedicated graph databases (Neo4j, FalkorDB, Memgraph, Neptune, SpannerGraph) after KuzuDB was archived in October 2025. The full evaluation, benchmarking results, and legal/procurement context are recorded in ADR 000: ClickHouse as graph storage.

View the Graph Query Engine design document for more details.

Motivation

Problem Statement

Modern software development operates across a complex web of repositories, issues, merge requests, CI/CD pipelines, deployment environments, infrastructure, and assets. Both code data and SDLC platform metadata are inherently interconnected network graphs. While GitLab is a single vehicle to deliver these collective features, our ability to consume and analyze this data is fragmented, forcing developers and AI agents to piece together context through dozens of API, GraphQL, and Agent-tool calls.

GitLab has hundreds of REST APIs and GraphQL Schema Elements. AI agents and data products need to be able to reason about GitLab data in a way that is impractical with traditional data-fetching techniques.

Solution

Orbit Remote runs outside GitLab Rails and exposes a unified data API for developers and AI agents to query across the software development lifecycle. Orbit Local brings the same Code Graph model to a repository on the user's machine without requiring the remote service.

Both products represent connected data as a Property Graph. Orbit Remote uses a ClickHouse-backed Graph Query Engine, while Orbit Local exposes its DuckDB graph through read-only SQL and MCP tools.

The current products index two types of conceptual data in property-graph form:

  • Code: The shared Code Graph engine indexes repositories, definitions, references, imports, and filesystem structure. Orbit Remote indexes default branches from GitLab, while Orbit Local indexes checked-out repositories.
  • SDLC: Orbit Remote indexes GitLab entities such as merge requests, pipelines, Work Items, groups, projects, and their relationships through the Data Insights Platform.

User-defined graph entities remain a possible future extension and are not part of the current product.

Why a Property Graph? Why not a REST and GraphQL layer?

While GitLab is powered by two primary data stores (Postgres and Git), GitLab data, including source code, can be represented in a network graph. See the diagram below as an example:

graph TD

    %% === STYLE DEFINITIONS ===
    classDef group fill:#ffeaea,stroke:#cc4444,stroke-width:2px,color:#aa2222,font-weight:bold;
    classDef project fill:#ffe8cc,stroke:#ff9900,stroke-width:2px,color:#cc6600,font-weight:bold;
    classDef epic fill:#fff5d6,stroke:#ffaa33,stroke-width:2px,color:#cc7700,font-weight:bold;
    classDef issue fill:#fff8e1,stroke:#ffb347,stroke-width:2px,color:#cc7a00,font-weight:bold;
    classDef mr fill:#fff1d6,stroke:#ff9900,stroke-width:2px,color:#cc6600,font-weight:bold;
    classDef file fill:#fff7e6,stroke:#ffaa33,stroke-width:2px,color:#cc7700,font-weight:bold;
    classDef codeGraph fill:#fffbe6,stroke:#ffcc33,stroke-width:2px,color:#cc9900,font-weight:bold;
    classDef classNode fill:#e7f1ff,stroke:#80aaff,stroke-width:1.5px,color:#004499;
    classDef methodNode fill:#ecffe7,stroke:#66cc66,stroke-width:1.5px,color:#006600;

    %% === GITLAB HIERARCHY ===
    subgraph "Parent Group"
        PG((Parent<br/>Group))
        EG((Epic A))
    end

    subgraph "Subgroup"
        SG((Subgroup))
        E2((Epic B))
    end

    subgraph "Project"
        PJ((User Service<br/>Project))
        IS1((Issue 1:<br/>Add Authentication))
        IS2((Issue 2:<br/>Fix Auth Bug))
        MR1((MR 1:<br/>Add Authentication))
        MR2((MR 2:<br/>Fix Auth Bug))
        F1((fileA.ts))
        F2((fileB.ts))
        CG((Code Graph))
    end

    %% === CODE GRAPH EXPANSION ===
    subgraph CODEGRAPH["Code Graph (Expanded)"]
        direction LR
        C1(("Class Foo"))
        M1(("hello()"))
        C2(("Class Bar"))
        M2(("myMethod()"))
        C1 -->|"defines"| M1
        C2 -->|"defines"| M2
        M1 -.->|"calls"| M2
    end

    %% === CONNECTIONS BETWEEN LAYERS ===
    PG --> SG
    PG --> EG
    SG --> E2
    SG --> PJ
    EG <-->|relates to| E2

    EG -->|contains| IS1
    E2 -->|contains| IS2

    PJ --> MR1
    PJ --> MR2
    MR1 -->|closes| IS1
    MR2 -->|closes| IS2

    MR1 --> F1
    MR1 --> F2
    MR2 --> F1
    MR2 --> F2
    IS1 --> |blocks| IS2

    F1 -->|belongs to| CG
    F2 -->|belongs to| CG
    CG --> CODEGRAPH

    %% === STYLE ASSIGNMENTS ===
    class PG,SG group
    class PJ project
    class EG,E2 epic
    class IS1,IS2 issue
    class MR1,MR2 mr
    class F1,F2 file
    class CG codeGraph
    class C1,C2 classNode
    class M1,M2 methodNode
Loading

1. AI Works Best with Property Graph Data Models

Our research and live prototypes showed that LLMs reliably generate property graph tool calls because their structure mirrors natural “find-things-connected-to-X” reasoning. The current Query DSL represents that structure with explicit node selectors, typed relationships, filters, and bounded hops.

By contrast, GraphQL and REST require schema introspection and nested field expansion. LLMs struggle to reason about variable-depth recursion or dynamic joins inside those structures.

2. We Need Arbitrary Neighbor Exploration and Path Finding (N-Hop Queries)

Many Orbit workloads involve exploring neighbors and path finding up to N levels deep—for example, finding “all pipelines triggered by MRs that close issues linked to epics under a group.” Neither REST nor GraphQL provides a clean or efficient way to express variable-length traversal:

  • REST would require chained requests or recursive pagination.
  • GraphQL can express limited nesting but not dynamic-depth traversal (*..N); resolvers explode in complexity and performance cost.

The Query DSL makes these traversals first-class through typed relationships and bounded hops, and the compiler turns them into authorization-scoped ClickHouse SQL.

3. Aggregations and Analytics Are Essential

Orbit is not just a document API—it is an analytical OLAP system. For example, this query counts work items by project within a group. Replace your-group/ with your group's full path before running it.

{
  "query": {
    "query_type": "aggregation",
    "nodes": [
      {
        "id": "p",
        "entity": "Project",
        "columns": ["name"],
        "filters": {"full_path": {"starts_with": "your-group/"}}
      },
      {"id": "w", "entity": "WorkItem"}
    ],
    "relationships": [
      {"type": "IN_PROJECT", "from": "w", "to": "p"}
    ],
    "group_by": ["p"],
    "aggregations": [
      {"count": "w", "as": "work_item_count"}
    ],
    "aggregation_sort": "-work_item_count",
    "limit": 10
  }
}

Implementing equivalent groupings via GraphQL or REST would either require bespoke endpoints or push heavy joins into the application layer. Orbit executes the compiled aggregation in ClickHouse, using adjacency-oriented graph tables and columnar execution.

4. Schema Flexibility and Evolution are Essential

Customers will eventually need to be able to add their own data to the graph. Additionally, Orbit’s schema must evolve rapidly as new GitLab SDLC entities (e.g., vulnerabilities, packages, runners) appear.

Orbit declares node types, relationships, properties, and pipelines in one ontology. The same declarations drive indexing, query validation, authorization metadata, and storage code generation, so existing Query DSL requests remain stable as the graph grows. User-defined data types remain a possible future extension.

This makes property graphs a flexible choice for both AI-driven analytics and human consumers.

5. Property Graphs are Standard

Property graphs are standardized by SQL 2023's ISO/IEC 9075-16:2023. The team evaluated Cypher and dedicated graph databases during the original storage and query-language design; those decisions remain documented in ADR 000. The implemented client contract is the JSON Query DSL, which expresses traversals, aggregations, neighbors, and path finding without exposing generated SQL.

See the querying design documents for the current query architecture.

Orbit is OLAP, not OLTP

Orbit is an OLAP application over an OLTP one. Orbit Remote is a read-only analytical data store and retrieval API for code and SDLC metadata; it provides point-in-time indexed results rather than transaction guarantees or real-time data. Orbit Local is also read-only at query time, while explicit indexing commands update its DuckDB graph.

Architecture Goals

The architecture is guided by the following goals. The linked design documents describe the implemented controls and any remaining gaps:

Security

  • Build a multi-tenant architecture so that supports isolating customer data in the database tables.
  • Add authorization layers and query filtering throughout the service to restrict access to graph nodes to only the necessary data based on the user's permissions.
  • Implement a thorough review process and security tests for SDLC metadata indexing algorithms to ensure customer data is not indexed incorrectly.
  • Sanitization, redaction, and validation of all input queries to prevent injection and DDoS attacks.

Please see the Security Design Document for more details.

Operational Observability

  • Integrate observability metrics into the service to monitor the service and database health and performance.
  • Add logging and tracing best practices (LabKit integration) to the service.
  • Provide observability to both self-managed and .com customers.

Please see the Observability Design Document for more details.

Distributed Scalability & Reliability

  • Build a multi worker deployment architecture to provide high availability and redundancy for the web service and indexing services.
  • Leverage the same architecture and codebase for both Code Indexing and SDLC Metadata Indexing.

For how we will achieve this, please see the following design documents:

FinOps & Maintainability

  • Aim for cost-effective storage and compute resource requirements for self-managed customers with resource and budget constraints.
  • For .com, aim to scale up and down based on demand.
  • Build a flexible data model configuration layer for:
    • Handling Rails database migrations and schema changes.
    • Adding new data source entities to the graph.

Please see the Orbit Data Model design document for more details on the data model.

Follow up MR: New section/page on deployment resource requirements, and plan for operations past day 30.

Maintainability & Delivery

  • Aim for independent upgrade & security patching without Rails as a dependency to enable continuous delivery for both .com and self-managed customers.
  • The design of the service should incorporate Cells architectural implications where applicable.
  • Build a data model configuration layer for SDLC metadata sources to support Rails database migrations and schema changes, and to add new data source entities.
  • Avoid requiring customers to run any scripts (Rake tasks) to fix data state issues.

Code Indexing vs SDLC Metadata Indexing

Our eventual goal is to have a unified Code Indexing and SDLC data graph. It's important to distinguish between the two to understand their conceptual differences and similarities. The primary point to understand is that both indexing pipelines and query services can reuse the same architecture and codebase.

Here are the differences and similarities between the two:

Differences

  • Where the data starts:
    • Code indexing reads repositories directly—no database required. Everything is inferred from the files themselves.
    • Namespace indexing listens to GitLab PostgreSQL through Siphon, stages CDC events and queries them in ClickHouse
  • What transformation looks like:
    • Code indexing is parser-driven and produces a call graph plus file system hierarchy.
    • Namespace indexing is SQL-on-lake-driven, calculating nodes for namespaces, projects, issues, merge requests, pipelines, runners, and vulnerabilities, along with their relationships.
  • How the load phase runs:
    • Code indexing writes ephemeral Parquet files, then creates or updates the indices in ClickHouse. Incremental updates are calculated by determining the ref's changed files and index deltas.
    • Namespace indexing will calculate the nodes and edges from GitLab data either through direct SQL ETL statements in ClickHouse, or perform ETL via ClickHouse queries and streaming, and insert the data back into ClickHouse.

Similarities

  • What they share:
    • Both indexing pipelines and query services can reuse the same architecture and codebase, as they can share the same graph technology, indexing patterns, and architectural components. Because we are leveraging the Data Insights Platform, we will be able to share the same ingesters, NATS JetStream, and database (ClickHouse). This allows us to have an event-driven platform for both Code Indexing and SDLC Metadata Indexing.
    • They will share the same codebase, observability patterns, and security patterns.

Iteration Plan

While the topics of Code Indexing and SDLC Indexing differ in how they index, the architecture above provides a platform for both Code Indexing and SDLC data.

Please see the Code Indexing and SDLC Metadata Indexing design documents for more details.

Phase 1 - SDLC Metadata Indexing and Project Level Code Graph Indexing

This is still the current implementation shape in the repository. SDLC and code data share the same codebase, ontology-driven graph model, and API layer, but they remain operationally distinct in how they are indexed and stored:

  • SDLC data is loaded into typed gl_* node tables plus ontology-configured edge tables (defaulting to gl_edge) using namespaced ETL driven by the ontology.
  • Code data is loaded into gl_branch, gl_directory, gl_file, gl_definition, gl_imported_symbol, and the ontology-configured edge table(s), keyed by traversal_path, project_id, and branch.
  • Cross-graph linkage happens through shared entity identifiers and shared relationship semantics rather than by collapsing everything into a single undifferentiated store.

Workstreams Roadmap

The first release of Orbit will be delivered through three parallel workstreams. The successful deployment of these components is primarily dependent on aligning cross-team resourcing, as the core technical components are largely complete.

  • Producer-Only Siphon Production Deployment: The goal is to deploy a producer-only Siphon to GitLab.com to validate end-to-end logical replication from PostgreSQL. This stream is currently blocked on DBRE and Infrastructure resourcing.

  • NATS Unit-Level Production Deployment: Deploy NATS at the cell level for localized event distribution. NATS is already underway for global usage billing event tracking but further progress is blocked on engineering resources.

  • ClickHouse Work (Consumers + Data Modeling): This involves deploying consumers to ingest data from NATS, operationalizing ClickHouse as the data lake and extending the schema to cover SDLC and CI/CD data. This work is blocked by the NATS unit-level deployment.

Dedicated & Self-Managed

Work for Dedicated and Self-Managed customers is blocked until the GitLab.com deployments of Siphon and NATS are complete.

Phase 2 - Multi-Project Code Graphs

In the second phase, we will allow users to query across multiple projects simultaneously. This is a challenging engineering problem, as we will need to handle the scale of the graph and the performance of queries based on the number of nodes and relationships.

We need to assess the benefits of cross-graph analysis before determining where MCP tools can handle most cases, or whether there are ways to create unified graphs in a federated manner.

Phase 3 - Unified Code and SDLC Metadata Graphs

Phase 3 has similar engineering challenges as Phase 2, but with the added complexity of handling both code and SDLC metadata in unified queries.