Zero-egress AI for the NOC: root-cause analysis, alarm-storm correlation, and human-gated self-healing for 5G networks — telemetry never leaves your machine. Built on LLM OS — the model, the reasoning, and the data all stay local.
Live, unedited: an injected backhaul storm is correlated to its root cause, the agent proposes a fix, an unauthorized user is DENIED, the NOC manager approves, execution heals the network (issues 6→5, alarms 9→7) — and the AI console answers through the audited kernel. Full video: docs/dashboard_demo.mp4.
Network telemetry — KPIs, alarms, cell topology, subscriber-adjacent logs — is regulated, sovereign data. Data-localization rules and operator security policy make cloud LLMs a non-starter in the NOC. TelecomOS runs the entire diagnostic and remediation loop on a local model, works air-gapped, and writes every decision into a tamper-evident audit chain.
Diagnose
- LangGraph 5-node RCA pipeline — Classify → Retrieve → Analyze → Recommend → Validate, on a local 7B model (no API keys, flat cost)
- Instant network-wide triage — deterministic rules, no LLM call (10 cells in ~12 ms)
- Alarm-storm correlation — separates the probable root alarm from downstream symptoms by site topology and causal ranking
- Cluster impact scoping — isolated cell vs site-wide vs area-wide (dispatch the field team, or don't)
- Trend analysis — which KPIs are worsening, and when each first breached threshold
Act (safely)
- Closed-loop remediation with human approval gates — the agent may propose, never execute. Execution is mechanically refused unless the action is
APPROVED, and approval requires thenoc_managerrole held by an authenticated console operator — the agent has no approval tool, so it cannot self-approve. Proposed → approved(by) → executed → verified is the audit record. If verification finds the cell has NOT recovered, the next catalogued action is auto-proposed — again behind the human gate — until the catalog is exhausted and it escalates. →docs/ENTERPRISE.md - Execution guardrails — approvals expire (TTL), per-site blast-radius caps, and
maintenance.jsonchange-freeze windows: every check a change-management process demands, mechanical and fail-closed - Live network simulator — injectable fault scenarios behind the same connector interface real OSS plugs into, so the whole loop is demonstrable end-to-end today
Know
- 📄 3GPP spec answers with clause citations — offline RAG over your spec library: "what does the spec say about RLF?" →
3GPP TS 38.331 § 5.3.10, air-gapped - 📝 Incident reports — full RCA formatted as a ticket-ready markdown document
- ⚙️ Config audit — 7 KPI baseline checks + alarm-to-config mapping, 0–100 score
- 🧠 Incident memory — past incident reports are indexed locally; diagnosis can cite precedent: "same signature as INC-0042 — the fix that worked was X"
- 🔁 Proactive monitoring — sweeps that record every degradation/recovery; "has this happened before?"
Integrate
- 📥 Bring-your-own-data — map any vendor KPI export (Ericsson/Nokia/Huawei column names, id formats, units) with one YAML:
examples/vendor_mapping.example.yaml - 📡 Real-data ingestion — file drop — point
TELECOMOS_PM_DROP_DIRat wherever your EMS already writes periodic performance files (3GPP TS 32.435 PM-XML or CSV) and every tool reads it live, no code changes, no network stack:examples/pm_xml_sample.xml·./scripts/filedrop_demo.sh - 22 MCP tools — plug into LLM OS (or any MCP host); every call digest-pinned, egress-monitored, hash-chain audited
- Cloud strictly opt-in — OpenAI/Bedrock still exist behind
LLM_BACKEND, but the default cannot leak
NOC dashboard (Streamlit) LLM OS kernel ── chat, routing,
grid · storms · approvals (local model) audit chain,
│ │ egress sentinel,
│ │ MCP model pinning
▼ ▼
┌──────────────────────────────────────────────────────────────────┐
│ TelecomOS │
│ │
│ mcp_server.py ── 22 tools ──┬─ diagnostics (5-node RCA) │
│ ├─ triage · storms · trends │
│ ├─ spec RAG (cited) │
│ └─ remediation: propose → APPROVE │
│ (RBAC) → execute │
│ │ │
│ connectors/ ◄────────────────────┘ │
│ • SimulatorConnector (shipped) │
│ • FileDropConnector — PM-XML/CSV + alarm-file ingestion (shipped)│
│ • SNMP · Kafka · OSS execution (dry-run-first) (shipped) │
│ │ │
│ data: KPI CSV · alarms JSON · logs · topology · specs │
└──────────────────────────────────────────────────────────────────┘
everything on this machine
./scripts/run_dashboard.sh storm # or: congestion | (no arg = recorded data)A dark console at localhost:8502 (loopback-only — nothing is exposed on your network):
| View | What it shows |
|---|---|
| Network | Site/cell grid with live status + PRB / throughput / BLER trend charts |
| Alarms & storms | Active alarms, and storms correlated to a probable root cause |
| Remediation queue | The closed loop: propose → approve (RBAC-gated) → execute → verify, with the network healing in the stat tiles |
| AI console | Chat through the LLM OS kernel; every answer shows the tool used and its audit id |
Switch identity in the sidebar (viewer1 has no roles — watch approval get
denied; dilip is a noc_manager) and drive live faults with the simulation
controls. The sidebar also surfaces the kernel's egress sentinel and
model-pinning status.
Add to your LLM OS mcp_servers.json (absolute paths):
"telecom-diagnostics": {
"command": "/path/to/telecomos/.venv/bin/python",
"args": ["/path/to/telecomos/mcp_server.py"]
}The kernel discovers all 22 tools and routes NOC questions to them:
| Ask the chat… | Routed to | Speed |
|---|---|---|
| "Which cells have problems?" | network_health_check (deterministic) |
~12 ms |
| "Why is cell 3 degraded?" | diagnose_cell (full 5-node RCA) |
~40 s |
| "Correlate these alarms — what's the root cause?" | correlate_alarm_storm |
instant |
| "Propose a fix for cell 2" | propose_remediation (needs approval) |
instant |
| "Execute ACT-123" (after a human approves in the console) | execute_remediation |
instant |
| "Did the fix work?" | verify_remediation (proposes a gated follow-up if not) |
instant |
| "Approve ACT-123" | (no agent tool — approval is a human action in the NOC console) | — |
| "What does the spec say about RLF?" | search_specs (cited: TS + clause) |
instant |
| "Is it just cell 3 or the whole site?" | analyze_cluster_impact |
instant |
| "When did cell 3 start degrading?" | get_kpi_trend |
instant |
| "Anything new since the last check?" | monitor_sweep |
instant |
| "Has cell 3 degraded before?" | get_status_history |
instant |
| "Write an incident report for cell 7" | generate_incident_report |
~40 s |
| "Audit the config of cell 1" | audit_cell_config |
instant |
| "Start the backhaul_storm simulation" | start_simulation · stop_simulation |
instant |
| "Is the PM file drop working?" | file_drop_status |
instant |
| "Has this happened before?" (precedent) | find_similar_incidents |
instant |
| "What's about to break?" (prediction) | forecast_cell |
instant |
| KPIs / alarms / logs for a cell | get_cell_kpis · get_active_alarms · get_recent_logs |
instant |
Cell ids are fuzzy: "cell 3", "CELL_003" and "3" all resolve.
Real answer from the pipeline, fully offline:
You: Why is cell 3 degraded? ⚙ routed to
diagnose_cell· audited Agent: Root cause: a traffic spike drove RRC connected users to 265 and PRB utilization to 91%, collapsing throughput to 8 Mbps (congestion, confidence 0.99). Recommended: traffic shaping + QoS prioritization, …
Wi-Fi radio disabled on camera → 10-cell triage → full RCA on cell 3 → alarm
drill-down → audit chain verified, hashes on screen. Reproduce: ./scripts/noc_demo.sh
(kills the Wi-Fi radio itself). Video: docs/noc_demo.mp4.
The closed-loop demo — fault → propose → denied → approved → executed → healed —
is scripted too: ./scripts/selfheal_demo.sh.
Scored with the LLM OS eval harness on a 24-prompt telecom set, including trap questions that must not trigger diagnostics (definitional questions may answer directly or cite the spec library — both count as correct):
| local model | telecom routing accuracy |
|---|---|
| qwen2.5-coder (7B) | 96% |
| llama3.2 (3B) | 88% |
Requires Ollama — no API keys:
git clone https://github.com/Indianinnovation/telecomos.git
cd telecomos
ollama pull qwen2.5-coder # the local reasoning model
ollama pull all-minilm # embeddings for the spec RAG
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# Run the engine hardened: loopback-only, vendor cloud features OFF
OLLAMA_HOST=127.0.0.1:11434 OLLAMA_NO_CLOUD=1 ollama serve # (see LLM OS README)
./scripts/run_dashboard.sh storm # NOC dashboard → localhost:8502
uvicorn app.main:app --port 8000 # optional: the REST APITo deliberately use a cloud backend: LLM_BACKEND=openai with OPENAI_API_KEY
(and pip install langchain-openai). The default backend is local and cannot leak.
| Scenario | Cell | Expected issue |
|---|---|---|
| Throughput drop | CELL_001 | Congestion |
| High BLER | CELL_002 | BLER issue |
| Cell congestion | CELL_003 | Congestion |
| Low SINR | CELL_005 | Interference |
| Link failure | CELL_007 | Link failure |
| Power degradation | CELL_009 | Hardware fault |
Live fault injection: start_simulation("cell_congestion") or ("backhaul_storm")
— telemetry drifts, alarms fire, and approved remediations actually resolve them.
telecomos/
├── app/
│ ├── connectors/ # NetworkConnector contract + live simulator
│ ├── graph/agent.py # LangGraph 5-node RCA (local-first LLM)
│ ├── tools/ # kpi · alarms · logs · triage · trends · cluster
│ │ # · storms · monitor · reports · config · spec RAG
│ │ # · remediation (approval gates + RBAC)
│ ├── api/routes.py # FastAPI endpoints
│ └── main.py
├── data/ # sample telemetry · topology · rbac
├── specs/ # 3GPP spec library (replace with official docs)
├── docs/ENTERPRISE.md # safety model · SSO · fleet · connector roadmap
├── mcp_server.py # 22 MCP tools → LLM OS
├── dashboard.py # the NOC console
└── scripts/ # run_dashboard · noc_demo · selfheal_demo
docs/ENTERPRISE.md covers the closed-loop safety model,
SSO/OIDC identity (production roles come from the session, never from chat), fleet
deployment, and the connector contract for SNMP / Kafka / 3GPP PM-XML / OSS
execution — with an honest list of what is shipped versus what awaits a design
partner's environment.
- LLM OS — local agentic kernel: routing, audit chain, model pinning, egress sentinel
- Ollama / qwen2.5-coder — local reasoning (cloud strictly opt-in)
- LangGraph — RCA pipeline orchestration
- ChromaDB — offline spec index (telemetry disabled by policy)
- FastAPI · Streamlit · Plotly · Pandas
TelecomOS is the telecom vertical of the LLM OS platform: the same auditable local kernel, loaded with telecom organs. Same pattern, any regulated industry.
MIT