Skip to content

README.md

Latest commit

 

History

History
391 lines (312 loc) · 18.7 KB

README.md

File metadata and controls

391 lines (312 loc) · 18.7 KB
title Full Single Cluster Blueprint
description Complete end-to-end deployment of an Arc-enabled Kubernetes cluster with optional Azure IoT Operations on a single node, including all components from VM creation to cluster setup
author Edge AI Team
ms.date 2026-02-17
ms.topic reference
keywords
azure iot operations
single node cluster
kubernetes
arc-enabled
terraform
bicep
edge computing
vm deployment
k3s cluster
dataflow graphs
schema registry
estimated_reading_time 5

Full Single Cluster Blueprint

This blueprint provides a complete end-to-end deployment of an Arc-enabled Kubernetes cluster on a single node, with optional Azure IoT Operations (AIO). It deploys all necessary components from VM creation to cluster setup, resulting in a fully functional edge computing environment that integrates with Azure cloud services. Set should_deploy_aio = false to deploy an Arc-connected cluster without AIO. Please follow general blueprint recommendations from blueprints README.md.

Architecture

This blueprint deploys:

  1. A Linux VM host in Azure
  2. A K3s Kubernetes cluster on the VM
  3. Azure Arc connection for the cluster
  4. Cloud resources (Key Vault, Storage, Observability, Messaging, ACR)
  5. Azure IoT Operations components (optional, controlled by should_deploy_aio)
  6. Schema registry with versioned message schemas (when AIO is enabled)
  7. Optional messaging and observability components
  8. Optional dataflow graphs for WASM-based data processing pipelines (in Terraform only)
  9. Optional Preview Connectors for asset discovery (in Terraform only)

The resulting architecture provides a unified edge-to-cloud solution with secure communication, data processing capabilities, and comprehensive monitoring.

Implementation Options

This blueprint is available in two implementation options:

  • Terraform - Infrastructure as Code using HashiCorp Terraform
  • Bicep - Infrastructure as Code using Azure Bicep

Choose the implementation that best fits your team's expertise and existing pipelines.

Terraform Structure

This blueprint consists of the following key components:

  • Main Configuration (main.tf): Orchestrates the deployment workflow and module dependencies
  • Variables (variables.tf): Defines input parameters with descriptions and defaults
  • Outputs (outputs.tf): Exposes important resource information for future reference
  • Versions (versions.tf): Specifies provider versions and requirements

Key Modules Used in Terraform

Module Purpose Source Location
cloud_resource_group Creates resource groups ../../../src/000-cloud/000-resource-group/terraform
cloud_security_identity Handles identity and security resources ../../../src/000-cloud/010-security-identity/terraform
cloud_observability Sets up monitoring infrastructure ../../../src/000-cloud/020-observability/terraform
cloud_data Creates data storage resources ../../../src/000-cloud/030-data/terraform
cloud_messaging Sets up messaging infrastructure ../../../src/000-cloud/040-messaging/terraform
cloud_vm_host Creates the VM host for the cluster ../../../src/000-cloud/051-vm-host/terraform
cloud_acr Azure Container Registry ../../../src/000-cloud/060-acr/terraform
edge_cncf_cluster Deploys K3s Kubernetes cluster ../../../src/100-edge/100-cncf-cluster/terraform
edge_iot_ops Installs Azure IoT Operations ../../../src/100-edge/110-iot-ops/terraform
edge_observability Sets up edge monitoring ../../../src/100-edge/120-observability/terraform
edge_messaging Deploys edge messaging components ../../../src/100-edge/130-messaging/terraform

Variable Reference in Terraform

Beyond the basic required variables, this blueprint supports advanced customization:

Variable Description Default Notes
environment Environment type Required "dev", "test", "prod", etc.
resource_prefix Prefix for resource naming Required Short unique alphanumeric string (max 8 chars recommended)
resource_group_name Name of resource group null When null, name is generated from prefix, env, and instance
location Azure region location Required "eastus2", "westus3", etc.
instance Deployment instance number "001" For multiple deployments
should_get_custom_locations_oid Auto-retrieve Custom Locations OID true Set to false when providing custom_locations_oid
custom_locations_oid Custom Locations SP Object ID null Required for Arc custom locations
should_deploy_aio Deploy Azure IoT Operations true Set to false for Arc-only deployment without AIO
should_create_anonymous_broker_listener Enable anonymous MQTT listener false For dev/test only, not secure for production
should_create_aks Create Azure Kubernetes Service false When true, deploys AKS in addition to the K3s cluster
should_create_acr_private_endpoint Enable ACR private endpoint false Creates a private endpoint for the Azure Container Registry
aio_features AIO feature configurations null Map of feature settings for Azure IoT Operations
schemas Schema registry schemas Default List of schemas with versions for the schema registry
dataflow_graphs Dataflow graph definitions [] List of dataflow graphs with nodes and connections

For additional configuration options, review the variables in variables.tf.

Note: The aio_features variable is a map that allows you to specify feature flags for Azure IoT Operations. This can be used to enable or disable specific features based on your deployment needs. For example, you can use the following format of variables to enable the preview feature OPC UA asset discovery:

should_deploy_resource_sync_rules     = true

aio_features = {
  connectors = {
    settings = {
      preview = "Enabled"
    }
  }
}

Bicep Structure

This blueprint also provides a Bicep implementation with the following components:

  • Main Template (bicep/main.bicep): The primary deployment template that orchestrates the overall solution
  • Types Definition (bicep/types.core.bicep): Defines core parameter types and structures used throughout the deployment

The Bicep implementation follows the same architecture as the Terraform version, providing a native Azure Resource Manager (ARM) approach to deploying the same resources.

Key Modules Used in Bicep

Module Purpose Source Location
resourceGroup Creates the resource group Inline resource in main.bicep
cloudResourceGroup Sets up cloud resources ../../../src/000-cloud/000-resource-group/bicep
cloudSecurityIdentity Handles identity and security ../../../src/000-cloud/010-security-identity/bicep
edgeVmHost Creates the VM host for the cluster ../../../src/100-edge/051-vm-host/bicep
edgeCncfCluster Deploys K3s Kubernetes cluster ../../../src/100-edge/100-cncf-cluster/bicep
edgeIotOps Installs Azure IoT Operations ../../../src/100-edge/110-iot-ops/bicep

Parameter Reference in Bicep

The Bicep implementation uses a streamlined parameter approach with a Common object type and additional parameters:

Parameter Description Default Notes
common.resourcePrefix Prefix for resource naming Required Short unique alphanumeric string (max 8 chars recommended)
common.location Azure region location Required "eastus2", "westus3", etc.
common.environment Environment type Required "dev", "test", "prod", etc.
common.instance Deployment instance number Required For multiple deployments
useExistingResourceGroup Use existing resource group false When true, looks up a resource group instead of creating it
resourceGroupName Name of resource group Generated When empty, name is generated from common parameters
resourceGroupName Resource group name Auto-generated Uses pattern: rg-{prefix}-{environment}-{instance}
adminPassword VM admin password Required Important: always pass this securely
customLocationsOid Custom Locations SP Object ID Required Needed for Arc custom locations feature
shouldCreateAnonymousBrokerListener Enable anonymous MQTT listener false For dev/test only

Prerequisites

Ensure you have the following prerequisites:

  • Sufficient quota for a VM in your target region
  • At least 8 GB of RAM per VM, recommended 16 GB of RAM per VM
  • Registered resource providers (see deployment instructions)
  • Appropriate permissions to create resources

Advanced Configuration

Schema Registry Configuration in Terraform

This blueprint deploys schemas into the Azure Device Registry schema registry through the cloud_data module. A default temperature schema is included, and you can customize or extend schemas via the schemas variable.

Each schema supports:

  • Multiple named versions with independent content
  • Configurable format (defaults to JsonSchema/draft-07)
  • Display name and description metadata

Quick Example:

schemas = [
  {
    name         = "temperature-schema"
    display_name = "Temperature Schema"
    description  = "Schema for temperature sensor data"
    format       = "JsonSchema/draft-07"
    type         = "MessageSchema"
    versions = {
      "1" = {
        description = "Initial version"
        content     = "{\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"type\":\"object\",\"properties\":{\"temperature\":{\"type\":\"number\"}}}"
      }
    }
  }
]

Dataflow Graphs in Terraform

This blueprint supports deploying dataflow graphs for WASM-based data processing pipelines through the edge_messaging module. Dataflow graphs define processing pipelines with source, graph (WASM operator), and destination nodes connected to route and transform data.

Key Benefits:

  • Declarative configuration using Terraform variables
  • WASM-based graph operators from Azure Container Registry
  • Configurable node connections with optional schema references
  • Version control for pipeline definitions

Quick Example:

dataflow_graphs = [
  {
    name = "temperature-processing"
    nodes = [
      {
        nodeType = "Source"
        name     = "temperature-source"
        sourceSettings = {
          endpointRef = "default"
          dataSources = ["raw"]
        }
      },
      {
        nodeType = "Graph"
        name     = "temperature-map-custom"
        graphSettings = {
          registryEndpointRef = "acr-myprefix"
          artifact            = "graph-simple-map-custom:1.0.0"
        }
      },
      {
        nodeType = "Destination"
        name     = "temperature-destination"
        destinationSettings = {
          endpointRef     = "default"
          dataDestination = "processed"
        }
      }
    ]
    node_connections = [
      {
        from = { name = "temperature-source" }
        to   = { name = "temperature-map-custom" }
      },
      {
        from = { name = "temperature-map-custom" }
        to   = { name = "temperature-destination" }
      }
    ]
  }
]

See dataflow-graphs.tfvars.example for a complete configuration example.

REST HTTP Connector Assets in Terraform

This blueprint supports deploying REST HTTP connector devices and assets using the 111-assets component. This provides Infrastructure as Code (IaC) management of REST endpoints with full state management.

Key Benefits:

  • ✅ Declarative configuration using Terraform variables
  • ✅ Version control for asset definitions
  • ✅ Integrated with blueprint deployment
  • ✅ Replaces manual kubectl and YAML management

Quick Example:

namespaced_devices = [
   {
    name    = "rest-auth-device"
    enabled = true
    endpoints = {
      outbound = { assigned = {} }
      inbound = {
        "auth-device-endpoint" = {
          endpoint_type = "Microsoft.Http"
          address       = "http://auth-device:8082"
          version       = "1.0"
          additionalConfiguration = "{\"tlsEnabled\":false,\"timeoutSeconds\":45}"
          authentication = {
            method = "UsernamePassword"
            usernamePasswordCredentials = {
              usernameSecretName = "device-username"
              passwordSecretName = "device-password"
            }
          }
          trustSettings = null
        }
      }
    }
  }
]

namespaced_assets = [
  {
    name         = "rest-auth-device-asset"
    display_name = "Authenticated REST Device"
    enabled      = true
    device_ref = {
      device_name   = "rest-auth-device"
      endpoint_name = "auth-device-endpoint"
    }
    description  = "Secure REST HTTP device with basic authentication"
    manufacturer = "SecureIoT"
    model        = "SecureDevice-1000"
    serial_number = "SD-003"
    attributes = {
      deviceType     = "secure-device"
      security_level = "basic-auth"
    }
    datasets = [
      {
        name                  = "device-status"
        data_source           = "api/device/status"
        dataset_configuration = "{\"samplingIntervalInMilliseconds\":20000}"
        data_points = []
        destinations = [
          {
            target = "Mqtt"
            configuration = {
              topic  = "telemetry/company/cloud/region/environment/auth-device-01/status"
              retain = "Never"
              qos    = "Qos1"
            }
          }
        ]
      }
    ]
    default_datasets_configuration = "{\"publishingInterval\":20000,\"samplingInterval\":20000,\"queueSize\":1}"
  }
]

See rest-connector-assets.tfvars.example for complete configuration examples.

Deploy Blueprint

Follow detailed deployment instructions from the blueprints README.md, Detailed Deployment Workflow

Testing and Validation

This blueprint includes comprehensive test suites for validating deployments and maintaining code quality.

Test Directory

Location: tests/

The tests directory contains contract tests and end-to-end deployment validation for both Terraform and Bicep implementations.

Key Components:

  • Contract Tests - Fast static validation ensuring output declarations match test expectations (zero cost, no Azure resources)
  • Deployment Tests - Full end-to-end deployment validation with infrastructure creation and functional testing
  • Helper Scripts - run-contract-tests.sh and run-deployment-tests.sh for simplified test execution

See: tests/README.md for complete testing documentation including setup, usage, and troubleshooting

For Maintainers

When modifying this blueprint:

Before Making Changes

  1. Run contract tests to establish baseline: cd tests && ./run-contract-tests.sh both
  2. Review test structure in tests/outputs.go to understand output contract

After Making Changes

  1. Update output contract if adding/removing/renaming outputs:

  2. Run contract tests to verify declarations: cd tests && ./run-contract-tests.sh both

  3. Update deployment tests if changing deployment behavior:

  4. Run deployment tests to validate changes: cd tests && ./run-deployment-tests.sh <framework>

    • Valid framework values: 'terraform', 'bicep', or 'both'
    • Set CLEANUP_RESOURCES=true to auto-delete resources after testing
    • Expected duration: 30-45 minutes for full deployment

  5. Update documentation if changing parameters, modules, or architecture:

    • Update this README.md with new parameters or module references
    • Update tests/README.md if test behavior changes

🤖 Crafted with precision by ✨Copilot following brilliant human instruction, then carefully refined by our team of discerning human reviewers.