Skip to content

Commit 8e000d6

Browse files
jkoronaAtCiscomx-psi
jkoronaAtCisco
and
mx-psi
authored
[chore] Add implementation details to Component Configuration Management RFC (#14486)
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Added implementation details and examples to RFC document Changes were announced on `#otel-collector-dev` slack channel. <!-- Issue number if applicable --> #### Link to tracking issue Extends #14433 --------- Co-authored-by: Pablo Baeyens <pablo.baeyens@datadoghq.com>
1 parent 699932f commit 8e000d6

3 files changed

Lines changed: 264 additions & 11 deletions

File tree

‎.github/workflows/utils/cspell.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -180,6 +180,7 @@
180180
"cumulativetodelta",
181181
"cumulativetodeltaprocessor",
182182
"customname",
183+
"custompkg",
183184
"dataloss",
184185
"datapoints",
185186
"debugexporter",
@@ -278,6 +279,8 @@
278279
"httpprovider",
279280
"httpsprovider",
280281
"httptest",
282+
"icmpcheck",
283+
"icmpcheckreceiver",
281284
"illumos",
282285
"incorrectclass",
283286
"incorrectcomponent",
@@ -327,6 +330,7 @@
327330
"mowies",
328331
"muehle",
329332
"multiclient",
333+
"multierr",
330334
"multimod",
331335
"mycert",
332336
"mycomponent",

‎.markdownlint.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,5 +31,7 @@ MD053: false # unused link definitions
3131
MD058: false # blanks around tables
3232
MD059: false # descriptive link text
3333
MD060: false # table column style
34+
MD010: # no hard tabs
35+
code_blocks: false # tabs are valid in Go/YAML code blocks
3436
MD028: false # blank line inside blockquote
3537
MD037: false # spaces inside emphasis

‎docs/rfcs/component-configuration-schema-roadmap.md‎

Lines changed: 258 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -66,9 +66,7 @@ Use of the `schemagen` tool is dictated by the modularity of the Collector compo
6666

6767
```yaml
6868
config:
69-
allOf:
70-
- $ref: "go.opentelemetry.io/collector/scraper/scraperhelper#/$defs/ControllerConfig"
71-
- $ref: "#/$defs/MetricsBuilderConfig"
69+
type: object
7270
properties:
7371
targets:
7472
type: array
@@ -81,23 +79,264 @@ config:
8179
ping_count:
8280
type: integer
8381
description: "Number of pings to send"
84-
default: 3
82+
default: 4
8583
ping_interval:
8684
type: string
87-
format: duration
88-
x-customType: "time.Duration"
8985
description: "Interval between pings"
90-
default: "1s"
91-
required: ["host"]
86+
format: duration
87+
default: 300ms
88+
ping_timeout:
89+
type: string
90+
description: "Timeout of ping request"
91+
format: duration
92+
default: 4s
93+
required: [ host ]
94+
minItems: 1
95+
custom_field:
96+
go:
97+
custom: ['type', 'default', 'validate']
98+
required: [ targets ]
99+
allOf:
100+
- $ref: 'go.opentelemetry.io/collector/scraper/scraperhelper.controller_config'
101+
- $ref: './internal/metadata.metrics_builder_config'
92102
```
93103
94-
`#/$defs/MetricsBuilderConfig` would be automatically generated by mdatagen with the same process used to generate the go structs and documentation today.
104+
#### Schema specification details
95105
96-
`go.opentelemetry.io/collector/scraper/scraperhelper#/$defs/ControllerConfig` would be generated by the new tool from the schema definition in the scraperhelper component.
106+
- **format** of config schema follows roughly JSON Schema specification (https://json-schema.org/)
107+
- **special types** like time.Duration or time.Time can be represented as strings with specific formats (e.g., "duration", "date-time").
108+
- **dependencies** are specified with `$ref` attribute to reference other schema definitions, either from external packages or internal definitions.
109+
To identify references we need full package paths (for external) or use local paths (for internal). Examples:
110+
- `go.opentelemetry.io/collector/scraper/scraperhelper.controller_config` full package path with type name after the last dot if we want to import schema from external repository.
111+
- `/config/confighttp.client_config` absolute path with repository root as a base for internal references to other schema defined in the same repository.
112+
- `./internal/metadata.metrics_builder_config` relative path for internal reference to a schema defined in the same repository.
113+
114+
Dependencies from external packages not covered with schemas will be replaced by `any` type with actual type annotation.
115+
- **default values** can be specified using the `default` attribute.
116+
- **validation** is possible using standard JSON Schema attributes, see [docs](https://json-schema.org/draft/2020-12/json-schema-validation).
97117

98118
#### Extensibility
99119

100-
The YAML schema specification can be extended with custom fields (e.g., `x-customType`) to capture domain-specific types and validation rules that are not natively supported in JSON schema. Additionally, we may introduce custom fields that generate fields that will produce references to structs or validation functions that require more complex logic and manual implementation.
120+
The YAML schema specification can be extended with custom annotations to capture domain-specific types and validation rules that are not natively supported in JSON schema. Additionally, we may introduce custom fields that generate fields that will produce references to structs or validation functions that require more complex logic and manual implementation.
121+
122+
One proposal is to have `go` annotation that could contain the following sub-attributes:
123+
- type [string] - allows specifying a custom Go type for the field in a format `<package>.<type_name>`.
124+
- pointer [bool] - if set to true, the generated field will be a pointer to the specified type.
125+
- optional [bool] - if set to true, the generated field will be wrapped in Optional[...] generic type.
126+
- custom [list of strings] - allows specifying custom code snippets to be injected into the generated code.
127+
The possible values are: `type`, `default`, `validate`.
128+
Example:
129+
```yaml
130+
some_optional_field:
131+
type: string
132+
go:
133+
type: com.github/custom/custompkg.CustomType
134+
pointer: true
135+
optional: true
136+
custom: ['validate']
137+
```
138+
139+
### Generated Go code example
140+
141+
See below for an example of the generated Go code from the above schema:
142+
143+
```go
144+
package icmpcheckreceiver
145+
146+
import (
147+
"time"
148+
149+
"go.opentelemetry.io/collector/component"
150+
"go.opentelemetry.io/collector/scraper/scraperhelper"
151+
"go.uber.org/multierr"
152+
153+
"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/icmpcheckreceiver/internal/metadata"
154+
)
155+
156+
type Config struct {
157+
scraperhelper.ControllerConfig `mapstructure:",squash"`
158+
metadata.MetricsBuilderConfig `mapstructure:",squash"`
159+
Targets []TargetsItemConfig `mapstructure:"targets"`
160+
CustomField CustomFieldType `mapstructure:"custom_field"`
161+
}
162+
163+
// subtype for targets item generated from schema
164+
type TargetsItemConfig struct {
165+
Host string `mapstructure:"host"`
166+
PingCount int `mapstructure:"ping_count"`
167+
PingInterval time.Duration `mapstructure:"ping_interval"`
168+
PingTimeout time.Duration `mapstructure:"ping_timeout"`
169+
}
170+
171+
func (c *Config) Validate() error {
172+
var err error
173+
174+
if len(c.Targets) == 0 {
175+
return multierr.Append(err, errMissingTarget)
176+
}
177+
178+
for _, target := range c.Targets {
179+
if target.Host == "" {
180+
err = multierr.Append(err, errMissingTargetHost)
181+
}
182+
}
183+
184+
err = multierr.Append(err, ValidateCustomField(c.CustomField))
185+
186+
return err
187+
}
188+
189+
func createDefaultConfig() component.Config {
190+
return &Config{
191+
// default values derived from schema
192+
CustomField: DefaultCustomFieldValue(),
193+
}
194+
}
195+
196+
```
197+
198+
#### Generated code details
199+
200+
- The generated `config.go` file will define Go structs that mirror the configuration schema. It's capable
201+
of handling nested objects and arrays and creating subtypes with auto-generated names.
202+
- The `Validate` method will include validation logic based on the schema (e.g., required fields).
203+
- A `createDefaultConfig` function will be generated to provide default values for the configuration.
204+
205+
#### Customization
206+
207+
Because schema defined field with custom type, validation and default value called `custom_field`, generated code will refer to placeholders that must be implemented in component's Go package.
208+
This creates an implementation hook for component developers to add the custom logic.
209+
210+
In the above example:
211+
1. `CustomFieldType` - a Go type for a `custom_field`
212+
2. `ValidateCustomField` - a function that provides validation of `custom_field` values
213+
3. `DefaultCustomFieldValue` - a function that returns default value(s) for `custom_field`
214+
215+
### Generated JSON Schema example
216+
217+
This file will be a direct translation of the `config` section from `metadata.yaml` into JSON Schema format.
218+
219+
Example:
220+
```json
221+
{
222+
"$schema": "https://json-schema.org/draft/2020-12/schema",
223+
"$id": "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/icmpcheckreceiver",
224+
"title": "receiver/icmpcheck component",
225+
"$defs": {
226+
"metadata.metric_config": {
227+
"description": "MetricConfig provides common config for a particular metric.",
228+
"type": "object",
229+
"properties": {
230+
"enabled": {
231+
"type": "boolean"
232+
}
233+
}
234+
},
235+
"metadata.metrics_builder_config": {
236+
"description": "MetricsBuilderConfig is a configuration for icmpcheckreceiver metrics builder.",
237+
"type": "object",
238+
"properties": {
239+
"metrics": {
240+
"$ref": "#/$defs/metadata.MetricsConfig"
241+
},
242+
"resource_attributes": {
243+
"$ref": "#/$defs/metadata.ResourceAttributesConfig"
244+
}
245+
}
246+
},
247+
"metadata.metrics_config": {
248+
"description": "MetricsConfig provides config for icmpcheckreceiver metrics.",
249+
"type": "object",
250+
"properties": { ... }
251+
},
252+
"metadata.resource_attribute_config": {
253+
"description": "ResourceAttributeConfig provides common config for a particular resource attribute.",
254+
"type": "object",
255+
"properties": { ... }
256+
},
257+
"metadata.resource_attributes_config": { ... },
258+
"scraperhelper.controller_config": {
259+
"description": "ControllerConfig defines common settings for a scraper controller configuration. Scraper controller receivers can embed this struct, instead of receiver.Settings, and extend it with more fields if needed.",
260+
"type": "object",
261+
"properties": {
262+
"collection_interval": {
263+
"description": "CollectionInterval sets how frequently the scraper should be called and used as the context timeout to ensure that scrapers don't exceed the interval.",
264+
"type": "string",
265+
"pattern": "^(?:[0-9]+(\\.[0-9]+)?(ns|us|µs|ms|s|m|h))+$",
266+
"default": "60s"
267+
},
268+
"initial_delay": {
269+
"description": "InitialDelay sets the initial start delay for the scraper, any non positive value is assumed to be immediately.",
270+
"type": "string",
271+
"pattern": "^(?:[0-9]+(\\.[0-9]+)?(ns|us|µs|ms|s|m|h))+$",
272+
"default": "1s"
273+
},
274+
"timeout": {
275+
"description": "Timeout is an optional value used to set scraper's context deadline.",
276+
"type": "string",
277+
"pattern": "^(?:[0-9]+(\\.[0-9]+)?(ns|us|µs|ms|s|m|h))+$",
278+
"default": "0s"
279+
}
280+
}
281+
}
282+
},
283+
"type": "object",
284+
"properties": {
285+
"targets": {
286+
"type": "array",
287+
"items": {
288+
"type": "object",
289+
"properties": {
290+
"host": {
291+
"type": "string"
292+
},
293+
"ping_count": {
294+
"type": "integer",
295+
"default": 4
296+
},
297+
"ping_interval": {
298+
"type": "string",
299+
"pattern": "^(?:[0-9]+(\\.[0-9]+)?(ns|us|µs|ms|s|m|h))+$",
300+
"default": "300ms"
301+
},
302+
"ping_timeout": {
303+
"type": "string",
304+
"pattern": "^(?:[0-9]+(\\.[0-9]+)?(ns|us|µs|ms|s|m|h))+$",
305+
"default": "4s"
306+
}
307+
},
308+
"required": [
309+
"host"
310+
],
311+
"minItems": 1
312+
},
313+
"required": [
314+
"targets"
315+
]
316+
},
317+
"custom_field": {
318+
"type": {} // any
319+
}
320+
},
321+
"allOf": [
322+
{
323+
"$ref": "#/$defs/scraperhelper.controller_config"
324+
},
325+
{
326+
"$ref": "#/$defs/metadata.metrics_builder_config"
327+
}
328+
]
329+
}
330+
```
331+
332+
#### Generated JSON Schema details
333+
334+
- The `config.schema.json` file will be a direct representation of the `config` section from `metadata.yaml`,
335+
with proper handling of `$ref` references to include definitions from external packages and internal definitions
336+
in a single JSON Schema file.
337+
- Each referenced schema will be embedded inline.
338+
- The `$id` field will be set to the component's import path for easy identification.
339+
- The custom types will be represented as `any` type in JSON Schema.
101340

102341
### Roadmap
103342

@@ -132,3 +371,11 @@ If existing config structs don't follow the established naming patterns produced
132371
**Success Criteria:**
133372
- All core and contrib components migrated to schema-first approach
134373
- All new components use schema-first tooling by default
374+
375+
#### Phase 4: Extend OCB
376+
377+
**Objective**: Use OCB to bind all components schemas to single schema that describes entire collector config file
378+
379+
**Success Criteria:**
380+
- Generated schema for collector config file includes all components
381+
- Schema can be used by internal and external tools for validation and autocompletion

0 commit comments

Comments
 (0)