[wip]feat(log): implement structured logging module

[GrapeBaBa]

Summary

Add a full-featured structured logging module for lodestar-z

Key features

• Comptime static dispatch pipeline: Filter → Diagnostic → Append, all resolved at compile time with zero runtime overhead for disabled log levels
• Three layout formats: TextLayout (human-readable with ANSI color), JsonLayout, LogfmtLayout
• Multiple appender types: WriterAppend (sync), AsyncAppend (lock-free ring buffer + background drain), RollingFileWriter (size/date rotation), OpenTelemetryAppend (OTLP JSON)
• Composable filters: LevelFilter, ScopeFilter, EnvFilter (parses RUST_LOG-style directives like warn,fork_choice=debug)
• Diagnostics: StaticDiagnostic (pre-bound attrs), ThreadLocalDiagnostic (per-thread context)
• std.log bridge: Drop-in std_options so existing std.log.scoped() calls route through the pipeline
• Global dispatcher builders: initConsoleDispatcher, initFileDispatcher, initCombinedDispatcher for common setups
• Microsecond timestamps: ISO 8601 UTC format (2024-08-11T22:44:57.172105Z)
• stderr by default: Following Unix convention (stdout = data, stderr = diagnostics)
• Extracted ring_buffer.zig: Generic lock-free SPSC ring buffer, reusable outside logging

Architecture

Logger(N) → AnyDispatcher → Dispatcher(Dispatches)
                               ├─ Dispatch(Filters, Diagnostics, Appends)
                               │    ├─ LevelFilter / ScopeFilter / EnvFilter
                               │    ├─ StaticDiagnostic / ThreadLocalDiagnostic
                               │    └─ WriterAppend(Layout) / AsyncAppend(Layout)
                               └─ Dispatch(...)

Module structure

Consolidated from 20+ separate files into 6 focused modules:

• record.zig — Scope, Level, Attr, Record, comptime interface checks
• filter.zig — LevelFilter, ScopeFilter, EnvFilter
• diagnostic.zig — StaticDiagnostic, ThreadLocalDiagnostic
• layout.zig — TextLayout, JsonLayout, LogfmtLayout, ISO 8601 formatting
• append.zig — WriterAppend, AsyncAppend, RollingFileWriter, OpenTelemetryAppend
• dispatch.zig — Dispatch, Dispatcher, AnyDispatcher, Logger

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements a robust, structured logging system designed for high-performance and flexibility. By leveraging Zig's comptime capabilities, the system provides a static dispatch pipeline that minimizes runtime overhead while supporting complex logging requirements such as asynchronous file rotation, structured JSON/Logfmt output, and OpenTelemetry integration. The changes include a complete infrastructure for global dispatchers and a bridge to ensure seamless integration with existing std.log usage.

Highlights

• Structured Logging Module: Introduced a new structured logging module in src/log/ featuring a comptime static dispatch pipeline (Filter → Diagnostic → Append).
• Flexible Dispatch and Layouts: Implemented multiple layouts (Text, JSON, Logfmt) and appenders (sync, async SPSC ring buffer, rolling file, OpenTelemetry) for versatile logging configurations.
• Global Dispatcher Infrastructure: Added global dispatcher infrastructure with async builders for console, file, and combined outputs, including a bridge for std.log migration.
• Performance and Reliability: Utilized a high-performance SPSC ring buffer for async logging with Small Buffer Optimization (SBO) to minimize allocations.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a structured logging library for the project, featuring a robust pipeline architecture with filters, diagnostic enrichment, and multiple appender types (console, file, and async). The implementation includes support for structured layouts (Text, JSON, Logfmt) and ensures performance through comptime-generic dispatch and small buffer optimization (SBO) for ring buffer messages. My review identified several instances where error handling was insufficient, specifically regarding silent error suppression in formatting and file operations, which violates the project's requirement that all errors must be handled.

[gemini-code-assist]

Throughout formatOtlp, write operations use catch return, which silently ignores errors. This can lead to incomplete or malformed OTLP JSON log entries being sent to the collector, which might reject them or parse them incorrectly. This violates the style guide's rule 160: "All errors must be handled."

Consider changing formatOtlp to return an error (!void) and using try for all write operations. The error can then be handled in the append function, for example by calling reportAppendError.

References

1. Rule 160 states that all errors must be handled. The current code uses catch return which silently ignores potential I/O errors during formatting, leading to potentially corrupt data. ^(link)

[gemini-code-assist]

Throughout the layout formatters (TextLayout, JsonLayout, LogfmtLayout), write operations use catch return. This silently ignores I/O errors and can result in partially written, corrupted log lines. This violates the style guide's rule 160: "All errors must be handled."

A better approach would be for format functions to return an error (!void) and use try for all write operations. The caller (the appender) could then handle the error appropriately, for instance by calling reportAppendError. This ensures that formatting failures are not silent.

References

1. Rule 160 states that all errors must be handled. The current code uses catch return which silently ignores potential I/O errors during formatting, leading to potentially corrupt log lines. ^(link)

[gemini-code-assist]

The errors from deleteFile are being ignored with catch {}. While this is an example file, the style guide requires all errors to be handled (Rule 160). Silently ignoring file system errors can hide underlying problems. It would be better to at least print the error to stderr.

    std.fs.cwd().deleteFile("smoke.log") catch |err| std.debug.print("Failed to delete smoke.log: {s}\n", .{@errorName(err)});
    std.fs.cwd().deleteFile("smoke_combined.log") catch |err| std.debug.print("Failed to delete smoke_combined.log: {s}\n", .{@errorName(err)});

References

1. Rule 160 states that all errors must be handled. The current code uses catch {} which silently ignores potential file system errors during cleanup. ^(link)

[gemini-code-assist]

Errors from rolling_writer.writer().writeAll are being silently ignored with catch return. This violates the style guide's rule 160: "All errors must be handled." These errors should be reported to stderr using the reportAppendError utility function, similar to how other appenders handle write errors.

                self.rolling_writer.writer().writeAll(written) catch |err| reportAppendError(err, "FileAppend.writeAll");

References

1. Rule 160 states that all errors must be handled. The current code uses catch return which silently ignores potential I/O errors. ^(link)

[gemini-code-assist]

Errors from rolling_writer.writer().writeAll are being silently ignored with catch return. This violates the style guide's rule 160: "All errors must be handled." These errors should be reported to stderr using the reportAppendError utility function, similar to how other appenders handle write errors.

            self.rolling_writer.writer().writeAll(list.items) catch |err| reportAppendError(err, "FileAppend.writeAll");

References

1. Rule 160 states that all errors must be handled. The current code uses catch return which silently ignores potential I/O errors. ^(link)

[gemini-code-assist]

The error from self.overrides.append is silently ignored with catch {}. If the number of overrides exceeds max_scopes (32), append will fail and the override will not be added. This failure is silent, which can be very confusing for a user who expects their configuration to be applied. According to the style guide (Rule 160), all errors must be handled. It would be better to log a warning if the limit is reached.

        self.overrides.append(.{ .scope_name = scope_name, .min_level = min_level }) catch |_| {
            std.debug.print("log.ScopeFilter: could not add override for '{s}', max_scopes ({d}) reached.\n", .{
                scope_name, max_scopes,
            });
        };

References

1. Rule 160 states that all errors must be handled. The current code uses catch {} which silently ignores the error when the maximum number of scope overrides is exceeded. ^(link)

[gemini-code-assist]

The fixed-size buffer buf: [8]u8 in parseLevel is brittle. While it's large enough for the current standard log levels, it makes the function less robust to future changes, such as the introduction of custom or longer log level names. A more robust approach would be to use std.ascii.eqlIgnoreCase to perform the comparison without creating a lowercased copy.

pub fn parseLevel(text: []const u8) ?Level {
    if (std.ascii.eqlIgnoreCase(text, "err")) return .err;
    if (std.ascii.eqlIgnoreCase(text, "error")) return .err;
    if (std.ascii.eqlIgnoreCase(text, "warn")) return .warn;
    if (std.ascii.eqlIgnoreCase(text, "warning")) return .warn;
    if (std.ascii.eqlIgnoreCase(text, "info")) return .info;
    if (std.ascii.eqlIgnoreCase(text, "debug")) return .debug;
    return null;
}