Skip to content

MAKING_A_NEW_SHADER.md

Latest commit

 

History

History
1217 lines (901 loc) · 43.5 KB

MAKING_A_NEW_SHADER.md

File metadata and controls

1217 lines (901 loc) · 43.5 KB
Raw

Creating New Shaders for Paper Cranes

This guide explains how to create music-reactive WebGL visualizations for the Paper Cranes system. These shaders run in the browser, react to audio input, and can be used to drive LED lights (Hue, Nanoleaf) through screen scraping.

Table of Contents

  1. System Overview
  2. Shader Basics
  3. Available Uniforms
  4. Utility Functions
  5. Audio Feature Deep Dive
  6. Choosing Unique Features — how to pick independent features for multi-element shaders
  7. Design Patterns
  8. Common Techniques
  9. Testing Your Shader
  10. Common Pitfalls
  11. Example Shaders to Study
  12. Debugging Twitchy Shaders — diagnosing and fixing jerky, flashy, or chaotic visuals

System Overview

Paper Cranes is a real-time music visualization system that:

  1. Captures audio from the microphone
  2. Performs FFT analysis and extracts 14 audio features
  3. Passes features as uniforms to GLSL fragment shaders
  4. Renders at 60fps with WebGL2
  5. Supports frame feedback (previous frame access) for temporal effects

The shader files live in /shaders/ and are loaded via query parameter: ?shader=filename (without .frag extension).

Going beyond the basics? Once you've got the fundamentals here, see advanced-shader-techniques.md for musical signal conditioning, ChromaDepth that pops, drag/pinch navigation, breaking tiling, and the verification workflow.

Shader Basics

File Structure

Create a .frag file in the /shaders/ directory. The system uses a ShaderToy-compatible format:

void mainImage(out vec4 fragColor, in vec2 fragCoord) {
    // fragCoord: pixel coordinates (0 to iResolution.xy)
    // fragColor: output color (RGBA, 0.0 to 1.0)

    vec2 uv = fragCoord / iResolution.xy;  // Normalized 0-1
    fragColor = vec4(uv.x, uv.y, 0.5, 1.0);
}

Coordinate Systems

// Normalized UV (0 to 1, origin bottom-left)
vec2 uv = fragCoord / iResolution.xy;

// Centered UV (-0.5 to 0.5, origin center)
vec2 centered = uv - 0.5;

// Aspect-corrected centered (-aspect to aspect, -1 to 1)
vec2 aspectCorrected = (fragCoord * 2.0 - iResolution.xy) / iResolution.y;

Available Uniforms

Core Uniforms

Uniform Type Description
iTime float Time in seconds since start
iResolution vec3 Screen resolution (width, height, 0)
iFrame int Frame counter
iMouse vec4 Mouse position and click state
time float Alias for iTime
resolution vec2 Screen size (x, y)
frame int Alias for iFrame

Texture Samplers

Uniform Description
iChannel0-3 Texture samplers (ShaderToy compatibility)
prevFrame Previous frame texture (for feedback effects)
initialFrame First frame / fallback texture

Audio Feature Uniforms

Each of the 14 audio features has 11 statistical variations, giving 154 audio uniforms total. See Audio Feature Deep Dive for details.

The 14 core features:

  • bass, mids, treble (frequency bands)
  • energy (overall loudness)
  • pitchClass (detected note 0-11, normalized to 0-1)
  • spectralCentroid, spectralSpread, spectralSkew, spectralKurtosis (spectral shape)
  • spectralFlux (rate of timbral change)
  • spectralRolloff (high frequency cutoff)
  • spectralRoughness (dissonance)
  • spectralEntropy (chaos/unpredictability)
  • spectralCrest (peakiness)

Statistical variations for each feature:

  • bass - raw value
  • bassNormalized - min-max normalized to 0-1
  • bassMean - historical average
  • bassMedian - historical median
  • bassMin, bassMax - historical range
  • bassStandardDeviation - variability
  • bassZScore - standardized (-1 to 1 roughly), detects anomalies
  • bassSlope - linear regression slope (is bass rising or falling over the history window?)
  • bassIntercept - regression intercept (predicted value at start of window)
  • bassRSquared - regression fit (0-1, how steady/linear the trend is)

Special Uniforms

Uniform Type Description
beat bool True when beat detected
iRandom float Random value that changes each frame
touch vec2 Touch/mouse position (normalized 0-1)
touched bool Whether user is touching/clicking
knob_1 to knob_200 float MIDI controller knobs (0-1)

Utility Functions

These are auto-injected and available in all shaders:

Color Conversion

Prefer Oklch/Oklab over HSL. Oklch produces perceptually uniform gradients — no unexpected dark bands between hues, no brightness shifts when rotating hue. HSL is available for compatibility but Oklch is the better default for new shaders.

// Oklch — polar perceptual color space (RECOMMENDED)
// vec3(L, C, h) where L=lightness(0-1), C=chroma(0-~0.37), h=hue(radians)
vec3 rgb2oklch(vec3 c)           // RGB to Oklch
vec3 oklch2rgb(vec3 lch)         // Oklch to RGB
vec3 oklchmix(vec3 c1, vec3 c2, float t)  // Mix with shortest-path hue interpolation

// Oklab — cartesian perceptual color space
vec3 rgb2oklab(vec3 c)           // RGB to Oklab (L, a, b)
vec3 oklab2rgb(vec3 lab)         // Oklab to RGB
vec3 oklabmix(vec3 c1, vec3 c2, float t)  // Mix in Oklab space

// Conversions between Oklab and Oklch
vec3 oklab2oklch(vec3 lab)       // Oklab to Oklch
vec3 oklch2oklab(vec3 lch)       // Oklch to Oklab

// HSL (legacy — prefer Oklch for new shaders)
vec3 hsl2rgb(vec3 hsl)           // HSL to RGB
vec3 rgb2hsl(vec3 rgb)           // RGB to HSL
vec3 hslmix(vec3 c1, vec3 c2, float t)  // Mix colors in HSL space

All color functions have vec4 overloads that pass alpha through.

When to Use Which Color Space

Task Best Space Why
Hue rotation Oklch Just add to h — perceptually uniform
Color mixing/gradients Oklch or Oklab No unexpected dark bands like HSL
Boost saturation Oklch Scale C without affecting lightness
Brighten without washing out Oklch Change L only
Create a color from L/C/hue Oklch oklch2rgb(vec3(0.7, 0.15, angle))
Quick additive glow RGB col += glow (oklch breaks with additive)
Simple lerp (mask on/off) RGB mix(bg, col, mask) is fine
Porting a ShaderToy shader HSL Match the original; convert to Oklch later if needed

Rule of thumb: Start with Oklch. Fall back to RGB for additive blending. Use HSL only when porting existing code that already uses it.

Oklch Quick Reference

// Rotate hue
vec3 lch = rgb2oklch(color);
lch.z += 1.0;  // rotate ~57 degrees
color = oklch2rgb(lch);

// Boost saturation
lch.y *= 1.5;

// Brighten
lch.x = min(lch.x + 0.1, 1.0);

// Create vivid color from scratch
vec3 col = oklch2rgb(vec3(0.7, 0.15, 1.0));  // bright, vivid, yellow-green hue

// Mix two colors with proper hue interpolation
vec3 result = oklchmix(red, blue, 0.5);  // goes through purple, not grey

Previous Frame Access

vec4 getLastFrameColor(vec2 uv)    // Sample previous frame (UV 0-1)
vec4 getInitialFrameColor(vec2 uv) // Sample initial frame

Random Numbers

float random(vec2 st, float seed)  // Deterministic random with seed
float random(vec2 st)              // Random using iRandom
float staticRandom(vec2 st)        // Consistent random (seed=0)

Value Mapping

float mapValue(float val, float inMin, float inMax, float outMin, float outMax)
// Remaps value from input range to output range with clamping

UV Utilities

vec2 centerUv(vec2 res, vec2 coord)  // Center coordinates
vec2 centerUv(vec2 coord)            // Using global resolution

Animation & Easing

float pingpong(float t)              // Oscillates 0→1→0
float animateSmooth(float t)         // Cubic smoothstep
float animateBounce(float t)         // Bouncing animation
float animatePulse(float t)          // Sine wave pulse

// Easing functions (all take t from 0-1):
float animateEaseInQuad(float t)
float animateEaseOutQuad(float t)
float animateEaseInOutQuad(float t)
float animateEaseInCubic(float t)
float animateEaseOutCubic(float t)
float animateEaseInOutCubic(float t)
float animateEaseInExpo(float t)
float animateEaseOutExpo(float t)
float animateEaseInOutExpo(float t)
float animateEaseInSine(float t)
float animateEaseOutSine(float t)
float animateEaseInOutSine(float t)
float animateEaseInElastic(float t)
float animateEaseOutElastic(float t)
float animateEaseInOutElastic(float t)

Audio Feature Deep Dive

When to Use Each Statistical Variation

Variation Use Case Example
Normalized Smooth 0-1 modulation bassNormalized * 0.5 for half-intensity effect
ZScore Detect spikes, drops, anomalies if (energyZScore > 0.5) for detecting loud moments
Mean/Median Baseline behavior, slow changes Background color based on spectralCentroidMean
Min/Max Historical context Scale effects relative to bassMax
StandardDeviation Detect stability vs volatility More chaos when energyStandardDeviation high
Slope Detect rising/falling trends energySlope > 0.0 = energy building up over time
Intercept Trend baseline/extrapolation Use with slope to predict where a feature is heading
RSquared Trend confidence (0-1) energyRSquared > 0.5 = steady trend, not chaotic noise
Raw value Direct analysis (less common) Usually prefer Normalized

Feature Independence

Good pairings (independent, use together for variety):

  • bass + treble (opposite frequency ranges)
  • spectralCentroid + spectralRoughness (pitch vs dissonance)
  • spectralEntropy + spectralCrest (chaos vs peakiness)
  • spectralFlux + spectralRolloff (change rate vs cutoff)
  • pitchClass + spectralKurtosis (pitch vs distribution shape)

Bad pairings (correlated, avoid using together):

  • energy + bass (both increase with loud low frequencies)
  • spectralCentroid + pitchClass (both relate to pitch)
  • spectralSpread + spectralEntropy (wider = more complex)

For a deeper guide on choosing unique features, energy decorrelation, hidden correlations, and a proven 8-feature set, see Choosing Unique Features.

Feature Domains

Choose features from DIFFERENT domains for variety:

Domain Features
Frequency Bands bass, mids, treble
Spectral Shape spectralCentroid, spectralSpread, spectralSkew, spectralKurtosis
Spectral Quality spectralRoughness, spectralEntropy, spectralCrest
Temporal spectralFlux
Tonal pitchClass, spectralRolloff
Energy energy

Design Patterns

Pattern 1: Slow Evolution (Recommended for LED sync)

For visuals that will drive LED lights, avoid rapid changes. Use heavy feedback blending:

// Get previous frame
vec3 prev = getLastFrameColor(uv).rgb;

// Compute new color
vec3 newColor = /* your fractal/effect */;

// Blend heavily toward previous (90% old, 10% new)
vec3 color = mix(prev, newColor, 0.1);

Pattern 2: Audio Affects Structure, Not Color

Map audio to structural parameters rather than directly to color for smoother results:

// GOOD: Audio affects fractal parameters
#define WARP_DEPTH mapValue(bassNormalized, 0., 1., 0.3, 0.8)
#define SPIRAL_TIGHTNESS mapValue(spectralCentroidNormalized, 0., 1., 1.2, 3.5)

// AVOID: Audio directly controlling color
vec3 color = vec3(bassNormalized, trebleNormalized, midsNormalized); // Too flashy

Pattern 3: Use ZScores for Events, Normalized for Modulation

// ZScore for detecting moments (drops, spikes)
if (energyZScore > 0.5) {
    // Something special happens
}

// Normalized for smooth continuous modulation
float radius = 0.5 + spectralCentroidNormalized * 0.3;

Pattern 4: Trend-Aware Evolution with Linear Regression

Use Slope, Intercept, and RSquared to make visuals that respond to where the music is heading, not just where it is now:

// Slope: positive = rising, negative = falling
// Use it to drive evolution direction
#define EVOLVE_DIRECTION (energySlope * 10.0)  // Scale up — raw slope values are small

// RSquared: 0 = chaotic, 1 = steady trend
// Use it to control confidence/stability of the visual response
#define TREND_CONFIDENCE (energyRSquared)

// Combine: only evolve strongly when trend is confident
#define CONFIDENT_EVOLUTION (energySlope * energyRSquared * 10.0)

// Detect musical sections:
// Confident build = positive slope + high rSquared
// Confident drop = negative slope + high rSquared
// Chaos/transition = low rSquared (any slope)
#define IS_BUILDING (energySlope > 0.001 && energyRSquared > 0.4)
#define IS_DROPPING (energySlope < -0.001 && energyRSquared > 0.4)
#define IS_CHAOTIC (energyRSquared < 0.2)

// Different features have different trend meanings:
#define GETTING_BRIGHTER (spectralCentroidSlope > 0.0)   // Timbre brightening
#define BASS_BUILDING (bassSlope > 0.0)                  // Low-end building
#define TEXTURE_EVOLVING (spectralEntropySlope)           // Complexity changing

Key insight: Slope values are small (they represent change per history-window index), so multiply by 5-20 to get useful visual ranges. RSquared is already 0-1 and works well as a confidence gate.

Pattern 5: Incommensurate Frequencies for Aperiodic Motion

Use irrational ratios (golden ratio, sqrt(2), etc.) to prevent repetitive loops:

#define PHI 1.61803398875

// These will never sync up, creating endless variation
float x = sin(iTime * 0.13);
float y = cos(iTime * 0.17 * PHI);
float z = sin(iTime * 0.11 * sqrt(2.0));

Pattern 6: Layered Fractals

Combine multiple fractal calculations at different scales:

vec4 layer1 = fractalFunction(uv, time);
vec4 layer2 = fractalFunction(uv * 2.1, time * PHI);
vec4 layer3 = fractalFunction(uv * 0.5, time * 0.7);

// Blend based on fractal depth/characteristics
vec3 color = mix(mix(color1, color2, blend2), color3, blend3);

Pattern 7: Smoothstep Thresholds for "Only React to the Loud Parts"

Raw normalized values respond linearly across the full 0-1 range, which means the fractal/visual is always moving, even during quiet passages. Use smoothstep to create a dead zone below a threshold, so the visual stays calm during normal moments and only comes alive when features push into their upper range:

// Linear: always reacting, even to small values
#define J_REAL_SHIFT (bassNormalized * 0.03)

// Thresholded: calm below 40th percentile, S-curved response above
#define J_REAL_SHIFT (smoothstep(0.4, 1.0, bassNormalized) * 0.03)

The S-curve from smoothstep also makes the onset of movement gentle rather than abrupt. Tune the lower edge (0.3-0.5) to control how much of the feature's range is "dead" vs "active."

This pattern works especially well for:

  • Fractal shape parameters (Julia set constants)
  • Camera/zoom movement
  • Drift and rotation speed
  • Any parameter where constant low-level motion is distracting
// Combine thresholded features from different domains for rich reactive movement
#define ZOOM_AUDIO (smoothstep(0.5, 1.0, bassNormalized) * 0.08)
#define ROT_AUDIO (smoothstep(0.3, 1.0, spectralFluxNormalized) * 0.12)
#define DRIFT_X (smoothstep(0.3, 1.0, spectralKurtosisNormalized) * 0.05)

Pattern 8: ZScore vs Normalized — Choosing Your Reactivity Character

The same audio feature tells a very different visual story depending on which statistical variation you use:

Variation Character Best For
ZScore Spiky, beat-reactive, flashy Event detection, beat pulses, dramatic moments
Normalized Smooth, flowing, continuous Fractal shape, color evolution, gradual movement
Slope * RSquared Trend-aware, anticipatory Building/dropping detection, confident evolution

Rule of thumb: If your visual parameter changes the overall feel of the image (shape, color, zoom), use Normalized or Slope * RSquared. If it should punctuate moments (flash, pulse, size bump), use ZScore.

A common mistake is using z-scores for everything — this makes the entire visual flashy and tiring to watch. Reserve z-scores for a few specific "event" parameters (mammoth size on beat, beat color shift) and use normalized values for the continuous movement (fractal shape, rotation, drift).

// GOOD: normalized for continuous movement, z-score for punctuation
#define FRACTAL_MORPH (smoothstep(0.3, 1.0, spectralKurtosisNormalized) * 0.03)
#define MAMMOTH_BEAT_PULSE (clamp(bassZScore, 0.0, 1.0) * 0.2)

// BAD: z-scores everywhere = constant visual spiking
#define FRACTAL_MORPH (spectralKurtosisZScore * 0.03)
#define MAMMOTH_BEAT_PULSE (bassZScore * 0.2)

Pattern 9: Taming Color Flashiness

Color changes are the most noticeable form of "flashiness." Multiple color-affecting parameters can stack up and produce jarring shifts even when each individual effect seems subtle.

Sources of color flash (audit all of these):

  • Hue shift on beat (e.g., bHSL.x -= 0.05)
  • Edge glow intensity scaling with audio
  • Depth-to-hue mapping range (how much of the color wheel is used)
  • Audio-driven hue drift over time
  • Saturation boost from audio features
  • Feedback blend amount (less feedback = faster color change)

Guidelines:

  • Keep beat hue shifts small (0.01-0.03, not 0.05+)
  • Keep beat lightness boosts subtle (1.02-1.05x, not 1.1+)
  • Cap additive glow terms: min(glow, vec3(0.25)) not vec3(0.5)
  • Use a narrower hue range in your color mapping (30-50% of the color wheel, not 75%)
  • Prefer Normalized over ZScore for any color-affecting parameter
  • Use Slope * RSquared for hue drift — it only shifts color during confident trends, not random fluctuations
// Subtle beat color shift
if (beat) {
    vec3 bHSL = rgb2hsl(max(col, vec3(0.001)));
    bHSL.x = fract(bHSL.x - 0.02);   // small hue nudge
    bHSL.z = min(bHSL.z * 1.03, 0.6); // gentle brightness, hard cap
    col = hsl2rgb(bHSL);
}

// Narrow, seed-varied palette instead of full rainbow
float hueRange = 0.3 + seed * 0.2;  // 30-50% of wheel
float hue = fract(depth * hueRange + seed2 * 0.5);

Pattern 10: Genre-Aware Feature Selection

Different music genres have different spectral signatures. Choose features that match what's most expressive in your target genre:

Dubstep / Bass Music:

  • spectralKurtosis — most dynamic feature (CoV=1.118), measures spectrum peakedness. Great for shape morphing.
  • spectralRoughness — the "wub" and grit. Drives distortion/warping effects.
  • bass — the sub-bass drops. Best for size/zoom pulses.
  • spectralFlux — transients and hits. Good for rotation/drift speed.

Ambient / Atmospheric:

  • spectralEntropy — chaos vs order. Drives complexity.
  • spectralCentroid — brightness shifts. Good for color temperature.
  • spectralSpread — harmonic width. Drives spatial effects.
  • Slope + RSquared variants — slow, confident evolution.

EDM / House:

  • spectralFlux — strongest beat correlate (2.37x spike on beats).
  • energy — overall loudness drives intensity.
  • bass + treble — opposing frequency bands for contrast.
  • beat — reliable pulse for rhythmic effects.

Common Techniques

2D Rotation Matrix

mat2 rot(float a) {
    float c = cos(a), s = sin(a);
    return mat2(c, -s, s, c);
}

// Usage
vec2 rotatedUV = rot(iTime * 0.1) * uv;

Complex Number Operations

vec2 cmul(vec2 a, vec2 b) {
    return vec2(a.x*b.x - a.y*b.y, a.x*b.y + a.y*b.x);
}

vec2 cdiv(vec2 a, vec2 b) {
    float d = dot(b, b);
    return vec2(a.x*b.x + a.y*b.y, a.y*b.x - a.x*b.y) / d;
}

vec2 cpow(vec2 z, float n) {
    float r = length(z);
    float theta = atan(z.y, z.x);
    return pow(r, n) * vec2(cos(n*theta), sin(n*theta));
}

Julia Set Fractal

vec2 julia(vec2 z, vec2 c, int maxIter) {
    for (int i = 0; i < maxIter; i++) {
        z = vec2(z.x*z.x - z.y*z.y, 2.0*z.x*z.y) + c;
        if (dot(z, z) > 4.0) break;
    }
    return z;
}

Orbit Trap Coloring

Track minimum distances during iteration for smooth coloring:

float minDist = 1e10;
vec2 z = startZ;
for (int i = 0; i < maxIter; i++) {
    z = iterate(z);
    minDist = min(minDist, abs(z.x));  // Line trap
    // or: minDist = min(minDist, length(z - trapPoint));  // Point trap
}
float color = exp(-minDist * 10.0);  // Glow effect

Domain Warping with FBM

float noise(vec2 p) { /* your noise function */ }

float fbm(vec2 p) {
    float value = 0.0;
    float amplitude = 0.5;
    for (int i = 0; i < 5; i++) {
        value += amplitude * noise(p);
        p *= 2.0;
        amplitude *= 0.5;
    }
    return value;
}

// Warp coordinates
vec2 warpedUV = uv + vec2(fbm(uv * 3.0), fbm(uv * 3.0 + 5.0)) * 0.3;

Cosine Palette

vec3 palette(float t, vec3 a, vec3 b, vec3 c, vec3 d) {
    return a + b * cos(6.28318 * (c * t + d));
}

// Rich preset
vec3 color = palette(t,
    vec3(0.5), vec3(0.5), vec3(1.0),
    vec3(0.0, 0.1, 0.2));

Kaleidoscope Effect

vec2 kaleidoscope(vec2 uv, float segments) {
    float angle = atan(uv.y, uv.x);
    float segment = 6.28318 / segments;
    angle = mod(angle + segment * 0.5, segment) - segment * 0.5;
    angle = abs(angle);
    return vec2(cos(angle), sin(angle)) * length(uv);
}

Smooth Minimum (for blending shapes)

float smin(float a, float b, float k) {
    float h = clamp(0.5 + 0.5*(b-a)/k, 0.0, 1.0);
    return mix(b, a, h) - k*h*(1.0-h);
}

Testing Your Shader

Local Development

npm run dev  # Starts server on localhost:6969

Access your shader: http://localhost:6969/?shader=your-shader-name

Query Parameters for Testing

Parameter Description Example
shader Shader file to load ?shader=fractal-abyss
noaudio Disable audio (use defaults) ?shader=test&noaudio
fullscreen Fullscreen mode ?fullscreen=true
fft_size FFT window size ?fft_size=2048
smoothing Audio smoothing factor ?smoothing=0.2
Any numeric param Becomes a float uniform ?bassMedian=0.5

Simulating Audio with Query Params

Any numeric query parameter automatically becomes a shader uniform, and query params override measured audio values. This means you can test exactly how your shader responds to different musical conditions without a microphone:

?shader=my-shader&noaudio=true&fullscreen=true&bassMedian=0.5&energyMedian=0.7

Test your shader against these profiles to catch issues like washout, dead zones, or lack of visual variety:

Suggested Test Profiles

Silent (baseline) — verify the shader looks good with no audio at all:

?shader=my-shader&noaudio=true&fullscreen=true

Heavy bass — deep, bassy music:

?shader=my-shader&noaudio=true&fullscreen=true&bassMedian=0.8&energyMedian=0.7&spectralCentroidMedian=0.2&spectralEntropyMedian=0.3&spectralRoughnessMedian=0.6

Bright / chaotic — busy, high-frequency content:

?shader=my-shader&noaudio=true&fullscreen=true&spectralCentroidMedian=0.8&spectralEntropyMedian=0.9&trebleSlope=0.5&trebleRSquared=0.8&energyMedian=0.5

Confident energy build — a steady crescendo:

?shader=my-shader&noaudio=true&fullscreen=true&energySlope=0.6&energyRSquared=0.9&energyMedian=0.6&bassMedian=0.5&bassSlope=0.4&bassRSquared=0.85&spectralCentroidMedian=0.5&spectralFluxZScore=1.5

Energy drop — fading out or post-drop:

?shader=my-shader&noaudio=true&fullscreen=true&energySlope=-0.5&energyRSquared=0.8&energyMedian=0.3&spectralCentroidMedian=0.6&spectralEntropyMedian=0.7

What to Look For

  • Washout: Do bright regions blow out to white? If so, reduce additive luminance terms or tighten your clamp.
  • Dead zones: Are there profiles where the shader looks nearly identical to silence? Your audio mapping may be too subtle.
  • Color variety: Does the shader look meaningfully different across profiles? If heavy bass and bright chaotic look the same, your color mapping may depend on too few features.
  • Contrast: Every profile should have visible dark regions. If you lose all contrast, your luminance contributions are stacking up.

Freezing Time with ?time=

Since time is just a uniform, you can override it with a query param to freeze the shader at a specific moment in seconds:

?shader=my-shader&noaudio=true&time=12.5

This renders the shader as if 12.5 seconds have elapsed — time stops advancing. Useful for:

  • Debugging time-dependent effects — freeze at the exact moment something looks wrong
  • Comparing states — screenshot at time=5, time=30, time=120 to see how the shader evolves
  • Testing periodicity — check if your shader loops cleanly by comparing time=0 with time=6.283 (2π)
  • Tuning constants — freeze time and adjust other params to isolate time-dependent behavior from audio-dependent behavior

Combine with audio overrides to test a specific shader state completely:

?shader=my-shader&noaudio=true&time=60&bassNormalized=0.8&energyZScore=0.5

Debug Tips

  1. Check browser console for shader compilation errors
  2. Start simple - get basic output before adding complexity
  3. Test with audio profiles (above) - don't just test with ?noaudio and call it done
  4. Visualize individual features to understand their behavior:
// Debug: visualize a single audio feature
fragColor = vec4(vec3(bassNormalized), 1.0);

Common Pitfalls

1. White-Out / Blown Highlights

Problem: Colors exceed 1.0 and clip to white.

Solution: Use tone mapping:

color = color / (1.0 + color * 0.3);  // Soft clamp
color = clamp(color, 0.0, 1.0);

2. Rapid Color Flashing

Problem: Colors change too quickly, especially with audio.

Solutions:

  • Use heavy feedback blending (90%+ previous frame)
  • Map audio to structure, not color directly
  • Use Normalized instead of ZScore for color
  • Slow down time: float t = iTime * 0.3;

3. Division by Zero

Problem: NaN or Inf from dividing by zero.

Solution: Always protect divisions:

float d = max(dot(z, z), 0.0001);
float result = x / d;

4. Log of Zero/Negative

Problem: log() of zero or negative produces NaN.

Solution:

float safeLog = log(max(value, 0.0001));

5. Feedback Accumulation Drift

Problem: Colors drift toward white or black over time with feedback.

Solution: Gently pull toward target values (Oklch keeps lightness and chroma independent):

vec3 prevLch = rgb2oklch(prev);
prevLch.y = mix(prevLch.y, 0.12, 0.01);  // Pull chroma toward vivid
prevLch.x *= 0.995;  // Slight darkening to prevent white-out
prev = oklch2rgb(prevLch);

6. Audio Features Not Varying

Problem: Audio uniforms seem static.

Causes:

  • Microphone not permitted
  • Audio muted at OS level
  • Using raw values instead of Normalized/ZScore

Debug:

// Visualize if audio is working
fragColor = vec4(energy, spectralFlux, bass, 1.0);

7. Everything is the Same Color (Depth-Based Coloring)

Problem: Using distance/depth for color but it's all one hue.

Causes:

  • Visible surfaces are at similar distances from camera
  • Depth range parameters don't match visible range
  • Using ray distance when world position would work better

Debug approach:

// Output raw depth as grayscale to see if it varies
float normDist = result.dist / MAX_DIST;
fragColor = vec4(vec3(normDist), 1.0);  // Should show gradient

Solutions:

  1. Tighten the depth range to match what's visible:
#define DEPTH_MIN 1.5   // Start of visible range
#define DEPTH_MAX 4.0   // End of visible range
float normDist = (dist - DEPTH_MIN) / (DEPTH_MAX - DEPTH_MIN);
  1. Try different depth sources - ray distance isn't always best:
// Ray distance (from camera)
float depth = result.dist / MAX_DIST;

// World Z position (depth into scene)
float depth = (p.z - Z_MIN) / (Z_MAX - Z_MIN);

// Orbit trap (varies with fractal structure!)
float depth = sqrt(trap.trapMin) * TRAP_SCALE;

// Distance from world origin
float depth = length(p) / MAX_RADIUS;
  1. Mix multiple sources for richer variation:
float hue = depthHue * 0.5 + trapHue * 0.3 + normalHue * 0.2;

Key insight: If your camera sees a "flat" view of a fractal (all surfaces at similar ray distances), use orbit traps or world position instead of ray distance for color variation.

Example Shaders to Study

/shaders/sexy/2.frag - Elegant Complex Exponential Fractal

  • Compact, elegant code
  • Orbit tracking for smooth coloring
  • Audio mapped to narrow parameter ranges
  • No feedback, pure mathematical beauty

Key technique: Angle-based coloring from orbit position:

P = sqrt(z + (z - z*z*z) * cos(atan(Z.y, Z.x) - vec4(0, 2.1, 4.2, 0)));

/shaders/moody-octopus.frag - Julia Set with Feedback

  • Julia set distortion
  • White-out prevention system
  • HSL color manipulation
  • Beat-reactive ripples

/shaders/fractal-abyss.frag - Layered Evolving Fractal

  • Multiple fractal layers at different scales
  • Heavy feedback for slow evolution
  • Domain warping with FBM
  • Rich jewel-tone palette
  • Audio affects structure, not color
  • Designed for LED light sync

Key techniques:

  • 92% feedback blend for slow evolution
  • Three fractal layers (2x complex exponential + Mandelbrot)
  • Incommensurate frequencies (PHI-based) for aperiodicity
  • Orbit trap glow effects

Workflow Summary

  1. Create file: /shaders/your-name.frag

  2. Start with template:

void mainImage(out vec4 fragColor, in vec2 fragCoord) {
    vec2 uv = (fragCoord * 2.0 - iResolution.xy) / iResolution.y;
    float t = iTime * 0.3;  // Slow time

    // Your visualization here — build color in Oklch for perceptual uniformity
    float hue = t * 0.5;                        // Hue in radians
    float chroma = 0.12 + bassNormalized * 0.05; // Saturation
    float lightness = 0.5 + energy * 0.2;        // Brightness
    vec3 color = oklch2rgb(vec3(lightness, chroma, hue));

    // Feedback for smooth evolution
    vec3 prev = getLastFrameColor(fragCoord / iResolution.xy).rgb;
    color = mix(prev, color, 0.1);

    fragColor = vec4(color, 1.0);
}

  1. Test: http://localhost:6969/?shader=your-name

  2. Iterate: Add complexity gradually, test frequently

  3. Polish: Add tone mapping, vignette, prevent white-out

The #define Swap Pattern (Simplest Approach)

The easiest way to toggle between constant values and audio-reactive uniforms is the comment swap pattern:

// ============================================================================
// TUNABLE PARAMETERS - swap constants for audio uniforms
// ============================================================================

// Active: audio-reactive
#define SCALE_MOD (-spectralEntropyZScore * 0.08)
// #define SCALE_MOD 0.0

// Active: constant (for testing)
// #define HUE_SHIFT (pitchClassNormalized * 0.3)
#define HUE_SHIFT 0.0

// Active: audio-reactive
#define BRIGHTNESS (1.0 + bassZScore * 0.15)
// #define BRIGHTNESS 1.0

Why this works:

  • Comment/uncomment one line to toggle
  • Test without audio: ?noaudio=true
  • Tune constant values before mapping to audio
  • Documentation stays inline with the code
  • Easy to see what's currently active at a glance

Grouping Parameters

Organize by function for clarity:

// ============================================================================
// COLOR PARAMETERS
// ============================================================================
#define HUE_BASE 0.0
#define HUE_RANGE 0.7
#define SATURATION 0.95

// ============================================================================
// STRUCTURE PARAMETERS
// ============================================================================
#define DEPTH_MIN 1.5
#define DEPTH_MAX 6.0

// ============================================================================
// AUDIO-REACTIVE PARAMETERS
// ============================================================================
#define ZOOM (spectralFluxZScore * 0.18)
// #define ZOOM 0.0

Knob-to-Audio Workflow (Alternative Pattern)

For more complex parameter relationships, use #ifdef blocks to switch entire parameter sets:

Why This Pattern Works

  1. Testable states - Use query params to test specific parameter combinations
  2. No microphone needed - Develop without audio setup
  3. Reproducible - Same URL = same visual state for debugging
  4. Playwright-friendly - Automated visual testing with different knob values
  5. Easy transition - Just flip a #define to switch to audio mode

The Pattern

// ============================================================================
// KNOB MODE: Use query params to test different states
// Example: ?shader=my-shader&knob_1=0.5&knob_2=0.8
// ============================================================================

// Uncomment to enable knob testing mode
// #define KNOB_MODE

#ifdef KNOB_MODE
    // Knob declarations
    uniform float knob_1; // PARAM_A: describe what 0 and 1 mean
    uniform float knob_2; // PARAM_B: describe what 0 and 1 mean

    // Map knobs to aesthetic ranges
    #define PARAM_A mapValue(knob_1, 0., 1., 0.3, 0.8)
    #define PARAM_B mapValue(knob_2, 0., 1., 1.0, 3.0)
#else
    // AUDIO MODE: Map audio features to the same parameters
    #define PARAM_A mapValue(bassNormalized, 0., 1., 0.3, 0.8)
    #define PARAM_B mapValue(spectralCentroidNormalized, 0., 1., 1.0, 3.0)
#endif

Development Steps

  1. Start with #define KNOB_MODE uncommented
  2. Test via URL: ?shader=my-shader&knob_1=0.2&knob_2=0.8&noaudio=true
  3. Find aesthetic ranges by adjusting knob values
  4. Document each knob with comments explaining what low/high values produce
  5. Switch to audio by commenting out #define KNOB_MODE
  6. Choose appropriate audio features from different domains (see Feature Domains in CLAUDE.md)

Testing with Playwright

Use Playwright to capture different states for visual comparison:

// Navigate with specific knob values
await page.goto('http://localhost:6969/?shader=my-shader&knob_1=0.0&knob_2=0.5&noaudio=true');
await page.screenshot({ path: 'state-low.png' });

await page.goto('http://localhost:6969/?shader=my-shader&knob_1=1.0&knob_2=0.5&noaudio=true');
await page.screenshot({ path: 'state-high.png' });

Example: fractal-abyss.frag

#ifdef KNOB_MODE
    uniform float knob_1; // WARP_DEPTH: 0=subtle, 1=heavy warping
    uniform float knob_2; // SPIRAL_TIGHTNESS: 0=loose, 1=tight spirals

    #define WARP_DEPTH mapValue(knob_1, 0., 1., 0.3, 0.8)
    #define SPIRAL_TIGHTNESS mapValue(knob_2, 0., 1., 1.2, 3.5)
#else
    #define WARP_DEPTH mapValue(bassNormalized, 0., 1., 0.3, 0.8)
    #define SPIRAL_TIGHTNESS mapValue(spectralCentroidNormalized, 0., 1., 1.2, 3.5)
#endif

Common Knob Assignments

Knob Range Typical Use
knob_1-7 Primary shader parameters
knob_70-79 Secondary/fine-tune parameters
knob_14-22 MIDI controller banks

/solo — Automated Knob/Audio Toggle

The /solo Claude Code skill automates the knob swap pattern. It can:

  • Knobify a shader (/solo knobs <path>) — extract all tunable parameters into #defines with knob_* uniforms, add commented-out audio alternatives, refactor any bare knob_* references into named #defines, and generate a preset URL.
  • Switch to audio mode (/solo audio <path>) — swap every #define pair so the audio-reactive line is active and the knob line is commented out.
  • Auto-toggle (/solo or /solo <path>) — detect the shader's current mode and switch to the other one. If it's in knob mode, switch to audio; if in audio mode, switch to knobs. If the shader hasn't been knobified yet, knobify it.

The skill defaults to the first .frag file modified in the git worktree, so you can often just run /solo with no arguments.

The pattern /solo produces:

// Shoulder shrug: 0=relaxed, 1=shrugged up
#define SHOULDER_Y      (-0.02 + knob_1 * 0.015)
// #define SHOULDER_Y      (-0.02 + max(bassZScore, 0.0) * 0.015)

This is a flat comment-swap pattern (no #ifdef). Each #define has exactly one knob line and one audio line — toggle by commenting/uncommenting. The /solo skill does this swap for every pair in the shader at once.

Creating ChromaDepth Versions of Existing Shaders

ChromaDepth 3D glasses use color to create a depth illusion: red appears closest, green is middle depth, and blue/violet appears farthest. Black is neutral. To create a ChromaDepth version of an existing shader, you replace its color palette with a depth-mapped rainbow while keeping its structure, audio reactivity, and visual effects intact.

ChromaDepth Color Mapping

The core function maps a 0-1 depth value to the chromadepth rainbow using HSL:

vec3 chromadepthColor(float t, float sat, float lit) {
    t = clamp(t, 0.0, 1.0);
    // Hue: 0=red(near) → 0.75=violet(far)
    float hue = fract(t * 0.75 + seed2 * 0.15);  // seed2 shifts the mapping
    return hsl2rgb(vec3(hue, sat, lit));
}

Use HSL, not Oklch for chromadepth. The chromadepth glasses work with the physical light spectrum (red→green→blue), so HSL's hue wheel maps directly. Oklch's perceptual uniformity isn't needed here — you want the raw spectral order.

Depth Mapping Strategy

The key decision is mapping your shader's visual structure to depth (0=near, 1=far):

For fractals (Julia/Mandelbrot):

  • Interior (didn't escape) → red/warm (depth 0.0-0.35) = near
  • Boundary (escaped slowly) → green/cyan (depth 0.35-0.55) = middle
  • Exterior (escaped fast) → blue/violet (depth 0.55-0.85) = far
vec3 fractalChromadepth(float tO, float tX, float tY, float tC, float iter, bool escaped) {
    tO = sqrt(tO); tX = sqrt(tX); tY = sqrt(tY);
    float depth, brightness;

    if (!escaped) {
        // Interior — orbit traps create structural detail
        float trapDetail = min(tX, tY);
        float trapBlend = mix(tO, trapDetail, 0.5 + seed * 0.3);
        depth = clamp(trapBlend * (0.35 + seed * 0.1), 0.0, 0.35);
        brightness = 0.45 + tO * 0.15 + trapDetail * 0.15;
    } else {
        // Exterior — iteration count maps to distance
        float escapeFrac = clamp(iter / 80.0, 0.0, 1.0);
        depth = mix(0.85, 0.4, pow(escapeFrac, 0.5 + seed * 0.3));
        brightness = mix(0.1, 0.6, pow(escapeFrac, 0.6));
    }

    float sat = 0.92 - depth * 0.06;
    float lit = clamp(brightness * 0.55, 0.05, 0.55);
    return chromadepthColor(depth, sat, lit);
}

For raymarching (3D fractals, SDFs):

  • Use ray distance, world position, or orbit traps for depth
  • Mix multiple depth sources for richer variation
  • See shaders/wip/chromadepth/1.frag for a Mandelbox example

Making Each Instance Unique with Seeds

Use seed, seed2, seed3, seed4 uniforms to make each device display a unique variant:

Seed Best Use Example
seed Fractal shape, orbit trap weighting mix(tO, trapDetail, 0.5 + seed * 0.3)
seed2 Hue offset in depth mapping fract(t * 0.75 + seed2 * 0.15)
seed3 Zoom level, rotation angle, drift phase seed3 * PI * 2.0 + iTime * 0.008
seed4 Secondary effects (tendrils, background) seed4 * PI * 2.0 + iTime * 0.025

Step-by-Step Conversion Checklist

  1. Copy the original shader and rename it (e.g., wooli/1.frag → wooli/chromadepth-1.frag)
  2. Replace the color palette function (e.g., icyColor()) with a chromadepthColor() + depth mapping function
  3. Convert oklch/oklab operations to HSL — feedback decay, hue aging, etc. should use rgb2hsl/hsl2rgb
  4. Make edge glow red/orange (hue ~0.0-0.1) so it pops forward in chromadepth
  5. Make background elements blue/violet (hue ~0.6-0.7) so they recede
  6. Beat flash: shift toward red instead of just brightening — bHSL.x = fract(bHSL.x - 0.05) makes the beat "pop forward"
  7. Keep all structural code unchanged — masks, fractal iteration, feedback, infinity zoom, etc.
  8. Update metadata tags — add chromadepth, 3d to @tags
  9. Clamp final output to 0-1 — chromadepth needs clean colors, no HDR blowout

ChromaDepth Best Practices

  • High saturation (~0.9+) — desaturated colors lose the depth effect
  • Moderate lightness (0.05-0.55) — too bright washes out; too dark loses color
  • No white — white has no depth in chromadepth; clamp lightness
  • Black = neutral — use black for areas with no depth information (background)
  • Vignette helps — darkening edges keeps the depth effect centered

Reference Shaders

Shader Type Notes
claude/chromadepth-julia.frag 2D Julia set Simple chromadepth, orbit traps
claude/chromadepth-mandelbrot.frag 2D Mandelbrot Deep zoom, feedback
wip/chromadepth/1.frag 3D Mandelbox Raymarching, multiple depth sources
wooli/chromadepth-1.frag Seeded Julia + image mask Full conversion from oklch palette
wooli/chromadepth-2.frag Seeded Julia + scrolling line Scrolling tapestry with chromadepth

Audio Pipeline & Latency

Understanding the audio processing pipeline helps explain why your shader might feel sluggish or jittery:

The Three Smoothing Layers

Audio data passes through three independent smoothing stages before reaching your shader:

  1. FFT window (fftSize, default 8192) — determines frequency resolution. Larger = better bass detail but more historical audio per frame. At 44.1kHz, 8192 samples ≈ 186ms of audio.

  2. AnalyserNode smoothing (smoothingTimeConstant, default 0.4) — browser-level exponential average on raw FFT bins. Each frame: output = old × 0.4 + new × 0.6.

  3. Feature smoothing (default 0.25, z-scores get 2.5x multiplier) — exponential smoothing applied to computed features before they become shader uniforms.

The statistical features (mean, median, z-score) computed over the 500-frame history window provide additional smoothing by their nature.

Startup Jitter

The first ~2 seconds of audio are jittery because statistical features (z-scores, normalized values) are unstable with few history samples. The system applies a warmup ramp that dampens z-scores and slopes toward 0, and normalized values toward 0.5, gradually fading in over ~120 frames.

If your shader looks wild on startup but settles down, this is expected behavior.

Mobile Audio Gotchas

Browser audio processing features designed for voice calls can destroy music visualization:

  • Auto Gain Control (AGC) — compresses dynamics, squashing the transients that beat detection needs. Disabled by default.
  • Noise Suppression — targets non-voice sounds like bass and transients. Disabled by default.
  • Echo Cancellation — can interfere with audio analysis. Disabled by default.

All three can be re-enabled via query params (?autoGainControl=true) if needed for specific scenarios (e.g., very quiet room).

Tips for LED Light Sync

When creating shaders for driving Hue/Nanoleaf lights via screen scraping:

  1. Avoid rapid changes - LEDs have latency, fast changes look bad
  2. Use saturated colors - Desaturated colors look washed out on LEDs
  3. Large color regions - Small details are lost in screen sampling
  4. Smooth gradients - Harsh edges create flickering
  5. Test at low resolution - Screen scraping often downsamples significantly