Skip to content

Migrate from Processing to p5.js #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
devin-ai-integration wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Migrate from Processing to p5.js #1

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ae24ce8
Migrate from Processing to p5.js
leandroestrella May 25, 2026
a01eed0
Fix frame availability check to mirror Processing's available()
leandroestrella May 25, 2026
bdf36c3
Support multiple video formats for cross-browser compatibility
leandroestrella May 26, 2026
dfeced1
Put MP4 first in video source order for widest compatibility
leandroestrella May 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 17 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,23 @@

![soothing chantal art cover](https://c2.staticflickr.com/6/5582/15102001878_8861ea499d.jpg)

*A generative lullaby made with processing*
*A generative lullaby made with [p5.js](https://p5js.org/)*

Documentation:
## Running

1. Place your video file at `data/lala.oggtheora.ogv`
2. Serve the project directory with any static HTTP server, for example:
```bash
# Python
python -m http.server 8000

# Node.js
npx serve .
```
3. Open `http://localhost:8000` in a browser and click to start

> A video file is required for the sketch to work. The original `.pde` Processing sketch is kept for reference.

## Documentation
- Video: http://vimeo.com/106598680 / https://youtu.be/1dVaOp7sOe4
- Audio: https://soundcloud.com/estrellaleandro/soothing-chantal
44 changes: 44 additions & 0 deletions index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>soothing chantal</title>
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.11.3/p5.min.js"></script>
<style>
html, body {
margin: 0;
padding: 0;
overflow: hidden;
background: #000;
}
canvas {
display: block;
}
#start-overlay {
position: fixed;
top: 0;
left: 0;
width: 100%;
height: 100%;
background: rgba(0, 0, 0, 0.85);
display: flex;
justify-content: center;
align-items: center;
z-index: 10;
cursor: pointer;
}
#start-overlay span {
color: #fff;
font-family: monospace;
font-size: 18px;
}
</style>
</head>
<body>
<div id="start-overlay">
<span>click to start &mdash; soothing chantal</span>
</div>
<script src="sketch.js"></script>
</body>
</html>
151 changes: 151 additions & 0 deletions sketch.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
/**
* "soothing chantal"
* a generative lullaby
* by leandro estrella
* cargocollective.com/leandroestrella
*
* Migrated from Processing to p5.js
*/

const MAX_MOVIES = 6;
const VIDEO_SRC = "data/lala.oggtheora.ogv";

let isPlaying = true;
let rand;
let oldX = 0;
let oldY = 0;

let firstClipVideo;
let myVideos = [];
let videosPlaying = [];
let started = false;
let lastFrameTime = -1;

function preload() {
// Videos are created in setup after user interaction
}

function setup() {
createCanvas(1100, 800);
background(0);
textFont("monospace");

rand = floor(random(MAX_MOVIES));

firstClipVideo = createVideo(VIDEO_SRC);
firstClipVideo.hide(); // hide the default HTML element
firstClipVideo.volume(0.5);

for (let i = 0; i < MAX_MOVIES; i++) {
myVideos[i] = createVideo(VIDEO_SRC);
myVideos[i].hide();
myVideos[i].volume(0);
}

videosPlaying.push(firstClipVideo);

// Handle click-to-start (required by browser autoplay policies)
let overlay = document.getElementById("start-overlay");
if (overlay) {
overlay.addEventListener("click", function () {
startPlayback();
overlay.style.display = "none";
});
}
}

function startPlayback() {
if (started) return;
started = true;
firstClipVideo.loop();
}

function draw() {
if (!started) {
background(0);
return;
}

// Detect new frame availability (mirrors Processing's Movie.available())
let currentTime = firstClipVideo.time();
let frameAvailable = currentTime !== lastFrameTime && firstClipVideo.width > 0;
Comment on lines +74 to +75

@devin-ai-integration devin-ai-integration Bot May 26, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Frame availability detection using time() comparison never triggers the else-if branch during playback

The currentTime !== lastFrameTime check on line 75 does not replicate Processing's Movie.available() semantics. HTML5 video's currentTime (exposed by p5's .time()) advances continuously on the media timeline — it changes on virtually every animation frame during playback, even between decoded visual frames. This means frameAvailable is always true during normal playback, and the else if (oldX <= 200) branch at line 109 effectively never fires.

In the original Processing code (soothingChantal.pde:54), available() returns true only when the decoder delivers a new visual frame (~24-30fps), making it false on roughly half of draw() calls at 60fps. The else-if branch firing regularly is what creates the generative additive video-layering effect — the core artistic feature of the piece. With the current p5.js implementation, this layering never occurs during playback.

Prompt for agents 
The frame availability detection at sketch.js:74-75 uses `firstClipVideo.time() !== lastFrameTime` to determine if a new frame is available. This doesn't replicate Processing's `Movie.available()` because HTML5 video's currentTime updates continuously (not per-decoded-frame). The else-if branch at line 109 (which adds generative video layers) essentially never fires during playback.

To fix this, you need to detect actual new visual frame delivery rather than timeline advancement. Options:

1. Use the HTMLVideoElement's `requestVideoFrameCallback()` API (modern browsers) to set a flag when a new decoded frame is available, and check/reset that flag in draw().

2. Use a threshold-based approach: treat a frame as 'new' only when currentTime has advanced by more than one video-frame-duration (e.g., 1/30 second), and set frameAvailable=false on intermediate draw() calls.

3. Draw the video to an off-screen graphics buffer and compare a hash/sample of pixels to detect actual visual changes.

Option 1 is the most faithful to the original Processing behavior. The pattern would be:
- In setup: register a callback via `firstClipVideo.elt.requestVideoFrameCallback(onNewFrame)` that sets a `newFrameReady = true` flag and re-registers itself.
- In draw: check `newFrameReady` instead of the time comparison, and reset it to false after processing.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.


if (frameAvailable) {
lastFrameTime = currentTime;
let vidWidth = firstClipVideo.width;
let vidHeight = firstClipVideo.height;
tint(255, 50);
image(firstClipVideo, 0, 0, width, height);

// Search for the brightest pixel
firstClipVideo.loadPixels();
if (firstClipVideo.pixels.length > 0) {
let brightestX = 0;
let brightestY = 0;
let brightestValue = 0;

for (let y = 0; y < vidHeight; y++) {
for (let x = 0; x < vidWidth; x++) {
let index = (y * vidWidth + x) * 4;
let r = firstClipVideo.pixels[index];
let g = firstClipVideo.pixels[index + 1];
let b = firstClipVideo.pixels[index + 2];
let pixelBrightness = (r + g + b) / 3;

if (pixelBrightness > brightestValue) {
brightestValue = pixelBrightness;
brightestY = y;
brightestX = x;
}
}
}
oldX = brightestX;
oldY = brightestY;
}
} else if (oldX <= 200) {
rand = floor(random(MAX_MOVIES));
videosPlaying.push(myVideos[rand]);
videosPlaying[videosPlaying.length - 1].loop();
}
Comment on lines +109 to +113

@devin-ai-integration devin-ai-integration Bot May 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Unbounded videosPlaying array growth causes progressive performance degradation

The else if block at sketch.js:109-113 pushes a video reference to videosPlaying on every frame where frameAvailable is false and oldX <= 200, but entries are never removed. Since draw() runs at ~60fps and frameAvailable is false whenever firstClipVideo.time() hasn't changed between frames (e.g., during buffering, at low playback speeds down to 0.0625x per line 125, or between video frames), this block can fire dozens of times per second. Notably, oldX initializes to 0 (sketch.js:19), which satisfies <= 200, so this fires continuously until the first video frame is processed. The drawing loop at sketch.js:116-122 iterates the entire array, calling image() for each entry. After running for even a minute, the array can accumulate thousands of redundant entries (all references to the same 6 video objects), causing the draw loop to perform thousands of image() calls per frame. The visual effect of repeated alpha-blended draws saturates after ~20 layers, so all additional draws are pure waste.

Impact at low playback speed

At speed 0.0625x (the minimum from map(oldX, 0, width, 0.0625, 2) when oldX is near 0), the video advances extremely slowly, so currentTime may report the same value across many consecutive draw() calls. This maximizes the rate of array growth, creating a feedback loop: low oldX → low speed → more frames with !frameAvailable → more pushes → more draws → worse performance.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.


// Draw all playing videos
for (let i = 0; i < videosPlaying.length; i++) {
let m = videosPlaying[i];
if (m.width > 0 && m.height > 0) {
tint(255, 50);
image(m, 0, 0, width, height);
}
}

// Main video speed (clamped to valid HTML5 range: 0.0625 to 16)
let newSpeed = map(oldX, 0, width, 0.0625, 2);
try {
firstClipVideo.speed(newSpeed);
} catch (e) {
// Ignore speed errors from browser restrictions
}

// Main video volume (normalized to 0-1)
let newVolume = constrain(map(oldX + oldY, 0, width, 0, 1), 0, 1);
firstClipVideo.volume(newVolume);

// Second+ video speed
let newSpeedB = map(oldY, 0, height, 0.0625, 2);
try {
myVideos[rand].speed(newSpeedB);
} catch (e) {
// Ignore speed errors from browser restrictions
}

// Visualizers (HUD overlay)
noTint();
textAlign(CENTER, TOP);
fill(255);
textSize(10);
text(nfc(newSpeed, 2) + " S", 30, 20);
text(nfc(oldY, 2) + " Y", 570, 20);
text(nfc(newSpeedB, 2) + " S", 1070, 20);
text(nfc(frameRate(), 2) + " F", 30, 785);
text(nfc(oldX, 2) + " X", 570, 785);
text(nfc(newVolume, 2) + " V", 1070, 785);
}