Added option to render a texture (e.g. Sensor output) as main image in the viewer

[daniela-hase]

Description

Adds the ability to render an arbitrary logged image — e.g. a SensorTiledCamera
color output — as the main viewer surface for a frame, instead of the 3D scene.

A new ViewerGL.log_main_image(name, image) method logs an image (via the existing
ImageLogger) and marks it as the frame's main surface. During end_frame,
ViewerGL then draws that image texture directly to the window and skips the 3D
scene render, while keeping the UI available on top. The selection is per-frame: if
a later frame does not call log_main_image(), the viewer returns to normal 3D
scene rendering automatically.

Supporting changes:

• RendererGL.render_texture() draws a texture (single image or a packed tile
  atlas) directly to the window, laying tiles out with compute_grid_layout and
  centering them, or clearing the window when no texture is available.
• The frame shader gains a uv_rect uniform so a sub-rectangle of an atlas texture
  can be sampled per tile (default identity rect preserves the existing full-frame
  blit).
• ImageLogger.get_texture() exposes live GL texture metadata
  (LoggedImageTexture), and ImageLogger.draw() accepts a hidden_name so the
  image currently shown as the main surface isn't also drawn as a floating overlay
  window.
• The example_sensor_tiled_camera example gains a "Sensor Color as Main View"
  toggle demonstrating the feature, defaulting on when the viewer supports it and
  falling back gracefully to overlay logging otherwise.

Checklist

• New or existing tests cover these changes
• The documentation is up to date with these changes
• For user-facing changes, a fragment has been added by following the
  changelog fragment instructions

Test plan

uv run --extra dev -m newton.tests -k test_viewer_main_image
uv run --extra dev -m newton.tests -k test_viewer_image_logger
uv run --extra dev -m newton.tests -k test_example_sensor_tiled_camera

• test_viewer_main_image covers the UV atlas flip, all three _update render
  branches (main-image texture, missing-image clear, normal scene), and the public
  log_main_image logging/selection and empty-name rejection.
• test_viewer_image_logger covers get_texture returning live metadata and
  returning None for missing/uninitialized images.
• test_example_sensor_tiled_camera covers the example's main-view toggle, overlay
  fallback, viewers without main-image support, and the GUI checkbox.

Manually verified in the GL viewer by running the sensor example and toggling
"Sensor Color as Main View".

New feature / API change

import newton

viewer = newton.viewer.ViewerGL()

# ... produce an RGBA image (wp.array or np.ndarray), e.g. a sensor output ...
viewer.begin_frame(time)
viewer.log_main_image("sensor", sensor_rgba)  # shown full-screen this frame
viewer.end_frame()

# A frame that omits log_main_image() renders the normal 3D scene again.

Summary by CodeRabbit

• New Features

  • Added support for displaying a logged image as the viewer’s main surface during a frame.
  • Supports tiled camera outputs while preserving UI overlays.
  • Added a tiled-camera option to show sensor color output as the main view.
  • Automatically returns to normal 3D rendering when no main image is selected.
  • Improved handling of missing or unavailable logged images.

• Documentation

  • Documented main-image display behavior and usage.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: 6a61db9a-e4f6-4065-a0ff-f7c38ecd7f9a

📥 Commits

Reviewing files that changed from the base of the PR and between 8b7e473 and b63566a.

📒 Files selected for processing (1)

• newton/tests/test_viewer_get_frame.py

🚧 Files skipped from review as they are similar to previous changes (1)

• newton/tests/test_viewer_get_frame.py

---

📝 Walkthrough

Walkthrough

ViewerGL can display a logged image as the main viewer surface for the current frame. The implementation exposes texture metadata, renders atlas tiles with configurable UVs, preserves UI rendering, restores 3D fallback behavior, and adds tiled-camera example support and tests.

Changes

Main image viewer rendering

Layer / File(s)	Summary
Texture metadata and atlas rendering newton/_src/viewer/gl/image_logger.py, newton/_src/viewer/gl/opengl.py, newton/_src/viewer/gl/shaders.py, newton/tests/test_viewer_image_logger.py, newton/tests/test_viewer_main_image.py	ImageLogger exposes live texture and atlas metadata. RendererGL.render_texture renders full textures and atlas tiles through configurable UV rectangles.
Frame-specific main image selection newton/_src/viewer/viewer_gl.py, newton/tests/test_viewer_main_image.py, newton/tests/test_viewer_get_frame.py	ViewerGL.log_main_image selects an image for one frame, replaces 3D rendering when available, hides its floating window, retains UI rendering, and resets the selection afterward.
Sensor main-view integration newton/examples/sensors/example_sensor_tiled_camera.py, newton/tests/test_example_sensor_tiled_camera.py	The tiled-camera example adds a capability-aware main-view toggle and routes sensor color output through log_main_image when enabled.
Main image API documentation docs/guide/visualization.rst, changelog/+viewer-main-image-18cc9933.added.md	The guide and changelog describe the frame-specific ViewerGL.log_main_image behavior.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

• newton-physics/newton#2578: Extends the shared ViewerGL image-logging infrastructure.
• newton-physics/newton#2733: Relates to ViewerGL.clear_model() and image-logger lifecycle handling.

Suggested reviewers: eric-heiden

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 42.19% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: rendering a texture, such as sensor output, as the viewer's main image.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@newton/_src/viewer/gl/opengl.py`:
- Around line 1392-1458: Update the main-image rendering path around the
framebuffer setup and texture-drawing loop to bind and render into
renderer._frame_fbo instead of the default framebuffer, so ViewerGL.get_frame()
reads the selected image. Preserve the existing screen presentation behavior as
needed after rendering, and add a headless get_frame() regression test covering
a main-image frame.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: 396b22bc-0b88-470f-9721-9e2c74894daf

📥 Commits

Reviewing files that changed from the base of the PR and between 05ca05a and 4a7f0b7.

📒 Files selected for processing (10)

• changelog/+viewer-main-image-18cc9933.added.md
• docs/guide/visualization.rst
• newton/_src/viewer/gl/image_logger.py
• newton/_src/viewer/gl/opengl.py
• newton/_src/viewer/gl/shaders.py
• newton/_src/viewer/viewer_gl.py
• newton/examples/sensors/example_sensor_tiled_camera.py
• newton/tests/test_example_sensor_tiled_camera.py
• newton/tests/test_viewer_image_logger.py
• newton/tests/test_viewer_main_image.py

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@newton/tests/test_viewer_get_frame.py`:
- Around line 117-121: Update the ViewerGL construction handling in the test
around newton.viewer.ViewerGL to catch only the specific exception(s) indicating
an unavailable graphics backend, preserving the existing skip behavior for those
cases. Remove the broad Exception catch so unexpected headless initialization
failures propagate and fail the test.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: 1fec9a5d-94cb-4546-9c14-be6aca5d6dcb

📥 Commits

Reviewing files that changed from the base of the PR and between 4a7f0b7 and c9b4c29.

📒 Files selected for processing (2)

• newton/_src/viewer/gl/opengl.py
• newton/tests/test_viewer_get_frame.py

[codecov]

Codecov Report

❌ Patch coverage is 98.13084% with 2 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
newton/_src/viewer/gl/image_logger.py	93.75%	1 Missing ⚠️
newton/_src/viewer/viewer_gl.py	95.83%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[mmacklin]

Instead of a new function could we make this an option on the viewer.log_image() method?

In general we should name things with common prefixes so they're grouped together, e.g.: viewer.log_image_main(), but in this case I think an option is better, e.g.: "maximize=True" or "docked=True".

[mmacklin]

Also a little surprised at the size of this change, if it's just adding a docked image log function then do we really need all the internal refactoring as well? Given the late stage of the PR it would be good to keep it as minimal as possible.