RFC: Thor - Human/Hammer Gesture Contract
#10014
Replies: 6 comments 2 replies
-
|
@ryanbaumann @chrisgervang @ibgreen @Pessimistress @georgioskarnas @ilyabo @connor-kress @craiglabenz |
Beta Was this translation helpful? Give feedback.
-
|
Wow that's a compelling demo. I think certainly there's a lot of room for improvement in our event handler and this does demonstrate an interesting capability that a typical built-in browser gesture handler wouldn't support. I am curious to collect the current issues we have with our event handler and understand better if there's kind of a surgical approach to what we might want to change or make it more extendable. I think largely the maintainers haven't changed event handling because we felt it's so core - we don't want to break things and it wasn't necessarily clear that there was a vision for how it needed to change. This definitely brings up some interesting ideas and it'd be worth discussing it further. |
Beta Was this translation helpful? Give feedback.
-
|
I'll do a thorough triage of all the issues, discussions and proposals and create the feature tree here |
Beta Was this translation helpful? Give feedback.
-
deck.gl RFC Lineage: Views, Gesture Managers, and the AI HorizonThor lands at a genuinely significant moment in deck.gl's interaction history. To help contextualize where this RFC sits in the full arc — and to directly address the three open questions — here is a synthesis of everything relevant in the RFC corpus, issue tracker, and PRs from v4 through today. Part I: The Gesture Manager Lineage1. EventManager RFC — v4.1 (2017)
The original sin that Thor is here to resolve: in v4.1, the EventManager RFC explicitly separated gesture handling into three layers — DOM event normalization, viewport-level controller events, and model/pick events — and chose hammer.js Recognizers as the abstraction for gestural events. The factory pattern with optional Singleton was introduced specifically to prevent multiple instances fighting over This RFC is why mjolnir.js exists. It was the right call in 2017. The problem is hammer.js has been effectively unmaintained for 10 years and the abstraction layer was designed for a world of pointer and touch events, not landmark-based recognition. Key tension the RFC documented: "touch events call Mouse event handlers, simplifying app code" — this unification was a feature then. Thor needs to extend that principle: Human gestures should fire into the same handler contract as Hammer gestures, so applications do not branch on input modality. 2. Mobile Platform Support RFC
The gesture gap on mobile was acknowledged here — specifically that tilting the map required a keyboard modifier (unavailable on mobile) and needed two/three-finger gesture alternatives. The RFC chose WebView over native SDKs. This is directly relevant to Thor's Q3 (touch-specific/device-specific gestures): the precedent is to normalize platform gestures into the unified event contract, not expose device APIs directly. The recommendation was opt-in extension, not first-class bifurcation. Part II: The Views Lineage3. View Class RFC — v5.2
The View class was introduced to decouple projection logic from layout configuration. Before this, Viewport was both a mathematical projection object and a layout descriptor — the RFC split them. This is the foundational reason Thor's proposal correctly targets "gesture directly on the deck.gl Views": each View now has a stable identity ( The 4. View Class Extensions 2 RFC (2018)
Proposed adding framebuffer rendering per View (parameters object + offscreen render targets). This matters for Thor if/when gesture recognition pipelines need to process video frames — the framebuffer infrastructure is already there for compositing MediaPipe's camera feed alongside deck.gl's WebGL context. 5. View State RFC — Never Merged
This one is instructive for what not to do. The RFC proposed a unified ViewState class enabling all controllers to share a common geospatial anchor — and was explicitly marked "highly controversial, no consensus yet." The failure mode was over-centralization: trying to make one object own all state for multiple independent viewports with different controller types. Thor's gesture contract should learn from this. The event routing layer should not own view state. Keep the gesture recognizer stateless relative to viewport math; emit normalized events that the existing controller/viewState chain can consume. 6. Data View Manager RFC (2023)The most recent views-related RFC. Proposed decoupling big data loading from layer rendering via dedicated data sources and views. The architectural move here — separating "what data is visible" from "how it renders" — is analogous to what Thor proposes for interaction: separating "what gesture occurred" from "how the viewport responds." Part III: The AI AngleThere are no prior AI-specific RFCs in the deck.gl corpus. Thor is the first proposal to integrate an ML inference pipeline directly into deck.gl's interaction stack. That is the headline: MediaPipe hand landmark detection is a neural network running at ~30fps in the browser, and Thor proposes making its outputs a first-class event type alongside pointer and touch. This has meaningful precedent across adjacent ecosystems:
The "web -> mobile -> video -> voice -> VR -> full avengers" progression maps cleanly onto the RFC history: the v4.1 EventManager handled web pointer events, the Mobile RFC addressed mobile touch, Thor adds video (camera landmark detection). Voice and VR are the natural next stages of the same pipeline. One architectural note for the AI integration: MediaPipe runs its inference in a Worker (or WASM on the main thread). The event contract should be designed so the inference stage is async and off the main thread by default, with the recognizer emitting into the main thread event loop only on gesture state changes — not on every frame. This prevents blocking deck.gl's render loop. Part IV: Addressing the Open Questions with Historical ContextQ1: Gesture Conflict Resolution — Priority vs. DelayThe v4.1 EventManager used hammer.js's built-in recognizer exclusion system ( For Human gestures, the delay-based approach is almost certainly correct — not because of latency, but because human body gestures have a natural "onset" period (MediaPipe's confidence score rises over multiple frames before a gesture is stable). Recommendation: threshold on MediaPipe's confidence score per landmark cluster, not on wall-clock delay. That gives conflict resolution without artificial latency: a gesture is "committed" when confidence exceeds a tunable threshold, and competing interpretations are resolved by which reaches threshold first. Q2: Custom Gesture API — Callback vs. Declarative vs. PluginThe v4.1 RFC landed on callback-based recognizers (hammer.js Recognizer subclasses). The Streams API reference in the RFC is the most forward-compatible option: gesture events as a
Recommendation: plugin architecture as the base contract, with declarative config as a higher-level layer. This mirrors how hammer.js Manager works — plugins (Recognizers) feed into a normalized event stream, and application code registers handlers declaratively. The plugin approach also lets the human gesture recognizer be tree-shaken out for applications that do not need it. Q3: Device-Specific Gestures — First-Class vs. Opt-InThe Mobile Platform RFC established the precedent: opt-in extension, not first-class bifurcation. Pressure sensitivity and device orientation are hardware capabilities that may or may not be present; making them first-class would break the event normalization principle from v4.1 (touch events should call the same handlers as mouse events). The right model is the same as Part V: The mjolnir.js Rename / thor Package QuestionThe 1M weekly hammer.js downloads vs. the actual library being mjolnir.js is a real discoverability problem. But renaming is a breaking change for everyone currently importing mjolnir.js. A middle path: publish SummaryThor is architecturally sound and lands at the right moment. The full RFC history supports the core proposal:
The thing deck.gl has never had is inference-in-the-loop — a recognition stage that runs a model at interaction time and produces semantic gesture events. Thor is that. It completes the interaction stack from "raw input normalization" (v4.1) through "semantic gesture recognition" (Thor) to "controller response" (existing ViewState chain). Happy to help triage existing gesture/view issues and sketch the feature roadmap. The Streams API angle and the thor package naming are the two things worth nailing down early — everything else follows from those two decisions. |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
|
this feels like a separate issue - I'd start by determining where the patches specifically take effect:
Open an issue in respective repo |
Beta Was this translation helpful? Give feedback.
-
Current DemoBuilding on the RFC and the lineage analysis above, proposing a concrete next step: Thor should should fully replace the mouse The hand landmark - and future addition - streams from MediaPipe is rich enough to subsume every pointer event the existing EventManager produces. A pinch is a click. An open palm drag is a pan. Two-hand spread is zoom. The mouse becomes a fallback input, not the primary one. To make this work, users need visual feedback of the gesture state — a persistent on-screen widget that renders the detected hand skeleton in real-time. This widget should:
Full ConnectionThe widget is not just visualization — it closes the feedback loop:
The widget makes Thor usable because without visual feedback of what the system "sees," users cannot calibrate their gestures. The mouse has implicit feedback (the cursor). Thor needs explicit feedback (the hammer). @chrisgervang the surgical approach you mentioned maps well here — the widget is self-contained and the pointer emulation layer slots into the existing EventManager without touching controller internals. |
Beta Was this translation helpful? Give feedback.
-
|
Development & Discussion will continue here: https://github.com/NEW-HEAT/thor.gl for now |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Thor
js/glhand-controller.mov
Proposal
Add Mediapipe to mjolnir.js to obey interactive "Hammer" and "Human" gestures directly on the deck.gl Views
Thor - a General purpose event manager for deck.gl with an agent driven open source JS gesture package evolution of - web -> mobile -> video -> voice -> vr -> full avengers.
Developer Reach
1M weekly hammerjs downloads - no update in 10 year -> forked into mjolnor.js years ago yet google ranks outdated higher
7M weekly mediapipe/vision downloads - (most) popular human gesture open source js
110K weekly deck.gl downloads - no increase since summit PROPOSAL.
Meanwhile - ThreeJS up 300% over the year on par with mediapipe
Thor is easier to spell than mjolnir too - better accessibility
have you ever tried to type
Open Questions
GestureRecognizerinterface vs. declarative config vs. plugin architecture? Streams API?References
DeckGL Interactivity /
The one shot demo PR - publishing soon
Beta Was this translation helpful? Give feedback.
All reactions