Resume Replay Feature Research
This document defines and evaluates design approaches for implementing Replay and Resume functionality in the simulator engine. These features are critical for supporting long-running simulations, fault tolerance, debugging, and reproducibility.
This wiki covers:
- Deterministic replay techniques
- State persistence strategies
- Diff frame vs key frame storage
- Resume mechanics from persisted simulation state
- UI/UX sketches
Replay refers to the ability to reconstruct a simulation’s execution history from persisted data, reproducing events, state transitions, and outputs while the user is unable to add any additional inputs while re-watching the simulator.
Replay may involve:
- Re-executing simulation logic
- Applying recorded state changes
- Replaying user inputs and external events
Resume refers to the ability to continue simulation execution from the last persisted state without reprocessing prior steps.
Resume implies:
- Restoring in-memory simulation state
- Rehydrating schedulers and queues
- Continuing time progression from a known point
- Front end page allowing users to resume.
https://notes.eatonphil.com/2024-08-20-deterministic-simulation-testing.html
Although we do not use deterministic simulation testing, it's core principals are being used to research how we will be implementing it for our own simulation engine that we are creating. Many of it's core functionality can be extended to how a simulator replays scenarios therefore researching it is great for this.
Deterministic Simulation Testing (DST) refers to structuring execution so that all sources for nondeterminism including randomness and time are controlled purely by the simulator rather than by the run time system. In the DST model, the article tells us that it typically uses a random seed derived by a function that the simulator supplies, and with this random seed, the simulator will have the exact same output/outcome every time it runs.
Key takeaways from this article:
- Controlled randomness: Random seeds can be generated to completely control randomness for a simulation engine. However in our case, we have user inputs which may affect how we will be doing this, since a random seed cannot capture events done by a user.
- Injected time: We must be able to inject our own time into the simulation rather than relying on the run time system. The underlying sim library w are using does this by allowing us to fast forward or set the .env of the sim, which is great!
- Simulated environment: To enforce determinism, even I/O events will be handled by simulation control which poses quite a struggle for us, since this is a huge functionality of our project. (knowing this we might need to alter the design from random seed)
Source:
What’s the big deal about Deterministic Simulation Testing? — Phil Eaton, August 20, 2024
Source:
https://rescale.com/documentation/main/rescale-advanced-features/checkpointing-long-simulations/
Checkpointing is the process of saving the necessary state of a running simulation so that at any point of simulation state running, we can resume whenever. This is very important for our project, since we rely on websocket connections and them usually wanting to do tasks simultaneously the chance of disconnecting is high, so a seamless resume is important! This article uses the notion of checkpointing but we use the nomenclature of KEYFRAMES
The primary goal of checkpointing is to:
- Enable simulation restart after interruption disconnect.
- Preserve progress for long-running jobs and not needing to restart if disconnected/failed.
- Provide safe recovery points during execution and for users.
Checkpointing is typically implemented at the application level, where the simulator writes restart files containing only the data required to reconstruct the simulation state. However as discussed in meetings, we will have a database table responsible for persisting keyframes for any given Simulator.
Plan Ahead
Many simulation tools require checkpointing or restart behavior to be explicitly enabled through configuration flags before execution begins, however in our case, the stakeholder has stated that by default they want every sim to be able to be resumed.
Checkpointing Too Frequently
- Increases storage consumption
- Introduces performance overhead
- Can interrupt execution and slow overall runtime
Checkpointing Too Infrequently
- Risks losing significant progress
- Increases recovery time after failures
We came up with a balanced strategy such that we save based on a configurable keyframe frequency: 1/N (By default 20 frames). This allows a balance between space and time. However once again, a key problem and complexity we face is that user inputs happen frequently so where do capture this user input in a keyframe? After deliberation with the team Brian and I came to conclusion that we will save every 1/N keyframes AS WELL AS on any sort of disconnect will persist the last frame as a keyframe.
These principles directly inform simulator resume functionality:
- Resume should be based on explicitly saved simulation state
- Checkpoints should contain only what is required to reconstruct execution
- Resume frequency must balance performance overhead with fault tolerance
This aligns with a key-frame–based resume strategy, where snapshots act as reliable restart anchors for long-running or interactive simulations.
Source:
https://users.cs.utah.edu/~regehr/papers/vee16-xentt.pdf
The deterministic principals described by Burtsev directly applies to our simulation engine but not at the hardware level he describes it at, but more for the non deterministic factors like how to handle I/O. The main point of reading this article and researching it was to gain a better understanding of how to map non deterministic behavior in a way that we can then later revisit it and always be the same output (deterministic). That was the main problem I was facing when researching this task.
Explicit handling of nondeterminism
The paper emphasizes that replay is feasible when all nondeterministic inputs are identified, classified, and mediated. Similarly, our simulator treats runtime user input and external triggers as first-class events that must be explicitly recorded or reflected in persisted state. This aligns with the paper’s distinction between deterministic execution and nondeterministic external influence.
Replay anchors vs full execution logging
Rather than recording every instruction or low-level event, our design relies on key frames as replay anchors, from which execution is deterministically rolled forward. We are using the idea of "low level events" from the paper and are just translating it to our equivalent keyframe! The same principals apply which is why it was such a good research article to use.
Controlled execution between events
As mentioned previously, a big part of this was figuring out how are we going to manage saving state of an input between keyframes, and after long discussion saving a keyframe on a user input might not be the best way and we might just need to save all diff frames into the database. I was looking for some answers on this topic but not much gained here!
Separation of deterministic and nondeterministic components
The paper’s three-part model (replayed system, deterministic environment, external world) maps naturally onto our architecture:
- The simulation core corresponds to the deterministic environment
- User input and external triggers correspond to the nondeterministic external world
- The replayed simulation instance corresponds to the replayed system
Source:
https://community.sisense.com/kb/data_sources/postgres-vs-mongodb-for-storing-json-data-%E2%80%94-which-should-you-choose/111
https://www.postgresql.org/docs/current/datatype-json.html
Since the possibility of saving entire diff frames of a simulator is a potential solution to replay feature, I wanted to do some research on possible options for other databases.
Historically, PostgreSQL and MongoDB served different purposes: PostgreSQL focused on relational, structured data, while MongoDB was designed as a native document store for JSON-like data. Over time, this distinction has blurred as PostgreSQL introduced robust support for JSON and JSONB, enabling it to store and query semi-structured data efficiently within a relational system.
I never really worked with postgress before so I decided to dive deep into their documentation in JSON types and how they are stored and what is their best way. Backend decided to just store it as a JSON string and not have much structure for our keyframes, and I advised for this because while researching this domain, I knew that the keyframe would be constantly changing. This is normal since it is all our first time building a full scale simulator and this flexibility saves us time, rather than constantly changing the data models.
JSON is a flexible, human-readable data format that allows unstructured and nested data without predefined schemas. However, JSON lacks efficient indexing. PostgreSQL’s JSONB format addresses this by storing JSON data in a binary representation, enabling indexing and faster query performance at the cost of slightly slower writes.
MongoDB uses BSON, a binary JSON-like format optimized for document storage and retrieval.
-
Schema & Constraints
- PostgreSQL supports constraints, validation, and ACID transactions, even for JSONB data.
- MongoDB uses flexible, schema-less documents and places responsibility for data correctness largely on the application.
-
Data Limits
- MongoDB restricts numeric values to 64-bit representations.
- PostgreSQL’s JSONB does not impose this limitation.
-
Scalability
- MongoDB provides built-in automatic sharding for horizontal scaling.
- PostgreSQL typically scales vertically; horizontal scaling is possible but more complex or requires external tooling.
-
Write Behavior
- MongoDB can improve write throughput by deferring disk writes, trading durability for performance.
- PostgreSQL prioritizes durability and consistency.
-
Hybrid Data Modeling
- PostgreSQL allows both structured relational data and unstructured JSONB data to coexist in the same database.
- MongoDB is optimized primarily for document-centric data models.
Although NoSQL systems are often assumed to be faster, benchmark results have shown PostgreSQL performing competitively—and in some cases outperforming MongoDB—on large-scale JSON workloads. PostgreSQL has demonstrated faster ingestion, selection, and insertion in certain benchmarks while using less disk space. MongoDB has since improved performance with newer storage engines, narrowing the gap.
As a result, performance alone is no longer a decisive factor when choosing between the two.
Due to the time constraint of the project and simplicity of iteration 2, I suggest continuing with Postgress (like we have now) but implemented the functionality in a modular way so that we can easily change the database in a future release if needed! We won't know for sure the performance concerns until the feature is finished so doing this is the best approach in my opinion.
Concept
Replay the simulation by re-running it from time t = 0 using:
- A fixed random seed
- Identical initial configuration
- Identical execution order
- Minimal storage requirements
- Conceptually simple
- Works well in purely deterministic environments
- Fails when runtime user input is introduced
- Sensitive to timing, async execution, and concurrency
- Breaks if logic or execution order changes
Examples of breaking inputs:
- WebSocket messages
- Manual user interactions
- Time-dependent or asynchronous events
Conclusion
Seed-based replay alone is insufficient for real-world simulations that accept external input. It can serve as a baseline assumption but not a complete replay strategy. For instance, a core functionality of our simulator is that a user can assign tasks at any point in time. This defeats the point of a random seed function since it would not be possible to reconstruct it. Additionally, we have future plans for traffic, thus making it even harder to replay due to the always changing traffic times. We spoke as a team and decided that if we are going to use additional persistence we might as well look into a pure persistence approach.
Concept
Persist all state changes (diffs) at every simulation step.
- Perfect replay fidelity
- Supports nondeterministic and user-driven behavior
- No reliance on deterministic execution
- High storage overhead
- Slower replay for long simulations
- Complex diff generation and application
- Difficult to version and migrate state schemas
Conclusion
This approach is technically robust but operationally expensive and difficult to maintain at scale due to the need to save every single diff frame. However, a diff frame is so low in size (approx. 4KB) such that it might be feasible to store let's say the last 7 weeks of simulators for their use or even keep them longer.
Concept
Persist:
- Periodic key frames (full state snapshots)
- Incremental diff frames between snapshots
Replay is performed by:
- Loading the nearest key frame
- Applying subsequent diff frames forward
- Balanced storage requirements
- Faster replay than full diff streams
- Supports user input and nondeterminism
- Enables resume functionality
- Requires snapshot serialization logic
- Requires schema versioning
- Requires deterministic diff application
Conclusion
this approach gives us a good balance between correctness and performance however does have a quite complex implementation process since during run time of the simulation, user inputs can occur. To preserve replay fidelity, user inputs must be treated as first-class events and captured as part of the diff stream. This would mean we would need to completely reconstruct the key frame model and how it works with user events. Failure to do so can result in divergent execution paths when applying diffs during replay.
Further more, user input may affect the internal state scheduling of simulator, event queues and execution paths within the simulation in ways that can cascade beyond a single simulation step. This increases the complexity of recreating sim state from a Diff Frame.
Knowing this, we discussed as a team that we will get metrics of how much space it would be to persist all diff frames of a single simulator, as this would be the easiest approach for us. If this is not feasible, then we will need to go with the Diff Frame + Keyframe hybrid 4.3 solution.
This diagram illustrates the replay execution flow using a batched approach and a REST API model instead of maintaining a websocket connection. The batch size will be configurable based on how smooth the skipping will be. We will need to test a sweet spot for this since to large of a batch might cause a slow response from the server, but the a to small batch size will not be enough frames for a smooth functionality. Nonetheless, the best approach for this would be to save every diff frame for maximum smoothness!
This design artifact explains how we will be resuming a simulator. Before starting this feature we knew that we need to maintain this websocket connection seen in (Step 8.) So we needed to reconstruct the sim state (roll forward) to wherever the sim stopped.(The full flow will be shown down below in the interaction diagram)
Above is our initial replay design where would fetch the keyframe assuming that the scenario was easily retrievable however after speaking with backend, I needed to make a service in order to fetch both those information to reconstruct the input parameters. As well as the, the start() method is out of scope of this diagram since we realized that we are leveraging an already existing resume end point (for active sims). Below the final version of the Resume Design. Also, for some reason I thought we needed a method to get a frame for specific tick, however I modified this so that we just retrieved the last persisted key frame which is a lot simpler! (this was done before we decided to save a keyframe on disconnect)
After iterative design and testing code and seeing what worked best, we came to a design as seen above. This final design includes additional services that would need to be added in order to fetch the correct information to restore the state of simulator. We also made a specific call to get the "last" scenario used for a simulation.
Before this ticket was being worked on and researched, all scenarios were being shared by the simulators, and if a user made a change to that scenario, all OLD sims that were based off that scenario would be modified. Therefore, we needed to persist a IMMUTABLE scenario for each simulator. This is captured by the get_latest_scenario()
Finally, a keyframe and scenario are then reconstructed into input parameters so that a sim can now restart at the keyframe is disconnected at.
A big part of the research and discussion was identifying how often we should save keyframes to save time and space (classic trade off). We came to the decision to save it every 1/N frames, such that N is configurable for the user. (default 20). However since user input is possible BEETWEEN the persistence of 2 key frames, where does that get captured? This means there was no way to restore a simulator from EXACTLY the moment it stopped, we would need to roll forward N frames on average. Therefore we came to an agreement to just save a keyframe on disconnect, which matches with the research I did prior to starting the design.
- Snapshot schema evolution
- How does this scale with traffic
- How does traffic effect the resume parser
- How does traffic effect the data models in place.
- Optimal key frame interval
- Snapshot compression strategy
- Diff frame compression strategy to just store entire diff frames?