Feature Request / Bug Report
Description
When using a multi-turn online evaluator with thread idle time configured, the evaluator fires correctly after the first idle timeout. However, if a user resumes the conversation in the same thread and the thread goes idle again, the evaluator does not re-run.
Steps to Reproduce
- Configure a multi-turn online evaluator (e.g. user sentiment) with 0.5 min thread idle time
- User sends negative messages → wait for idle timeout → evaluator fires → scores negative ✓
- User resumes conversation with positive messages in the same thread → wait for idle timeout → evaluator does not fire again ✗
Expected Behavior
The evaluator should re-evaluate the full thread each time new traces are added and the thread goes idle again. Each evaluation should be stored as a separate score, preserving the history — not overwriting the previous one.
Why This Matters
In production, users frequently resume conversations. Common scenarios:
- Customer starts frustrated, gets issue resolved, leaves happy
- Customer starts happy, encounters a problem, ends frustrated
- Customer says "sorry I was rude" — but the underlying frustration still happened
Both data points are valuable. The negative moment reveals where the experience broke down. The positive follow-up shows the agent recovered. Together they enable metrics like "sentiment recovery rate" — what % of negative conversations end positively.
If the evaluator only fires once, you get a frozen snapshot that may not reflect the full interaction, and you lose insight into how your agent handles escalation and recovery.
Suggested Behavior
- Re-run the multi-turn evaluator every time the thread goes idle after new traces are added
- Store each evaluation as a new score (don't overwrite the previous one)
- This preserves the sentiment journey and enables trajectory analysis
- Optionally, surface this in the UI as a timeline of scores per thread
Documentation Gap
The current docs describe thread idle time as "time to wait after the last message before running an evaluation" but do not clarify:
- Whether the evaluator re-runs when a conversation resumes
- Whether scores are replaced or appended
- Best practices for long-lived or resumed threads
Environment
- LangSmith (cloud, March 2026)
- Python SDK: langsmith>=0.6.0
- Multi-turn evaluator with 0.5 min idle time
Feature Request / Bug Report
Description
When using a multi-turn online evaluator with thread idle time configured, the evaluator fires correctly after the first idle timeout. However, if a user resumes the conversation in the same thread and the thread goes idle again, the evaluator does not re-run.
Steps to Reproduce
Expected Behavior
The evaluator should re-evaluate the full thread each time new traces are added and the thread goes idle again. Each evaluation should be stored as a separate score, preserving the history — not overwriting the previous one.
Why This Matters
In production, users frequently resume conversations. Common scenarios:
Both data points are valuable. The negative moment reveals where the experience broke down. The positive follow-up shows the agent recovered. Together they enable metrics like "sentiment recovery rate" — what % of negative conversations end positively.
If the evaluator only fires once, you get a frozen snapshot that may not reflect the full interaction, and you lose insight into how your agent handles escalation and recovery.
Suggested Behavior
Documentation Gap
The current docs describe thread idle time as "time to wait after the last message before running an evaluation" but do not clarify:
Environment