Skip to content

Commit bbcba45

Browse files
julienldclaude
julienld
and
claude
authored
feat: Improve bug report clarity and add agent behavior feedback (#401)
* fix(ci): add debug output and re-check draft status before publishing The workflow was not automatically publishing draft releases because the draft status check was happening before uploading binaries. This adds: 1. Debug output showing draft status before and after upload 2. Re-check draft status after uploading binaries 3. Use the post-upload status for the publish decision 4. Clearer messaging about what's happening This ensures releases will be automatically published after binaries are uploaded, fixing the issue where v5.1.0 and v6.0.0 remained as drafts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: improve bug report clarity and add agent behavior feedback ## Bug Report Templates - Split bug_report.md into 2 specialized templates: - runtime_bug.md: For bugs while ha-mcp is running - startup_bug.md: For installation/startup errors - Add emojis for better visual scanning - Reorganize sections for clarity (description → reproduce → expected vs actual) - Environment section now at top for quick context ## Agent Behavior Feedback - Add new template: agent_behavior_feedback.md - For reporting AI agent inefficiency, wrong tool usage, workflow suggestions - NOT for ha-mcp bugs, but for improving agent-tool interactions - Add announcement doc in local/ ## ha_bug_report Tool Improvements - Increase default tool_call_count: 3 → 10 (better log context) - Increase max limit: 50 → 100 - Auto-extract error messages from logs (new _extract_error_messages function) - Pre-fill Error Messages section automatically - Match template format EXACTLY (runtime_bug.md) - Startup logs now optional/conditional (only if non-empty) - Add emojis to generated template for clarity - Improved multiplier for log entries: 4x tool_call_count ## Changes Summary - 🎯 Better clarity: Emojis + reorganized sections - 📊 More verbose logs: 10 tool calls default (was 3) - 🚨 Auto-filled errors: No more empty error sections - 🤖 New feedback type: Agent behavior improvements - ✅ Format alignment: Tool output === GitHub template 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: rename ha_bug_report to ha_report_issue with dual template generation ## Breaking Change - Tool renamed: `ha_bug_report` → `ha_report_issue` - Return structure changed: - `bug_report_template` → `runtime_bug_template` + `agent_behavior_template` - `issue_url` removed (URLs now in templates) ## Dual Template Generation - Tool now generates BOTH templates: - `runtime_bug_template`: For ha-mcp bugs/errors - `agent_behavior_template`: For AI agent inefficiency feedback - Agent auto-selects based on conversation context - Clear instructions in return for agent to choose correctly ## Template Adaptations - runtime_bug.md: Updated to reference ha_report_issue - agent_behavior_feedback.md: Updated with auto-collection notice - Both templates now mention agent auto-detection ## Agent Auto-Detection Logic Instructions guide agent to analyze conversation: - "Error", "not working", "failed" → runtime_bug_template - "You should have", "inefficient", "wrong tool" → agent_behavior_template - If unclear, agent asks user ## Tests Updated - All 8 unit tests passing ✅ - Updated assertions for new template structure - Fixed log multiplier test (2x → 4x) - Verified both templates generated correctly ## Benefits - One tool for all report types - Agent chooses appropriate template automatically - No user confusion about which tool to call - Extensible for future report types 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: address Gemini Code Assist review comments ## Changes 1. **Fix tool_call_count limit (le=100 → le=16)** - Previous limit allowed requesting 1200 logs (100*4*3) - But ring buffer is capped at 200 entries - New limit: 16 (16*4*3=192, fits buffer) - Updated description to explain constraint 2. **Make startup_logs check robust** - Before: Checked string "(No startup logs available)" - After: Check `startup_logs` list directly - Pass list to _generate_runtime_bug_template - More robust, won't break if string changes 3. **Fix typo: ha_bug_report → ha_report_issue** - Template still referenced old tool name - Fixed in runtime bug template comment All tests passing ✅ (8/8) Addresses: #401 (comment) Addresses: #401 (comment) Addresses: #401 (comment) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
1 parent fc66001 commit bbcba45

8 files changed

Lines changed: 680 additions & 167 deletions

File tree

‎.github/ISSUE_TEMPLATE/agent_behavior_feedback.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
---
2+
name: AI Agent Behavior Feedback
3+
about: Report AI agent inefficiency, wrong tool usage, or suggest workflow improvements
4+
title: '[AGENT] '
5+
labels: agent-behavior, enhancement
6+
assignees: julienld
7+
8+
---
9+
10+
## 💡 Use the `ha_report_issue` Tool!
11+
12+
> **Ask your AI assistant:** *"You should have used a different tool"* or *"That was inefficient"*
13+
>
14+
> The agent will call `ha_report_issue()` which auto-collects tool call history and generates a template.
15+
>
16+
> The agent will automatically determine if this is agent behavior feedback vs a bug report.
17+
18+
---
19+
20+
## 🤖 What Did the AI Agent Do?
21+
22+
<!-- Describe what the AI agent did that could be improved -->
23+
<!-- Examples: -->
24+
<!-- - Used the wrong tool initially, then corrected itself -->
25+
<!-- - Provided invalid parameters to a tool -->
26+
<!-- - Made multiple unnecessary tool calls -->
27+
<!-- - Missed an obvious shortcut or better approach -->
28+
<!-- - Misinterpreted tool output -->
29+
30+
31+
## 🎯 What Should the Agent Have Done?
32+
33+
<!-- Describe the more efficient or correct approach -->
34+
35+
36+
## 📝 Conversation Context
37+
38+
<!-- Provide context about what you were trying to do -->
39+
<!-- Example: "I asked the agent to create an automation that..." -->
40+
41+
42+
---
43+
44+
## 🔧 Tool Calls Made
45+
46+
<!-- If you used ha_report_issue, the tool call sequence is auto-filled below -->
47+
48+
<details>
49+
<summary>Click to expand tool call sequence</summary>
50+
51+
```
52+
<!-- Sequence of tools the agent called -->
53+
<!-- Format: timestamp | tool_name | OK/FAIL | exec_time -->
54+
<!-- ha_report_issue automatically collects this -->
55+
```
56+
57+
</details>
58+
59+
---
60+
61+
## 💡 Suggested Improvement
62+
63+
<!-- How could the agent be improved? Options: -->
64+
65+
- [ ] **Tool documentation** - Tool description or examples need clarification
66+
- [ ] **Error messages** - Tool should return better guidance on failure
67+
- [ ] **Tool design** - Tool should accept different parameters or return more info
68+
- [ ] **Agent prompting** - System prompt should guide agent differently
69+
- [ ] **New tool needed** - Missing functionality requires a new tool
70+
- [ ] **Other** - Describe below
71+
72+
**Details:**
73+
<!-- Explain your suggestion -->
74+
75+
76+
---
77+
78+
## 📊 Environment (Optional)
79+
80+
<!-- If relevant, provide: -->
81+
- **ha-mcp Version:**
82+
- **AI Client:** (Claude Desktop / Claude Code / Other)
83+
- **Home Assistant Version:**
84+
85+
---
86+
87+
## 📎 Additional Context
88+
89+
<!-- Screenshots, conversation logs, or other helpful info -->
90+
91+
92+
---
93+
94+
**Note:** This is NOT for reporting bugs in ha-mcp itself. This is for improving how AI agents interact with ha-mcp tools.
95+
96+
If you're experiencing a bug (errors, crashes, incorrect behavior), please use the [Runtime Bug](?template=runtime_bug.md) or [Startup Bug](?template=startup_bug.md) template instead.

‎.github/ISSUE_TEMPLATE/bug_report.md‎

Lines changed: 0 additions & 64 deletions
This file was deleted.

‎.github/ISSUE_TEMPLATE/runtime_bug.md‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
---
2+
name: Runtime Bug
3+
about: Report a bug that occurs while ha-mcp is running
4+
title: '[BUG] '
5+
labels: bug
6+
assignees: julienld
7+
8+
---
9+
10+
## 🚨 Use the `ha_report_issue` Tool First!
11+
12+
> **Ask your AI assistant:** *"I want to report a bug"* or *"This isn't working"*
13+
>
14+
> The agent will call `ha_report_issue()` which auto-collects environment info, logs, and generates a ready-to-paste template.
15+
>
16+
> **Only fill this manually if** you cannot access the tool.
17+
18+
---
19+
20+
## 📋 Bug Description
21+
<!-- ONE clear sentence: What went wrong? -->
22+
23+
24+
## 🔄 Steps to Reproduce
25+
1.
26+
2.
27+
3.
28+
29+
## ✅ Expected vs ❌ Actual Behavior
30+
31+
**Expected:**
32+
<!-- What should have happened? -->
33+
34+
35+
**Actual:**
36+
<!-- What actually happened? -->
37+
38+
39+
---
40+
41+
## 🔧 Environment
42+
43+
<!-- IF YOU USED ha_report_issue: The tool already filled this section. Paste it here. -->
44+
45+
<!-- IF MANUAL: Fill these details -->
46+
- **ha-mcp Version:**
47+
- **Installation Method:** (pypi/docker/addon/git/binary)
48+
- **Platform:** (e.g., Linux, macOS, Windows)
49+
- **Python Version:**
50+
- **Home Assistant Version:**
51+
- **Connection Status:**
52+
- **Entity Count:**
53+
54+
---
55+
56+
## 🚨 Error Messages
57+
58+
```
59+
<!-- Paste error messages here -->
60+
<!-- ha_report_issue pre-fills this from logs -->
61+
```
62+
63+
---
64+
65+
## 📊 Recent Tool Calls
66+
67+
<details>
68+
<summary>Click to expand recent tool calls (ha_report_issue auto-fills this)</summary>
69+
70+
```
71+
<!-- Tool call logs showing the sequence of operations -->
72+
<!-- Format: timestamp | tool_name | OK/FAIL | exec_time -->
73+
<!-- ha_report_issue automatically collects this -->
74+
```
75+
76+
</details>
77+
78+
---
79+
80+
## 🚀 Startup Logs (if relevant)
81+
82+
<details>
83+
<summary>Click to expand startup logs</summary>
84+
85+
```
86+
<!-- Only include if the bug relates to server startup -->
87+
<!-- ha_report_issue includes this when needed -->
88+
```
89+
90+
</details>
91+
92+
---
93+
94+
## 💡 Additional Context
95+
96+
<!-- Any other relevant information: -->
97+
<!-- - Suggested fixes -->
98+
<!-- - Workarounds you found -->
99+
<!-- - Related issues -->
100+
<!-- - Configuration snippets -->
101+
102+
103+
---
104+
105+
**Privacy reminder:** Please review and anonymize sensitive information (tokens, IPs, personal names) before submitting.

‎.github/ISSUE_TEMPLATE/startup_bug.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
---
2+
name: Startup/Installation Bug
3+
about: Report a bug preventing ha-mcp from starting or installing
4+
title: '[BUG] '
5+
labels: bug, startup
6+
assignees: julienld
7+
8+
---
9+
10+
## 📋 Bug Description
11+
<!-- What error prevents ha-mcp from starting? -->
12+
13+
14+
## 🔄 Steps to Reproduce
15+
1.
16+
2.
17+
3.
18+
19+
## ✅ Expected vs ❌ Actual Behavior
20+
21+
**Expected:**
22+
<!-- Server should start successfully -->
23+
24+
25+
**Actual:**
26+
<!-- What error or failure occurs? -->
27+
28+
29+
---
30+
31+
## 🔧 Environment
32+
33+
**Fill manually (server isn't running, so ha_bug_report cannot help):**
34+
35+
- **Installation method:** (pypi/docker/addon/git clone/binary)
36+
- **OS:** (e.g., Ubuntu 24.04, Windows 11, macOS 15)
37+
- **Python version:** (e.g., 3.13)
38+
- **ha-mcp version:** (e.g., 5.0.6)
39+
- **Home Assistant version:** (e.g., 2025.1.0)
40+
41+
---
42+
43+
## 🚨 Error Messages / Logs
44+
45+
```
46+
<!-- Paste the full error output -->
47+
<!-- Include terminal output, Docker logs, or add-on logs -->
48+
```
49+
50+
---
51+
52+
## 📝 Configuration Files
53+
54+
<details>
55+
<summary>Click to expand relevant configuration</summary>
56+
57+
```yaml
58+
# Docker Compose, MCP config, or other relevant config
59+
# REMOVE TOKENS AND SECRETS before pasting
60+
```
61+
62+
</details>
63+
64+
---
65+
66+
## 💡 Additional Context
67+
68+
<!-- Any other relevant information: -->
69+
<!-- - Installation method tried -->
70+
<!-- - Network configuration (proxy, firewall) -->
71+
<!-- - Previous versions that worked -->
72+
<!-- - System-specific details (WSL, VM, etc.) -->
73+
74+
75+
---
76+
77+
**Privacy reminder:** Please REMOVE all tokens, passwords, and secrets from config files before submitting.

0 commit comments

Comments
 (0)