Feat: add missing AI search crawlers and serve robots.txt at root

[immortalcodes]

Done

• Add missing AI search crawlers as suggested in the ticket https://warthogs.atlassian.net/browse/WD-38045
• Serve robots.txt at the root (no redirect)
  Previously /robots.txt was a 302 redirect to /static/files/robots.txt, which tells crawlers to re-check the original location every crawl cycle (a wasted request per cycle).

QA

• Open the https://canonical-com-2887.demos.haus/robots.txt
• Ensure there is no redirect in networks tab
• Ensure the suggested crawlers are now part of robots.txt

Fixes

WD-38045.

[webteam-app]

Demo

Jenkins

demos.haus

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.14%. Comparing base (f92e270) to head (496e81f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2887   +/-   ##
=======================================
  Coverage   74.14%   74.14%           
=======================================
  Files          19       19           
  Lines        2352     2352           
=======================================
  Hits         1744     1744           
  Misses        608      608           

Flag	Coverage Δ
python	74.14% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This PR updates crawler directives and delivery for robots.txt, aiming to improve AI-crawler discoverability while eliminating the previous /robots.txt redirect behavior.

Changes:

• Add additional AI-related user agents to robots.txt and refresh the “Last updated” header.
• Remove the redirects.yaml entry that redirected /robots.txt to /static/files/robots.txt.
• Ensure robots.txt is included in the built rock image.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
webapp/views.py	Updates the Google Ads verification docstring to reflect “dedicated route” behavior (no longer referencing robots redirects).
rockcraft.yaml	Packages robots.txt into the rock image (flask/app/robots.txt).
robots.txt	Adds missing AI crawler user agents and updates the last-updated date.
redirects.yaml	Removes the /robots.txt redirect rule.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.