Update to robots

[karlcarstensen]

Updates Robots.txt

Summary by CodeRabbit

• Chores
  • Expanded robots configuration to support comprehensive crawler policies across major search engines and AI platforms (Google, Anthropic, OpenAI, Meta, Perplexity, and others).
  • Updated sitemap configuration and enabled trailing slash normalization.
  • Added noindex headers for LLM model manifest files to control indexing behavior.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
terragrunt-docs	Ready	Preview, Comment	May 8, 2026 11:25pm

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: a2c61d75-0f19-4a7e-abe6-e19d4033cf3a

📥 Commits

Reviewing files that changed from the base of the PR and between a4f808a and d0ba1c5.

📒 Files selected for processing (2)

• docs/public/robots.txt
• docs/vercel.json

---

📝 Walkthrough

Walkthrough

This PR expands the documentation site's crawler access policy and routing configuration. The robots.txt file is enhanced from a minimal ruleset to an explicit allow-all policy covering traditional search engines, AI model companies, platforms, training data crawlers, and social bots. The Vercel configuration is updated to route sitemap requests and apply a noindex header to LLM-specific manifest files.

Changes

Documentation Site Configuration

Layer / File(s)	Summary
Robots.txt Header and Global Policy docs/public/robots.txt	Adds timestamped header comment and explicit Allow: / for User-agent: *.
Traditional Search Engine Crawlers docs/public/robots.txt	Enumerates allow rules for Googlebot, Bingbot, DuckDuckGo, Yahoo Slurp, Baidu, Yandex, and Applebot.
AI Model Company Crawlers docs/public/robots.txt	Enumerates allow rules for Anthropic Claude, OpenAI GPTBot/ChatGPT, and Google AI/Gemini user agents.
Platform and Emerging AI Crawlers docs/public/robots.txt	Enumerates allow rules for Meta, Perplexity, Applebot-Extended, Azure AI, Amazon, Cohere, Mistral, DeepSeek, and xAI crawlers.
Training Data and Research Platform Crawlers docs/public/robots.txt	Enumerates allow rules for CCBot, AI2Bot, Diffbot, DataForSeoBot, Seekr, peer39_crawler, and img2dataset.
Content Intelligence and Social Crawlers docs/public/robots.txt	Enumerates allow rules for marketing intelligence bots (AddSearchBot, Meltwater, SemrushBot, Omgili) and social link preview crawlers (Twitterbot, LinkedInBot, Facebook, Slack, Discord); includes additional emerging AI crawlers.
Vercel Routing and Header Configuration docs/vercel.json	Enables trailing slashes, rewrites /sitemap.xml to /sitemap-index.xml, and sets X-Robots-Tag: noindex header for llms.txt and LLM manifest size variants.

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The pull request description is minimal and incomplete, lacking required sections from the template such as issue reference, detailed change description, TODOs checklist, and backwards compatibility confirmation.	Complete the description using the repository template: add issue reference, detailed explanation of robots.txt and vercel.json changes, and fill out all TODO items.
Title check	❓ Inconclusive	The title 'Update to robots' is vague and generic, using non-descriptive phrasing that doesn't convey the specific nature or scope of the changes to robots.txt and related configuration.	Provide a more specific title that clearly describes the main changes, such as 'Update robots.txt and Vercel configuration for comprehensive bot crawling policy'.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch robots

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}