Don't disallow llms.txt and .md in robots.txt

They already send X-Robots-Tag: noindex, which keeps them out of search results while staying readable by AI agents. A robots.txt Disallow blocks fetching entirely (so the noindex is never seen) and blocks agents from llms.txt, defeating its purpose.

Author: akorchev

RadzenBlazorDemos.Tools/Program.cs

@@ -1528,8 +1528,6 @@ static void WriteRobotsTxt(string filePath)
         sb.AppendLine("User-agent: *");
         sb.AppendLine("Content-Signal: ai-train=yes, search=yes, ai-input=yes");
         sb.AppendLine($"Sitemap: {BaseUrl}/sitemap.xml");
-        sb.AppendLine("Disallow: /llms.txt");
-        sb.AppendLine("Disallow: /*.md$");
         File.WriteAllText(filePath, sb.ToString(), new UTF8Encoding(false));
     }
 }