test: comprehensive test suite for RobotsTxtManager

Test coverage:
* can_fetch() with allow/disallow rules, overrides, wildcards, edge cases
* get_crawl_delay() with float parsing, None fallback, fetch errors
* get_request_rate() with tuple parsing, None fallback
* get_sitemaps() with multi-sitemap, empty list, error fallback
* Caching: single fetch per domain, shared cache across methods, separate cache per domain+sid
* URL construction: scheme preservation, port handling, path-independence
* Encoding: non-UTF-8 decoding, bytes handling
* Cache management: clear_all(), clear by domain, clear by sid, clear by both
* Concurrency: double-checked locking, per-domain fetching, consistent results under load

Author: AbdullahY36