User-agent: * Disallow: /search User-agent: * Disallow: /emailprefs User-agent: Googlebot-News Disallow: /sponsored/ User-agent: Amazonbot Disallow: / User-agent: amazon-kendra Disallow: / User-agent: bedrockbot Disallow: / User-agent: Anthropic-ai Disallow: / User-agent: AwarioRssBot Disallow: / User-agent: AwarioSmartBot Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: Cohere-ai Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: FacebookBot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: meta-externalfetcher Disallow: / User-agent: Google-Extended Disallow: / User-agent: Gemini-Deep-Research Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Magpie-crawler Disallow: / User-agent: Omgili Disallow: / User-agent: Omgilibot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: YouBot Disallow: / User-agent: huggingface Disallow: / User-agent: NewsNow Disallow: / User-agent: news-please Disallow: / User-agent: Diffbot Disallow: / User-agent: Scrapy Disallow: / User-agent: TurnitinBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: AI2Bot Disallow: / User-agent: MistralAI-User Disallow: / User-agent: ChatGLM-Spider Disallow: / User-agent: Cotoyogi Disallow: / User-agent: DeepSeekBot Disallow: / User-agent: EchoboxBot Disallow: / User-agent: iaskspider Disallow: / User-agent: ICC-Crawler Disallow: / User-agent: LinerBot Disallow: / User-agent: PanguBot Disallow: / User-agent: ProRataInc Disallow: / User-agent: QualifiedBot Disallow: / User-agent: SeekrBot Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: WARDBot Disallow: / User-agent: WPBot Disallow: / User-agent: WRTNBot Disallow: / User-agent: ZanistaBot Disallow: / User-Agent: * sitemap: https://www.usnews.com/news/editorial-sitemap.xml sitemap: https://www.usnews.com/news-sitemap.xml sitemap: https://deals.usnews.com/sitemap.xml