Google's John Mueller has confirmed that the Cloudflare-invented "Content Signals" robots.txt directive has "no effects whatsoever for any crawler or LLM." If you added it hoping to influence how Google, Bing, or AI engines treat your content, it isn't doing anything except cluttering your robots.txt file. Here's what actually controls crawling and AI usage of your site — and what to check before Cloudflare's September 15, 2026 default changes hit.

What changed

On July 6, 2026, Google's John Mueller responded to a Reddit thread in r/TechSEO where a site owner was testing whether "Content-Signal" headers and llms.txt files help with entity disambiguation. His answer was blunt:

  • Google does not use llms.txt or llms-author.txt. He's not aware of any other crawler or LLM confirming they use these either (outside of some SEO tools).
  • No crawler or LLM uses the content-signal robots.txt directives, as far as Google knows. The directive was "made up by a CDN" (Cloudflare). Using it "just adds bloat and future maintenance to your robots.txt file."

The "Content Signals" system (documented at contentsignals.org) was introduced by Cloudflare to let sites express preferences about how their content is used — with three classifications: Search, Training (AI model training), and Agent (AI agents/answer engines). The idea is a machine-readable "please don't use my content to train AI" flag layered into robots.txt.

The catch: a robots.txt directive only does something if the crawler on the other end chooses to honor it. Mueller is saying Google reads the directives it supports and ignores everything else — and it does not support Content Signals.

Important context on timing: Cloudflare has set a deadline of September 15, 2026, when it will apply new defaults. For all new domains onboarding to Cloudflare, the Training and Agent categories will be blocked by default on pages that display ads, while Search will remain allowed by default. Cloudflare powers roughly 21% of all websites, so this default shift will quietly touch a huge slice of the web.

Why it matters for rankings on Google, Bing, and other search engines

  • False sense of control. If you (or a plugin, or your host) added Content Signals expecting it to keep your content out of AI Overviews or ChatGPT, it isn't. Google grounds AI Overviews and AI Mode using its normal index — Content Signals doesn't change that.
  • The real risk is over-blocking, not under-blocking. When Cloudflare flips its September defaults, a misconfigured setting could block legitimate crawlers or hurt how ad-supported pages are accessed. The bigger ranking danger is accidentally blocking something you needed indexed.
  • llms.txt is not a ranking file. Despite the hype, Google has repeatedly said llms.txt neither helps nor hurts. Time spent hand-maintaining it is time not spent on content and technical fundamentals that do move rankings.
  • robots.txt hygiene is a real signal of technical health. A bloated robots.txt with unsupported directives increases the chance of a typo or conflicting rule that blocks Googlebot or Bingbot from something important.

What to do about it

  1. Audit your current robots.txt today. Open https://yourdomain.com/robots.txt and read every line. Note anything referencing Content-Signal, content-signal, llms.txt, or AI-training directives.
  2. Confirm nothing important is blocked. In Google Search Console, use the URL Inspection tool on your key pages and the robots.txt report to verify Googlebot can crawl them. Do the same in Bing Webmaster Tools.
  3. Decide your actual AI stance deliberately. If you genuinely want to limit AI use of your content, use the mechanisms that engines actually honor:
    • Google-Extended user-agent in robots.txt to opt out of Gemini/Vertex AI training (this is respected by Google, and is separate from Search indexing).
    • The nosnippet, max-snippet, and data-nosnippet directives to control how much of your text can appear in snippets and AI Overviews.
    • Google Search Console's Search generative AI controls (Settings → Search generative AI) to influence whether your content can appear in and ground AI Overviews, AI Mode, and generative AI in Discover.
  4. Check your Cloudflare AI settings before September 15, 2026. If you use Cloudflare, log into the dashboard and review the AI/bot and content-usage settings now. Confirm the Search category stays allowed and that any Training/Agent blocking matches your intent — especially on ad-supported pages.
  5. Keep robots.txt lean. Remove directives that no crawler honors. Every line should have a purpose you can explain.
  6. Re-test after any change. Fetch the live robots.txt again and re-inspect key URLs so a cleanup doesn't accidentally block a section of the site.

Common mistakes / what to avoid

  • Assuming a directive works just because a vendor added it. A CDN or plugin can write anything into robots.txt; only directives the crawler supports have any effect.
  • Blocking crawlers to "protect" content from AI, then losing Search visibility. If you block Googlebot itself, you lose classic rankings too — AI opt-out and Search indexing are different levers.
  • Treating llms.txt as an SEO deliverable. It's optional, unsupported by Google, and not a ranking factor.
  • Ignoring the September 15 Cloudflare change and being surprised later when default AI-blocking behavior changes on your ad pages.
  • Editing robots.txt without re-testing. One stray Disallow: can deindex an entire directory.

Quick-win checklist

  • Read your live robots.txt line by line
  • Remove unsupported Content-Signal / content-signal lines
  • Stop hand-maintaining llms.txt as a ranking tactic
  • Verify Googlebot & Bingbot can crawl your money pages (GSC + Bing WMT)
  • Set a real AI stance with Google-Extended, nosnippet/data-nosnippet, and GSC AI controls
  • Review Cloudflare AI/content settings before September 15, 2026
  • Re-test robots.txt and key URLs after any edit

Sources