# Lync HumanEdge
#
# /api/ is the contact endpoint: POST only, so a crawler gets a 405 and nothing
# useful. Excluded so nobody spends a crawl budget finding that out.
#
# The eight pitch concepts are NOT listed here, and that is deliberate.
# .vercelignore keeps public/concepts/ out of the upload entirely, so the build
# never sees them and /concepts/ is a 404 on the live site. A Disallow for a
# path that does not exist is noise, and it was worse than noise while they were
# deployed: a disallowed crawler never fetches the page, so it never sees the
# X-Robots-Tag: noindex that was paired with it, and a URL somebody linked could
# still be indexed with no description at all. The two rules cancelled.
User-agent: *
Disallow: /api/

# AI search and answer crawlers, allowed on purpose.
#
# This business wants to be CITED by name in AI Overviews, ChatGPT search and
# Perplexity, and blocking a retrieval crawler removes exactly the thing the
# Insights section exists to win. Every figure on this site is traced to a named
# published source, which is precisely the shape a model can attribute, so being
# readable by these agents is an asset rather than a leak.
#
# Several user-agents share one group because they share one answer. Listing
# them separately would be ten near-identical blocks to keep in step.
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Allow: /
Disallow: /api/

# Blocked deliberately: no citation channel that reaches a UK managing,
# operations or finance director, and a documented history of ignoring
# robots.txt elsewhere. No upside, some downside. Revisit if that changes.
User-agent: Bytespider
Disallow: /

Sitemap: https://lynchumanedge.co.uk/sitemap.xml