# Allow All Search Engines, Meta Pixel, Cloudflare, AI Search Engines & LLM Bots # Crawl vs index: robots.txt cannot deindex a URL. Login, register, social # redirects, /ur, and tracking query strings stay crawlable so Google can see # noindex or 301. Disallow only admin/API/secrets. This clears GSC # "Indexed, though blocked by robots.txt". # ========================================== # Default Rules for All Crawlers & Bots # ========================================== User-agent: * Allow: / Disallow: /admin/ Disallow: /auth/ Disallow: /api/ Allow: /cdn-cgi/speculation Allow: /cdn-cgi/l/email-protection Allow: /cdn-cgi/challenge-platform/ Disallow: /cdn-cgi/ Allow: /manifest.json Allow: /ai/ Allow: /ai/*.json Disallow: /*.json Disallow: /*.sql Disallow: /*.db Disallow: /node_modules/ Disallow: /vendor/ Disallow: /storage/ Disallow: /.env Disallow: /.git # Allow Static Assets Allow: /*.css Allow: /*.js Allow: /*.woff2 Allow: /images/ Allow: /css/ Allow: /js/ # ========================================== # Google Search Console & Ad Services # ========================================== User-agent: Googlebot User-agent: Googlebot-Mobile User-agent: Googlebot-Image User-agent: Googlebot-Video User-agent: Googlebot-News User-agent: Google-InspectionTool User-agent: AdsBot-Google User-agent: Mediapartners-Google User-agent: FeedFetcher-Google User-agent: Google-Extended User-agent: GoogleOther Allow: / Disallow: /admin/ Disallow: /auth/ Disallow: /api/ Allow: /cdn-cgi/speculation Allow: /cdn-cgi/l/email-protection Allow: /cdn-cgi/challenge-platform/ Disallow: /cdn-cgi/ Allow: /manifest.json Allow: /ai/ Allow: /ai/*.json Disallow: /*.json Disallow: /*.sql Disallow: /*.db Disallow: /node_modules/ Disallow: /vendor/ Disallow: /storage/ Disallow: /.env Disallow: /.git # ========================================== # Microsoft Bing & LinkedIn # ========================================== User-agent: Bingbot User-agent: msnbot User-agent: LinkedInBot Allow: / Disallow: /admin/ Disallow: /auth/ Disallow: /api/ Allow: /cdn-cgi/speculation Allow: /cdn-cgi/l/email-protection Allow: /cdn-cgi/challenge-platform/ Disallow: /cdn-cgi/ Allow: /manifest.json Allow: /ai/ Allow: /ai/*.json Disallow: /*.json Disallow: /*.sql Disallow: /*.db Disallow: /node_modules/ Disallow: /vendor/ Disallow: /storage/ Disallow: /.env Disallow: /.git # ========================================== # AI Search & RAG Crawlers (Explicitly Allowed) # Citation bots must be named here, not only via User-agent: * # OAI-SearchBot, Claude-SearchBot, PerplexityBot, GrokBot, xAI-SearchBot, Googlebot, Applebot # ========================================== User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Claude-Web User-agent: Applebot-Extended User-agent: Amazonbot User-agent: Bytespider User-agent: cohere-ai User-agent: CCBot User-agent: Meta-ExternalAgent User-agent: Meta-ExternalFetcher User-agent: facebookexternalhit User-agent: Facebot User-agent: FacebookBot User-agent: Claude-SearchBot User-agent: GrokBot User-agent: xAI-SearchBot User-agent: Claude-User User-agent: Google-CloudVertexBot User-agent: Applebot User-agent: DuckAssistBot User-agent: PetalBot User-agent: YouBot User-agent: AI2Bot User-agent: AI2Bot-Dolma User-agent: xAI-Bot Allow: / Disallow: /admin/ Disallow: /auth/ Disallow: /api/ Allow: /cdn-cgi/speculation Allow: /cdn-cgi/l/email-protection Allow: /cdn-cgi/challenge-platform/ Disallow: /cdn-cgi/ Allow: /manifest.json Allow: /ai/ Allow: /ai/*.json Disallow: /*.json Disallow: /*.sql Disallow: /*.db Disallow: /node_modules/ Disallow: /vendor/ Disallow: /storage/ Disallow: /.env Disallow: /.git # Sitemaps (Google Search Console + Bing Webmaster Tools) # sitemap-index.xml is the only advertised sitemap. It points at sitemap.xml. # All values inside remain https://www.freightcorner.com.pk/... Sitemap: https://www.freightcorner.com.pk/sitemap-index.xml # IndexNow (Bing, Yandex, Seznam, Naver) # Key file: https://www.freightcorner.com.pk/d9a9fcb301414b438b0936e788c03e3a.txt # LLM & AI Search Engine Knowledge Bases # Primary LLM Summary: https://www.freightcorner.com.pk/llms.txt # Full Technical Corpus: https://www.freightcorner.com.pk/llms-full.txt