robots.txt - Инструкции и секреты настройки

Новый шаблон 2.3 v4:

robots.txt - XF 2.3 (v4)

Код:Скопировать

Disallow: /account/
Disallow: /admin.php
Disallow: /attachments/
Disallow: /birthdays/
Disallow: /cdn-cgi/
Disallow: /conversations/
Disallow: /featured/
Disallow: /forums/*/page-*
Disallow: /goto/
Disallow: /help/
Disallow: /lfs/
Disallow: /login/
Disallow: /lost-password/
Disallow: /members/
Disallow: /misc/
Disallow: /online/
Disallow: /posts/*/reactions
Disallow: /register/
Disallow: /resources/*/field
Disallow: /resources/*/filters
Disallow: /resources/*/history
Disallow: /resources/*/reviews
Disallow: /resources/*/update/*/reactions
Disallow: /resources/*/updates
Disallow: /resources/*?prefix_id=
Disallow: /resources/authors/*/
Disallow: /resources/categories/*/featured
Disallow: /search/
Disallow: /tags/
Disallow: /threads/*/reply
Disallow: /threads/*/who-replied/
Disallow: /threads/*?prefix_id=
Disallow: /whats-new/

Sitemap: https://ВАШ_ДОМЕН/sitemap.xml

User-agent: Googlebot-News
Allow: /forums/-/index.rss

User-agent: YandexNews
Allow: /forums/-/index.rss

User-agent: adbeat_bot
User-agent: adsbot-google
User-agent: AhrefsBot
User-agent: AhrefsSiteAudit
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: BLEXBot
User-agent: BuzzSumot
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: Cliqzbot
User-agent: cohere-ai
User-agent: DataForSeoBot
User-agent: DeepCrawl
User-agent: Diffbot
User-agent: dotbot
User-agent: DotBot
User-agent: FacebookBot
User-agent: FlipboardProxy
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ia_archiver
User-agent: MegaIndex
User-agent: Mediapartners-Google
User-agent: Meta-ExternalAgent
User-agent: MJ12bot
User-agent: OAI-Embedder
User-agent: PetalBot
User-agent: PocketParser
User-agent: rogerbot
User-agent: Screaming Frog SEO Spider
User-agent: SemrushBot
User-agent: seobots
User-agent: SEOkicks
User-agent: SiteBulb
User-agent: spbot
Disallow: /

• Удален блок Yandex, потому что утратил смысловое значения по уникальности учёта дублей. Он способен всё же учитывать основной блок, но с тем условием, если его блок удален.
• Удалены избыточные правила.
• Текущая настройка более чистая и правильна по последним меркам и тестам. На текущий момент предлагаю от неё отталкиваться.
• Даже пусть у нас появится какие-то дубли, то нужно их уже внимательно проанализировать и уже после принимать меры.

Новый шаблон 2.3 v3:

robots.txt - XF 2.3 (v3)

Код:Скопировать

User-agent: *
Disallow: /account/
Disallow: /attachments/
Disallow: /admin.php
Disallow: /birthdays/
Disallow: /cdn-cgi/
Disallow: /conversations/
Disallow: /forums/*/page-*
Disallow: /featured/
Disallow: /goto/
Disallow: /help/
Disallow: /lfs/
Disallow: /login/
Disallow: /lost-password/
Disallow: /members/
Disallow: /misc/
Disallow: /misc/cookies
Disallow: /misc/style-variation
Disallow: /misc/style?*
Disallow: /resources/authors/*/
Disallow: /resources/*/history
Disallow: /resources/*/reviews
Disallow: /resources/*/updates
Disallow: /resources/*/filters
Disallow: /resources/*/update/*/reactions
Disallow: /resources/categories/*/featured
Disallow: /resources/*?prefix_id=
Disallow: /online/
Disallow: /posts/*/reactions
Disallow: /register/
Disallow: /search/
Disallow: /tags/
Disallow: /threads/*/reply
Disallow: /threads/*/who-replied/
Disallow: /whats-new/
Disallow: /*?prefix_id=
Allow: /css/
Allow: /js/
Allow: /styles/

Sitemap: https://ВАШ_ДОМЕН/sitemap.xml

User-agent: Yandex
Disallow: /account/
Disallow: /attachments/
Disallow: /admin.php
Disallow: /forums/*/page-*
Disallow: /birthdays/
Disallow: /cdn-cgi/
Disallow: /conversations/
Disallow: /featured/
Disallow: /goto/
Disallow: /help/
Disallow: /lfs/
Disallow: /login/
Disallow: /lost-password/
Disallow: /members/
Disallow: /misc/
Disallow: /misc/cookies
Disallow: /misc/style-variation
Disallow: /resources/authors/*/
Disallow: /resources/*/history
Disallow: /resources/*/reviews
Disallow: /resources/*/updates
Disallow: /resources/*/filters
Disallow: /resources/*/update/*/reactions
Disallow: /resources/categories/*/featured
Disallow: /resources/*?prefix_id=
Disallow: /online/
Disallow: /posts/*/reactions
Disallow: /register/
Disallow: /search/
Disallow: /tags/
Disallow: /threads/*/reply
Disallow: /threads/*/who-replied/
Disallow: /whats-new/
Disallow: /*?prefix_id=
Allow: /css/
Allow: /js/
Allow: /styles/

User-agent: Googlebot-News
Allow: /forums/-/index.rss

User-agent: YandexNews
Allow: /forums/-/index.rss

User-agent: adbeat_bot
User-agent: adsbot-google
User-agent: AhrefsBot
User-agent: AhrefsSiteAudit
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: BLEXBot
User-agent: BuzzSumot
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: Cliqzbot
User-agent: cohere-ai
User-agent: DataForSeoBot
User-agent: DeepCrawl
User-agent: Diffbot
User-agent: dotbot
User-agent: DotBot
User-agent: FacebookBot
User-agent: FlipboardProxy
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ia_archiver
User-agent: MegaIndex
User-agent: Mediapartners-Google
User-agent: Meta-ExternalAgent
User-agent: MJ12bot
User-agent: OAI-Embedder
User-agent: PetalBot
User-agent: PocketParser
User-agent: rogerbot
User-agent: Screaming Frog SEO Spider
User-agent: SemrushBot
User-agent: seobots
User-agent: SEOkicks
User-agent: SiteBulb
User-agent: spbot
Disallow: /

• Устранены ошибки, что ругался Google.
• Забудьте про правило Disallow: /threads/*/page-*, оказывается есть учёт смены заголовка при смене страницы. Моя ошибка, не внимательный Если пост у вас закреплен в теме, то будет ругаться на дубли мета-описания, тут без плагина не решить уже проблему дубля.
• Убраны все лишние Get-параметры, а Clean-param и Crawl-delay удалены из-за Google. Из параметров остался t и prefix_id т.к. часто встречаются. По остальным вопросы возникли об их актуальности. Многие перемеренные от 2.2 просто отсутствуют на 2.3, потому сразу не замечалось.

Новый шаблон 2.3:

robots.txt - XF 2.3 (v2)

Код:Скопировать

User-agent: *
Disallow: /account/
Disallow: /attachments/
Disallow: /admin.php
Disallow: /birthdays/
Disallow: /cdn-cgi/
Disallow: /conversations/
Disallow: /forums/*/page-*
Disallow: /featured/
Disallow: /goto/
Disallow: /help/
Disallow: /lfs/
Disallow: /login/
Disallow: /lost-password/
Disallow: /members/
Disallow: /misc/
Disallow: /misc/cookies
Disallow: /misc/style-variation
Disallow: /misc/style?*
Disallow: /resources/authors/*/
Disallow: /resources/*/history
Disallow: /resources/*/reviews
Disallow: /resources/*/updates
Disallow: /resources/*/filters
Disallow: /resources/*/update/*/reactions
Disallow: /resources/categories/*/featured
Disallow: /resources/*?prefix_id=
Disallow: /online/
Disallow: /posts/*/reactions
Disallow: /register/
Disallow: /search/
Disallow: /tags/
Disallow: /threads/*/reply
Disallow: /threads/*/who-replied/
Disallow: /whats-new/
Disallow: /*?accept=
Disallow: /*?_debug=
Disallow: /*?reject=
Disallow: /*?update=
Allow: /css/
Allow: /js/
Allow: /styles/

Sitemap: https://ВАШ_ДОМЕН/sitemap.xml

User-agent: Yandex
Crawl-delay: 1.5
Clean-param: content&user_id&prefix_id&desc&page&download_count&reject&accept&update&_debug&direction&order&tab_id&t&rating
Disallow: /account/
Disallow: /admin.php
Disallow: /forums/*/page-*
Disallow: /birthdays/
Disallow: /cdn-cgi/
Disallow: /conversations/
Disallow: /featured/
Disallow: /goto/
Disallow: /help/
Disallow: /lfs/
Disallow: /login/
Disallow: /lost-password/
Disallow: /members/
Disallow: /misc/
Disallow: /misc/cookies
Disallow: /misc/style-variation
Disallow: /resources/authors/*/
Disallow: /resources/*/history
Disallow: /resources/*/reviews
Disallow: /resources/*/updates
Disallow: /resources/*/filters
Disallow: /resources/*/update/*/reactions
Disallow: /resources/categories/*/featured
Disallow: /resources/*?prefix_id=
Disallow: /online/
Disallow: /posts/*/reactions
Disallow: /register/
Disallow: /search/
Disallow: /tags/
Disallow: /threads/*/reply
Disallow: /threads/*/who-replied/
Disallow: /whats-new/

User-agent: Googlebot-News
Allow: /forums/-/index.rss

User-agent: YandexNews
Allow: /forums/-/index.rss

User-agent: adbeat_bot
User-agent: adsbot-google
User-agent: AhrefsBot
User-agent: AhrefsSiteAudit
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: BLEXBot
User-agent: BuzzSumot
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: Cliqzbot
User-agent: cohere-ai
User-agent: DataForSeoBot
User-agent: DeepCrawl
User-agent: Diffbot
User-agent: dotbot
User-agent: DotBot
User-agent: FacebookBot
User-agent: FlipboardProxy
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ia_archiver
User-agent: MegaIndex
User-agent: Mediapartners-Google
User-agent: Meta-ExternalAgent
User-agent: MJ12bot
User-agent: OAI-Embedder
User-agent: PetalBot
User-agent: PocketParser
User-agent: rogerbot
User-agent: Screaming Frog SEO Spider
User-agent: SemrushBot
User-agent: seobots
User-agent: SEOkicks
User-agent: SiteBulb
User-agent: spbot
Disallow: /

User-agent: Applebot
Crawl-delay: 1

User-agent: Ask
Crawl-delay: 2

User-agent: Baiduspider
Crawl-delay: 3

User-agent: bingbot
Crawl-delay: 1.5

User-agent: DuckDuckBot
Crawl-delay: 1

User-agent: FindSounds
Crawl-delay: 2

User-agent: Googlebot
Crawl-delay: 0.5

User-agent: Mail.Ru
Crawl-delay: 2

User-agent: PerplexityBot
Crawl-delay: 2

User-agent: Phind
Crawl-delay: 2

User-agent: StartPage
Crawl-delay: 2

User-agent: TinEye
Crawl-delay: 2

User-agent: trendictionbot
Disallow: /

User-agent: Waldo
Crawl-delay: 2

User-agent: Wolfram
Crawl-delay: 3

User-agent: YaCy
Crawl-delay: 5

User-agent: Yeti
Crawl-delay: 2

User-agent: YouBot
Crawl-delay: 2

Упорядочен список ботов, переписаны запреты, удалено лишнего с общего блока и продублированы некоторые запреты в блок яндекса, потому что было подтверждено, что яндекс игнорит многое с общего блока.