Современный облачный хостинг провайдер | Aéza
Контакты ООО "АЕЗА ГРУПП" ИНН 2312304914 ОГРН 1212300056430
robots.txt - Инструкции и секреты настройки

robots.txt - Инструкции и секреты настройки

Обзор Обновления (2) Обсуждение

Новая стратегия XF 2.3 v3

Новый шаблон 2.3 v3:
Код: 
User-agent: *
Disallow: /account/
Disallow: /attachments/
Disallow: /admin.php
Disallow: /birthdays/
Disallow: /cdn-cgi/
Disallow: /conversations/
Disallow: /forums/*/page-*
Disallow: /featured/
Disallow: /goto/
Disallow: /help/
Disallow: /lfs/
Disallow: /login/
Disallow: /lost-password/
Disallow: /members/
Disallow: /misc/
Disallow: /misc/cookies
Disallow: /misc/style-variation
Disallow: /misc/style?*
Disallow: /resources/authors/*/
Disallow: /resources/*/history
Disallow: /resources/*/reviews
Disallow: /resources/*/updates
Disallow: /resources/*/update/*/reactions
Disallow: /resources/categories/*/featured
Disallow: /resources/*?prefix_id=
Disallow: /online/
Disallow: /posts/*/reactions
Disallow: /register/
Disallow: /search/
Disallow: /tags/
Disallow: /threads/*/page-*
Disallow: /threads/*/reply
Disallow: /threads/*/who-replied/
Disallow: /whats-new/
Disallow: /*?prefix_id=
Disallow: /*?t=
Allow: /css/
Allow: /js/
Allow: /styles/

Sitemap: https://ВАШ_ДОМЕН/sitemap.xml

User-agent: Yandex
Disallow: /account/
Disallow: /admin.php
Disallow: /forums/*/page-*
Disallow: /birthdays/
Disallow: /cdn-cgi/
Disallow: /conversations/
Disallow: /featured/
Disallow: /goto/
Disallow: /help/
Disallow: /lfs/
Disallow: /login/
Disallow: /lost-password/
Disallow: /members/
Disallow: /misc/
Disallow: /misc/cookies
Disallow: /misc/style-variation
Disallow: /resources/authors/*/
Disallow: /resources/*/history
Disallow: /resources/*/reviews
Disallow: /resources/*/updates
Disallow: /resources/*/update/*/reactions
Disallow: /resources/categories/*/featured
Disallow: /resources/*?prefix_id=
Disallow: /online/
Disallow: /posts/*/reactions
Disallow: /register/
Disallow: /search/
Disallow: /tags/
Disallow: /threads/*/page-*
Disallow: /threads/*/reply
Disallow: /threads/*/who-replied/
Disallow: /whats-new/
Disallow: /*?prefix_id=
Disallow: /*?t=

User-agent: Googlebot-News
Allow: /forums/-/index.rss

User-agent: YandexNews
Allow: /forums/-/index.rss

User-agent: adbeat_bot
User-agent: adsbot-google
User-agent: AhrefsBot
User-agent: AhrefsSiteAudit
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: BLEXBot
User-agent: BuzzSumot
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: Cliqzbot
User-agent: cohere-ai
User-agent: DataForSeoBot
User-agent: DeepCrawl
User-agent: Diffbot
User-agent: dotbot
User-agent: DotBot
User-agent: FacebookBot
User-agent: FlipboardProxy
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ia_archiver
User-agent: MegaIndex
User-agent: Mediapartners-Google
User-agent: Meta-ExternalAgent
User-agent: MJ12bot
User-agent: OAI-Embedder
User-agent: PetalBot
User-agent: PocketParser
User-agent: rogerbot
User-agent: Screaming Frog SEO Spider
User-agent: SemrushBot
User-agent: seobots
User-agent: SEOkicks
User-agent: SiteBulb
User-agent: spbot
Disallow: /
  • Устранены ошибки, что ругался Google.
  • Если считаете, что нужно индексировать страницы сообщений в темах на каждой странице, то удалите из шаблона Disallow: /threads/*/page-*, но учтите, тут такая же пагинация (дубли), у вас везде будет один и тот же заголовок, но разные мета описания в зависимости от того, какой 1 пост будет в начале каждой страницы, но если у вас 1 пост закреплен на все страницы, то у вас дубли будут всегда на заголовок и мета описание. Потому, лишний раз подумайте, стоит ли открывать.
  • Убраны все лишние Get-параметры, а Clean-param и Crawl-delay удалены из-за Google. Из параметров остался t и prefix_id т.к. часто встречаются. По остальным вопросы возникли об их актуальности. Многие перемеренные от 2.2 просто отсутствуют на 2.3, потому сразу не замечалось.

Новая стратегия XF 2.3 v2

Новый шаблон 2.3:
Код: 
User-agent: *
Disallow: /account/
Disallow: /attachments/
Disallow: /admin.php
Disallow: /birthdays/
Disallow: /cdn-cgi/
Disallow: /conversations/
Disallow: /forums/*/page-*
Disallow: /featured/
Disallow: /goto/
Disallow: /help/
Disallow: /lfs/
Disallow: /login/
Disallow: /lost-password/
Disallow: /members/
Disallow: /misc/
Disallow: /misc/cookies
Disallow: /misc/style-variation
Disallow: /misc/style?*
Disallow: /resources/authors/*/
Disallow: /resources/*/history
Disallow: /resources/*/reviews
Disallow: /resources/*/updates
Disallow: /resources/*/update/*/reactions
Disallow: /resources/categories/*/featured
Disallow: /resources/*?prefix_id=
Disallow: /online/
Disallow: /posts/*/reactions
Disallow: /register/
Disallow: /search/
Disallow: /tags/
Disallow: /threads/*/page-*
Disallow: /threads/*/reply
Disallow: /threads/*/who-replied/
Disallow: /whats-new/
Disallow: /*?accept=
Disallow: /*?_debug=
Disallow: /*?reject=
Disallow: /*?update=
Allow: /css/
Allow: /js/
Allow: /styles/

Sitemap: https://ВАШ_ДОМЕН/sitemap.xml

User-agent: Yandex
Crawl-delay: 1.5
Clean-param: content&user_id&prefix_id&desc&page&download_count&reject&accept&update&_debug&direction&order&tab_id&t&rating
Disallow: /account/
Disallow: /admin.php
Disallow: /forums/*/page-*
Disallow: /birthdays/
Disallow: /cdn-cgi/
Disallow: /conversations/
Disallow: /featured/
Disallow: /goto/
Disallow: /help/
Disallow: /lfs/
Disallow: /login/
Disallow: /lost-password/
Disallow: /members/
Disallow: /misc/
Disallow: /misc/cookies
Disallow: /misc/style-variation
Disallow: /resources/authors/*/
Disallow: /resources/*/history
Disallow: /resources/*/reviews
Disallow: /resources/*/updates
Disallow: /resources/*/update/*/reactions
Disallow: /resources/categories/*/featured
Disallow: /resources/*?prefix_id=
Disallow: /online/
Disallow: /posts/*/reactions
Disallow: /register/
Disallow: /search/
Disallow: /tags/
Disallow: /threads/*/page-*
Disallow: /threads/*/reply
Disallow: /threads/*/who-replied/
Disallow: /whats-new/

User-agent: Googlebot-News
Allow: /forums/-/index.rss

User-agent: YandexNews
Allow: /forums/-/index.rss

User-agent: adbeat_bot
User-agent: adsbot-google
User-agent: AhrefsBot
User-agent: AhrefsSiteAudit
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: BLEXBot
User-agent: BuzzSumot
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: Cliqzbot
User-agent: cohere-ai
User-agent: DataForSeoBot
User-agent: DeepCrawl
User-agent: Diffbot
User-agent: dotbot
User-agent: DotBot
User-agent: FacebookBot
User-agent: FlipboardProxy
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ia_archiver
User-agent: MegaIndex
User-agent: Mediapartners-Google
User-agent: Meta-ExternalAgent
User-agent: MJ12bot
User-agent: OAI-Embedder
User-agent: PetalBot
User-agent: PocketParser
User-agent: rogerbot
User-agent: Screaming Frog SEO Spider
User-agent: SemrushBot
User-agent: seobots
User-agent: SEOkicks
User-agent: SiteBulb
User-agent: spbot
Disallow: /

User-agent: Applebot
Crawl-delay: 1

User-agent: Ask
Crawl-delay: 2

User-agent: Baiduspider
Crawl-delay: 3

User-agent: bingbot
Crawl-delay: 1.5

User-agent: DuckDuckBot
Crawl-delay: 1

User-agent: FindSounds
Crawl-delay: 2

User-agent: Googlebot
Crawl-delay: 0.5

User-agent: Mail.Ru
Crawl-delay: 2

User-agent: PerplexityBot
Crawl-delay: 2

User-agent: Phind
Crawl-delay: 2

User-agent: StartPage
Crawl-delay: 2

User-agent: TinEye
Crawl-delay: 2

User-agent: trendictionbot
Disallow: /

User-agent: Waldo
Crawl-delay: 2

User-agent: Wolfram
Crawl-delay: 3

User-agent: YaCy
Crawl-delay: 5

User-agent: Yeti
Crawl-delay: 2

User-agent: YouBot
Crawl-delay: 2
Упорядочен список ботов, переписаны запреты, удалено лишнего с общего блока и продублированы некоторые запреты в блок яндекса, потому что было подтверждено, что яндекс игнорит многое с общего блока.
  • Мне нравится
Реакции: Efremov
Назад
Сверху Снизу