Reply to thread

There are some issues here...


The first major one is you say you want Google to index attachments, but in robots.txt you are specifically disallowing them from crawling the /attachments/ directory.


You have numerous references to directories/links only seen by logged in users. Search engines will never see these, unless you specifically give them login credentials which is uncommon.


Recommend putting directories in alphabetical order, this will help eliminate duplicates.


Your current robots.txt with notes...


[CODE]User-agent: PetalBot

User-agent: AspiegelBot

User-agent: AhrefsBot

User-agent: SemrushBot

User-agent: SemRush

User-agent: DotBot

User-agent: MauiBot

User-agent: MJ12bot

Disallow: /


User-agent: Amazonbot

Disallow: /threads/*/reply    unnecessary, these urls only show if logged in


User-agent: *

Disallow: /whats-new/

Disallow: /whats-new/*    unnecessary exact same affect as /whats-new/

Disallow: /conversations/    old reference should be /direct-messages/

Disallow: /find-threads/    unnecessary, these urls only show if logged in

Disallow: /*/create-thread    unnecessary, these urls only show if logged in

Disallow: /*/post-thread    unnecessary, these urls only show if logged in

Disallow: /login/

Disallow: /logout/    unnecessary, these urls only show if logged in

Disallow: /lost-password/

Disallow: /misc/

Disallow: /online/

Disallow: /profile-posts/    not sure this directory exists (likely now a subdirectory of /whats-new/ or /members/ which are already covered)

Disallow: /*/add-reply    unnecessary, these urls only show if logged in

Disallow: /*/approve    unnecessary, these urls only show if logged in

Disallow: /*/draft    unnecessary, these urls only show if logged in

Disallow: /*/latest    not positive, but I think this is all covered by /whats-new/

Disallow: /*/post    unnecessary, these urls only show if logged in

Disallow: /*/reply    unnecessary, these urls only show if logged in

Disallow: /*/unread     unnecessary, these urls only show if logged in

Disallow: /account/

Disallow: /members/

Disallow: /attachments/    remove if you want attachments indexed

Disallow: /goto/

Disallow: /help/    if you have any custom terms, privacy policy, or help pages you may want to remove this

Disallow: /posts/

Disallow: /login/    this is a duplicate

Disallow: /search/

Allow: /    unnecessary, this is already inferred


Sitemap: https://www.nnn.com/sitemap.xml[/CODE]


I would recommend you start with this and then make any additional adjustments:

[CODE]Sitemap: https://www.nnn.com/sitemap.xml


User-agent: PetalBot

User-agent: AspiegelBot

User-agent: AhrefsBot

User-agent: SemrushBot

User-agent: SemRush

User-agent: DotBot

User-agent: MauiBot

User-agent: MJ12bot

Disallow: /


User-agent: *

Disallow: /admin.php

Disallow: /account/

Disallow: /direct-messages/

Disallow: /goto/

Disallow: /help/

Disallow: /login/

Disallow: /lost-password/

Disallow: /members/

Disallow: /misc/

Disallow: /online/

Disallow: /posts/

Disallow: /register/

Disallow: /search/

Disallow: /whats-new/[/CODE]


Back
Top Bottom