robots.txt

Any issues with the site? IF YOU ARE HAVING LOGIN ISSUES COME HERE!
Post Reply
User avatar
dannykansas
Posts: 34
Joined: Sat Nov 03, 2007 3:17 pm
Location: XRoads, KC

Thu Apr 18, 2013 11:27 am

Could someone summon youthsonic? I'd like to ask that he switch up the robots.txt to Disallow everything? I believe we simply change it to:

Code: Select all

User-agent: *
Disallow: /
Image
User avatar
youthsonic
Urban Perversion
Posts: 1031
Joined: Tue Jul 05, 2005 4:58 pm
Location: KCMO
Contact:

Sun May 05, 2013 10:43 am

I modified the existing one, and un-commented the Disalow: * line.

It might be of some use, but not all spiders honor this, obviously. Which is why I am also adjusting the "Robots" area of the admin panel to also not allow any archiving through the form software itself
Post Reply