robots.txt files...

general issues...
User avatar
spaceace
Site Owner
Site Owner
Posts: 917
Joined: Wed Dec 16, 2009 8:30 pm
Location: Ontario, Canada
Flag: Canada

robots.txt files...

Post by spaceace » Sat Jan 03, 2015 9:25 am

robots.txt can be your friend, or an enemy. if you want to block search engine indexing of certain areas of your website or forum, then this is what you use to control what are called friendly bots/spiders/crawlers. mess the contents of this file up and you can block your entire site from being indexed :roll:

i was getting about 60 pages in the error log a week for bots looking for things like the portal page that no longer exists and a couple other pages. i looked things up on the net to find out what to do with the robots.txt file as i had never had one before. here's what i found from many searches...

all settings were on one line... this is wrong. here's what was in my robots.txt file as the result of many searches and even google's webmaster tools would not display any errors or warnings
Code: Select all
User-agent: * Disallow: /portal.php Disallow: /donate/ Disallow: /downloadcentre/ Disallow: /tracker.php
now, here's how this file is supposed to look and now actually works :D
Code: Select all
User-agent: * 
Disallow: /portal.php
Disallow: /tracker.php
Disallow: /donate/
Disallow: /downloadcentre/

Sitemap: http://www.livemembersonly.com/sitemap.xml
everything on it's own line. i also have the path to the sitemap added as some bots were looking for sitemap.txt ;)

if you like my work and would like to contribute to my development of styles, please donate by using the donate button in the header.

Image

Image

User avatar
Reid
Newly Registered User
Newly Registered User
Posts: 1
Joined: Mon Sep 26, 2016 2:37 am
Flag: United States of America

Re: robots.txt files...

Post by Reid » Tue Sep 27, 2016 1:39 am

Mine was all on one line as well. I've put mine like yours so hopefully I wont be getting so many error logs now. I also get quite a few 501 errors lately as well, not sure why.
Last edited by Reid on Wed Apr 05, 2017 10:41 am, edited 1 time in total.

Who is online

Users browsing this forum: CommonCrawl [Bot] and 0 guests