robots.txt files...

[spaceace]

robots.txt can be your friend, or an enemy. if you want to block search engine indexing of certain areas of your website or forum, then this is what you use to control what are called friendly bots/spiders/crawlers. mess the contents of this file up and you can block your entire site from being indexed

i was getting about 60 pages in the error log a week for bots looking for things like the portal page that no longer exists and a couple other pages. i looked things up on the net to find out what to do with the robots.txt file as i had never had one before. here's what i found from many searches...

all settings were on one line... this is wrong. here's what was in my robots.txt file as the result of many searches and even google's webmaster tools would not display any errors or warnings

Code: Select all

User-agent: * Disallow: /portal.php Disallow: /donate/ Disallow: /downloadcentre/ Disallow: /tracker.php

now, here's how this file is supposed to look and now actually works

Code: Select all

User-agent: * 
Disallow: /portal.php
Disallow: /tracker.php
Disallow: /donate/
Disallow: /downloadcentre/

Sitemap: http://www.livemembersonly.com/sitemap.xml

everything on it's own line. i also have the path to the sitemap added as some bots were looking for sitemap.txt

[Reid]

Mine was all on one line as well. I've put mine like yours so hopefully I wont be getting so many error logs now. I also get quite a few 501 errors lately as well, not sure why.

[Kroeku39]

Yes, Robots.txt file plays an important role in on page seo improvement. I have been learning all these skills recently and getting very good results. I am going to use the professional SEO and facebook ads Los Angeles services for the off page campaigns because they need proper dedication of time and efforts. I am sure that the results will be good.