robots.txt files...

general issues...
Post Reply
User avatar
spaceace
Site Owner
Posts: 991
Joined: Wed Dec 16, 2009 8:30 pm
Location: Ontario, Canada
Flag: Canada
Contact:

robots.txt files...

Post by spaceace »

robots.txt can be your friend, or an enemy. if you want to block search engine indexing of certain areas of your website or forum, then this is what you use to control what are called friendly bots/spiders/crawlers. mess the contents of this file up and you can block your entire site from being indexed :roll:

i was getting about 60 pages in the error log a week for bots looking for things like the portal page that no longer exists and a couple other pages. i looked things up on the net to find out what to do with the robots.txt file as i had never had one before. here's what i found from many searches...

all settings were on one line... this is wrong. here's what was in my robots.txt file as the result of many searches and even google's webmaster tools would not display any errors or warnings
Code: Select all
User-agent: * Disallow: /portal.php Disallow: /donate/ Disallow: /downloadcentre/ Disallow: /tracker.php
now, here's how this file is supposed to look and now actually works :D
Code: Select all
User-agent: * 
Disallow: /portal.php
Disallow: /tracker.php
Disallow: /donate/
Disallow: /downloadcentre/

Sitemap: http://www.livemembersonly.com/sitemap.xml
everything on it's own line. i also have the path to the sitemap added as some bots were looking for sitemap.txt ;)

if you like my work and would like to contribute to my development of styles, please donate by using the donate button in the header.

Image

Need hosting? Click this text to check out ICDSOFT

User avatar
Reid
Newly Registered User
Posts: 1
Joined: Mon Sep 26, 2016 2:37 am
Flag: United States of America

Re: robots.txt files...

Post by Reid »

Mine was all on one line as well. I've put mine like yours so hopefully I wont be getting so many error logs now. I also get quite a few 501 errors lately as well, not sure why.
Last edited by Reid on Wed Apr 05, 2017 10:41 am, edited 1 time in total.
I recommend this Airsnore review if you want to find out how it works.

User avatar
Kroeku39
Newly Registered User
Posts: 1
Joined: Fri Jun 21, 2019 3:27 am
Flag: United States of America

Re: robots.txt files...

Post by Kroeku39 »

Yes, Robots.txt file plays an important role in on page seo improvement. I have been learning all these skills recently and getting very good results. I am going to use the professional SEO and facebook ads Los Angeles services for the off page campaigns because they need proper dedication of time and efforts. I am sure that the results will be good.
Last edited by Kroeku39 on Mon Jun 24, 2019 6:39 am, edited 2 times in total.

Post Reply