Best javascript editor debugger     Freeware editor 
↑


Main Page

203

The

robots.txt

file usually looks something like this:

User-agent: googlebot

Disallow: /private-files/

Disallow: /more-private-files/

User-agent: *

Disallow: /cgi-scripts/

In this example, Google’s spider (called Googlebot) is excluded from indexing files

within the two directories called

private-files

and

more-private-files

, and all robots (signi-

fied by a wild-card asterisk

*

) are excluded from indexing the directory called

cgi-scripts

.

There are numerous websites that will walk you through building and saving

your

robots.txt

file. A very clear tutorial can be found here:

www.searchengineworld.com/robots/robots_tutorial.htm

. Answers to just about any

question you could think of about robots are here:

www.robotstxt.org

. And we are

particularly fond of the regularly updated listing of robot names, available here:

www.jafsoft.com/searchengines/webbots.html

.

If you are feeling any doubt about whether your

robots.txt

file is written properly,

don’t

post it. The last thing you want to do is inadvertently shut out the search engines.

Here’s a fun experiment—what do your Big Five competitors have on

their

robots.txt

files?

Robots Meta Tags

A robots meta tag serves a similar purpose as the

robots.txt

file, but it is placed

within individual pages on your site rather than in your root directory. A robots

meta tag affects only the page it resides on. Chances are you don’t need to use this

type of tag, but here’s a quick overview in case you do. You might choose to use a

robots meta tag rather than a

robots.txt

file because you have only one or two pages

you wish to exclude on the site, or maybe you only want to do a brief, temporary

exclusion. Another possible reason is that you do not have access to the root direc-

tory on your site.

To exclude the robots from a page using the robots meta tag, simply include the

following code in the HTML head of the page:

<meta name=”robots” content=”noindex, nofollow”>

Now:

Create your

robots.txt

file and save it in the root directory of your website, or request that your

webmaster do so.

xtra

cred

c08. 8:08 203

Best javascript editor debugger     Freeware editor 
→