WalleyMilano235

Very few web master take some time to use a robots.txt file for his or her site. For search engine spiders that use the robots.txt to see what sites to search through, the robots.txt report can be extremely useful in keeping the spiders indexing your perhaps not other data and actual pages, such as looking through your figures!

The robots.txt report is useful to keep your lions from accessing elements folders and files in your hosting directory that are completely unrelated to your real website content. You can decide to have the lions kept out of parts that contain programming that search engines can't parse effectively, and to help keep them out of the net stats percentage of your site.

Many search engines cannot view dynamically generated content effectively, mostly developed by programming languages, such as PHP or ASP. If you have an online store set in your hosting account, and it's in a directory, you'd be a good idea to stop the lions from this directory so relevant information is only found by it.

The robots.txt file must certanly be placed in the directory where your main records for the hosting are situated. Which means you will be advised to produce a clear text file, and save yourself it as robots.txt, and then publish it to your hosting to the same index your index.htm file is located.

Here is samples of the utilization of the robots.txt file:

To filter an index in a robots.txt report, like a subdirectory for your online store called /store/ you'd do the following: Disallow: /store/

Another case to filter your stats directory: Disallow: /stats/

You could also want to disallow specific files that you don't want explored by the search engines. For example you dont want search.php to be parsed by the Search Engines. To achieve this you type in these on its own line:

Disallow: /search.php

After the principles discussed and making the robots.txt document, you will hold search engine spiders out of unwanted files and directories, and letting them feel the important files to see what your on line site is about! hosted ftp service