Search engine spiders are simply robot computer programs, sent out over the internet by the various search engines, to crawl websites. Their purpose is to find as many internet pages as possible, and add them to the search engine’s data base.
Don’t fear the search engine spiders. Welcome them.
Are you making the robot spiders feel welcome at your website?
How to get the spiders to visit your site
Spiders don’t simply arrive at random at your site. They must be invited, and made to feel at home. The easier you can make their work, the longer they will stay at your site, the more frequently they will visit, and the deeper into your files they will travel.
Each robot has a different timetable for visiting your website. Some can visit a site almost daily, as the Googlebot often does, while others may not crawl very frequently at all. You want to set up your site to be as inviting for as many of the spiders as possible.
If you have a new site, you want the spiders to visit you the first time to index your website in their data base. There are a few ways to send out your invitation.
One method is to find someone with a website, already indexed in the search engine data base, to link to your site. As the spider traverses their outgoing links, your incoming link will be discovered. The spider travels down the link, and finds your brand new site, and indexes it within the search engine files. Your site is then added to the list of sites to be crawled again at regular intervals.
Another method to attract the spiders is to submit your site directly to the search engines themselves. While a slow process, and often taking weeks to months for your site to receive a robot visit, search engine submissions are an option.
A third way to invite the spiders into your website parlor is to by submitting your website to the many internet directories available. Whether you choose, DMOZ, Yahoo, JoeAnt, Gimpsy, Zeal/Looksmart, or one of many lesser known directories, inclusion in them will get your website crawled by a spider. Since the various internet directories are crawled on a very regular basis, you can be almost certain a spider will travel to your website.
Welcoming the spiders
It’s important that you make your website a spider friendly place to visit.
One way to do create an inviting place for spiders to drop by is to provide them with a site map. A site map is simply a page of your website that forms an overall outline of all of your web pages. By providing the spider (and any visitors to your site for that matter) with a fully linked table of contents, you enable free and easy movement around your site.
Since all of the pages on your site should be linked, to easily enter and exit, a robot can crawl your entire site. It will happily index, in the search engine data base, all of the pages you make available. Don’t leave any dead ends, or the spiders may not return to that page.
How do I know if a spider has visited?
Because an internet spider is a simply a visitor, like anyone else, a record of their arrival will be noted in your site visitor logs. Be sure to closely examine your traffic logs, if you don’t monitor them closely already. They contain a wealth of information on all of your website traffic, not restricted simply to search engine spiders.
Knowing when the various search engine spiders visit your website enables you to know the frequency of their visits. It also helps you to know approximately when crawls of your website are being scheduled.
Feeding the spiders
Search engine spiders thrive on fresh website content. That includes all of the text, title tags, alt tags, CSS tags, and photo tags on your website. Every time you add new web pages, and updated content to your site, there is a reason for the spiders to return. Making even small changes to a file can result in a visit to your newly updated pages.
Because they are computer programs, as long as they can read and devour the content food you offer, the spiders will crawl your site. To help keep the search engine spiders coming back for more, be sure to update and add fresh new text content to your pages.
Don’t forget to keep adding new pages as well, and link them properly to your site map. The spiders can’t find and index them, if there is no open space to crawl through. As you add new content to your site, the search engines will crawl the updated pages.
Because of the importance of being crawled by the spiders from as many search engine spiders as possible, you should do everything to make them feel at home. As you add new web pages to your site, or fresh new content to already existing web pages, make sure they are spider friendly.
If you keep the incoming links to your website open, and the internal paths within your website clear, the robot spiders will visit and index them all.
A happy spider, is an welcomed spider, who is kept well fed with lot of yummy spider food.
Don’t fear the spiders. Invite them into your website.
Make them feel at home and they will return again and again.
