To Catch a Spider

You may be wondering what I'm talking about, so I'll let you in on the secret. It's a mysterious creature known as the Search Engine Spider. Until this morning, even I only had a vague awareness of what they actually do. Since I've recently been educated, I thought I'd pass on my knowledge.

So we are all (hopefully) aware that search engine spiders are the vehicle that gets you to the first page of Google, but what exactly are they? What do they see when they look at your website?

Your Friendly, Neighborhood Spider
A search engine spider is a cleverly written, always changing program that the various search engines send out to continuously look over website pages. The information that they come in contact with is then stored based on the ranking, relevancy and variables set by the different search engines. Not all spiders are created equal, though, and different engines have different criteria.

What They See
A spider really doesn't care for the design of your website. They want to go straight for the heart of it, the content. They are always searching for new information and have some definite dislikes:

• Flash pages - a little judicious use of Flash is ok, but the spiders can't read anything on a website constructed entirely in Flash.
• JavaScript
• Text in Images
• Dynamic pages - identified by #, &, ? and = in the url
  

Some Likes
What makes the spiders happy? A handful of things really get them excited.

• TITLE tags
• META tags
• Heading tags
• Relevant content with good keyword phrases
  

A few things to also consider when designing and optimizing a website: duplicate content, keywords, and pleasing design. Duplicate content is a huge hang-up that I run into. If you think you can get away with it, think again. You may be successful for a while, but eventually it will catch up to you. I've seen websites results slip slowly, plummet, or even be black-listed by search engines. And these spiders are getting smarter everyday. You used to be able to just stuff keywords and locations anywhere in a sentence and the spiders would be fine with that. Now, however, they can recognize that. You need to be thoughtful and careful about where you place your keywords; they need to flow with the rest of the content. Lastly, pleasing design. You aren't just out to impress the spiders, you have to think of the people that are viewing the website. You can be top dog on Google, but if you site is poorly designed, you won't get any business out of it.