How Spiders Are Like People
One of the most rewarding things I've done over the past year is to rediscover the world of search engine optimization. Having first encountered SEO in the mid-to-late 90s, I have to admit I viewed search engine marketers as being roughly on par with pet psychics, professionally speaking. I've been amazed and gratified to learn just how much the SEO world has changed, and how relevant those changes are to usability.
One of my more pleasant discoveries is the many ways in which search engine spiders (the programs that "crawl" the web, looking for content) have been increasingly designed to mimic human behavior. The upshot is that, more and more, the same tactics that are good for usability are also good for SEO. So, what do spiders and people have in common?
Spiders & People Like Words
Spiders were designed to parse text in HTML files, and words are still what they digest most easily. People, even as they enjoy more rich media, are still drawn to text, as evidenced by the seemingly never-ending rise in blogging and user-driven content. It's been said many times by many people, but content is still king, and good content attracts both human visitors and search spiders.
Spiders & People Follow Links
As young as the web is, it's easy to forget that hypertext links were very recently a revolutionary concept. The inventors of the worldwide web tackled the difficult hurdle of creating a document language that could be used cross-platform, but the real genius was in the idea of making content active, and building a structure in which documents could be linked in a contextual and non-linear fashion. Links are still fundamental to the web, and they remain the primary way that both people and spiders travel from site to site and document to document.
Spiders & People Like New Things
We're all suckers for novelty, and spiders are no exception. Both the frequency and depth with which spiders visit a site are affected by how often new content is created, one of the reasons that blogs tend to fare well in search results. Likewise, visitors come back to sites that have regularly (and predictably) updated content.
Spiders & People Like What's Popular
It's ok to admit it: you've occasionally, "accidentally" clicked on a link about Britney Spears' downfall or OJ's latest arrest. Turns out, spiders are suckers for gossip, too. The landscape changed when Google first introduced its
PageRank algorithm, which attempted to map how sites were linked together to develop a crude measure of popularity. Although Google's algorithms have become much more complex, the core ideas of popularity and even trust have become gradually more codified, and spiders tend to follow content that attracts the most people.
The Good News for Usabillity
The upshot of all of this is that you no longer have to choose whether to build for people or build for search engines. There are, of course, specialized tactics in both SEO and usability, but designing a usable website goes a long way towards improving visibility to search spiders.
Next time: the obvious companion piece, "How Spiders Are NOT Like People".
