10 Steps to a Spider Friendly Website
July 20th, 2006The 10 steps below will ensure that GoogleBot and other search engine spiders are able to crawl your site with little effort when they visit you. The search engines job is to index as many pages as possible from the web and they do this by trying to find the most efficient pages to process so that they may move on to the next. You can ensure their favor by helping them to index your site efficiently and effectively.
Proper Nesting
Many spiders are built to parse the HTML on your page. One of the things that may confuse a parser or cause it to work harder is improper nesting of HTML elements on a page. These are tags that are opened and closed out of order such as abc…cab versus abc…cba In order to have the spiders crawl through your page flawlessly, visit validator.w3.org and use their HTML validator specifically looking for error messages that say “end tag for ‘####’ omitted, but its declaration does not permit this” or “end tag for element ‘####’ which is not open”. These are errors that signify improper nesting and while they may allow some spiders to visit your website, your site will lose valuable SEO points because the spiders cannot visit your site as efficiently as it can visit other sites. Remember, search engines need to spider many pages quickly to stay in their game.
Use of Frames
This is a spider killer if there ever was one. You simply cannot link to a specific page in a framed site with any effectiveness. If you link directly to a page within the frame, the rest of the framed page disappears. If you link to the main frames page, there is no way to link to the subpages and only your homepage(s) will show. I-Frames offer a slight advantage as they are contained within a page that has it’s own URL, but many times the content within an I-Frame is inaccessible. Don’t place your navigation or main content into an I-Frame.
SessionID’s in Your URLs
These lead to a seemingly unending list of variations on each URL that have no real effect on the page. Every time a spider visits your pages, they get more and more versions of the same ‘other’ pages, but with different URLs each time because of the sessionID. If you want the spiders to feel at home on your site, then the barbed-wire fields of sessionIDs are not going to work. One way around this is to use a sitemap creation service like AutoMapIt.com that allow you to ‘ignore’ certain URL keys like the sessionID or keys used for sorting lists. This way, they have at least one link to each page that doesn’t include a sessionID. Even if they are turned off at your sitemap, when the spiders visit, they will find the IDs again.
SessionID’s may show up in otherwise friendly URLs. If your users have to log-in on your site or you use some types of tracking software, sessionID’s are what helps your site to keep visitors logged in. They are usually stored in cookies through your browser and don’t normally show up in your URL. If cookies are not available (like on most spiders), the sessionID then gets passed through the URL. You may want to turn cookies off in your browser and surf your site… do the sessionIDs show up?
Search-Friendly URLs
This goes beyond the sessionID’s in the URL and appeal more to the actual URLs being used. According to Google Webmaster Guidelines, “It helps to keep the parameters short and the number of them few”. Apache makes this easy through mod_rewrite and Windows servers offer Isapi-Rewrite to accomplish turning ‘ugly’ URLs into ’static’ URLs. You should NEVER use id= in your URLs as Google does not to spider those pages. Change it to cid= or lmnopid=, but if you use id=, your pages are already doomed from the start.
Proper Meta Tags
Despite their value decreasing, and a lot of talk about them being of no use now, meta tags are anything but dead. Google uses your meta-description as the description on their SERPs. Yahoo uses description tags and prefer sites that have them. Many other search engines use the meta keywords as well. Even if they only verify that it is present and correct, many search engines won’t touch your site without them. The meta robots tag is unnecessary unless it’s restricting access to a page. It is the spiders natural state to index a page and follow all links from it. Using it to ‘allow’ spiders only increases your code:text ratio needlessly and may hurt your potential rank by a slight amount.
Javascript/Images/Flash
Use these sparingly to accentuate your site for users. You should never rely on these for critical website functions like navigation, content, or other vital page info. Even though 98% of the internet uses javascript, turning it off along with images and multimedia in your browser (or using the Lynx browser) will allow you to visit your site as the search engines do. This 20 minute task will alert you to potential problems that your site will have when the search engines visit.
Links
Most everyone knows that links into your website bring spiders to visit and help your rank, but the links within your site to itself and other websites are also very important. Google limits you to 100 links per page and most other engines are rather close (but perhaps more generous) and allow you 150 or so links per page. This is total for links within your site and links to other sites.
Keyword stats and code:text ratio
Your goal should be to have the most content possible (human readable) while limiting the amount of code used to deliver this. Keywords should be limited to a few select phrases and possibly some variations on those phrases. If you are at all considering any meaningful SEO work on your site, you should really invest your first few dollars into finding the right keywords. There are many free keyword tools such as Wordtracker.com to at least give you a start, but the few extra dollars for the upgrades are usually well worth it if you want to target the correct markets and find terms that you weren’t expecting.
Standards Compliant/Accessible
This builds on the Validator tool used earlier from W3C. By making your website completely standards compliant, you remove any possible errors in the code that may hang-up a spider. While complete compliance is not as critical as parsing errors from open tags and truly broken code, it certainly helps to ease the process of being spidered.
Branding
The links within your site should all use the same www. or non-www. branding. Many search engines see those variants as two separate sites and Google will even apply different PR to a page based on www or non-www. This can be further enforced by changing your htaccess file to automatically switch versions if the other is visited. If you visit the non-www version of my sites, it will automatically be forwarded to the www version. This is not a critical step, but it is a major enhancement for not a lot of work involved. Why split your value in half when you can focus it like a laser? Just be sure to request links, add your link to profiles, and link to your own site from within your site using one version or the other, but not both.
Following these ten steps will help to ensure that your site is well-received by ALL of the search engine spiders and will not limit you to the bare essentials for only one spider (or bad ranking with all of the spiders equally). Let’s face it, if you have ever tried to get better rank on a search engine, you know that any break you can get will help. These steps should be performed at the creation of your web site and continue throughout it’s life-cycle to ensure that the spiders are always able to visit your site with as little effort on their part as possible.
