AutoMapIt Sitemap Creation Service

AutoMapIt now uses If-Modified-Since headers with the crawlers. What this means for you is that pages that haven’t been updated since the last time this system crawled your website won’t need to be loaded again. This saves bandwidth and means faster crawls of your website.

Google has used these headers when requesting your pages for years and I’ve written about how they can help your website before, but the crawlers at AutoMapIt haven’t adopted this practice until now.

Before you can take advantage of this bandwidth saving feature, you will need to make sure that your site supports this HTTP header request. The Header Checker here at AutoMapIt has been re-designed to show you whether your server supports this feature or not. The bottom line of the header report states whether your website will ~likely~ support this because depending on the URL that you check, it may or may not hold true for other pages of yours.

As long as the Header Checker reports that your site supports the If-Modified-Since header, there is nothing else for you to do. When AutoMapIt crawls your pages, we will automatically ask if the page has been modified since the last time we checked it. If there have been no changes, the data from the last crawl will be used and we will not download the page from your site. The extreme scenario is that this will help you save up to 7.5 GB of bandwidth per update, but this will depend on how many URLs you have and how large your files are.

What gets in the way?

Apache is built to handle these headers automatically, no tweaking needed… unless your URLs are parsed through the PHP interpreter or other server-side ‘page creators’. I know for a fact that PHP breaks this ability on Apache… not sure of other servers or pre-processing languages (ASP, PERL, ColdFusion, RUby, Python, etc.). I have a fix for this in PHP by downloading my ServHead zip file (unzip it and include() the file into your pages at the top of the page).

This file is intended to be included into your pages and works with static files passed through the PHP interpreter. If your pages are entirely generated from a database (php-nuke, osCommerce, Joomla, phpBB, and many many more), then this script will require a tweak to get the lastmod datestamp from your database. Of course, if you can’t get this to work yourself, I can be contracted to make it work for a small fee The code in that zip is free for you to use and modify, but it is my code… I’m keeping the rights to it so that I can continue to offer it for free… personal or commercial use.

This entry was posted on Wednesday, September 19th, 2007 at 11:56 am and is filed under AutoMapIt News. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.