http://www2009.eprints.org/100/1/p991.pdf

To put this as simple as possible… use sitemaps if you have any interest in being indexed faster.

Another interesting note from this study is how \”a search engine\” would use a signal-to-noise ratio to determine better quality (more trustable) sitemaps. The \’signal\’ is how many pages progress through their search system and are eventually clicked on. The \’noise\’ is how many URLs fall short of that goal and at what point they are eliminated from the process.

The lessons learned here can be achieved by using filters on your domain settings at AutoMapIt. Remove URLs from your sitemap that don\’t add content to your site… buy now, view cart, make post, reply to post, sort up/down, calendars with no events that go on forever… and plenty of others similar to these. They are very useful links to have on your site, but when was the last time you saw Google list a \’view cart\’ link? By stripping out these types of URLs from your sitemaps, you will be improving your signal-to-noise ratio and improving the chances that your most important pages are getting indexed quickly.

This news means BIG things for AutoMapIt customers since FTP is no longer a necessity… you don\’t need to host the sitemap on your own site, you just need to add a link to your robots.txt file that points to your sitemap on AutoMapIt\’s servers! No need for FTP and no need to remember to make your own uploads… Once you signup and activate your domain, your site will be picked up by the search engines!

It has never been easier to make sitemaps for your website than it is at AutoMapIt right now!

Google has used these headers when requesting your pages for years and I\’ve written about how they can help your website before, but the crawlers at AutoMapIt haven\’t adopted this practice until now.

Before you can take advantage of this bandwidth saving feature, you will need to make sure that your site supports this HTTP header request. The Header Checker here at AutoMapIt has been re-designed to show you whether your server supports this feature or not. The bottom line of the header report states whether your website will ~likely~ support this because depending on the URL that you check, it may or may not hold true for other pages of yours.

As long as the Header Checker reports that your site supports the If-Modified-Since header, there is nothing else for you to do. When AutoMapIt crawls your pages, we will automatically ask if the page has been modified since the last time we checked it. If there have been no changes, the data from the last crawl will be used and we will not download the page from your site. The extreme scenario is that this will help you save up to 7.5 GB of bandwidth per update, but this will depend on how many URLs you have and how large your files are.

What gets in the way?

Apache is built to handle these headers automatically, no tweaking needed… unless your URLs are parsed through the PHP interpreter or other server-side \’page creators\’. I know for a fact that PHP breaks this ability on Apache… not sure of other servers or pre-processing languages (ASP, PERL, ColdFusion, RUby, Python, etc.). I have a fix for this in PHP by downloading my ServHead zip file (unzip it and include() the file into your pages at the top of the page).

This file is intended to be included into your pages and works with static files passed through the PHP interpreter. If your pages are entirely generated from a database (php-nuke, osCommerce, Joomla, phpBB, and many many more), then this script will require a tweak to get the lastmod datestamp from your database. Of course, if you can\’t get this to work yourself, I can be contracted to make it work for a small fee The code in that zip is free for you to use and modify, but it is my code… I\’m keeping the rights to it so that I can continue to offer it for free… personal or commercial use.

Most session IDs use a 32 character string of numbers 0-9 and letters A-F (hex-based). This means 16 possibilities for each character to the 32 (character) power or 3 with 38 zeroes after it… yup, a lot of potential page with different URLs all saying the exact same thing as each other. By stripping out the session IDs, these pages are consolidated into one.

Another issue that a tool like this could help to fix would be the ability to exclude URLs with certain keys or key/value pairs altogether. Yes, AutoMapIt has these as well. These types of filters are handy for skipping past pages such as action=buynow or redirect_to=htpp:///xxxxxx which typically all look the same and don\’t really contain any unique content to the other pages like these…

Imagine a 1,000 page catalog… every item description page has a link to a \’print version\’, \’buy now\’, \’tell-a-friend\’… it usually doesn\’t matter which product is shown… the tell-a-friend has the same form on it with a unique identifier. The only difference between one page and the other is the product ID code. Aside from possible a 6-8 digit ID number, those are 1,000 tell-a-friend pages that are exactly the same as each other. Plus 1,000 buy now pages that are exactly like each other… and 1,000 pages worth of print versions exactly like their parent pages.

That\’s a TON of bandwidth that can be saved by using filters and tools like Dynamic URL Rewriting from Yahoo! Well, Yahoo has taken the first step toward this bright future and hopefully webmasters can continue to help Yahoo help the webmasters.

If you haven\’t done so already, get over to Yahoo! and setup your filters… then come back to AutoMapIt and use the same filters AND MORE to help out the other search engines.

tags: Dynamic URL Rewriting, sitemap, seo

AutoMapIt contains many advanced features and stats. Sometimes it helps to ask what something means or how to make changes to improve your site. This Live Help system will keep me in closer contact with you for those times that you need a little more information than what\’s written on the page.

After this, my spiders will visit your website and analyze the data it collects. You will have 6 sitemap formats uploaded to your website automatically, or available for download within minutes. As always, AutoMapIt will continue to spider your website from time to time in order to create updated sitemaps regularly. We still offer site search for your website with representative images for each of your pages in the search results. Just add a small bit of code to your pages and AutoMapIt will show your product next to the link in the search results.

tags: google sitemaps, sitemap generator

AutoMapIt has long used a simple spider to handle all of the information that I try to cram into your site stats. The philosophy behind this new technology uses ants as inspiration. The new ants each work individually from cached copies of your pages that the scout ants bring back to the anthill. Once we have your pages stored in the anthill, an army of statistic-grabbing ants sinks their mandibles into your content and determines keyword analysis, populates the site search engine that I offer, checks for broken links, webmaster guidelines violations, and overall spiderability and health of your site.

This new technology allows me to produce sitemaps for you in a fraction of the time that it used to take meaning a shorter time to wait for your sitemaps and a more regular spidering schedule. because of the architecture, I can now deliver your sitemaps to your server before AutoMapIt finishes calculating the additional stats!

The server load has overall been reduced due to this upgrade from about 10MB per running spider down to about 1MB per ant which allows me to coordinate the timing better and shift the number of ants crawling sites to increase the stats-ants when there are fewer sites to be crawled.

I have also added an adaptive crawling feature since the spider was sped up so fast. This technology senses when your server is having a difficult time trying to keep up with the spider and slows the rate of page requests down to match your servers rate of comfortability. The system adds a longer delay between crawling your pages whenever your server slows down, automatically!

The goal of the inbound link checker is to allow you to look at who is pointing at you and what their link is saying about you in order to help you guide your link building campaigns using multiple class \’C\’ IP addresses and varying text that applies to your site.

Stay tuned for updates on this feature which will only be available to upgraded members. upgraded members receive access to all of our tools, increased update frequency for your sitemaps, and a whole lot of extra goodies!

This service is free, ad-supported, and available to any member of the AutoMapIt Sitemap Creation Service. There is an upgrade available for those who wish to use this as an ad-free service. In order to keep the search results current with your website, the contents of your site are updated automatically by our spider on a daily, weekly, or monthly basis depending on your membership level.

Sign Up for AutoMapIt service or visit our Search Box forums.

Tags: Our new technology allows you to offer search for your own website content

Site-Wide Theme as of 22Jun2006:
website optimization comptrio site services google seo can web url will

is the theme for Comptrio.com which specializes in what??? Website Optimization Services

Site-Wide Theme as of 22Jun2006:
automapit posts sitemap site forum service map images topics cannot

is the theme for AutoMapIt.com Sitemap Creation Service.

Hovering over those keywords on the SEO Tool page tells me how many \’points\’ are assigned to each of those words. Now I know not only how each page is setup as far as keywords, but I know what my entire site is saying from one end to the other!

Tags: AutoMapIt has a new keyword analysis tool for all webmasters in their SEO efforts.