AutoMapIt Sitemap Creation Service

AutoMapIt is proud to announce a more streamlined signup process. Several steps have been fully automated making it easier than ever to start getting your website crawled for sitemaps and tons of other stats related to your websites health. We now ask for basic information such as your domain, your email address to let you know when the spider completes, and whether this will be an upgraded or free account. That’s it!

After this, my spiders will visit your website and analyze the data it collects. You will have 6 sitemap formats uploaded to your website automatically, or available for download within minutes. As always, AutoMapIt will continue to spider your website from time to time in order to create updated sitemaps regularly. We still offer site search for your website with representative images for each of your pages in the search results. Just add a small bit of code to your pages and AutoMapIt will show your product next to the link in the search results.

tags: google sitemaps, sitemap generator

The long-awaited arrival of an entirely new spidering system is beginning to take shape!!

AutoMapIt has long used a simple spider to handle all of the information that I try to cram into your site stats. The philosophy behind this new technology uses ants as inspiration. The new ants each work individually from cached copies of your pages that the scout ants bring back to the anthill. Once we have your pages stored in the anthill, an army of statistic-grabbing ants sinks their mandibles into your content and determines keyword analysis, populates the site search engine that I offer, checks for broken links, webmaster guidelines violations, and overall spiderability and health of your site.

This new technology allows me to produce sitemaps for you in a fraction of the time that it used to take meaning a shorter time to wait for your sitemaps and a more regular spidering schedule. because of the architecture, I can now deliver your sitemaps to your server before AutoMapIt finishes calculating the additional stats!

The server load has overall been reduced due to this upgrade from about 10MB per running spider down to about 1MB per ant which allows me to coordinate the timing better and shift the number of ants crawling sites to increase the stats-ants when there are fewer sites to be crawled.

I have also added an adaptive crawling feature since the spider was sped up so fast. This technology senses when your server is having a difficult time trying to keep up with the spider and slows the rate of page requests down to match your servers rate of comfortability. The system adds a longer delay between crawling your pages whenever your server slows down, automatically!

While the broken link checker has been around on AutoMapIt for a while now, I figured there had to be a good way to check inbound links coming in to your site as well. We are in the middle stages of creating a backlink checker. We find up to 1,000 or more of your backlinks and check them for a variety of stats including their rank, the number of links that are on their page, their IP, and any backlink text that is associated with the link to your site.

The goal of the inbound link checker is to allow you to look at who is pointing at you and what their link is saying about you in order to help you guide your link building campaigns using multiple class ‘C’ IP addresses and varying text that applies to your site.

Stay tuned for updates on this feature which will only be available to upgraded members. upgraded members receive access to all of our tools, increased update frequency for your sitemaps, and a whole lot of extra goodies!

Recently I was helping a client of mine with her website and noticed some strange things happening as AutoMapIt spidered her site(details excluded to protect the innocent). Many pages were coming up as ‘OK’ while the text on them read something like “Page not found, please return to the index page and try again”. Although the automated check that AutoMapIt provides for this showed that her SERVER was returning the correct 404 for ridiculous files such as quijibo869.htmlqrst, if the script that listed the articles was given a bum article, the page said “error” to a human, but returned “OK” to the search engines… after all an error page WAS being found and returned instead of a themed article.

After some URLs changed, the old ones in my system began to take on a new life of their own. All of a sudden the site-wide theme tool on AutoMapIt was showing that her site was about “page not found”. When I checked the HTTP headers coming from some of her pages, it showed a status code of 200 (OK, Page Found).

what can you expect after fixing this sort of problem? You will likely see the count of your pages fall dramatically in the search engines after this, but your ranking for ‘real’ terms may come up as a result of them no longer seeing your site as having a theme related to “page not found” (or whatever page text you use). The pages that drop off the search engine will not be real pages, but an endless string of pages that tell humans “Error” but tell the search engines “ok”.

If you are after “quality above quantity”, please use the HTTP Header Checker on this site to test your error pages. If you don’t see a 404, but instead see a 200… you’re in trouble. I know that in PHP, this can be fixed by adding a header() function that sends a proper 404 error code on your error pages before any text is output to the browser. The key point here is “before any text is output to the browser”.

I don’t think that it is the use of Google sitemaps that hurts sites so badly. Many of the sites that I’ve seen talking about being reduced are in the mega-numbers… ok, more than 100 pages and usually MUCH more.

One site that I worked on a few months back had nearly 300,000 pages indexed in Google until I made some changes to the scripting. When the site: command returned 3,000 pages, the site owner flipped!

What I was doing to the site was setting up ‘proper’ 404 redirects instead of the ‘hosting Control Panel’ variety that redirects to the homepage and returns an HTTP Status code of 200 (Found).

What had happened was they lost all of the OLD URLs that weren’t being used anymore. Once Google crawled the site and picked up the bad URLs (some were REAL badly broken), it kept returning over time. Two years later and 300,000 pages strong, this website finally ‘confessed’ to Google that most of the indexed pages were actually no longer on the server.

Enough history… When you setup an account at Google Sitemaps, they require you to ‘fix’ your 404’s that report as a 200 (Found). Once they spider their old cached URLs, they realize that many of your pages are no longer around and drop them from their index.

In the case of my client, their ranks for current pages soared after this. It seems they may have also been hit with dupe content penalties for having 297,000 copies of their homepage!

This may not be the case for all pages dropped from the index, but it explains why it happens shortly after people sign up (and fix their 404’s).