New spiders at AutoMapIt!
The long-awaited arrival of an entirely new spidering system is beginning to take shape!!
AutoMapIt has long used a simple spider to handle all of the information that I try to cram into your site stats. The philosophy behind this new technology uses ants as inspiration. The new ants each work individually from cached copies of your pages that the scout ants bring back to the anthill. Once we have your pages stored in the anthill, an army of statistic-grabbing ants sinks their mandibles into your content and determines keyword analysis, populates the site search engine that I offer, checks for broken links, webmaster guidelines violations, and overall spiderability and health of your site.
This new technology allows me to produce sitemaps for you in a fraction of the time that it used to take meaning a shorter time to wait for your sitemaps and a more regular spidering schedule. because of the architecture, I can now deliver your sitemaps to your server before AutoMapIt finishes calculating the additional stats!
The server load has overall been reduced due to this upgrade from about 10MB per running spider down to about 1MB per ant which allows me to coordinate the timing better and shift the number of ants crawling sites to increase the stats-ants when there are fewer sites to be crawled.
I have also added an adaptive crawling feature since the spider was sped up so fast. This technology senses when your server is having a difficult time trying to keep up with the spider and slows the rate of page requests down to match your servers rate of comfortability. The system adds a longer delay between crawling your pages whenever your server slows down, automatically!
Powered by FireStats