by MaxPowers » May 8th, 2009, 1:57 pm
Hello Mary,
What you are looking at are the HTTP Headers sent by your server.
When a browser (or spider) tries to request a page from a webserver, there is some communication that normally runs behind the scenes, hidden from you as a web-surfer, but clearly visible and even critical to how your browser handles the URL and what it gets back as far as a webpage.
The same holds true for what happens when a web crawler visits your pages... the request is formatted so that web server understands what the spider wants. The server will respond in the cryptic code you see below which tells the spider how to handle the connection, if there is a server-side redirect involved (not a meta tag in the HTML, but a redirect defined from within the web server).
The headers you see below contain all of the info used to identify the server, the file, and the status of the connection. The status of the connection is one of the most important things to watch when trying to improve the spider-friendliness of your server. It is the very first line below "HTTP/1.1 200 OK" and this one tells the browser (or spider) that the URL is good and the file was found... the server will begin to send your HTML code immediately after the rest of the headers are sent. It is important in server side languages, such as PHP or ASP, that any headers are sent before any output is sent to the browser and this is why... headers before HTML.
Date: is when the request was made
Server: is an identifier... your server is introducing itself, they have ego's as well.
X-(anything): are custom headers that are created... PHP has an ego as well.
Connection: tells the 'thing' connecting to the server what to do with the connection after it's "finished". In your case, it simply closes the connection and will re-open a new one if there is another request to be made.
Transfer-Encoding: indicates how your server has packaged up the content to be sent to ensure that it makes the trip from server to browser safely. In your case, it has been 'chunked' into bite-size pieces of the same length. If any fail, they can be re-requested by the browser.
Content-Type: tells the browser how to interpret what it's about to receive... 'text' is used for HTML, text, csv, and many other 'text-based' files. 'text' could be replaced with 'image', 'application', or several other broad types of files. The second half, 'html' describes what type of text file it is... html, css, javascript. The charset= part defines which character set used in the text... basically, arabic letters, chinese characters, umlauts, russian characters, and a number of other types of specific character sets like the ones I mentioned offhand.
There are plenty of other headers covering all kinds of aspects on how data is sent, how caches are controlled, what proxies should do (if used), and what the client (browser or spider) should do next.
In short, your HTTP Status code of 200 is good and I can see that the URL you entered did not return a last modified timestamp. When it sends a timestamp and supports the If-Modified-Since headers, you could save a lot of bandwidth and help the search engines to find content on your site that has changed since they last visited it... not the same old content visit after visit.
-Shawn