In the first joint and open initiative to improve the Web crawl process for search engines, Google, Yahoo and Microsoft, have announced support for Sitemaps 0.90 (www.sitemaps.org), a free and easy way for webmasters to notify search engines about their websites and be indexed more comprehensively and efficiently, resulting in better representation in search indices. For users, Sitemaps enables higher quality, fresher search results. An initiative initially driven by Yahoo and Google, Sitemaps builds upon the pioneering Sitemaps 0.84, released by Google in June of 2005, which is now being adopted by Yahoo and Microsoft to offer a single protocol to enhance Web crawling efforts.

Together, the sponsoring companies will continue to collaborate on the Sitemaps protocol and publish enhancements on a jointly maintained website www.sitemaps.org, which provides all of the details about the Sitemaps protocol.

How Sitemaps Work
A Sitemap is an XML file that can be made available on a website and acts as a marker for search engines to crawl certain pages. It is an easy way for webmasters to make their sites more search engine friendly. It does this by conveniently allowing webmasters to list all of their URLs along with optional metadata, such as the last time the page changed, to improve how search engines crawl and index their websites.

Sitemaps enhance the current model of Web crawling by allowing webmasters to list all their Web pages to improve comprehensiveness, notify search engines of changes or new pages to help freshness, and identify unchanged pages to prevent unnecessary crawling and save bandwidth. Webmasters can now universally submit their content in a uniform manner. Any webmaster can submit their Sitemap to any search engine which has adopted the protocol.

The Sitemaps protocol used by Google has been widely adopted by many Web properties, including sites from the Wikimedia Foundation. Any company that manages dynamic content and a lot of web pages can benefit from Sitemaps. For example, if a company that utilizes a content management system (CMS) to deliver custom web content – (i.e., pricing, availability and promotional offers) – to thousands of URLs places a Sitemap file on its web servers, search engine crawlers will be able discover what pages are present and which have recently changed and to crawl them accordingly. By using Sitemaps, new links can reach search engine users more rapidly by informing search engine "spiders" and helping them to crawl more pages and discover new content faster. This can also drive online traffic and make search engine marketing more effective by delivering better results to users.

For companies looking to improve user experience while keeping costs low, Sitemaps also helps make more efficient use of bandwidth. Sitemaps can help search engines find a company’s newest content more efficiently and avoid the need to revisit unchanged pages. Sitemaps can list what is new on a site and quickly guide crawlers to that new content.

"At industry conferences, webmasters have asked for open standards just like this," said Danny Sullivan, editor-in-chief of Search Engine Watch. "This is a great development for the whole community and addresses a real need of webmasters in a very convenient fashion. I believe it will lead to greater collaboration in the industry for common standards, including those based around robots.txt, a file that gives Web crawlers direction when they visit a website."