[whatwg] Metalink: higher availability and self repairing downloads

[Anthony Bryan]

Hi,

I saw http://wiki.whatwg.org/wiki/Link_Hashes on the wiki and thought
some people might be interested in Metalink (it was mentioned on the
discussion page), which has been used to automate file verification.

Metalink is an XML format that lists multiple ways
(FTP/HTTP/P2P/rsync) to retrieve files, along with checksums and other
metadata, to increase availability and guarantee integrity in
downloads. It is used by download programs such as download managers,
P2P & FTP programs, but was designed for web browsers.

Around 21 programs have Metalink support. Metalink is frequently used
for large downloads, such as Linux/BSD ISOs (600 megs to multiple
gigs). Projects/sites such as OpenOffice.org, cURL, openSUSE, DOFUS,
Eiffel Software, and others use it.

DownThemAll! 1.0 Beta [1] is a Firefox extension, and the easiest way
to try Metalink out. It supports download selection and whole file
verification.

Some basic features the information in a metalink can be used for:
- Listing multiple mirrors for higher availability.
- Automatic error recovery and repair of corrupted downloads.
- More information about a download for users. [2]
- Add a file or multiple files to a download queue.
- Share a library of files between filesharing applications.
- Simplified download pages.

"Metalink solves the first problem — how to find the most speedy way
to download a file — by grouping different download protocols into one
protocol. This enables Metalink clients to automatically switch
between different mirror servers without explicit user instructions.
Not only multiple server information is described but also meta-data
(e.g., the location of the server, document tags and license)." [3]

An example .metalink:

<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" generator="Metalink Editor version 1.0.0"
xmlns="http://www.metalinker.org/">
  <publisher>
    <name>Xi Software</name>
    <url>http://www.xi-soft.com/</url>
  </publisher>
  <identity>Net Transport</identity>
  <version>2.43</version>
  <copyright>2005-2007</copyright>
  <description>Net Transport (NetXfer) is a downloading tool.</description>
  <files>
    <file name="NXSetup_multi.zip">
      <size>2014232</size>
      <os>Windows-x86</os>
      <verification>
        <hash type="md5">18148ed6dc0fc6a48f345579f099d6bd</hash>
        <hash type="sha1">01586b2c0ec5d8e985138204404878f5ecbeef58</hash>
        <pieces type="sha1" length="262144">
          <hash piece="0">ce50319f58e846e3e0c66a9ada9015e601126864</hash>
          <hash piece="1">c9e73e336831a364cedcd889259b7330d48f90fd</hash>
          <hash piece="2">a7f5281c629ea63c791f320db3b0df85fdb01861</hash>
          <hash piece="3">e6cf5571185db0ec79b551222596462db445bdd6</hash>
          <hash piece="4">8982e850d2e3ea005119beea3cc05daca28de474</hash>
          <hash piece="5">a0e7a3a3a46f1ae5fb4eb8529b6fb8ee82fa43a8</hash>
          <hash piece="6">96540cfade1ce9d3846f41510fbed46965b29568</hash>
          <hash piece="7">87108128d4dde5ba62ffec1f1cc3fc292fc298ca</hash>
        </pieces>
      </verification>
      <resources>
        <url type="http"
preference="100">http://www.xi-soft.com/downloads/NXSetup_multi.zip</url>
        <url type="http" location="gb"
preference="100">http://nettransport.abangadek.com/NXSetup_multi.zip</url>
        <url type="http" location="hu"
preference="100">http://www.axitech.hu/nettransport/downloads/NXSetup_multi.zip</url>
        <url type="ftp" location="cn"
preference="100">ftp://nettransport:nettransport@202.107.248.221/NXSetup_multi.zip</url>
      </resources>
    </file>
  </files>
</metalink>

For more information: http://www.metalinker.org and
http://en.wikipedia.org/wiki/Metalink

Thanks,
-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

[1] http://www.downthemall.net/main/install-it/downthemall-10-beta/
[2] http://code.downthemall.net/maierman/metaselect4.png
[3] http://www.geospatialsemanticweb.com/2007/02/25/metalink-unifies-internet-downloads