libchop, tools & library for data backup and distributed storage
Libchop is a set of utilities and library for data backup and
distributed storage. Its main application is chop-backup, an
encrypted backup program with several salient features:
- Encrypted. The backup data can safely be stored at an untrusted site without compromising its confidentiality.
- Tamper-proof. The backup's integrity is checked upon recovery.
- Distributable. Backup data can be written to more than one store.
- Shareable. Each directory/file in a snapshot is identified by a "tuple", which is necessary and sufficient to retrieve it. A tuple can be shared with others, which gives them access to the corresponding file/directory and only it.
- Versioned. The history of directory snapshots is recorded, at little cost.
- Compressed. Similar data among files or versions are coalesced. For each file type an appropriate compression method is chosen.
- Evolutive. The application is not bound to any storage, hash, encryption, or compression method. In fact, all these parameters can vary from file to file within a snapshot.
The library itself was initially designed as a building block for a peer-to-peer, cooperative backup system. In such a system, data has to be sent by pieces, incrementally, and it may be scattered across several participating nodes. In addition, participating nodes may be untrusted, which puts data confidentiality, integrity, and availability at risk. Read more…
Libchop was initially developed as part of a PhD thesis in the MoSAIC project.
Releases
Releases are available from your nearest Savannah mirror (or from Savannah itself).
Documentation
- The (incomplete) reference manual is part of the source distribution in GNU Texinfo form. It can also be browsed on-line.
- The slides of a talk at the 2010 GNU Hackers Meeting in The Hague.
- Storage Tradeoffs in a Collaborative Backup Service for Mobile Devices presents the motivation behind libchop along with experimental results comparing several storage strategies.
- Chapter 4 of Cooperative Data Backup for Mobile Devices provides additional details and results, for the insatiable.
- A bibliography relative to cooperative backup and peer-to-peer storage.
- Report (in French) about the use of libchop for a tentative peer-to-peer backup system in modem-routers for LDN, a French non-profit, neutral Internet service provider (ISP). June 2012.
Mailing List
A mailing list for bug reports and general discussion is available at <libchop-devel@nongnu.org>. You can subscribe to the mailing list and view the list archive.
Development
- Git repository at
git://git.sv.gnu.org/libchop.git - Savannah project page
- continuous integration
Related Work
- Plan 9's Venti provides unencrypted content-addressable storage and archival.
- GNUnet uses content-hash keys for its file sharing service (like
chop-backup). Some of the early papers on its data encoding scheme (ECRS) were instrumental in the design of parts of libchop andchop-backup. - Tahoe-LAFS is a distributed file system, where data is encrypted but
remains selectively shareable; it handles redundancy automatically.
It has a
backupsub-command. - Ugarit, a nice content-addressable backup & archival system written
in Chicken Scheme. The main differences compared to
chop-backupare that it does not use content-hash keys for encryption (a symmetric encryption key must be kept in augarit.conffile), and lacks adaptive compression and content-based chopping (it doesn't implement the storage pipeline found in libchop.)
