CernVM-FS Shrinkwrap Utility
The CernVM-FS Shrinkwrap utility provides a means of exporting CVMFS repositories. These exports may consist of the complete repository or contain a curated subset of the repository.
The CernVM-FS shrinkwrap utility uses
libcvmfs to export repositories
to a POSIX file tree. This file tree can then be packaged and exported in
several ways, such as SquashFS, Docker layers, or TAR file.
cvmfs_shrinkwrap utility supports multithreaded copying to increase
throughput and a file specification to create a subset of a repository.
cvmfs_shrinkwrap utility is packaged for Red Hat based and Debian based
platforms in the
In order to compile
cvmfs_shrinkwrap from sources, use the
-DBUILD_SHRINKWRAP=on cmake option.
CernVM-FS Shrinkwrap Layout
The structure used in the Shrinkwrap output mirrors that used internally
by CernVM-FS. The visible files are hard linked to a hidden data directory.
cvmfs_shrinkwrap builds in a base directory (
where a directory exists for each repository and a
containing the content-addressed files for deduplication.
The shrinkwrap output directory should be formatted with XFS. The ext file systems limit the number of hard links to 64k.
Default base directory Single mount point that can be used to package repositories, containing both the directory tree and the data directory.
Repository file tree Directory containing the visible structure and file names for a repository.
File storage location for repositories Content-addressed files in a hidden directory.
Storage location for provenance
Hidden directory that stores the provenance
The specification file allows for both positive entries and exclusion statements. Inclusion can be specified directly for each file, can use wildcards for directories trees, and an anchor to limit to only the specified directory. Directly specify file :
Specify directory tree :
Specify only directory contents :
Negative entries will be left out of the traversal :
Creating an image for ROOT
Start out with either building
cvmfs_shrinkwrap, adding it to your path,
or locating it in your working directory.
Create a file specification to limit the files subject to being shrinkwrapped.
Here is an example for ROOT version 6.10 (~8.3 GB). For our example put this in
a file named
/lcg/releases/ROOT/6.10.04-4c60e/x86_64-centos7-gcc7-opt/* /lcg/contrib/binutils/2.28/x86_64-centos7/lib/* /lcg/contrib/gcc/* /lcg/releases/gcc/* /lcg/releases/lcgenv/*
libcvmfs configuration file that will be used for
cvmfs_shrinkwrap puts heavy load on servers. DO NOT configure it to read
from production Stratum 1s!
cvmfs_shrinkwrap at CERN please use
and for OSG please use
Here is an example that uses the CERN server, written to
CVMFS_REPOSITORIES=sft.cern.ch CVMFS_REPOSITORY_NAME=sft.cern.ch CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch CVMFS_SERVER_URL='http://cvmfs-stratum-zero-hpc.cern.ch/cvmfs/sft.cern.ch' CVMFS_HTTP_PROXY=DIRECT # Avoid filling up any local squid's cache CVMFS_CACHE_BASE=/var/lib/cvmfs/shrinkwrap CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch # Need to be provided for shrinkwrap CVMFS_SHARED_CACHE=no # Important as libcvmfs does not support shared caches CVMFS_USER=cvmfs
Keys will need to be provided. The location in this configuration is the default used for CVMFS with FUSE.
Using the cvmfs repository
sudo cvmfs_shrinkwrap -r sft.cern.ch -f sft.cern.ch.config -t sft.cern.ch.spec --dest-base /tmp/cvmfs -j 16
Creating an image in user space
Start by using the above setup.
Alternatively, shrinkwrap images can be created in user space. This is achieved using
the UID and GID mapping feature of
libcvmfs. First mapping files need to be written.
Example (Assuming UID 1000). Write
* 1000 into
Add this rule
The same is done with GID into
Using the cvmfs repository
cvmfs_shrinkwrap -r sft.cern.ch -f sft.cern.ch.config -t sft.cern.ch.spec --dest-base /tmp/cvmfs -j 16
Using a shrinkwrap image
Shrinkwrap was developed to address similar restrictions as the CVMFS Preloader. Having created an image from your specification there are a number of ways this can be used and moved around.
Having a fully loaded repository, including the hard linked data, the image can be exported to a number of different formats and packages. Some examples of this could be ZIP, tarballs, or squashfs. The recommendation is to use squashfs as it provides a great amount of portability and is supported for directly mounting on most OS.
If tools for creating squashfs are not already available try :
apt-get install squashfs-tools
– or –
yum install squashfs-tools
After this has been installed a squashfs image can be created using the above image :
mksquashfs /tmp/cvmfs root-sft-image.sqsh
This process may take time to create depending on the size of the shrinkwrapped image. The squashfs image can now be moved around and mounted using :
mount -t squashfs /PATH/TO/IMAGE/root-sft-image.sqsh /cvmfs
Bind mounting an image
The shrinkwrap image can also be directly moved and mounted using bind mounts.
mount --bind /tmp/cvmfs /cvmfs
This provides a quick method for testing created images and verifying the contents will run your expected workload.
Important note on use
Shrinkwrap images mirror the data organization of CVMFS. As such it is important that the data and the file system tree be co-located in the file system/mountpoint. If the data is separated from the file system tree you are likely to encounter an error.