Release Notes for CernVM-FS 2.4.1¶
CernVM-FS 2.4 is a feature release that comes with performance improvements, new functionality, and bugfixes. We would like to thank Brian Bockelman (U. Nebraska), Dave Dykstra (FNAL), and Tom Downes (U. Wisconsin) for their contributions to this release!
There are several substantial improvements in this release, which are further described below.
- A plugin interface for the client cache together with an in-memory plugin
- A built-in tiered cache manager
- Instant access to named snapshots through the hidden .cvmfs/snapshots directory
- Support for branching and diffing in CernVM-FS’ internal versioning
- Faster propagation of repository updates
- Support for Yubikey 4 & NEO for signing CernVM-FS repository
- Improved apt repository structure for Debian/Ubuntu packages
- New platforms: Fedora 25 and 26 on x86_64, Debian 8 and 9, gcc >= 6, OpenSSL >= 1.1
As with previous releases, upgrading should be seamless just by installing the new package from the repository. As usual, we recommend to update only a few worker nodes first and gradually ramp up once the new version proves to work correctly. Please take special care when upgrading a client in NFS mode.
For Stratum 0 servers, all transactions must be closed before upgrading. After
the software upgrade, the directory layout on the release manager needs to be
adjusted by a call to
cvmfs_server migrate for each repository.
For Stratum 1 server, there should be no running snapshots during the upgrade.
Note: if the configuration of the Stratum 0/1 server is handled by a configuration management system (Puppet, Chef, ...), please see Section Manual Migration from 2.3.5 Release Manager Machines and Stratum 0/1 Web Servers.
Note: on Debian/Ubuntu platforms, please read Section New apt Repositories regarding hotpatching the client.
Every CernVM-FS client is configured to use a directory as local cache of data and meta-data. Instead of this directory, the task of maintaining a local cache can optionally be performed by an external process, a “CernVM-FS Cache Plugin” (CVM-1054). This allows for special-purpose cache managers in non-standard deployments, for instance on supercomputers. Cache plugins can be developed and deployed independently from the CernVM-FS client itself. CernVM-FS 2.4 provides one such plugin, an in-memory cache that uses a fixed amount of RAM as a cache (CVM-1044).
Together with support for cache plugins, there is now support for a multi-tier client cache (CVM-1050, CVM-1183). A tiered cache can combine two other caches and organize them as an upper cache layer and a lower cache layer. Data is first searched for in the upper layer. Upon an upper layer cache miss, data is copied from the lower layer into the upper layer. Tiered caches can be used to combine a small cache on fast storage (e.g. SSD, memory) with a large cache on slower storage (e.g. HDD, network drive).
See Section Advanced Cache Configuration for configuration and use of a tiered cache.
Instant Access to Named Snapshots¶
A new server parameter,
CVMFS_VIRTUAL_DIR=[true,false], can be used to
control the existance of the hidden top-level directory
a repository (CVM-1062). If
enabled, the file system state referred to by the named tags can be browsed
.cvmfs/snapshots/$tagname. This feature requires a CernVM-FS 2.4
client, older clients show an empty
See Section Instant Access to Named Snapshots for further information.
cvmfs_server checkout command can be used to branch off a certain
named snapshot in order to publish a fix for a previous repository state
(CVM-1197). This feature makes
most sense for repositories that use the instant snapshot access (see above).
See Section Branching for further information.
Faster Propagation of Repository Updates¶
Several improvements have been made to reduce the time to propagate changes from the release manager machine to clients.
- The default repository time-to-live is reduced from 15 minutes to 4 minutes (CVM-1336). Unless the
CVMFS_REPOSITORY_TTLparameter is explicitly set, the first
cvmfs_server publishcommand with version 2.4 reduces the time-to-live value. Thus clients are instructed to check every 4 minutes for repository updates.
- On RHEL 7 and newer, clients can actively evict old entries from kernel buffers (CVM-1041). When clients see a new repository revision, they hence get rid of a 60 seconds delay to passively wait for local kernel buffers to expire.
- The new server parameter
CVMFS_GENERATE_LEGACY_BULK_CHUNKS=nocan be used to omit creation of unchunked objects for large files (CVM-640). This is most interesting for repositories hosting many files that are larger than 4MB. For those repositories, the speed of the publication process is improved by more than a factor of two. This setting requires clients newer than version 2.1.7. Note for garbage collected repositories: Besides the release manager machine, all stratum 1s need to run version 2.4, too. Otherwise they will delete the chunks of files with no bulk hash during garbage collection.
This release supports maintaining the repository master key on a Yubikey smart card device (CVM-1259). If the masterkey is stored on such devices, it cannot be stolen even if the computer hosting the repositories is compromised.
See Section Master keys for further information.
New apt Repositories¶
Starting with this release, the apt repositories that provide deb packages for
Ubuntu and Debian are restructured. So far, all Debian based platforms got
packages built for Ubuntu 12.04. These packages are still used if the platform
is not recognized by the
cvmfs-release package. For Debian stable platforms
and Ubuntu LTS releases, packages built for the specific platform are used
For Ubuntu 16.04 and Debian 8, the CernVM-FS apt repositories contain a fixed
version of the
autofs package which is necessary to support the CernVM-FS
Note on client hotpatching: packages from the new apt repository cannot seamlessly upgrade previous cvmfs clients. In order to upgrade the client, please
cvmfs_config umountto unmount all active repositories
- Upgrade to the cvmfs-release 2.X package and run
- Update the cvmfs client package.
This is a one-time migration. The next CernVM-FS release will again upgrade seamlessly.
- Client: fix small memory leak during remount of root catalog
- Client: fix handling of file:// url in CVMFS_SERVER_URL
- Client: fix
cvmfs_config reloadunder root environment with dependencies into /cvmfs (CVM-1352)
- Client: fix mount helper for very long lines in /etc/group (CVM-1304)
- Client: fix mount helper if repository name resolves to local path (CVM-1106)
- Client: fix shell errors when required config repo cannot be mounted (CVM-1300)
- Client / macOS: fix cache size reporting in ‘df’ (CVM-1286)
- Client / macOS: fix
- Client / X509 Auth: Use default X509_CERT_DIR also if it is empty string (CVM-1083)
- Server: fix potential deadlock during catalog commit phase (CVM-1360)
- Server: do not abort resiging on negative repository health check (CVM-1358)
- Server: resolve SElinux conflict on port 8000 with soundd on RHEL 7 (CVM-1308)
- Server / S3: fix authentication timeouts for large transactions on Ceph (CVM-1339)
- Client: allow for config repository on Ubuntu >= 16.04, Debian >= 8 (CVM-771)
- Client: cache proxy settings in workspace directory (CVM-1156)
- Client: improve stratum 1 geo sorting with active fallback proxy (CVM-769)
- Client: add support for CVMFS_OOM_SCORE_ADJ to adjust the out-of-memory priority (CVM-1092)
- Client: add support for revoking repository revisions up to a threshold in the blacklist (CVM-992)
- Client: perform fail-over when whitelist or manifest is corrupted (CVM-837)
- Client: add
cvmfs_talk remount synccommand
- Cient: use cache for fetching history database on mount
- Client: show all
- Client: add
cvmfs_config showconfig -soption to show only non-empty parameters
- Client: add
ncleanup24xattr and Nagios check for cleanup rate (CVM-1097)
- Client / macOS: use built-in LibreSSL on macOS (CVM-1112)
- Server: add
cvmfs_server gc -aoption to garbage collect all applicable repositories (CVM-1095)
- Server: make
cvmfs_server catalog-chowncommand public (CVM-1077)
- Server: add
cvmfs_server resign -wfor stand-alone whitelist resigning (CVM-1265)
- Server: add
cvmfs_server resign -pcommand to facilitate repository key rotation (CVM-1140)
- Server: add
cvmfs_server resign -doption to change whitelist expiration duration (CVM-1279)
- Server: add
cvmfs_server check -rcommand to repair reflog checksum (CVM-1240)
- Server: allow ext3 as spool file system on RHEL 7.3 / overlayfs (CVM-1186)
- Server: Optionally ignore special files with a warning on publish with
- Server: increase maximum repostory name from ~30 chars to 60 chars (CVM-1173)
- Server: trim trailing whitespaces from .cvmfsdirtab entries (CVM-1061)
- Server / rsync: use rsync’s “perishable” feature instead of list-catalogs (CVM-1199)
- Server: allow for Apache 2.4 style access controls on repositories (CVM-1255)
- Server: add support for
CVMFS_ENFORCE_LIMITSto set publish limits (CVM-1094, CVM-1123)
- Server: improve error reporting (CVM-1241, CVM-1246, CVM-1267,)
Manual Migration from 2.3.5 Release Manager Machines and Stratum 0/1 Web Servers¶
If you do not want to use
cvmfs_server migrate to automatically upgrade,
release manager machines that maintain Stratum 0 repositories as well as web
servers serving stratum 0/1 repositories can be migrated from version 2.3.5 with
the following steps:
- Ensure that there are no open transactions and no active replication or garbage collection processes before updating the server software and during the repository layout migration.
- Install the
The Apache configuration on the release manager machine (resp. stratum 0) and
on stratum 1 repositories, as well as the configuration for the meta-data area
/cvmfs/info, should be adjusted as follows:
AllowOverride Limit AuthConfig
- Reduce the cache expiry for files of type
application/jsonfrom 2 minutes to 61 seconds
- Add to the
<Directoy>directive for the repository
<FilesMatch "^[^.]*$"> ForceType application/octet-stream </FilesMatch>
Reload the Apache service and perform the following steps for all repositories:
- Only on release manager machines: remove the
CVMFS_CATALOG_ENTRY_WARN_THRESHOLDparameter. If it was set to a value other than 500000, set
CVMFS_NESTED_KCATALOG_LIMIT=500. Consider setting a lower limit for
- Update /etc/cvmfs/repositories.d/<REPOSITORY>/server.conf and set
On release manager machines, in agreement with the repository owner it’s recommended to make a test publish
cvmfs_server transaction <REPOSITORY> cvmfs_server publish <REPOSITORY>
before resuming normal operation.