A filesystem manages how information is stored and recovered. Information is grouped in files, which are structured hierarchically, in order to facilitate their sorting.
There are many kinds of filesystems, and userspace (FUSE) ones are quite peculiar. Userspace filesystems allow non-privileged users to run the code of a filesystem in user space. The FUSE module acts as the only "bridge" to the kernel.
The filesystem's capacity is related with the size of the disk partition it is managing, for storing information. As big as this space could possibily be, it is nonetheless limited, and it is often the case that it becomes saturated while storing information. It gets flooded by data which must be kept over time, and also by data valuable only in the short term, which will eventually lose purpose as time goes by. App logs and backup files belong to this second category of data.
App logs are the summary of all actions taken by an app. Backups, however, are copies of other data, done for safety purposes. Both these kinds of information have the peculiarity of being marked temporally, with a timestamp indicating their moment of writing, which is why they actually grow old over time. It would be quite useless to keep a backup file from last year.
If available space becomes completely saturated, the application will crash, since it will no longer be able to write information on the disk. The worst-case scenario is the partial writing of a file, causing its complete corruption.
Monitoring systems keep the level of occupied space on the partitions, and therefore on the filesystems, under control. Every time occupied space reaches a certain threshold, the monitoring system activates an alarm, for notifying the user of the critical situation, since, as we said before, we don't want the service to get freezed, with the risk of corrupting the files.
The system administrator will have to free up saturating space, tentatively he won't be able to delete useful information, thus he will end up deleting files of little to no value, meaning the oldest ones among temporally marked files.
This is a repetitive action, often implemented through dedicated scripts, to be launched periodically and either manually or automatically. If the entire process is efficiently performed, the monitoring system will be configured to launch them automatically, before free space becomes completely exhausted. This last solution also proves of little elegance, in fact, it is necessarily related to the launch of a script which should be well implemented and thoroughly tested, but most of all, should also be tailored for the system running it.
Usually, scripts for cleaning up older files act as follows: they will search for all files older than a given number of days, deleting them immediately. In this case, if information written within a day were to substantially increase in order to keep a stable level of occupation, we should reduce the number of retention days, otherwise an error will be reported. Then, the number of days will be increased as soon as the writing comes back to normal.
A fix for all these problems is offered by limit-fs, which gives us an all-in-one solution for all kinds of environments, without the need of developing code. As we'd expect, it acts on the filesystem itself, since that's what the problem targets, solving its problems in a transparent and automatic way.
limit-fs is indeed a filesystem, with several unique features which are useful for avoiding space saturation.
Its feature make it ideal for storing backup files and app logs.
The features making limit-fs special are:
Limit-fs verifies the occupation level of used space. Since it has full control of operations on all files, it doesn't run the check periodically, but only when it really becomes necessary. Indeed, it checks occupation levels every time the writing of a file is finished. This optimizes the resources needed for monitoring.
At the moment of mounting, it is possible to define a threshold for occupied space, which if not specified otherwise, will be of 80% by default. If the level of occupied space was to exceed the set threshold, older files will be sought after and deleted, until the occupation threshold once again falls down within the limit.
What if we used limit-fs on a non-empty directory? No problem, limit-fs can be mounted on a directory with files and folders already within. After the mount, limit-fs will not act like other filesystems, preserving the entirety of the content present before the mount, instead. Once limit-fs is unmounted, all files written through it will remain stored in the same position. This is why it can be defined as clear, or transparent, it's because it doesn't consume space on the partition, and all reading and writing operations are re-routed to the underlying filesystem.
What happens if we use limit-fs on a path with important files or even system files? How can we avoid their accidental deletion? It is actually quite simple, since limit-fs is a userspace filesystem (FUSE), which means it's mounted by a simple user who will not be able to delete files belonging to either the system or other users.
git clone https://github.com/piuma/limit-fs.git
cd limit-fs
./setup.sh
./configure
make
sudo make installsudo mkdir -p /mnt/tmpfs
sudo mount -t tmpfs --options size=1000m none /mnt/tmpfs
mkdir /mnt/tmpfs/limitfs
limit-fs /mnt/tmpfs/limitfsfor i in $(seq 40); do
dd if=/dev/zero of=/mnt/tmpfs/limitfs/sample\_100m\_00$i bs=100M count=1 2>/dev/null
sleep 0.5
printf '%.0s=' {1..64}; echo
df -h /mnt/tmpfs/limitfs/
echo
ls -l /mnt/tmpfs/limitfs/
done[...]
================================================================
Filesystem Size Used Avail Use% Mounted on
limit-fs 1000M 700M 300M 70% /mnt/tmpfs/limitfs
total 716800
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_001
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_002
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_003
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_004
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_005
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_006
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_007
================================================================
Filesystem Size Used Avail Use% Mounted on
limit-fs 1000M 800M 200M 80% /mnt/tmpfs/limitfs
total 819200
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_001
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_002
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_003
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_004
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_005
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_006
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_007
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_008
================================================================
Filesystem Size Used Avail Use% Mounted on
limit-fs 1000M 800M 200M 80% /mnt/tmpfs/limitfs
total 819200
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_002
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_003
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_004
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_005
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_006
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_007
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_008
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_009
================================================================
Filesystem Size Used Avail Use% Mounted on
limit-fs 1000M 800M 200M 80% /mnt/tmpfs/limitfs
total 819200
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_0010
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_003
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_004
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_005
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_006
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_007
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_008
-rw-rw-r-- 1 piuma piuma 104857600 May 30 15:38 sample_100m_009
[...]It's an elegant solution, for avoiding the uncontrolled growth of space occupied by backup and log files.