Saturday, May 4, 2024
Homepage · apt
 Popular · Latest · Hot · Upcoming
13
rated 0 times [  13] [ 0]  / answers: 1 / hits: 4923  / 1 Year ago, sat, january 14, 2023, 11:13:55

I am running Ubuntu with ZFS on root, and my bpool seems to be almost full.


Output of zpool list bpool:


NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT

bpool  1.88G  1.69G   191M        -         -    60%    90%  1.00x    ONLINE  -

Output of df -h /boot:


Filesystem                                        Size  Used Avail Use% Mounted on

bpool/BOOT/ubuntu_s2c09x                          367M  303M   64M  83% /boot

This is causing problems with completing the installation of kernel updates, e.g. running sudo apt upgrade:


Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

Calculating upgrade... Done

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

2 not fully installed or removed.

After this operation, 0 B of additional disk space will be used.

Do you want to continue? [Y/n] 

Requesting to save current system state

ERROR couldn't save system state: Minimum free space to take a snapshot and preserve ZFS performance is 20%.

Free space on pool "bpool" is 10%.

Please remove some states manually to free up space. 

Setting up linux-firmware (1.197.5) ...

update-initramfs: Generating /boot/initrd.img-5.11.0-41-generic

I: The initramfs will attempt to resume from /dev/nvme0n1p2

I: (UUID=abdb40da-2d0b-4a45-b024-1c7245ea27ea)

I: Set the RESUME variable to override this.

Error 24 : Write error : cannot write compressed block 

E: mkinitramfs failure cpio 141 lz4 -9 -l 24

update-initramfs: failed for /boot/initrd.img-5.11.0-41-generic with 1.

dpkg: error processing package linux-firmware (--configure):

 installed linux-firmware package post-installation script subprocess returned error exit status 1

Setting up linux-image-5.11.0-34-generic (5.11.0-34.36) ...

Processing triggers for linux-image-5.11.0-34-generic (5.11.0-34.36) ...

/etc/kernel/postinst.d/dkms:

 * dkms: running auto installation service for kernel 5.11.0-34-generic

Error! Your kernel headers for kernel 5.11.0-34-generic cannot be found.

Please install the linux-headers-5.11.0-34-generic package,

or use the --kernelsourcedir option to tell DKMS where it's located

   ...done.

/etc/kernel/postinst.d/initramfs-tools:

update-initramfs: Generating /boot/initrd.img-5.11.0-34-generic

I: The initramfs will attempt to resume from /dev/nvme0n1p2

I: (UUID=abdb40da-2d0b-4a45-b024-1c7245ea27ea)

I: Set the RESUME variable to override this.

Error 24 : Write error : cannot write compressed block 

E: mkinitramfs failure cpio 141 lz4 -9 -l 24

update-initramfs: failed for /boot/initrd.img-5.11.0-34-generic with 1.

run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1

dpkg: error processing package linux-image-5.11.0-34-generic (--configure):

 installed linux-image-5.11.0-34-generic package post-installation script subprocess returned error exit status 1

Errors were encountered while processing:

 linux-firmware

 linux-image-5.11.0-34-generic

ZSys is adding automatic system snapshot to GRUB menu

I've tried destroying some bpool snapshots, but zfs list -t snapshot | grep bpool doesn't list any over 100K. I have also tried removing some kernels, see output of dpkg -l linux-{image,headers}-"[0-9]*":


un  linux-headers-3.0               <none>         <none>       (no description available)

un  linux-headers-5.10.0-1045-oem   <none>         <none>       (no description available)

un  linux-headers-5.11.0-34-generic <none>         <none>       (no description available)

un  linux-headers-5.11.0-35-generic <none>         <none>       (no description available)

un  linux-headers-5.11.0-36-generic <none>         <none>       (no description available)

un  linux-headers-5.11.0-37-generic <none>         <none>       (no description available)

un  linux-headers-5.11.0-38-generic <none>         <none>       (no description available)

ii  linux-headers-5.11.0-40         5.11.0-40.44   all          Header files related to Linux kernel version 5.11.0

ii  linux-headers-5.11.0-40-generic 5.11.0-40.44   amd64        Linux kernel headers for version 5.11.0 on 64 bit x86 SMP

ii  linux-headers-5.11.0-41         5.11.0-41.45   all          Header files related to Linux kernel version 5.11.0

ii  linux-headers-5.11.0-41-generic 5.11.0-41.45   amd64        Linux kernel headers for version 5.11.0 on 64 bit x86 SMP

un  linux-headers-686-pae           <none>         <none>       (no description available)

ic  linux-image-5.10.0-1045-oem     5.10.0-1045.47 amd64        Signed kernel image oem

un  linux-image-5.11.0-27-generic   <none>         <none>       (no description available)

iF  linux-image-5.11.0-34-generic   5.11.0-34.36   amd64        Signed kernel image generic

rc  linux-image-5.11.0-35-generic   5.11.0-35.37   amd64        Signed kernel image generic

rc  linux-image-5.11.0-36-generic   5.11.0-36.40   amd64        Signed kernel image generic

rc  linux-image-5.11.0-37-generic   5.11.0-37.41   amd64        Signed kernel image generic

rc  linux-image-5.11.0-38-generic   5.11.0-38.42   amd64        Signed kernel image generic

ii  linux-image-5.11.0-40-generic   5.11.0-40.44   amd64        Signed kernel image generic

ii  linux-image-5.11.0-41-generic   5.11.0-41.45   amd64        Signed kernel image generic

The oldest currently installed one seems to be linux-image-5.11.0-34-generic, however sudo apt purge linux-image-5.11.0-34-generic suggests that it wants to also install linux-image-unsigned-5.11.0-34-generic, taking up more space.


What else can I try? Is it conceivably an option to resize/grow my bpool? I am pretty new to using ZFS.


Edit: I did some more investigating. zfs list bpool/BOOT/ubuntu_s2c09x -o space -r reports


NAME                                                             AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD

bpool/BOOT/ubuntu_s2c09x                                          229M  1.51G     1.22G    303M             0B         0B

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-11-19-1344        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-21-1339         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-22-1336         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-23-1340         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-24-1335         -    56K         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-25-1339         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-26-1338         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-11-26-1343        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-27-1338         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-28-1336         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-29-1338         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-30-1340         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-01-1340         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-02-1340         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-03-1339         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-12-03-1344        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-04-1337         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-05-1338         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-06-1338         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-07-1339         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-08-1337         -    80K         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-09-0645         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@autozsys_a2u0ly                             -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-10-1336         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-12-10-1341        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-11-1339         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-12-1335         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@autozsys_omna1o                             -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@autozsys_5q7106                             -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-13-1336         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@autozsys_fwalez                             -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-14-1336         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@autozsys_dsm4th                             -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-15-1335         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@autozsys_4t769s                             -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-16-1336         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@autozsys_z8aa09                             -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-17-1339         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-12-17-1344        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@autozsys_dqsnib                             -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@autozsys_7w0l6s                             -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@autozsys_323q08                             -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@autozsys_gqbsa5                             -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-18-1338         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-19-1339         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0217        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0317        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0417        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0517        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0617        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0717        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0817        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0917        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1017        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1117        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1217        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1317        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-21-1337         -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1417        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1517        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1617        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1717        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1817        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1917        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2017        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2117        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2217        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2317        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-22-0017        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0100      -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0115      -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-22-0117        -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0130      -     0B         -       -              -          -

bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0145      -     0B         -       -              -          -

I don't understand how it can claim that 1.22 GB are used by snapshots, but none of the listed bpool snapshots are using any space?


More From » apt
 Answers
5

Uninstalling old and unused kernels and associated files is the correct action to take; to avoid the suggestion to install an unsigned kernel, remove packages that depend on linux-image-5.11.0-34-generic via:


sudo apt purge linux*5.11.0-34*

The second step required to free up space is to remove old snapshots referencing the files that were removed; list the snapshots via:


zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT

You can see in the REFER column how much data is referenced. Identical numbers often coincide with minimal USED values; the snapshots are effectively copies of previous snapshots.


Manually remove the five oldest snapshots via:


zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT | head -n 5 | cut -c 35-40 | xargs -n 1 sudo zsysctl state remove --system

Keep removing snapshots until you see space being freed up.


(Source of the last one-liner and more info: https://github.com/ubuntu/zsys/issues/155)


Notes:



  1. Growing a ZFS pool is possible if there is unallocated space, but Ubuntu allocates the remaining space to rpool. Shrinking a ZFS pool in-place is apparently not possible.



  2. I see auto-snapshotting is turned on for bpool, but unless you expect any files to change outside of APT, ZSys snapshotting should be sufficient and would reduce the number of snapshots you have to consider.



  3. To avoid this problem in the future, stay on top of kernel upgrades and remove old ones regularly. Also consider reducing the ZSys garbage cleaning defaults in /etc/zsys.conf. For an example, see: https://github.com/ubuntu/zsys/issues/155#issuecomment-758902487




[#919] Monday, January 16, 2023, 1 Year  [reply] [flag answer]
Only authorized users can answer the question. Please sign in first, or register a free account.
shadowoof
Add To Favorites
Follow
Total Points: 293
Total Questions: 112
Total Answers: 137
Location: Burkina Faso
Member since Sun, Nov 21, 2021
3 Years ago
shadowoof questions
Wed, Dec 7, 22, 03:59, 1 Year ago
Thu, Sep 9, 21, 15:29, 3 Years ago
Sun, May 8, 22, 22:01, 2 Years ago
Fri, Apr 29, 22, 23:54, 2 Years ago
Wed, Jul 7, 21, 18:54, 3 Years ago
View All
;