I decided to change the setup of my ol' homeserver, a Nuc with core i3 3410u.
Motivation
I have been using this machine for quite a while with an internal ssd and some external hard drive as the main storage, acting like NAS and running an instance of Nextcloud and Emby.
However I got tired of the cables / external stuff and found a nice passively cooled case by akasa. Along with the adapter they provide the old internal mSata connector can be turned into a regular Sata connector.
So I decided to buy a second harddrive to run as raid 1 and put the operating system onto an external 500gb ssd I already have. I have used it before as an external drive to run linux from, but I could use an usb stick for that, if I really ever need to do it again. I just hope that there will be a tiny bit of room left inside the case to put that ssd inside and sacrifice one of the front usbs.
Anyway, the goal is set, let’s get to it.
Whatever, take me to the TR:DL already
Formating the usb ssd
For this task, I used fdisk. There was already an EFI partition, because of the use as a external boot disk, but the current layout wasn’t what I’d like. first we find out what the drive got enumerated as. Find it using lsblk
❯ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 71.3M 1 loop /snap/lxd/16100
loop1 7:1 0 55M 1 loop /snap/core18/1880
loop2 7:2 0 71.3M 1 loop /snap/lxd/16044
loop3 7:3 0 96.5M 1 loop /snap/core/9436
loop5 7:5 0 9.1M 1 loop /snap/canonical-livepatch/95
loop6 7:6 0 55M 1 loop /snap/core18/1754
loop7 7:7 0 97M 1 loop /snap/core/9665
sda 8:0 0 4.6T 0 disk
sdb 8:16 0 59.6G 0 disk
├─sdb1 8:17 0 512M 0 part /boot/efi
├─sdb2 8:18 0 50G 0 part /
└─sdb3 8:19 0 9.1G 0 part [SWAP]
sdc 8:32 0 465.8G 0 disk
├─sdc1 8:33 0 366.2M 0 part /media/efi
├─sdc2 8:34 0 455G 0 part /media/rootfs
└─sdc3 8:35 0 10.4G 0 part
In my case, the the external disk is sdc. this terminal output shows the final result.
Next, run fdsik /dev/sdc. Useful commands are:
mshow helpplist partitionsllist partition types, the ones we need are 1 (EFI) 19 (SWAP) and 20 (Linux FS)ddelete an existing partitionnadd a new partitionwwrite any changes to diskqquite discarding any changes
Using the tool is pretty straight forward with a lot of onscreen explanations, I won’t go into to much detail.
As I already had a GPT on this harddisk I only used d a few times to delete the old partitions and then a w to actually write the changes. After that I used n two times to create the main rootfs and a generous swap area.
When asked for the size of the partitions, just enter ++455G and I used the remaining space for swap next.
The final step here was to format the root partitions using a simple mkfs.btrfs /dev/sdc2. That’s it.
Moving the file system
To achieve this, I will attempt to just copy the existing efi partition and btrfs snapshot of the current root. After that, adjust the UUID in the fstab of the new root. I am not sure if I need to reinstall grub at this point. However, since I don’t modify the existing boot drive, Nothing can go wrong anyway.
as root user just bashing at the keyboard
mkdir /media/boot /media/rootfs
mount /dev/sdc1 /media/boot
mount /dev/sdc2 /media/rootfs
rm -r /media/boot/*
rsync -a /boot/ /media/boot/
And to copy the rootfs with will use btrfs send & receive. Preferably In a byobu terminal, as the ssh could drop and we have to start over again.
cd /
btrfs sub snap -r . @rootfs
btrfs send @rootfs | btrfs rec /media/rootfs
This leaves us with a read only version of the old root fs, but I noticed that my lxd containers didn’t get moved along as those where more btrfs subvolumes. I also noticed that plexmediaserver left a 5gb folder after it’s deinstallation…
❯ btrfs sub list -o /var
ID 297 gen 181903 top level 5 path var/lib/lxd/storage-pools/default
ID 340 gen 181942 top level 5 path @rootfs
Now turn the rootfs snapshot into a writable subvolume and mount it somewhere for what’s next.
❯ cd /media/rootfs
❯ mv @rootfs @rootfs_ro
❯ btrfs sub snap @rootfs_ro @rootfs
Create a snapshot of '@rootfs_ro' in './@rootfs'
❯ ls
@rootfs @rootfs_ro
❯ mkdir /media/newroot
❯ mount -o subvol=/@rootfs /dev/sdc2 /media/newroot
Now the tricky part is, btrfs doesn’t recurse all the subvolumes in question, which is annoying. I will attempt to just recreate the emtpy ones, otherwise send them again.
❯ du -hd1 /var/lib/lxd/storage-pools/default
962M /var/lib/lxd/storage-pools/default/containers
0 /var/lib/lxd/storage-pools/default/snapshots
2.6G /var/lib/lxd/storage-pools/default/images
0 /var/lib/lxd/storage-pools/default/custom
3.6G /var/lib/lxd/storage-pools/default
❯ btrfs subvolume list --sort=path /
ID 340 gen 181942 top level 5 path @rootfs
ID 297 gen 182094 top level 5 path var/lib/lxd/storage-pools/default
ID 298 gen 181903 top level 297 path var/lib/lxd/storage-pools/default/containers
ID 312 gen 181903 top level 298 path var/lib/lxd/storage-pools/default/containers/utor
ID 301 gen 181903 top level 297 path var/lib/lxd/storage-pools/default/custom
ID 300 gen 182064 top level 297 path var/lib/lxd/storage-pools/default/images
ID 307 gen 162420 top level 300 path var/lib/lxd/storage-pools/default/images/8aaca3494acf
ID 338 gen 181905 top level 300 path var/lib/lxd/storage-pools/default/images/baef6615ff69
ID 299 gen 181903 top level 297 path var/lib/lxd/storage-pools/default/snapshots
First create the empty subvolumes
❯ cd /media/newroot/var/lib/lxd/storage-pools
❯ btrfs sub create default
Create subvolume './default'
❯ btrfs sub create default/custom
Create subvolume 'default/custom'
❯ btrfs sub create default/snapshots
Create subvolume 'default/snapshots'
❯ btrfs sub create default/images
Create subvolume 'default/images'
❯ btrfs sub create default/containers
Create subvolume 'default/containers'
Then send the read-only subvolumes
cd /
❯ btrfs sub list -r .
ID 307 gen 162420 top level 300 path var/lib/lxd/storage-pools/default/images/8aaca3494acf
ID 338 gen 181905 top level 300 path var/lib/lxd/storage-pools/default/images/baef6615ff69
ID 340 gen 181942 top level 5 path @rootfs
❯ cd var/lib/lxd/storage-pools/default/images
❯ btrfs send /var/lib/lxd/storage-pools/default/images/8aaca3494acf5cf | btrfs rec ./
At subvol /var/lib/lxd/storage-pools/default/images/8aaca3494acf
At subvol 8aaca3494acf
❯ btrfs send /var/lib/lxd/storage-pools/default/images/baef6615ff69 | btrfs rec ./
At subvol /var/lib/lxd/storage-pools/default/images/baef6615ff69
At subvol baef6615ff69
Currently I only have a single container. Create a ro snapshot, send it and finally convert it back to a rw subvolume.
❯ btrfs sub snap -r /var/lib/lxd/storage-pools/default/containers/utor /utor_ro
Create a readonly snapshot of '/var/lib/lxd/storage-pools/default/containers/utor' in '//utor_ro'
❯ cd /media/newroot/var/lib/lxd/storage-pools/default/containers
❯ btrfs send /utor_ro | btrfs rec .
At subvol /utor_ro
At subvol utor_ro
❯ btrfs sub snap utor_ro utor
Create a snapshot of 'utor_ro' in './utor'
❯ btrfs sub del utor_ro
Delete subvolume (no-commit): '/media/newroot/var/lib/lxd/storage-pools/default/containers/utor_ro'
Setting up fstab with the new UUIDs
When starting out running blkid I noticed my swap partition had no UUID. But that was just because I didn’t create the swap space yet. So no problem.
❯ mkswap /dev/sdc3
Setting up swapspace version 1, size = 10.4 GiB (11169431552 bytes)
no label, UUID=6dfa501b-df5b-459c-92a9-0dd04e7407a3
❯ blkid
/dev/sda: UUID="2a94831d-0606-47dc-b212-1d342725c1e4" TYPE="crypto_LUKS"
/dev/sdb1: UUID="016E-E748" TYPE="vfat" PARTUUID="381d023c-2a82-4e3e-aa8a-4f375ace8ebd"
/dev/sdb2: UUID="042dd718-251d-4727-9820-e6789b89da44" UUID_SUB="51964a56-eb4d-4016-9dd2-7f091ffe62c9" TYPE="btrfs" PARTUUID="fb6469f4-b0d6-457f-9ff1-2186e7149197"
/dev/sdb3: UUID="5ed2aed7-ff00-4cc2-a505-c7e413bc58c7" TYPE="swap" PARTUUID="d0bc8d57-82a2-4283-bc9b-9551c7455a99"
/dev/sdc1: SEC_TYPE="msdos" UUID="D660-B8B5" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="e2418736-eeed-43d2-81af-7c309befc701"
/dev/sdc2: UUID="3376ef79-d15c-4858-adb4-8c47020fdf04" UUID_SUB="ad77a0d2-f268-4c41-b2ce-c346f6726878" TYPE="btrfs" PARTUUID="956c5302-f8ef-3e43-97a5-1b421258e2c3"
/dev/sdc3: UUID="6dfa501b-df5b-459c-92a9-0dd04e7407a3" TYPE="swap" PARTUUID="e29e35a5-63b2-b74b-b080-a64b6483f912"
Now edit the values of the newroot fstab with the UUIDs shown here. Also add the subvol option for the rootfs, as after this operation, I will run it as a subvolume, not clutered about under the main volume.
# /etc/fstab: static file system information.
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/3376ef79-d15c-4858-adb4-8c47020fdf04 / btrfs subvol=/@rootfs,defaults 0 0
/dev/disk/by-uuid/6dfa501b-df5b-459c-92a9-0dd04e7407a3 none swap sw 0 0
# /boot/efi was on /dev/sda1 during curtin installation
/dev/disk/by-uuid/D660-B8B5 /boot/efi vfat defaults 0 0
# mount the base filesystem too, so we can take regular snapshots of the root system easily
/dev/disk/by-uuid/3376ef79-d15c-4858-adb4-8c47020fdf04 /mnt/btrfs btrfs defaults 0 0
none /home/synsi/ramdisk tmpfs nodev,nosuid,noexec,nodiratime,size=5G 0 0
Trying to boot
Since the setup should be complete, it’s time to head into the bios and try to boot from the usb harddrive. If everything works out, this part should be done quite fast. Fingers crossed
…
Well, of course it didn’t work. The bios did not recognize the usb-ssd as an uefi bootable drive, only in legacy mode. But it refused to boot from it in that mode as well. I have to properly move the boot drive, I guess.
This guy saved my life, or at least, a few minutes of it.
sudo mount -subvol=/@rootfs /dev/sdb2 /mnt
sudo mount /dev/sdb1 /mnt/boot/efi
for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /mnt$i; done
sudo chroot /mnt
grub-install /dev/sdb
update-grub
Now the usb-ssd appeared properly in the bios and was even set as the default boot option. Everything looks fine. The lxc container also starts without any issue.
❯ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 59.6G 0 disk
├─sda1 8:1 0 512M 0 part
├─sda2 8:2 0 50G 0 part
└─sda3 8:3 0 9.1G 0 part
sdb 8:16 0 465.8G 0 disk
├─sdb1 8:17 0 366.2M 0 part /boot/efi
├─sdb2 8:18 0 455G 0 part /mnt/btrfs
└─sdb3 8:19 0 10.4G 0 part [SWAP]
❯ ls /mnt/btrfs
@rootfs @rootfs_ro
TL;DR
- format another drive
- send all btrfs subvolumes
- chroot and reinstall grub
- done
