Lucas Werkmeister

@LucasWerkmeistr

And / is btrfs, so apt creates a subvolume before doing the dist-upgrade, presumably as a backup. Doesn’t sound like a bad idea…

7/1/2017, 6:10:36 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

…but apparently, that subvolume + all the actions apt/dpkg did during the upgrade completely filled up the metadata space of the filesystem.

7/1/2017, 6:11:41 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

So I had a filesystem with no support for creating new files, or doing much of anything… in the middle of a major package upgrade. /o\

7/1/2017, 6:12:18 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Of course, my natural reaction to “no space left on device” was to delete some non-essential stuff.

7/1/2017, 6:12:55 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

But of course that doesn’t do jack shit while everything’s still referenced by the subvolume (except /var/cache/apt, I guess).

7/1/2017, 6:13:27 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

(Guess how many files I deleted before I realized that :( but as I said, non-essential files.)

7/1/2017, 6:13:57 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Fortunately, as it turns out, btrfs lets you add another device to a live filesystem! This page was very helpful: http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html

7/1/2017, 6:14:29 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

So I added a spare HD I had lying around (with data on it, but also non-essential) and added it to the root file system.

7/1/2017, 6:14:54 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

So the filesystem had enough metadata space available again (it was a big external HD), and I could continue.

7/1/2017, 6:15:50 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

dpkg --reconfigure -a, then apt dist-upgrade, resumed the broken upgrade without problems. (apt told me I needed the dpkg command first.)

7/1/2017, 6:16:53 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

(So big compliment to dpkg+apt, that this breakage in the middle of a big upgrade was completely recoverable! Very pleasant.)

7/1/2017, 6:17:37 PM

Favs: 0

Retweets: 1

Lucas Werkmeister

@LucasWerkmeistr

Then I had to rebalance the root filesystem some more before I could remove the external HD again (btrfs balance -ddevid=1 -dusage=50 /)…

7/1/2017, 6:18:33 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

…and then btrfs device delete /dev/sdb1 / worked (took about a minute to shift everything to the internal disc). And there you are.

7/1/2017, 6:19:45 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

So, my system broke because of apt / btrfs (not sure which is more to blame), but I was also able to fix it thanks to them.

7/1/2017, 6:20:45 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

Overall, I’m not mad. Just a bit miffed because I’d planned to spend this weekend on getting + setting up a new server (Debian Stretch).

7/1/2017, 6:21:54 PM

Favs: 0

Retweets: 0

Lucas Werkmeister

@LucasWerkmeistr

But I can still start with that and have fun with all the improvements in Stretch (e. g. newer systemd!) and see how far I get :) /end

7/1/2017, 6:22:51 PM

Favs: 0

Retweets: 0