Preventing a btrfs Nightmare | LAS 320

Preventing a btrfs Nightmare | LAS 320

What happens when btrfs goes bad? After rescuing our system from a massive crash, we’ll share what went wrong, how you can prevent problems, reclaim gigabytes of space, and optimize your Linux box for an SSD drive.

Plus a quick look at Ubuntu MATE Remix, the NSA targets Linux users, solving the distro hopping bug….

AND SO MUCH MORE!

All this week on, The Linux Action Show!

Thanks to:


DigitalOcean


Ting

Download:

HD Video | Mobile Video | WebM Torrent | MP3 Audio | Ogg Audio | YouTube | HD Torrent

RSS Feeds:

HD Video Feed | Large Video Feed | Mobile Video Feed | MP3 Feed | Ogg Feed | iTunes Feeds | Torrent Feed

— Show Notes: —

How I saved myself from a btrfs nightmare:


System76

Brought to you by: System76

We asked: Do you trust btrfs?

It really started about three weeks ago, when I was doing a big package upgrade. The upgrade installs started to fail reporting out of space.

df reported more than 30GBs free on / partition. Despite df’s optimized summarization of my situation I cleared my package cache, deleted some unneeded VMs , and about 20 Steam games that I had stashed in /opt.

Fast forward to this Friday and my system won’t boot. I knew it was a risk, but I was desperate. I decided to compress all the things. Based on my quick reading of a few wiki pages, I could compress files on demand but necessarily compress the entire file system all the time. IE compress a lot of crap small libraries, reduce the amount of blocks they take up, and free up some space.

btrfs has built in support for doing this. I have ran compression on my /home file system since day one of this install. I opt for LZO compression. The ration of compression is lower than other options, but the performance is fantastic. I set off compressing the messy areas of my / (like /var /usr/lib).

But after a sanity check reboot, and many errors about failing to find a file, it was clear my understanding was wrong, and the mount options in my /etc/fstab needed to be updated.

I dd’ed the latest Antergos ISO onto a USB 3.0 thumb drive and booted the Bonobo into the live environment, fired up gparted to remind my self which physical device my rootfs was on (hey it’s been over a year!) and then promptly created a mount point for it under /mnt/fix.

I jumped into my /mnt/fix folder and immediately was back at home on my Bonobo’s root file system. A quick nano (YES) of my /mnt/fix/etc/fstab file and I added the compress flag to my root file system’s mount options.

Unmounted and rebooted the Bonobo, and the 1+ year Arch install fired right up.

I then set off to figure out how to better use btrfs on my system, and specifically tune the file system for my SSD drives.

Why btrfs?

btrfs

“a new copy on write (CoW) filesystem for Linux aimed at implementing advanced features while focusing on fault tolerance, repair and easy administration.”

  • Early days still, but its getting a lot closer and I wanted to have some real time under my belt with it.
  • The features it brings to Linux are going to be seen as minimum requirements in the future. CoW, snapshot, checksum, volume management.
  • SSD (Flash storage) awareness (TRIM/Discard for reporting free blocks for reuse) and optimizations.
  • Background scrub process for finding and fixing errors on files with redundant copies.
  • Online file system defragmentation.

OSCON Speaker Benchmarks LZO compression

  • **Compression **with modern hardware (like SSDs, multi-core CPUs) is a serious solution. It not only offers more value out of your SSD drive, but because the disk has to read less data overall, and the CPU is generally just waiting for I/O you can actually see an improvement on transfer speeds.

  • btrfs Stability Status

The filesystem disk format is no longer unstable, and it’s not expected to change unless there are strong reasons to do so. If there is a format change, file systems with a unchanged format will continue to be mountable and usable by newer kernels.

It kept telling me my device was full, but I had 30GB free. btrfs balance to the rescue.

  • Package updates would fail, reporting no space left on device.
  • Checking my system with df -h clearly reported 30GB free on my /
  • Checking with btrfs filesystem df showed a different story.
  • I used ncdu to sniff out the biggest files I could delete or move to give me some emergency wiggle room.

Why the different DF?

  • General linux userspace tools such as df will inaccurately report free space on a Btrfs partition.
  • df does not take into account space allocated for and used by the metadata. It is recommended to use /usr/bin/btrfs to query a btrfs partition.

  • Run the btrfs command to get a sense of what it can do. You want to perform a filesystem function and show stats about a device.

btrfs filesystem df /

  • Also:

btrfs filesystem show /dev/sda3

So, in general, it is impossible to give an accurate estimate of the amount of free space on any btrfs filesystem. Yes, this sucks. If you have a really good idea for how to make it simple for users to understand how much space they’ve got left, please do let us know, but also please be aware that the finest minds in btrfs development have been thinking about this problem for at least a couple of years, and we haven’t found a simple solution yet.

Step 1: Rebalance, it’s not just for RAID arrays anymore!

Rebalance to realign the blocks on the disk

  • Balance does a defragmentation, but not on a file level rather on the block group level. It can move data from less used block groups to the remaining ones, eg. using the usage balance filter.

btrfs balance start / -v

In general usage, no. A full unfiltered balance typically takes a long time, and will rewrite huge amounts of data unnecessarily. You may wish to run a balance on metadata only (see Balance_Filters) if you find you have very large amounts of metadata space allocated but unused, but this should be a last resort. At some point, this kind of clean-up will be made an automatic background process.

Step 2: Defragment that disk

btrfs filesystem defragment -r -v /

  • Optional if you’re really hard pressed for space consider defragmenting the metadata too:

find / -xdev -type d -print -exec btrfs filesystem defragment ‘{}’ \;

Step 3: Compress that disk’s file system.

SSD Workload with LZO

What are the differences between compression methods?

There’s a speed/ratio trade-off:

  • ZLIB — slower, higher compression ratio (uses zlib level 3 setting, you can see the zlib level difference between 1 and 6 https://code.google.com/p/lz4/ here).
  • LZO — faster compression and decompression than zlib, worse compression ratio, designed to be fast

The differences depend on the actual data set and cannot be expressed by a single number or recommendation. Do your own benchmarks. LZO seems to give satisfying results for general use.

  • This is running another defragmentation pass. Yes this means two defragmentation runs. But if you’re really tight on space, you need to free some up first before you can compress. The system will need some temporary space while it writes compressed versions of the files.

btrfs filesystem defragment -r -v -clzo /

  • Important: Update your /etc/fstab to include compress=lzo. For example:

LABEL=rootfs / btrfs defaults,compress=lzo 0 1

Step 4: Optimize for SSD
  • Best results with Linux 3.14 and up
  • I have added these flags to my fstab mount for both my / and my /home

noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache

Example:

LABEL=homefs /home btrfs defaults,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache 0 1

btrfs gotchas?

This page lists problems one might face when trying btrfs, some of these are not really bugs, but rather inconveniences about things not yet implemented, or yet undocumented design decisions.


– Picks –

Runs Linux

This GSM Base Station Powered by the BeagleBone Black, Runs Linux
  • Debian Wheezy 7
  • Upgraded to the 3.15.2 Linux Kernel
  • USRP B200 ($675 USD) The USRP B200 provides a fully integrated, single board, Universal Software Radio Peripheral platform with continuous frequency coverage from 70 MHz –6 GHz. Designed for low-cost experimentation, it combines a fully integrated direct conversion transceiver providing up to 56MHz of real-time bandwidth, an open and reprogrammable Spartan6 FPGA, and fast and convenient bus-powered.
  • OpenBTS is the software that provides us with the software part of a cellular station.
  • sipauthserve, for cellular subscriber registration

Desktop App Pick

New Linux Podcast App ‘Vocal’

The project’s developer, Nathan Dyer, has made beta builds — still unstable and not feature complete — available for testing through a dedicated PPA for Ubuntu 14.04 LTS and 14.10.

Weekly Spotlight

Tails – Privacy for anyone anywhere

Tails is a live operating system, that you can start on almost any computer from a DVD, USB stick, or SD card. It aims at preserving your privacy and anonymity

HTTPS Everywhere is a Firefox, Chrome, and Opera extension that encrypts your communications with many major websites, making your browsing more secure. Encrypt the web: Install HTTPS Everywhere today.


— NEWS —

NSA targets the privacy-conscious | Das Erste – Panorama – Meldungen

Xkeyscore

The investigation discloses the following:

  • Two servers in Germany – in Berlin and Nuremberg – are under surveillance by the NSA.
  • Merely searching the web for the privacy-enhancing software tools outlined in the XKeyscore rules causes the NSA to mark and track the IP address of the person doing the search. Not only are German privacy software users tracked, but the source code shows that privacy software users worldwide are tracked by the NSA.
  • Among the NSA’s targets is the Tor network funded primarily by the US government to aid democracy advocates in authoritarian states.
  • The XKeyscore rules reveal that the NSA tracks all connections to a server that hosts part of an anonymous email service at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) in Cambridge, Massachusetts. It also records details about visits to a popular internet journal for Linux operating system users called “the Linux Journal – the Original Magazine of the Linux Community”, and calls it an “extremist forum”.
If you read Boing Boing, the NSA considers you a target for deep surveillance

What is XKEYSCORE?

Tor and Tails have been part of the mainstream discussion of online security, surveillance and privacy for years. It’s nothing short of bizarre to place people under suspicion for searching for these terms.

One expert suggested that the NSA’s intention here was to separate the sheep from the goats — to split the entire population of the Internet into “people who have the technical know-how to be private” and “people who don’t” and then capture all the communications from the first group.

Another expert said that s/he believed that this leak may come from a second source, not Edward Snowden, as s/he had not seen this in the original Snowden docs; and had seen other revelations that also appeared independent of the Snowden materials.

Schneier on Security: NSA Targets the Privacy-Conscious for Surveillance

Jake Appelbaum et. al, are reporting on XKEYSCORE selection rules that target users — and people who just visit the websites of — Tor, Tails, and other sites. This isn’t just metadata; this is “full take” content that’s stored forever.

Dear NSA, Privacy is a Fundamental Right, Not Reasonable Suspicion | Electronic Frontier Foundation

Even the U.S. Foreign Intelligence Surveillance Court recognizes this, as the FISA prohibits targeting people or conducting investigations based solely on activities protected by the First Amendment. Regardless of whether the NSA is relying on FISA to authorize this activity or conducting the spying overseas, it is deeply problematic.

NSA: Linux Journal is an “extremist forum” and its readers get flagged for extra surveillance | Linux Journal

The Ultra-Simple App That Lets Anyone Encrypt Anything | Threat Level | WIRED

Nadim Kobeissi

Wired reports that Nadim Kobeissi will release a bet aversion of an all-purpose file encyrption browser plugin called miniLock at the HOPE hacker conference in New York. The free and open source plugin is meant to make it easy to drag and drop files to encrypt so that no one but the intended recipient can unscramble them.

Wayland in Fedora Update

So the summary is that while we expect to have a version of Wayland in Fedora Workstation 21 that will be able to run a fully functional desktop, there are some missing pieces we now know that will not make it. Which means that since we want to ship at least one Fedora release with a feature complete Wayland as an option before making it default, that means that Fedora Workstation 23 is the earliest Wayland can be the default.

The KDE Improv Project Has Announced Its End

Improv Board

Carl Symons on the behalf of Aaron Seigo and the rest of the Improv crew have sent out an email to the backers saying the project is over, they will issue partial refunds, etc. It’s only a partial refund right now as they had already invested some money into buying long lead times with their Chinese manufacturer.

There were also credit card processing fees, etc, but they’re working out a path for full reimbursement. It’s also said Aaron invested $200k USD into the project.

There was simply not enough support to make the project work, despite having fully functional, production ready devices and a strong commitment to succeed. The Free software community does not seem ready at this point to make a concerted stand on the pressing issue of hardware freedom

The group of folks behind the KDE desktop environment have been trying for a few years to deliver a tablet with wouldn’t rely on proprietary software. More recently they unveiled the Improv Board, a small, cheap computer module designed to ship with Mer Linux.

At this point, all the team is promising are partial refunds since part of the money has already been spent. But the goal is to eventually provide full refunds to folks who have put their faith (and money) into the project.

In a statement, the team suggests “the Free software community does not seem ready at this point to make a concerted stand on the pressing issue of hardware freedom,”
the Improv board was expected to sell for around $75

Ubuntu MATE Remix

Ubuntu MATE Edition

Objectives

The Ubuntu MATE Remix has the following primary goal:

  • Use Ubuntu to create a solid foundation on which to build a pure MATE desktop.

Ubuntu MATE Remix has some secondary objectives:

  • Increase both Ubuntu and MATE user adoption.
  • Restore the halcyon days of Ubuntu before indicators, Unity and scopes were introduced.
  • Be the go to Ubuntu alternative for computers that aren’t powerful enough to run a composited desktop, as well as those that are.
  • Make use of existing Ubuntu themes and artwork wherever possible so Ubuntu MATE Remix is immediately familiar.
  • When ever possible submit new or revised packages to Debian so both the Debian and Ubuntu communities benefit.
  • Package selection will favour functionality and stability over lightness and whimsy.
  • Provide a refuge for Linux users who prefer a traditional desktop experience.
  • Get adopted as an official Ubuntu “flavour”.

Feedback:

— Chris’ Stash —

Hang in our chat room:

irc.geekshed.net #jupiterbroadcasting

— What’s Matt Doin? —

— Find us on Google+ —

— Find us on Twitter —

— Follow the network on Facebook: —

— Catch the show LIVE Sunday 10am Pacific / 1pm Eastern / 6pm UTC: —

Question? Comments? Contact us here!