Wiping Disks

How fast can you wipe a complete disk? For this I dumped zeros to a MS Windows partition based on an SSD formatted with NTFS.

[root@i7 ~]# time dd if=/dev/zero of=/dev/sda1 bs=1M
dd: error writing '/dev/sda1': No space left on device
81920+0 records in
81919+0 records out
85898297344 bytes (86 GB, 80 GiB) copied, 182.699 s, 470 MB/s

real    3m2.703s
user    0m0.033s
sys     0m43.470s

It was a real pleasure to get rid of MS Windows after just 3 minutes 😉

Once more, second partition on SSD formatted with NTFS.

[root@i7 ~]# time dd if=/dev/zero of=/dev/sda2 bs=1M
dd: error writing '/dev/sda2': No space left on device
147015+0 records in
147014+0 records out
154155352064 bytes (154 GB, 144 GiB) copied, 326.108 s, 473 MB/s

real    5m26.110s
user    0m0.097s
sys     1m21.503s

Now a hard disk, i.e., moving parts, also formatted with NTFS.

[root@i7 ~]# time dd if=/dev/zero of=/dev/sdb1 bs=4M
128459+0 records in
128459+0 records out
538796097536 bytes (539 GB, 502 GiB) copied, 5404.54 s, 99.7 MB/s     <--- from kill -10
dd: error writing '/dev/sdb1': No space left on device
238467+0 records in
238466+0 records out
1000202043392 bytes (1.0 TB, 932 GiB) copied, 11876.6 s, 84.2 MB/s

real    197m56.647s
user    0m0.000s
sys     15m49.763s

From the man-page of dd:

Sending a USR1 signal to a running ‘dd’ process makes it print I/O statistics to standard error and then resume copying.

For example, in above scenario I used

kill -10 12523

Signal numer 10 is SIGUSR1, see /usr/include/bits/signum.h, or

$ kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT      7) SIGBUS       8) SIGFPE       9) SIGKILL     10) SIGUSR1
11) SIGSEGV     12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM
16) SIGSTKFLT   17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU     25) SIGXFSZ
26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGIO       30) SIGPWR
31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX

Make USB Stick Bootable

In various forums one reads that one should use dd to copy a ISO image to an USB stick. Although this works, more often you do not want to use a ISO image but rather copy a Linux system at hand. First you mount the root filesystem of the new USB stick, then mount /boot within a chroot. Finally use grub commands. I.e., type

mount /dev/sdc2 /mnt/stick
arch-chroot /mnt/stick
mount /dev/sdc1 /boot           <--- /boot is local to chroot!
grub-install --target=i386-pc --boot-directory=/boot /dev/sdc
grub-mkconfig -o /boot/grub/grub.cfg
umount /boot                    <--- umount "local" /boot

Also see GRUB in the Arch Wiki.

Setting the bootable flag in the partition table one either uses gparted, a graphical tool, or, as the task at hand is so simple, one just uses parted.

parted /dev/sdc
set 1 boot on
print

Unrelated, but often useful. Just in case you changed something in the initial RAM disk, use

mkinitcpio -p linux

Switching from ext4 to btrfs

Hearing all these wonder stories about btrfs I decided to give btrfs a try. Additionally I encrypted my SSD using LUKS.

The relevant entries in the Arch WIKI are: Setting up LUKS and btrfs. Here is the list of commands I used.

1. I saved my data from SSD using tar and saved tar-file on hard-disk:

cd /
time tar cf /mnt/c/home/klm/ssd1.tar bin boot etc home opt root srv usr var

Alternatively one can use cp -a to copy directories with symbolic links. Option -a is the same as -dR --preserve=all.

2. I used gparted to partition the SSD, creating /boot and / (root). For /boot I directly enabled btrfs in gparted.

3. Encrypt the partition.

cryptsetup -y -v -s 512 luksFormat /dev/sdc2

Then open it and give it an arbitrary name:

cryptsetup luksOpen /dev/sdc2 cryptssd

4. Create filesystem using btrfs. This is the reason for all this effort. Although this is the easiest.

mkfs.btrfs /dev/mapper/cryptssd

5. Adapt /etc/fstab, e.g., with genfstab -U /mnt >> /mnt/etc/fstab

UUID=3b8bb70c-390a-4a9e-9245-ea19af509282       /       btrfs   rw,relatime     0 0
UUID=a8d6c185-0769-4ec5-9088-2c7087815346       /boot   ext4    rw,data=ordered,relatime        0 2

Check results with lsblk -if.

6. Chroot into new system using arch-chroot and put GRUB on it, as usual. Add required directories, first.

mkdir boot proc run sys tmp

Then edit the configuraton file for GRUB:

vi /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="cryptdevice=UUID=5a74247e-75e8-4c05-89a7-66454f96f974:cryptssd:allow-discards root=/dev/mapper/cryptssd"

grub-install --target=i386-pc /dev/sdb
grub-mkconfig -o /boot/grub/grub.cfg

Keyword :allow-discards after cryptssd is for SSD TRIM. For a mechanical hard drive this keyword should be omitted.

7. Install btrfs utilities and programs on new system if not already installed. Add the btrfs executable to the initial RAM disk, i.e., set the entry for BINARIES.

pacman -S btrfs-progs

vi /etc/mkinitcpio.conf
. . .
BINARIES="/usr/bin/btrfs"
. . .
mkinitcpio -p linux

8. Extracting back from the tar-file.

time tar xf /mnt/c/home/klm/ssd1.tar

9. Adding TRIM for SSD:

systemctl status fstrim
systemctl enable fstrim.timer
systemctl start fstrim.timer

Show timers (like crontab -l):

systemctl --type=timer

10. A simple benchmark, as indicated by above time before tar, does not show any performance benefits so far. But performance was not the main motivator, but rather the added functionality, especially taking snapshots.

Downgrading PulseAudio with downgrade.sh

Just upgraded PulseAudio from 9.0-1 => 10.0-2, which was no good, as I was left with no sound.

The bash script downgrade.sh given in Github saved the day. Sound is back. The script is for Arch Linux.

Usage:

downgrade.sh libpulse pulseaudio pulseaudio-bluetooth

From now on I have to use

pacman -Syu --ignore pulseaudio --ignore libpulse --ignore pulseaudio-bluetooth

to not lose the good version and getting the infected version again.

slurmd confused of pid-file

I wrote on SLURM used on Ubuntu here. As I moved to Arch Linux I of course use the fine AUR package slurm-llnl maintained by Gordian Edenhofer.

After upgrading to version 16.05.7-1 I noticed the following: If slurmd is started by systemd then the pid-file specified for systemd in /lib/systemd/system/slurmd.service (PIDFile) should match the pid-file location (SlurmdPidFile) given in /etc/slurm-llnl/slurm.conf.

By the way, if one gets an error

slurm_receive_msg: Zero Bytes were transmitted or received

check your date and time of your nodes.

Bluetooth Headphones in Arch Linux

There is a big difference between noise-cancelling headphones, and classical headphones without noise-cancelling ability! Especially when you use them in a noisy environment, like a plane or a large office bureau. Inspired by a positive review of the Bose headphones by Marques Brownlee, I bought them.

Here is the review:

Pairing with my OnePlus One smartphone was completely automatic and works like a charm. No further explnation is required.

Paring with a PC/laptop running Arch Linux needed a little more effort. Some advice in Bluetooth headset helped me alot.

One-time configuation:

systemctl start bluetooth.service
hciconfig hci0 up piscan
pacmd list-sinks | grep index:
bluetoothctl

In bluetoothctl enter

help
show
devices
pair xx:yy:...
trust xx:yy:...
paired-devices

I had to delete the directory below /var/lib/bluetooth. Apparently something was stored there which shouldn’t have been there.

Once the pairing works, as described above, I just use:

systemctl start bluetooth.service
hciconfig hci0 up piscan

for starting bluetooth and making my PC with bluetooth visible. I switch on the headphone, which normally finds the PC in less than a second. Then I have to set the right sink via pacmd:

pacmd set-default-sink `pacmd list-sinks | grep index: | tail -1 | cut -d " " -f6`

Checking that all is well is:

pacmd list-sinks | grep index:

Once paired with your Linux machine, yo have to repeat set-default-sink if you lose the connection to your Linux machine, for example by walking too far away. If you lose Bluetooth connection apparently sound output will go to the regular speakers of your Linux machine. In case you are working in a large office other people will hear your music. Of course, you can mute the regular loudspeakers of your Linux machine using

pacmd set-sink-volume 0 0

assuming sink #0 is regular loudspeaker.

Running bacman in parallel

The mailing list for pacman-dev contains an interesting thread when rebuilding packages from source, i.e., running bacman in parallel. Gordian Edenhofer ran some tests on performance when running one, two, three, up to six jobs in parallel.

The results, below, clearly show that using all your cores is of great benefit:

simple_benchmark

Very similar to CPU Usage Time Is Dependant on Load.

There is an AUR alternative for bacman called fakepkg written by Gordian Edenhofer which supports this parallelism.

The measurements are as follows:

# Computer:
## nuc
CPU: i5-4250U CPU @ 1.30GHz
Mem: 8GB

# Packages: base base-devel

* Jobs: 1
real    5m40.325s
user    4m35.710s
sys     0m37.270s

* Jobs: 2
real    4m39.047s
user    4m50.913s
sys     0m35.717s

* Jobs: 3
real    3m9.770s
user    6m5.137s
sys     0m46.153s

* Jobs: 4
real    2m51.163s
user    7m25.950s
sys     0m30.090s

* Jobs: 5
real    2m41.124s
user    8m6.687s
sys     0m26.627s

* Jobs: 6
real    2m36.724s
user    8m12.370s
sys     0m24.470s

# Packages: base base-devel gnome

* Jobs: 1
real    10m34.284s
user    7m33.677s
sys     1m25.223s

* Jobs: 2
real    8m18.592s
user    8m45.407s
sys     1m45.633s

* Jobs: 3
real    5m48.511s
user    11m40.323s
sys     1m46.967s

* Jobs: 4
real    5m8.346s
user    13m47.353s
sys     1m16.640s

* Jobs: 5
real    4m52.659s
user    14m34.700s
sys     1m7.933s

* Jobs: 6
real    4m58.652s
user    14m25.353s
sys     1m6.863s

# Packages: all 1021

* Jobs: 1
real    80m56.509s
user    70m8.780s
sys     9m29.813s

* Jobs: 2
real    66m42.653s
user    75m13.030s
sys     9m57.040s

* Jobs: 3
real    43m25.701s
user    92m52.983s
sys     8m57.147s

* Jobs: 4
real    38m9.114s
user    110m5.763s
sys     6m43.427s

* Jobs: 5
real    36m12.302s
user    118m32.186s
sys     6m31.533s

* Jobs: 6
real    36m4.872s
user    118m54.449s
sys     6m35.830s

# Packages: base base-devel

* XZ_OPT="-T 1"
real    5m32.345s
user    4m22.960s
sys     0m28.610s

* XZ_OPT="-T 2"
real    4m27.433s
user    4m34.683s
sys     0m26.403s

* XZ_OPT="-T 3"
real    4m8.689s
user    4m59.597s
sys     0m20.943s

* XZ_OPT="-T 4"
real    4m7.828s
user    5m20.103s
sys     0m21.717s

* XZ_OPT="-T 5"
real    4m6.304s
user    5m20.757s
sys     0m21.817s

* XZ_OPT="-T 6"
real    4m5.932s
user    5m19.480s
sys     0m21.237s

* XZ_OPT="-T 0"
real    4m5.851s
user    5m18.137s
sys     0m20.823s

# Packages: base base-devel gnome

* XZ_OPT="-T 1"
real    10m20.164s
user    6m36.933s
sys     0m57.647s

* XZ_OPT="-T 2"
real    9m8.262s
user    6m52.057s
sys     0m57.357s

* XZ_OPT="-T 3"
real    8m53.265s
user    7m30.650s
sys     0m57.827s

* XZ_OPT="-T 4"
real    8m53.075s
user    8m1.787s
sys     0m59.480s

* XZ_OPT="-T 5"
real    8m48.173s
user    7m49.223s
sys     0m58.567s

* XZ_OPT="-T 6"
real    8m48.970s
user    7m47.837s
sys     0m56.570s

* XZ_OPT="-T 0"
real    8m49.713s
user    7m47.470s
sys     0m56.613s

# Packages: all 1021 packages

* XZ_OPT="-T 1"
real    79m42.006s
user    65m42.950s
sys     7m32.397s

* XZ_OPT="-T 2"
real    63m27.457s
user    68m57.536s
sys     6m56.100s

* XZ_OPT="-T 3"
real    60m37.071s
user    79m6.113s
sys     7m24.263s

* XZ_OPT="-T 4"
real    59m16.447s
user    85m22.746s
sys     7m35.783s

* XZ_OPT="-T 5"
real    59m2.436s
user    86m7.093s
sys     7m47.653s

* XZ_OPT="-T 6"
real    59m1.516s
user    86m18.973s
sys     7m34.320s

* XZ_OPT="-T 0"
real    58m36.010s
user    84m17.283s
sys     7m25.270s

The corresponding R-program to print the values is:

# Computer: NUC
## CPU: i5-4250U CPU @ 1.30GHz
## Mem: 8GB

# Measurement vectors
jobs = seq(1, 6)
# base and base-devel group (64 unique packages)
time_base = c(340.325, 279.047, 189.770, 171.163, 161.124, 156.724)
# base, base-devel and gnome group (111 unique packages)
time_gnome = c(634.284, 498.592, 348.511, 308.346, 292.659, 298.652)
# all packages installed at the NUC (1021 unique packages)
time_all = c(4856.509, 4002.653, 2545.701, 2289.114, 2172.302, 2164.872)
# base and base-devel group using XZ_OPT instead of true parallelization (64 unique packages)
time_xzopt_base = c(332.345, 267.433, 248.689, 247.828, 246.304, 245.932)
# base, base-devel and gnome group using XZ_OPT instead of true parallelization (111 unique packages)
time_xzopt_gnome = c(620.164, 548.262, 533.265, 533.075, 528.173, 528.970)
# all packages installed at the NUC using XZ_OPT instead of true parallelization (1021 unique packages)
time_xzopt_all = c(4782.006, 3807.457, 3637.071, 3556.447, 3542.436, 3541.516)

# Export drawing as vector graphic suitable for printing with A4
#svg("bacman: simple benchmark.svg", width=1*10, height=sqrt(2)*10)

# Define initial window size
par(mfrow=c(3, 1))

# Plot points and lines for base
plot(time_all ~ jobs, main="all (1021 pkgs)", ylim=c(0, max(time_all)*1.1), xlab="Jobs/Count", ylab="Time/s", col="blue")
lines(time_all ~ jobs, col="blue")
points(time_xzopt_all ~ jobs, col="black")
lines(time_xzopt_all ~ jobs, col="black")
legend(x="topright", legend=c("Parallele Jobs", "XZ_OPT=\"-T JOBS\""), col=c("blue", "black"), pch=c(19, 19))

# Plot points and lines for gnome
plot(time_gnome ~ jobs, main="base, base-devel, gnome (111 pkgs)", ylim=c(0, max(time_gnome)*1.1), xlab="Jobs/Count", ylab="Time/s", col="blue")
lines(time_gnome ~ jobs, col="blue")
points(time_xzopt_gnome ~ jobs, col="black")
lines(time_xzopt_gnome ~ jobs, col="black")
legend(x="topright", legend=c("Parallele Jobs", "XZ_OPT=\"-T JOBS\""), col=c("blue", "black"), pch=c(19, 19))

# Plot points and lines for all
plot(time_base ~ jobs, main="base, base-devel (64 pkgs)", ylim=c(0, max(time_base)*1.1), xlab="Jobs/Count", ylab="Time/s", col="blue")
lines(time_base ~ jobs, col="blue")
points(time_xzopt_base ~ jobs, col="black")
lines(time_xzopt_base ~ jobs, col="black")
legend(x="topright", legend=c("Parallele Jobs", "XZ_OPT=\"-T JOBS\""), col=c("blue", "black"), pch=c(19, 19))

# Write drawing to file
#dev.off()