Blocking IP addresses with ipset

In Chinese Hackers I mentioned that I block ssh attackers with the help of fail2ban. Unfortunately, fail2ban uses iptables to create firewall rules in “Chain f2b-SSH” for each individual IP address. For a modern processor this is no problem, even if you have thousands of these rules. While for a low powered ARM processer this can have a noticeable influence on network performance. In Using Odroid as IP Router I wrote:

Added 10-Jan-2019: I previously added ca. 3000 iptables rules for blocking IP address ranges which attacked me on port 22 (ssh). That many rules will deteriorate your network performance significantly. My download speed went down from 100 MBit/s to 20 MBit/s.Added 10-Jan-2019: I previously added ca. 3000 iptables rules for blocking IP address ranges which attacked me on port 22 (ssh). That many rules will deteriorate your network performance significantly. My download speed went down from 100 MBit/s to 20 MBit/s.

An alternate or additional approach to this slowdown due to iptables is to use ipset, Arch package ipset. ipset maintains hash tables of IP addresses or networks. At the time when I still used the Odroid, then the main sshd-server copied the blocked IP addresses to the Odroid, which in turn populated the hash table with the IP addresses to be blocked.

Setting up ipset and populating the hash tables from fail2ban can be done as below.

sqlite3 -csv /var/lib/fail2ban/fail2ban.sqlite3 "select distinct ip from bips order by ip" |    \
        perl -e 'BEGIN {print "create reisTmp hash:ip family inet hashsize 65536 maxelem 65536\n"; }
                print "add reisTmp $_" while (<>);'     \
        > ~/tmp/reisTmp

The name reisTmp is arbitrary. This way we have a file which looks like this:

create reisTmp hash:ip family inet hashsize 65536 maxelem 65536
add reisTmp
add reisTmp
add reisTmp
add reisTmp
add reisTmp
add reisTmp
. . .

This file is transfered from the sshd/fail2ban-server to our router, if required, and is the input to be run through ipset:

ipset restore -f ~klm/tmp/reisTmp
ipset swap reisTmp reisbauer
ipset destroy reisTmp

This assumes you have created a hash table in ipset called reisbauer. Again, this name is arbitrary. Above scenario assumes that populating this hash table takes some time, then just swap hash tables, which is almost instant.

To re-initiate the hash table after reboot, you use

systemctl enable ipset
systemctl start ipset

This basically does

ipset -f /etc/ipset.conf restore

Splitting and anti-merging vCard files

Sometimes vCard files need to be split into smaller files, or the file needs to be protected against merging in another application.

1. Splitting. Below Perl script splits the input file into as many files as required. Output files are named adr1.vcf, adr2.vcf, etc. You can pass a command line argument “-n” to specify the number of card records per file. Splitting a vCard file is provided in palmadrsplit on GitHub:

use Getopt::Std;

my %opts;
my ($i,$k,$n) = (1,0,950);
$n = ( defined($opts{'n'}) ? $opts{'n'} : 950 );

open(F,">adr.$i.vcf") || die("Cannot open adr.$i.vcf for writing");
while (<>) {
        if (/BEGIN:VCARD/) {
                if (++$k % $n == 0) {   # next address record
                        close(F) || die("Cannot close adr.$i.vcf");
                        ++$i;   # next file number
                        open(F,">adr.$i.vcf") || die("Cannot open adr.$i.vcf for writing");
        print F $_;
close(F) || die("Cannot close adr.$i.vcf");

This is required for Google Contacts, as Google does not allow to import more than 1,000 records per day, see Quotas for Google Services.

2. Anti-Merge. Inhibiting annoying merging is given in file palmantimerge on GitHub. Overall logic is as follows: Read entire vCard file and each card, delimited by BEGIN:VCARD and END:VCARD, is put on a hashmap. Each hashmap entry is a list of vCards. Hash key is the N: entry, i.e., the concatentation of lastname and firstname. Once everything is hashed, then walk through hash. Those hash entries, where the list contains just one entry, can be output as is. Where the list contains more than one entry, then these entries would otherwise be merged, and then the N: part is modified by using the ORG: field.

use strict;
my @singleCard = ();    # all info between BEGIN:VCARD and END:VCARD
my ($name) = "";        # N: part, i.e., lastname semicolon firstname
my ($clashes,$line,$org) = (0,"","");
my %allCards = {};      # each entry is list of single cards belonging to same first and lastname, so hash of array of array

while (<>) {
        if (/BEGIN:VCARD/) {
                ($name,@singleCard) = ("", ());
                push @singleCard, $_;
        } elsif (/END:VCARD/) {
                push @singleCard, $_;
                push @{ $allCards{$name} }, [ @singleCard ];
        } else {
                push @singleCard, $_;
                $name = $_ if (/^N:/);

for $name (keys %allCards) {
        $clashes = $#{$allCards{$name}};
        for my $sglCrd (@{$allCards{$name}}) {
                if ($clashes == 0) {
                        for $line (@{$sglCrd}) { print $line; }
                } else {
                        $org = "";
                        for $line (@{$sglCrd}) {
                                $org = $1 if ($line =~ /^ORG:([ \-\+\w]+)/);
                        for $line (@{$sglCrd}) {
                                $line =~ s/;/ \/${org}\/;/ if ($line =~ /^N:/);
                                print $line;

Every lastname is appended with “/organization/” if the combination of firstname and lastname is not unique. For example, two records with Peter Miller in ABC-Corp and XYZ-Corp, will be written as N:Miller /ABC-Corp/;Peter and N:Miller /XYZ-Corp/;Peter.

This way Simple Mobile Tools Contacts will not merge records together which it shouldn’t. Issue #446 for this is on GitHub.

J-Pilot Plugin For SQLite Export

In SQL Datamodel For J-Pilot I described the SQLite datamodel. I wrote a J-Pilot plugin which can export the below entities and write them to an SQLite database file. The direction is one-way: from J-Pilot to SQLite.

  1. Address
  2. Datebook
  3. Memo
  4. To-Do
  5. Expense
  6. Various categories for above entities

Adding more entities is pretty easy. For example, if people need the Calendar Palm database exported, this can be implemented quickly. We use the usual SQLite API with sqlite3_exec(), and sqlite3_prepare(), sqlite3_bind(), sqlite3_step(), and finally sqlite3_finalize().

The general mechanics of a J-Pilot plugin are described by Judd Montgomery, the author of J-Pilot, in this document. I took the Expense/expense.c source code from the Expense plugin as a guide.

The plugin provides the following functionality:

  1. Create new database from scratch, it is called jptables.db
  2. Export above mentioned entities
  3. In debug mode you can use J-Pilot‘s search to search in the SQLite database

If you call jpilot -d then debug-mode is activated.


  1. Compile single source code file jpsqlite.c
  2. Copy library (.so file) in plugin directory ($HOME/.jpilot/plugins)
  3. Copy datamodel SQL file jptables.sql in plugin directory

Compilation is with below command:

gcc `pkg-config -cflags-only-I gtk+-2.0` -I <J-Pilot src dir> -s -fPIC -shared jpsqlite.c -o -lsqlite3

For this to work you need the Pilot-Link header files and the J-Pilot (AUR) source code at hand.

Running the plugin: go to the plugins menu by main-menu selection or function key (F7 in my case), then press SQL button. All previous data is completey erased in the database, then all data is written to database within a single transaction.

In debug mode and in debug mode only, the J-Pilot search also searches through all entities in the SQLite database.

The long-term goal is that SQLite is the internal data structure for J-Pilot, thereby abandoning the binary files entirely.

SQL Datamodel For J-Pilot

Currently J-Pilot stores its data in binary form which is compatible with the original Palm data format. See Palm File Format Specification. Reading these binary formats is not simple, see for example pdbrd.c. Portion of the internal structure looks something like this:

typedef struct {        // header of record list
        LocalID nextRecordListID;
        UInt16 numRecords;
        UInt16 firstEntry;
} RecordListType;

typedef struct {        // single element of record list
        LocalID localChunkID;   // offset, start of actual data
        UInt8 attributes;       // 1st nibble: ?/private, 2nd: this is the category
        UInt8 uniqueID[3];
} RecordEntryType;

typedef struct {
        UInt16 renamedCategories;
        char categoryLabels[16][16];
        UInt8 categoryUniqIDs[16];
        UInt16 lastUniqID_pad;  //UInt8 lastUniqID; UInt8 padding;
} AppInfoType;

In 2013 I proposed to use SQLite as internal data format instead, see Possible Enhancements to J-Pilot.

Below datamodel closely follows the field declaration in J-Pilot and Pilot-Link, e.g., /usr/include/pi-datebook.h or utils.h.

Continue reading

Sharing Terminal Screen Using Tmux

Sharing your terminal input and output can be done in different ways.

  1. kibitz, see Linux commands: expect and kibitz
  2. VNC
  3. Skype / Teams / Zoom, etc.
  4. tmux

1. tmux essentials. tmux (tmux wiki) uses the following vocabulary and hierarchy.

  1. server
  2. session-group
  3. session
  4. client
  5. window
  6. pane

The tmux-server listens on a socket. The tmux-client talks to this socket. A tmux-server may have 0, 1, …, n tmux-sessions. A tmux-client connects to a tmux-session. Two or more different tmux-clients may connect to the same tmux-session (that’s what this post will lead to). Each tmux-session has 1, …, n tmux-windows (so at least one). Each tmux-window has 1, …, n tmux-panes (so at least one). Sessions in the same group share the exact same set of windows, except the current window, i.e., closing or creating new windows in this group affect all sessions in this group.

Below .tmux.conf is my preferred configuration and is in no way required for sharing terminal screens. Your file .tmux.conf can be empty or absent at all.

# The default key is C-b because the prototype of tmux was originally
# developed inside screen and C-b was chosen not to clash with
# the screen meta key.
set -g prefix C-a
unbind C-b
bind C-a send-prefix

# Set a  scrollback buffer limit
set -g history-limit 100000

# Enable mouse control (clickable windows, panes, resizable panes)
set -g mouse on

# Enable vi keys for working with buffers
setw -g mode-keys vi
# Constrain window only if another session is looking at it
setw -g aggressive-resize on
# Always resize windows to the larger/smallest session containing the window
#setw -g window-size largest
setw -g window-size smallest

new-session -s klm -n gen zsh
#new-window -t0 -n gen zsh
new-window -t1 -n bin zsh
new-window -t2 -n etc zsh
new-window -t3 -n man zsh
new-window -t4 -n root zsh
new-window -t5 -n log zsh
new-window -t6 -n ssh zsh
new-window -t7 -n music zsh
new-window -t8 -n chrome zsh
new-window -t9 -n Dwn zsh
split-window -h -t2
split-window -h -t8
select-window -t0

Starting tmux is now

tmux start-server \; attach-session

or in abbreviated form

tmux start \; a

Below screenshot shows all those 10 windows in action. One of those windows, here number 8, has two panes.

2. Screen sharing. If two different users A and B on a single Linux machine want to share their terminal screen, they have to do the following:

  1. User A: open a tmux using a “special” socket: tmux -S /tmp/userA_socket new-session -s userA_session
  2. User A: making socket accessible to user B: chmod 777 /tmp/userA_socket
  3. User B: access tmux from user A: tmux -S /tmp/userA_socket attach -t userA_session

So two users spawn two clients which connect to the same session managed by one server.

Instead of creating a new socket, /tmp/userA_socket, one can also use the default socket used by tmux, which is /tmp/tmux-/default. If chmod 777 is too risky, then use ACL:

setfacl -m u:UserB:rwx /tmp/userA_socket

You can achieve a similar effect with GNU screen, although one of the participants needs to setuid-root the screen program. If you do not have root access to the machine in question, then GNU screen is not an option.

J-Pilot Data on Android Phone: Contacts

Keeping J-Pilot data in sync with Android smartphone is quite a challenge. See my post about Google calendar. Copying contact data from J-Pilot to Android in the past went like this:

  1. Export data from J-Pilot in vCard (vcf) format
  2. If required, fiddle with this vcf file with some Perl script
  3. Stopping contact app in Android and stopping WiFi. Deleting all data in Android app.
  4. Manually delete contacts in Google Contacts in batches of ca. 500 records (Added 10-May-2020: Meanwhile there is no longer a limit here, so you can delete all your contacts in one go.)
  5. Import vCard data in Google Contacts
  6. Turn on WiFi on Android, starting contact-app

As one can easily see, this was very cumbersome. To make this even more difficult, now Google has limited the number of imported records in vCard file to 1,000 records per day. If you have more than that, then you are stuck. The limits are documented in quotas. Continue reading

slurmctld showing nodes as ‘drain’

Recently one node was changed from octacore to hexacore. Since that time sinfo showed this node in STATE drain. Stopping and restarting slurmctld did not resolve the issue. Log file /var/log/slurm-llnl/slurmctld.log showed

[2019-12-31T23:57:52.588] error: Node X appears to have a different slurm.conf than the slurmctld.  This could cause issues with communication and functionality.  Please review both files and make sure they are the same.  If this is expected ignore, and set DebugFlags=NO_CONF_HASH in your slurm.conf.

This hinted that some caching is the culprit. Entry


shows where slurmctld stores state. Removing the files in that directory and restarting slurmctld solves the problem.

Balking ARM ArchLinux Update

Had an issue with update of ARM ArchLinux which I use here. Error messages like:

(95/95) checking keys in keyring                                                          [####################################################] 100%
(95/95) checking package integrity                                                        [####################################################] 100%
error: readline: signature from "Arch Linux ARM Build System " is unknown trust
:: File /var/cache/pacman/pkg/readline-7.0.005-1-armv7h.pkg.tar.xz is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] n
error: bash: signature from "Arch Linux ARM Build System " is unknown trust
:: File /var/cache/pacman/pkg/bash-4.4.023-1-armv7h.pkg.tar.xz is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] ^C

Found forum post: many corrupted packages/invalid PGP signatures for aarch64? Suggestion there:

pacman-key --init
pacman-key --populate archlinuxarm

This helped, so could proceed with pacman -Syu.

Also try

pacman -Sy archlinux-keyring

if above is not sufficient.

Using Scooter Software Beyond Compare

Beyond Compare is a graphical file comparison tool sold by Scooter Software. Its open-source competitors are mainly vimdiff, and kdiff3. Its advantage is ease-of-use. While comparing files they can be edited instantly. You can diff complete directory trees.

It is written in Delphi Object Pascal, the source code is not open-source. It runs on Windows, x86 Linux, and OS X. It does not run on ARM, like Raspberry Pi or Odroid, see support for arm processors – like the raspberry pi. The “Standard Edition” costs $30, the “Pro Edition” costs $60. The software is in AUR.

1. Root User Problem. When using it as root-user you must use:


When running

DIFFPROG=bcompare pacdiff

the screen looks like this:

2. Git Usage. To use Beyond Compare with git difftool you have to do two things: First you must create an alias bc3 for bcompare.

[root /bin]# ln -s bcompare bc3

Second add the following lines to your ~/.gitconfig file:

        tool = bc3
        prompt = false
        bc3 = trustExitCode
        tool = bc3
        bc3 = trustExitCode

Alternatively to above changes in the ~/.gitconfig file, use the following commands:

git config --global diff.tool bc3
git config --global difftool.bc3.trustExitCode true
git config --global merge.tool bc3
git config --global mergetool.bc3.trustExitCode true

Instant Messaging Client Pidgin and Skype / Lync

One can use the instant messaging client Pidgin and Skype. Skype for Business was previously called Lync.

Install Pidgin

pacman -S pidgin

and install the SIP/Simple protocol for Skype for Business/Office 365/Lync

pacman -S pidgin-sipe

Setting up the account: Use your company mail-address. Protocol is: Office communicator.

In Set up Pidgin messenger and Office 365 Lync one finds the crucial hint that one has to use a special user-agent information string

User Agent :  UCCAPI/15.0.4420.1017 OC/15.0.4420.1017

In Pidgin configuration this looks like this:

The post from Gary Woodfine from above states that you also have to specify server, port, and authentification scheme. But you don’t have to. You can simply leave these entries empty, or having their default values.

Remote Unlocking of Encrypted Disks

1. Problem statement. You have an encrypted disk and want to decrypt the disk during boot while not sitting in front of your computer.

Solution is sketched and indicated in dm-crypt/Specialties. Below is a little bit more explanation. For the following you must be root.

2. Required software packages. Install the following packages: dropbear from repo “Community”. Then install the following AUR-packages:

  1. mkinitcpio-netconf
  2. mkinitcpio-utils
  3. mkinitcpio-dropbear

3. Populate root_key. First mkdir /etc/dropbear and populate root_key file with public ssh keys which should be able to log into your machine, similar to authorized_keys for OpenSSH. I.e., you must know the private keys on the corresponding machines you intend to use for unlocking.

4. Set-up networking in Grub. Edit /etc/default/grub and set

GRUB_CMDLINE_LINUX_DEFAULT="cryptdevice=UUID=5a74247e-75e8-4c05-89a7-66454f96f974:cryptssd:allow-discards root=/dev/mapper/cryptssd ip="

Then issue

grub-mkconfig -o /boot/grub/grub.cfg

to re-generate grub.cfg. The specification for “ip=” is given in Mounting the root filesystem via NFS (nfsroot). Its most important parts are:

  1. client-ip: IP address of the client
  2. server-ip: IP address of the NFS server
  3. gateway-ip: IP address of a gateway
  4. netmask: Netmask for local network interface
  5. hostname: Name of the client
  6. device: Name of network device to use
  7. autoconf: Method to use for autoconfiguration

5. Configure mkinitcpio. Finally, the main task. Edit /etc/mkinitcpio.conf and set

HOOKS="base udev block keymap keyboard autodetect modconf netconf dropbear encryptssh filesystems fsck"

Now call

mkinitcpio -p linux

See Arch Wiki mkinitcpio. Output of mkinitcpio looks something like this:

  -> Running build hook: [dropbear]
Key is a ssh-rsa key
Wrote key to '/etc/dropbear/dropbear_rsa_host_key'
Key is a ssh-dss key
Wrote key to '/etc/dropbear/dropbear_dss_host_key'
Key is a ecdsa-sha2-nistp256 key
Wrote key to '/etc/dropbear/dropbear_ecdsa_host_key'
dropbear_rsa_host_key : sha1!! e1:11:51:ce:0b:07:2b:c7:66:37:c0:b9:de:f3:80:56:64:69:cc:fd
dropbear_dss_host_key : sha1!! ca:75:42:85:f9:96:6d:db:fd:15:d1:7a:4a:ee:19:b1:ff:91:14:bb
dropbear_ecdsa_host_key : sha1!! b9:b3:c4:ee:c4:af:21:87:52:39:e8:b6:c2:a3:b7:53:0e:52:f1:85
   -P, --allpresets             Process all preset files in /etc/mkinitcpio.d
   -r, --moduleroot <dir>       Root directory for modules (default: /)
   -S, --skiphooks <hooks>      Skip specified hooks, comma-separated, during build
   -s, --save                   Save build directory. (default: no)
   -d, --generatedir <dir>      Write generated image into <dir>
   -t, --builddir <dir>         Use DIR as the temporary build directory
   -V, --version                Display version information and exit
   -v, --verbose                Verbose output (default: no)
   -z, --compress <program>     Use an alternate compressor on the image
  -> Running build hook: [encryptssh]
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
==> Image generation successful

Content in /etc/dropbear is then

$ ls -l /etc/dropbear
total 16
-rw------- 1 root root  458 Apr  1 13:24 dropbear_dss_host_key
-rw------- 1 root root  140 Apr  1 13:24 dropbear_ecdsa_host_key
-rw------- 1 root root  806 Apr  1 13:24 dropbear_rsa_host_key
-rw------- 1 root root 1572 Apr  1 12:25 root_key

6. Usage. Use ssh root@YourComputer to connect to your previously configured dropbear server and type in the password for the encrypted disk. The connection will then close, and dropbear disappears. By the way, dropbear does not look at your configuration for OpenSSH, so if you block root access via OpenSSH, this is of no concern for dropbear.

7. Limitations. Above set-up just works for unlocking the root-device. If there are other encrypted devices, for example devices given in /etc/crypttab, these cannot be unlocked by above procedure.

8. Further reading. See LUKS encrypted devices remote über Dropbear SSH öffnen (in German), Remote unlocking LUKS encrypted LVM using Dropbear SSH in Ubuntu Server 14.04.1 (with Static IP).

Set-Up “Let’s Encrypt” for Hiawatha Web-Server

Google announced that starting with Chrome version 68 they will gradually mark HTTP-connections as “not secure”. “Let’s Encrypt” is a free service for web-masters to obtain certificates in an easy manner. Work on “Let’s Encrypt” started in 2014.

Setting up “Let’s Encrypt” with Hiawatha web-server is quite easy, although there are some pitfalls. I used the ArchLinux package for Hiawatha. There is also a ArchWiki page for Hiawatha.

Another detailed description is: Let’s Encrypt with Hiawatha by Chris Wadge.

1. Unpacking and production-server setting. After installing the ArchLinux package I unpacked the file /usr/share/hiawatha/letsencrypt.tar.gz. You have to edit letsencrypt.conf at three places:

ACCOUNT_EMAIL_ADDRESS = your@mail.address
LE_CA_HOSTNAME =           # Production

I struggled with the last variable LE_CA_HOSTNAME. This has to be the productive “Let’s Encrypt” server. Although you might register with the testing-server, you apparently cannot do anything else with the testing-server. So delete the testing-server. The rest of the configuration file is obvious to change.

2. Configuration file. Now check your hiawatha.conf file:

Binding {
        Port = 443
        #TLScertFile = tls/hiawatha.pem
        TLScertFile = /etc/hiawatha/tls/
        Interface =
        MaxRequestSize = 2048
        TimeForRequest = 30
VirtualHost {
        Hostname =,,,,,,,, borussia

Continue reading