Moved Blog To eklausmeier.goip.de

This post has moved to eklausmeier.goip.de/blog/2021/05-18-moved-blog-to-eklausmeier-goip-de.

This blog here is no longer maintained. I moved to https://eklausmeier.goip.de. During migration I corrected a couple of minor typos and dead links.

Main reasons:

  1. This new editor put the last nail in the coffin, existing content is garbled once you edit it
  2. Math typesetting is quite arduous, would like to have full MathJax support
  3. Overall slow
  4. Importing does not allow to overwrite posts

Nevertheless, I started blogging here since January 2008, really used it starting in 2012. Collected more than 120 thousand views and almost 100 thousand visitors.

PHP extension seg-faulting

This post has moved to eklausmeier.goip.de/blog/2021/03-29-php-extension-seg-faulting.

Task at hand: Call Cobol (=GnuCobol) from PHP. I used FFI for this:

<?php
        $cbl = FFI::cdef("int phpsqrt(void); void cob_init_nomain(int,char**); int cob_tidy(void);", "/srv/http/phpsqrt.so");
        $ffj0 = FFI::cdef("double j0(double);", "libm.so.6");

        $cbl->cob_init_nomain(0,null);
        $ret = $cbl->phpsqrt();
        printf("\tret = %d<br>\n",$ret);
        echo "Before calling cob_tidy():<br>\n";
        echo "\tReturn: ", $cbl->cob_tidy(), "<br>\n";
        printf("j0(2) = %f<br>\n", $ffj0->j0(2));
?>

The Cobol program is:

000010 IDENTIFICATION DIVISION.
000020 PROGRAM-ID.   phpsqrt.
000030 AUTHOR.       Elmar Klausmeier.
000040 DATE-WRITTEN. 01-Jul-2004.
000050
000060 DATA DIVISION.
000070 WORKING-STORAGE SECTION.
000080 01 i       PIC 9(5).
000090 01 s       usage comp-2.
000100
000110 PROCEDURE DIVISION.
000120*    DISPLAY "Hello World!".
000130     PERFORM VARYING i FROM 1 BY 1 UNTIL i > 10
000140         move function sqrt(i) to s
000150*        DISPLAY i, " ", s
000160     END-PERFORM.
000170
000180     move 17 to return-code.
000190     GOBACK.
000200

Config in php.ini file has to be changed:

extension=ffi
ffi.enable=true

To call GnuCobol from C, you have to first call cob_init() or cob_init_nomain(), which initializes GnuCobol. I tried both initialization routines, and both resulted in PHP crashing after running above program, i.e., segmentation fault.

I created a bug for this: FFI crashes with segmentation fault when calling cob_init().

1. I compiled PHP 8.0.3 from source. For this I had to add below packages:

pacman -S tidy freetds c-client

I grep’ed my current configuration:

php -i | grep "Configure Comman"
Configure Command =>  './configure'  '--srcdir=../php-8.0.3' '--config-cache' '--prefix=/usr' '--sbindir=/usr/bin' '--sysconfdir=/etc/php' '--localstatedir=/var' '--with-layout=GNU' '--with-config-file-path=/etc/php' '--with-config-file-scan-dir=/etc/php/conf.d' '--disable-rpath' '--mandir=/usr/share/man' '--enable-cgi' '--enable-fpm' '--with-fpm-systemd' '--with-fpm-acl' '--with-fpm-user=http' '--with-fpm-group=http' '--enable-embed=shared' '--enable-bcmath=shared' '--enable-calendar=shared' '--enable-dba=shared' '--enable-exif=shared' '--enable-ftp=shared' '--enable-gd=shared' '--enable-intl=shared' '--enable-mbstring' '--enable-pcntl' '--enable-shmop=shared' '--enable-soap=shared' '--enable-sockets=shared' '--enable-sysvmsg=shared' '--enable-sysvsem=shared' '--enable-sysvshm=shared' '--with-bz2=shared' '--with-curl=shared' '--with-db4=/usr' '--with-enchant=shared' '--with-external-gd' '--with-external-pcre' '--with-ffi=shared' '--with-gdbm' '--with-gettext=shared' '--with-gmp=shared' '--with-iconv=shared' '--with-imap-ssl' '--with-imap=shared' '--with-kerberos' '--with-ldap=shared' '--with-ldap-sasl' '--with-mhash' '--with-mysql-sock=/run/mysqld/mysqld.sock' '--with-mysqli=shared,mysqlnd' '--with-openssl' '--with-password-argon2' '--with-pdo-dblib=shared,/usr' '--with-pdo-mysql=shared,mysqlnd' '--with-pdo-odbc=shared,unixODBC,/usr' '--with-pdo-pgsql=shared' '--with-pdo-sqlite=shared' '--with-pgsql=shared' '--with-pspell=shared' '--with-readline' '--with-snmp=shared' '--with-sodium=shared' '--with-sqlite3=shared' '--with-tidy=shared' '--with-unixODBC=shared' '--with-xsl=shared' '--with-zip=shared' '--with-zlib'

To this I added --enable-debug. Command configure needs two minutes. Then make -j8 needs another two minutes.

I copied php.ini to local directory, changed it to activated FFI. Whenever I called

$BUILD/sapi/cli/php

I had to add -c php.ini, when I called an extension written by me, stored in ext/.

2. The fix for segmentation fault is actually pretty easy: Just set environment variable ZEND_DONT_UNLOAD_MODULES:

ZEND_DONT_UNLOAD_MODULES=1 $BUILD/sapi/cli/php -c php.ini -r 'test1();'

Reason for this: see valgrind output below.

3. Before I had figured out the “trick” with ZEND_DONT_UNLOAD_MODULES, I wrote a PHP extension. The extension is:

/* {{{ void test1() */
PHP_FUNCTION(test1)
{
        ZEND_PARSE_PARAMETERS_NONE();

        php_printf("test1(): The extension %s is loaded and working!\r\n", "callcob");
        cob_init(0,NULL);
}
/* }}} */

Unfortunately, running this extension resulted in:

Module compiled with build ID=API20200930,NTS
PHP    compiled with build ID=API20200930,NTS,debug
These options need to match

I solved this by adding below string:

/* {{{ callcob_module_entry */
zend_module_entry callcob_module_entry = {
        STANDARD_MODULE_HEADER,
        //sizeof(zend_module_entry), ZEND_MODULE_API_NO, 1, USING_ZTS,
        "callcob",                                      /* Extension name */
        ext_functions,                                  /* zend_function_entry */
        NULL,                                                   /* PHP_MINIT - Module initialization */
        NULL,                                                   /* PHP_MSHUTDOWN - Module shutdown */
        PHP_RINIT(callcob),                     /* PHP_RINIT - Request initialization */
        NULL,                                                   /* PHP_RSHUTDOWN - Request shutdown */
        PHP_MINFO(callcob),                     /* PHP_MINFO - Module info */
        PHP_CALLCOB_VERSION,            /* Version */
        STANDARD_MODULE_PROPERTIES
        ",debug"
};
/* }}} */

I guess this is not the recommend approach.

4. Valgrind shows the following:

==37350== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==37350==  Access not within mapped region at address 0x852AD20
==37350==    at 0x852AD20: ???
==37350==    by 0x556EF7F: ??? (in /usr/lib/libc-2.33.so)
==37350==    by 0x5570DCC: getenv (in /usr/lib/libc-2.33.so)
==37350==    by 0x76BB43: module_destructor (zend_API.c:2629)
==37350==    by 0x75EE31: module_destructor_zval (zend.c:782)
==37350==    by 0x7777A1: _zend_hash_del_el_ex (zend_hash.c:1330)
==37350==    by 0x777880: _zend_hash_del_el (zend_hash.c:1353)
==37350==    by 0x779188: zend_hash_graceful_reverse_destroy (zend_hash.c:1807)
==37350==    by 0x769390: zend_destroy_modules (zend_API.c:1992)
==37350==    by 0x75F582: zend_shutdown (zend.c:1078)
==37350==    by 0x6C3F17: php_module_shutdown (main.c:2359)
==37350==    by 0x84E46D: main (php_cli.c:1351)
==37350==  If you believe this happened as a result of a stack
==37350==  overflow in your program's main thread (unlikely but
==37350==  possible), you can try to increase the size of the
==37350==  main thread stack using the --main-stacksize= flag.
==37350==  The main thread stack size used in this run was 8388608.
. . .
zsh: segmentation fault (core dumped)  valgrind $BUILD/sapi/cli/php -c $BUILD/php.ini -r 'test1();'

As shown above, the relevant code in question is Zend/zend_API.c in line 2629. This is shown below:

void module_destructor(zend_module_entry *module) /* {{{ */
{
. . .
        module->module_started=0;
        if (module->type == MODULE_TEMPORARY && module->functions) {
                zend_unregister_functions(module->functions, -1, NULL);
        }

#if HAVE_LIBDL
        if (module->handle && !getenv("ZEND_DONT_UNLOAD_MODULES")) {
                DL_UNLOAD(module->handle);
        }
#endif
}
/* }}} */

It is the DL_UNLOAD, which is a #define for dlclose, which actually provokes the crash.

According PHP Internals Book — Zend Extensions:

Here, we are loaded as a PHP extension. Look at the hooks. When hitting MSHUTDOWN(), the engine runs our MSHUTDOWN(), but it unloads us just after that ! It calls for dlclose() on our extension, look at the source code, the solution is as often located in there.

So what happens is easy, just after triggering our RSHUTDOWN(), the engine unloads our pib.so ; when it comes to call our Zend extension part shutdown(), we are not part of the process address space anymore, thus we badly crash the entire PHP process.

What is still not understood: Why does FFI not crash with those simple functions, like printf(), or sqrt()?

Add Disjoint IP Addresses To SSHGuard Blacklist

This post has moved to eklausmeier.goip.de/blog/2021/03-15-add-disjoint-ip-addresses-to-sshguard-blacklist.

Problem at hand: There are multiple machines running SSHGuard. Each of these machines accumulates different sets of blacklists. Task: Add disjoint IP addresses from one machine to another machine’s blacklist.

1. Copy from “master” machine:

scp -p master:/var/db/sshguard/blacklist.db blacklist_master.db

This blacklist looks like this:

1615278352|100|4|59.46.169.194
1615278438|100|4|45.144.67.47
1615279294|100|4|122.155.47.9
1615279795|100|4|106.12.173.237
1615284110|100|4|103.152.79.161
1615284823|100|4|79.255.172.22
1615286299|100|4|106.12.171.76

The first entry is time in time_t format, second entry is service, in our case always 100=ssh, third entry is either 4 for IPv4, or 6 for IPv6, fourth entry is actual IP address, see Analysis And Usage of SSHGuard.

2. Create difference set: Run script sshgadd:

sshgadd /var/db/sshguard/blacklist.db blacklist_master.db

Script sshgadd is:

[ -z "$1" ] && exit 11
[ -z "$2" ] && exit 12
[ -f "$1" ] || exit 13
[ -f "$2" ] || exit 14

comm -23 <(cut -d\| -f4 $1 | sort) <(cut -d\| -f4 $2 | sort)        \
        | perl -ane 'print "1613412470|100|4|$_"'

The comm command can suppress common columns:

       -1     suppress column 1 (lines unique to FILE1)
       -2     suppress column 2 (lines unique to FILE2)
       -3     suppress column 3 (lines that appear in both files)

This “<(list)” construct is called process substitution.

3. Stop SSHGuard on machine and add output of sshgadd to blacklist via any editor of your choice, or use cat and mv.

Lesser Known Static Site Generators

This post has moved to eklausmeier.goip.de/blog/2021/03-07-lesser-known-static-site-generators.

Well known static site generators are Hugo (written in Go), Pelican (written in Python), Grav (written in PHP), or Eleventy (written in JavaScript). For a list of static site generators see Jamstack (322 generators listed) or Static Site Generators (460 generators listed).

The following three static site generators unfortunately are not listed in before given Jamstack overview.

  1. mkws: a minimalistic static site generator, written in 31 lines of sh and 400 lines of C
  2. Saaze: an easy to use generator which is able to generate static or dynamic sites, written in PHP/Symfony
  3. Franklin: easy embedding of math and Julia notebooks, written in Julia

To better understand what makes above list stand out against the more popular ones: Hugo’s daily use is a little bit cumbersome and every release of Hugo gets more complex, additions to the Hugo source code are unwelcome.

1. mkws was covered in mkws – Static Site Generation With The Shell. The main loop in the 31 lines of shell code are:

for t in "$srcdir"/*.upphtml
do
        echo "Making $(basename "${t%.upphtml}".html)"
        pp "$sharedir"/l.upphtml "$t" "$1" > \
                "$(basename "${t%.upphtml}".html)"
done

I.e., read files with suffix upphtml and run them through the simple pre-processor program pp. pp is like php, but way simpler, although not necessarily faster. pp pumps everything between #! lines through a shell.

Posts in mkws have to be written in HTML, unless you add another helper program to the equation. mkws is to be considered a proof-of-concept.

2. Saaze has been written by Gilbert Pellegrom, who also wrote PicoCMS. Directory layout of Saaze is as below:

saaze/
├── build/
├── cache/
├── content/
│   ├── pages/
│   |   └── example-page.md
│   └── pages.yml
├── public/
│   └── index.php
└── templates/
    ├── collection.blade.php
    ├── entry.blade.php
    ├── error.blade.php
    └── layout.blade.php

The “build” directory will contain all HTML files, if one generates entirely static pages with the command

php saaze build

The “public” directory is used, if instead, you want your pages fully dynamic, i.e., with possible PHP code embedded.

The “templates” directory is used for templates, which are based on Laravel Blade.

Posts in Saaze are written in Markdown, and each post is prepended with a short Yaml frontmatter header, e.g.,

--- 
title: An Example Post
date: "2020-10-13"
--- 
This is an **example** with some _markdown_ formatting.

Posts in Saaze are called “entries”. These called “entries” are collected together in what Saaze calls “collection”.

Saaze can be extended to match ones own Markdown tags. This is described in Extending Saaze.

3. Franklin is used for the entire Julia documentation and shines at mathematics. It was written by Thibaut Lienart. For example, the Julia web-site is generated by Franklin.

The directory stucture of Franklin is:

.
├── _assets/
├── _layout/
├── _libs/
├── __site
│     ├── index.html
│     ├── folder
│     │   └── subpage
│     │       └── index.html
│     └── page
│     └── index.html
├── config.md
├── index.md
├── folder
│   └── subpage.md
└── page.md

The “__site” directory and its corresponding sub-folders are created by running

newsite("TestWebsite"; template="vela")

from within the Julia REPL. The “__site” directory is then to become the web-root of your web-presence.

From the three generators, Franklin is obviously the most powerful one. It has good integration to site-search using lunr.js, with math, with source code, even live Julia code, or plots.

Each post in Franklin is a plain Markdown file.

Testing J-Pilot feature-gtk3 branch

This post has moved to eklausmeier.goip.de/blog/2021/03-02-testing-j-pilot-feature-gtk3-branch.

J-Pilot still relies on GTK+ 2, which is heading towards planned deprecation. The entire work to migrate to GTK+ 3 is done by volunteers, Judd Montgomery and David Malia. This post is about testing this new experimental branch feature-gtk3 from GitHub.

Installing this branch is straightforward:

  1. Download zip-file and unpack
  2. Run autogen.sh, then run make
  3. Create virgin directory $HOME/tmp/gtk3/ and copy empty/*.pdb to $HOME/tmp/gtk3/.jpilot, otherwise no records will be shown in GUI

Below is a list of warnings, errors and crashes for version d558d56 from 25-Feb-2021.

W1. Starting

JPILOT_HOME=$HOME/tmp/gtk3/ ./jpilot

shows

(jpilot:42580): Gtk-CRITICAL **: 14:22:57.546: gtk_list_store_get_path: assertion 'iter->stamp == priv->stamp' failed

(jpilot:42580): Gtk-CRITICAL **: 14:22:57.546: gtk_tree_selection_select_path: assertion 'path != NULL' failed
(jpilot:42580): Gtk-CRITICAL **: 14:22:57.546: gtk_list_store_get_path: assertion 'iter->stamp == priv->stamp' failed

(jpilot:42580): Gtk-CRITICAL **: 14:22:57.546: gtk_tree_selection_select_path: assertion 'path != NULL' failed

E1. Importing from a hidden directory no longer works, as hidden directory is not shown. In “old” J-Pilot you could specify the hidden directory in edit-field and then press TAB. (No longer the case with ba8354f.)

E2. Searching crashes. (No longer the case with ba8354f.)

segmentation fault (core dumped)  JPILOT_HOME=$HOME/tmp/gtk3/ ./jpilot

Running J-Pilot from within gdb and then searching results in:

Thread 1 "jpilot" received signal SIGSEGV, Segmentation fault.
0x00007ffff71facd6 in g_type_check_instance_cast () from /usr/lib/libgobject-2.0.so.0

Executable has debug symbols:

$ file jpilot
jpilot: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=745cb795eccaf42d04e2920ca592534a72dbf1f4, for GNU/Linux 4.4.0, with debug_info, not stripped

E3. Address book entries look “awful”, i.e., entries are too close together. (No longer the case with ba8354f.)

Address records:

Date records:

Also, entering entirely new address record doesn’t show separation lines betweeen fields:

U1. To quickly get “mass”-data for a possible buggy J-Pilot software, I prepared a Perl script palmgencsv to generate address- and datebook-CSV files. For example:

$ palmgencsv -dn12000 > d12k.csv

generates 12-thousand records for datebook. Likewise:

$ palmgencsv -an15000 > a15k.csv

generates 15-thousand address records.

C1. One admirable feature, which does work with GTK+ 3, is using the “Broadway backend“:

broadwayd &
export JPILOT_HOME=$HOME/tmp/gtk3/
GDK_BACKEND=broadway ./jpilot

Alternatively, one could use broadwayd --address 0.0.0.0. Fire up a web-browser and visit http://localhost:8080/. This will show J-Pilot running within the web-browser:

How cool is that?

Added 21-Mar-2021: I retested with ba8354f = “Merge pull request #27 from dmalia1/fix-unfocused-color-theme”.

1. Importing from hidden directories: Works.
Tested importing 23-thousand address records, 21-thousand date records from hidden directory.

2. Search does no longer crash immediately (sounds worse than it really is)
As noted in the blog-post I created a Perl-script to generate masses of data. I generated 12-thousand date records (palmgencsv -dn12000). In this case, I created date records Event 1, Event 2, …, Event 12000. When I search for “Event” (no number), search-window will show me all 12-thousand entries correctly. But when I narrow the search to Event 12000 in the same search-window, then J-Pilot hangs. I tested the same with J-Pilot 1.8.2, which does not exhibit this behaviour. I agree that this is extreme. When I repeat with fewer search matches, e.g., just hundred or just thousand search results and then narrowing, then this hang does not occur. The hang shows no CPU time — it just hangs, and I lost patience and closed search-window, which also closed the main J-Pilot window.

3. Entries in address book look better now.

Starting J-Pilot shows:
(jpilot:49290): Gtk-CRITICAL **: 21:05:20.459: gtk_list_store_get_path: assertion ‘iter->stamp == priv->stamp’ failed
(jpilot:49290): Gtk-CRITICAL **: 21:05:20.459: gtk_tree_selection_select_path: assertion ‘path != NULL’ failed
But otherwise this does not seem to indicate any trouble.

Analysis And Usage of SSHGuard

This post has moved to eklausmeier.goip.de/blog/2021/02-28-analysis-and-usage-of-sshguard.

To ban annoying ssh access to your Linux box you can use fail2ban. Or, alternatively, you can use SSHGuard. SSHGuard’s installed size is 1.3 MB on Arch Linux. Its source code, including all C-files, headers, manuals, configuration, and makefiles is 8 KLines. In contrast, for fail2ban just the Python source code of version 0.11.2 is 31 KLines, not counting configuration files, manuals, and text files; its installed size is 3.3 MB. fail2ban is also way slower than SSHGuard. For example, one one machine fail2ban used 7 minutes of CPU time, where SSHGuard used 11 seconds. I have written on fail2ban in “Blocking Network Attackers“, “Chinese Hackers“, and “Blocking IP addresses with ipset“.

SSHGuard is a package in Arch Linux, and there is a Wiki page on it.

1. Internally SSHGuard maintains three lists:

  1. whitelist: allowed IP addresses, given by configuration
  2. blocklist: list of IP addresses which are blocked, but which can become unblocked after some time, in-memory only
  3. blacklist: permanently blocked IP addresses, stored in cleartext in file

SSHGuard’s main function is summarized in below excerpt from its shell-script /bin/sshguard.

eval $tailcmd | $libexec/sshg-parser | \
    $libexec/sshg-blocker $flags | $BACKEND &
wait

There are four programs, where each reads from stdin and writes to stdout, and does a well defined job. Each program stays in an infinite loop.

  1. $tailcmd reads the log, for example via tail -f, which might contain the offending IP address
  2. sshg-parser parses stdin for offending IP’s
  3. sshg-blocker writes IP addresses
  4. $BACKEND is a firewall shell script which either uses iptables, ipset, nft, etc.

sshg-blocker in addition to writing to stdout, also writes to a file, usually /var/db/sshguard/blacklist.db. This is the blacklist file. The content looks like this:

1613412470|100|4|39.102.76.239
1613412663|100|4|62.210.137.165
1613415749|100|4|39.109.122.173
1613416009|100|4|80.102.214.209
1613416139|100|4|106.75.6.234
1613418135|100|4|42.192.140.183

The first entry is time in time_t format, second entry is service, in our case always 100=ssh, third entry is either 4 for IPv4, or 6 for IPv6.

SSHGuard handles below services:

enum service {
    SERVICES_ALL            = 0,    //< anything
    SERVICES_SSH            = 100,  //< ssh
    SERVICES_SSHGUARD       = 110,  //< SSHGuard
    SERVICES_UWIMAP         = 200,  //< UWimap for imap and pop daemon
    SERVICES_DOVECOT        = 210,  //< dovecot
    SERVICES_CYRUSIMAP      = 220,  //< cyrus-imap
    SERVICES_CUCIPOP        = 230,  //< cucipop
    SERVICES_EXIM           = 240,  //< exim
    SERVICES_SENDMAIL       = 250,  //< sendmail
    SERVICES_POSTFIX        = 260,  //< postfix
    SERVICES_OPENSMTPD      = 270,  //< OpenSMTPD
    SERVICES_COURIER        = 280,  //< Courier IMAP/POP
    SERVICES_FREEBSDFTPD    = 300,  //< ftpd shipped with FreeBSD
    SERVICES_PROFTPD        = 310,  //< ProFTPd
    SERVICES_PUREFTPD       = 320,  //< Pure-FTPd
    SERVICES_VSFTPD         = 330,  //< vsftpd
    SERVICES_COCKPIT        = 340,  //< cockpit management dashboard
    SERVICES_CLF_UNAUTH     = 350,  //< HTTP 401 in common log format
    SERVICES_CLF_PROBES     = 360,  //< probes for common web services
    SERVICES_CLF_LOGIN_URL  = 370,  //< CMS framework logins in common log format
    SERVICES_OPENVPN        = 400,  //< OpenVPN
    SERVICES_GITEA          = 500,  //< Gitea
};

2. A typical configuration file might look like this:

LOGREADER="LANG=C /usr/bin/journalctl -afb -p info -n1 -t sshd -o cat"
THRESHOLD=10
BLACKLIST_FILE=10:/var/db/sshguard/blacklist.db
BACKEND=/usr/lib/sshguard/sshg-fw-ipset
PID_FILE=/var/run/sshguard.pid
WHITELIST_ARG=192.168.178.0/24

Furthermore one has to add below lines to /etc/ipset.conf:

create -exist sshguard4 hash:net family inet
create -exist sshguard6 hash:net family inet6

Also, /etc/iptables/iptables.rules and /etc/iptables/ip6tables.rules need the following link to ipset respectively:

-A INPUT -m set --match-set sshguard4 src -j DROP
-A INPUT -m set --match-set sshguard6 src -j DROP

3. Firewall script sshg-fw-ipset, called “BACKEND”, is essentially:

fw_init() {
    ipset -quiet create -exist sshguard4 hash:net family inet
    ipset -quiet create -exist sshguard6 hash:net family inet6
}

fw_block() {
    ipset -quiet add -exist sshguard$2 $1/$3
}

fw_release() {
    ipset -quiet del -exist sshguard$2 $1/$3
}

...

while read -r cmd address addrtype cidr; do
    case $cmd in
        block)
            fw_block "$address" "$addrtype" "$cidr";;
        release)
            fw_release "$address" "$addrtype" "$cidr";;
        flush)
            fw_flush;;
        flushonexit)
            flushonexit=YES;;
        *)
            die 65 "Invalid command";;
    esac
done

The “BACKEND” is called from sshg-blocker as follows:

static void fw_block(const attack_t *attack) {
    unsigned int subnet_size = fw_block_subnet_size(attack->address.kind);

    printf("block %s %d %u\n", attack->address.value, attack->address.kind, subnet_size);
    fflush(stdout);
}

static void fw_release(const attack_t *attack) {
    unsigned int subnet_size = fw_block_subnet_size(attack->address.kind);

    printf("release %s %d %u\n", attack->address.value, attack->address.kind, subnet_size);
    fflush(stdout);
}

SSHGuard is using the list-implementation SimCList from Michele Mazzucchi.

4. sshg-parser uses flex (=lex) and bison (=yacc) for evaluating log-messages. An introduction to flex and bison is here. Tokenization for ssh using flex is:

"Disconnecting "[Ii]"nvalid user "[^ ]+" "           { return SSH_INVALUSERPREF; }
"Failed password for "?[Ii]"nvalid user ".+" from "  { return SSH_INVALUSERPREF; } 

Actions based on tokens using bison is:

%token SSH_INVALUSERPREF SSH_NOTALLOWEDPREF SSH_NOTALLOWEDSUFF

msg_single:
    sshmsg            { attack->service = SERVICES_SSH; }
  | sshguardmsg       { attack->service = SERVICES_SSHGUARD; }
  . . .
  ;

/* attack rules for SSHd */
sshmsg:
    /* login attempt from non-existent user, or from existent but non-allowed user */
    ssh_illegaluser
    /* incorrect login attempt from valid and allowed user */
  | ssh_authfail
  | ssh_noidentifstring
  | ssh_badprotocol
  | ssh_badkex
  ;

ssh_illegaluser:
    /* nonexistent user */
    SSH_INVALUSERPREF addr
  | SSH_INVALUSERPREF addr SSH_ADDR_SUFF
    /* existent, unallowed user */
  | SSH_NOTALLOWEDPREF addr SSH_NOTALLOWEDSUFF
  ;

Once an attack is noticed, it is just printed to stdout:

static void print_attack(const attack_t *attack) {
    printf("%d %s %d %d\n", attack->service, attack->address.value,
           attack->address.kind, attack->dangerousness);
}

5. For exporting fail2ban’s blocked IP addresses to SSHGuard one would use below SQL:

select ip from (select ip from bans union select ip from bips)

to extract from /var/lib/fail2ban/fail2ban.sqlite3.

6. In case one wants to unblock an IP address, which got blocked inadvertently, you can simply issue

ipset del sshguard4 <IP-address>

in case you are using ipset as “BACKEND”. If this IP address is also present in the blacklist, you have to delete it there as well. For that, you must stop SSHGuard.

ssh as SOCKS server

This post has moved to eklausmeier.goip.de/blog/2021/02-21-ssh-as-socks-server.

Assume three computers A, B, and C. A can connect to B via ssh, but A cannot connect to C, but B can connect to C.

A -> B -> C

On A open ssh as SOCKS-server with

ssh -N -D 9020 user@B

Now on A one can use

brave --proxy-server="socks5://localhost:9020"

The browser will then show up as if directly surfing on B thereby circumventing the limitations on A.

Instead of the brave browser, one can use Chromium, or Firefox. Option “-N”: Do not execute a remote command.

See How to Set up SSH SOCKS Tunnel for Private Browsing, or SOCKS.

Poisson Log-Normal Distributed Random Numbers

This post has moved to eklausmeier.goip.de/blog/2021/02-09-poisson-log-normal-distributed-random-numbers.

Task at hand: Generate random numbers which follow a lognormal distribution, but this drawing is governed by a Poisson distribution. I.e., the Poisson distribution governs how many lognormal random values are drawn. Input to the program are \lambda of the Poisson distribution, modal value and either 95% or 99% percentile of the lognormal distribution.

From Wikipedia’s entry on Log-normal distribution we find the formula for the quantile q for the p-percentage of the percentile (0<p<1), given mean \mu and standard deviation \sigma:

q = \exp\left( \mu + \sqrt{2}\,\sigma\, \hbox{erf}^{-1}(2p-1)\right)

and the modal value m as

m = \exp\left( \mu - \sigma^2 \right).

So if q and m are given, we can compute \mu and \sigma:

\mu = \log m + \sigma^2,

and \sigma is the solution of the quadratic equation:

\log q = \log m + \sigma^2 + \sqrt{2}\,\sigma\, \hbox{erf}^{-1}(2p-1),

hence

\sigma_{1/2} = -{\sqrt{2}\over2}\, \hbox{erf}^{-1}(2p-1) \pm\sqrt{ {1\over2}\left(\hbox{erf}^{-1}(2p-1)\right)^2 - \log(m/q) },

or more simple

\sigma_{1/2} = -R/2 \pm \sqrt{R^2/4 - \log(m/q) },

with

R = \sqrt{2}\,\hbox{erf}^{-1}(2p-1).

For quantiles 95% and 99% one gets R as 1.64485362695147 and 2.32634787404084 respectively. For computing the inverse error function I used erfinv.c from lakshayg.

Actual generation of random numbers according Poisson- and lognormal-distribution is done using GSL. My program is here: gslSoris.c.

Poisson distribution looks like this (from GSL documentation):
Poisson distribution

Lognormal distribution looks like this (from GSL):
Lognormal distribution

Performance comparison Ryzen vs. Intel vs. Bulldozer vs. ARM

This post has moved to eklausmeier.goip.de/blog/2021/02-01-performance-comparison-ryzen-vs-intel-vs-bulldozer-vs-arm.

For comparing different machines I invert the Hilbert matrix

H = \left(\begin{array}{ccccc}  1 & {1\over2} & {1\over3} & \cdots & {1\over n} \\                                 {1\over2} & {1\over3} & {1\over4} & \cdots & {1\over n+1} \\                                 {1\over3} & {1\over4} & {1\over5} & \cdots & {1\over n+2} \\                                 \vdots    & \vdots    & \vdots    & \ddots & \vdots \\                                 {1\over n} & {1\over n+1} & {1\over n+2} & \cdots & {1\over2n-1}         \end{array} \right)         = \left( {\displaystyle {1\over i+j-1} } \right)_{ij}

This matrix is known have very high condition numbers. Program xlu5.c stores four double precision matrices of dimension n. Matrix H and A store the Hilbert matrix, X is the identity matrix, Y is the inverse of H. Finally the maximum norm of I-H\cdot H^{-1} is printed, which should be zero. These four double precision matrices occupy roughly 1.6 MB for n=230.

1. Runtime on Ryzen, AMD Ryzen 5 PRO 3400G with Radeon Vega Graphics, max 3.7 GHz, as given by lscpu.

$ time xlu5o3b 230 > /dev/null
        real 0.79s
        user 0.79s
        sys 0
        swapped 0
        total space 0

Cache sizes within CPU are:

L1d cache:                       128 KiB
L1i cache:                       256 KiB
L2 cache:                        2 MiB
L3 cache:                        4 MiB

Required storage for above program is 4 matrices, each having 230×230 entries with double (8 bytes), giving 1692800 bytes, roughly 1.6 MB.

2. Runtime on AMD FX-8120, Bulldozer, max 3.1 GHz, as given by lscpu.

$ time xlu5o3b 230 >/dev/null 
        real 1.75s
        user 1.74s
        sys 0
        swapped 0
        total space 0

Cache sizes within CPU are:

L1d cache:                       64 KiB
L1i cache:                       256 KiB
L2 cache:                        8 MiB
L3 cache:                        8 MiB

3. Runtime on Intel, Intel(R) Core(TM) i5-4250U CPU @ 1.30GHz, max 2.6 GHz, as given by lscpu.

$ time xlu5o3b 230 > /dev/null
        real 1.68s
        user 1.67s
        sys 0
        swapped 0
        total space 0

Cache sizes within CPU are:

L1d cache:                       64 KiB
L1i cache:                       64 KiB
L2 cache:                        512 KiB
L3 cache:                        3 MiB

Apparently the Ryzen processor can outperform the Intel processor on cache, higher clock frequency. But even for smaller matrix sizes, e.g., 120, the Ryzen is two times faster.

Interestingly, the error in computations are different!

AMD and Intel machines run ArchLinux with kernel version 5.9.13, gcc was 10.2.0.

4. Runtime on Raspberry Pi 4, ARM Cortex-A72, max 1.5 GHz, as given by lscpu.

$ time xlu5 230 > /dev/null
        real 4.37s
        user 4.36s
        sys 0
        swapped 0
        total space 0

Linux 5.4.83 and GCC 10.2.0.

5. Runtime on Odroid XU4, Cortex-A7, max 2 GHz, as given by lscpu.

$ time xlu5 230 > /dev/null
        real 17.75s
        user 17.60s
        sys 0
        swapped 0
        total space 0

So the Raspberry Pi 4 is clearly way faster than the Odroid XU4.

dumpe2fs: When was my hard-drive first formatted?

This post has moved to eklausmeier.goip.de/blog/2021/01-23-dumpe2fs-when-was-my-hard-drive-first-formatted.

I repeatedly forget to remember when my hard-drive or SSD was first formatted.

Command for this is dumpe2fs. This command is part of package e2fsprogs. Example:

# dumpe2fs -h /dev/sda1
dumpe2fs 1.45.6 (20-Mar-2020)
Filesystem volume name:   <none>
Last mounted on:          /boot
Filesystem UUID:          83a1bedb-6fd3-46d0-8900-e4e09536168e
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode dir_index filetype sparse_super
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              62248
Block count:              248832
Reserved block count:     12441
Free blocks:              126393
Free inodes:              61933
First block:              1
Block size:               1024
Fragment size:            1024
Reserved GDT blocks:      256
Blocks per group:         8192
Fragments per group:      8192
Inodes per group:         2008
Inode blocks per group:   251
RAID stride:              4
RAID stripe width:        4
Filesystem created:       Mon Apr 21 13:45:32 2014
Last mount time:          Sun May 31 14:18:09 2020
Last write time:          Mon Jun  1 00:40:25 2020
Mount count:              35
Maximum mount count:      -1

You must be root to use this command. It does not work for encrypted disks (LUKS) or volume groups.

Unitymedia und SIP VoIP

This post has moved to eklausmeier.goip.de/blog/2021/01-16-unitymedia-und-sip-voip.

Folgendes schrieb ich an die Firma Unitymedia, jetzt Vodafone:

ich möchte gerne meine Rufnummer XXX über meine YYY Telefonanlage via VoIP betreiben. Ich verwende den von Ihnen bereitgestellten Router “Vodafone Station”. Meine Kundennummer lautet ZZZ. Ich benötige nun folgende sieben Informationen:

1. Domain
2. Registrar
3. STUN-Server
4. Outbound-Proxy
5. SIP-UDP Port
6. Benutzername
7. Passwort

Antwort von Unitymedia nach einmaliger Erinnerung:

Unitymedia Telefon arbeitet mit der Technologie “PacketCable”. Im Gegensatz zu konkurrierenden, auf DSL-basierenden Internettelefonie-Angeboten, werden die Sprachdaten bzw. -pakete dabei nicht über das Internet transportiert. Die IP-basierte Übermittlung erfolgt ausschließlich im gesicherten Unitymedia-Netz, welches vom eigenen Kontrollzentrum (Unitymedia Network Operation Center in Kerpen) permanent überwacht und gesteuert wird. So können wir eine hohe Dienstequalität (“Quality of Service”) und Verfügbarkeit garantieren, die bei internetbasierten Voice-over-IP-Lösungen nicht zugesagt werden können.

Die SIP Daten werden nur rausgegeben wenn Sie sich einen eigenen Router anschaffen. Um die Daten dann in Ihrer Fritzbox (o.ä) eintragen. Bei unseren Routern werden die SIP Daten von unseren Servern generiert. Deswegen können wir die SIP Daten Ihnen nicht geben.

Vielleicht hilft dies anderen bei ähnlichen Anliegen.

Compiling Java source to binary (native)

This post has moved to eklausmeier.goip.de/blog/2020/12-19-compiling-java-source-to-binary-native.

With GraalVM you can now fully compile a Java file to a native binary. This is also called AOT, ahead-of-time compilation. Compilation is very slow, and resulting binary is huge as it must contain all code which might be referenced. In contrast the class file usually is quite small. Though, it is advantageous that the resulting binary starts way faster.

From the GraalVM web-page:

The Native Image builder or native-image is a utility that processes all classes of an application and their dependencies, including those from the JDK. It statically analyzes these data to determine which classes and methods are reachable during the application execution. Then it ahead-of-time compiles that reachable code and data to a native executable for a specific operating system and architecture. This entire process is called building an image

Assume the simple program:

public class hello {
    public static void main(String argv[]) {
        System.out.println("Hello world.");
    }
}

Compiling this simple program is quite slow:

$ time /usr/lib/jvm/java-11-graalvm/bin/native-image hello
[hello:90709]    classlist:   1,173.69 ms,  0.96 GB
[hello:90709]        (cap):     746.78 ms,  0.96 GB
[hello:90709]        setup:   2,072.73 ms,  0.96 GB
[hello:90709]     (clinit):     214.77 ms,  1.22 GB
[hello:90709]   (typeflow):   5,433.03 ms,  1.22 GB
[hello:90709]    (objects):   4,402.72 ms,  1.22 GB
[hello:90709]   (features):     281.83 ms,  1.22 GB
[hello:90709]     analysis:  10,615.01 ms,  1.22 GB
[hello:90709]     universe:     486.71 ms,  1.71 GB
[hello:90709]      (parse):   1,237.17 ms,  1.71 GB
[hello:90709]     (inline):   1,174.69 ms,  1.71 GB
[hello:90709]    (compile):   7,934.95 ms,  2.35 GB
[hello:90709]      compile:  10,857.38 ms,  2.35 GB
[hello:90709]        image:   1,052.94 ms,  2.35 GB
[hello:90709]        write:     174.08 ms,  2.35 GB
[hello:90709]      [total]:  26,598.83 ms,  2.35 GB
real 27.23s
user 145.40s
sys 0
swapped 0
total space 0

Compilation will start a huge number of threads. Resulting binary is ca. 9 MB.

In ArchLinux the native compiler is contained in native-image-jdk11-bin, which in turn needs jdk11-graalvm-bin.

Running the binary is way faster than starting the class file within JVM. Starting the binary.

$ time ./hello
Hello world.
real 0.00s
user 0.00s
sys 0
swapped 0
total space 0

Starting the class file in JVM.

$ time java hello
Hello world.
real 0.07s
user 0.07s
sys 0
swapped 0
total space 0

For building native images on Windows follow the instructions here: Prerequisites for Using Native Image on Windows.

Contrary to initial conception, GraalVM is not automatically faster during runtime. It is clearly way faster during startup. Michael Larabel conducted a performance test on GraalVM 20.1, OpenJDK 11, OpenJDK 14.0.1, OpenJDK 15, and others. Result: Sometimes GraalVM is faster, sometimes not.

null

SciMark #2