Splitting and anti-merging vCard files

Sometimes vCard files need to be split into smaller files, or the file needs to be protected against merging in another application.

1. Splitting. Below Perl script splits the input file into as many files as required. Output files are named adr1.vcf, adr2.vcf, etc. You can pass a command line argument “-n” to specify the number of card records per file. Splitting a vCard file is provided in palmadrsplit on GitHub:

use Getopt::Std;

my %opts;
getopts('n:',\%opts);
my ($i,$k,$n) = (1,0,950);
$n = ( defined($opts{'n'}) ? $opts{'n'} : 950 );

open(F,">adr.$i.vcf") || die("Cannot open adr.$i.vcf for writing");
while (<>) {
        if (/BEGIN:VCARD/) {
                if (++$k % $n == 0) {   # next address record
                        close(F) || die("Cannot close adr.$i.vcf");
                        ++$i;   # next file number
                        open(F,">adr.$i.vcf") || die("Cannot open adr.$i.vcf for writing");
                }
        }
        print F $_;
}
close(F) || die("Cannot close adr.$i.vcf");

This is required for Google Contacts, as Google does not allow to import more than 1,000 records per day, see Quotas for Google Services.

2. Anti-Merge. Inhibiting annoying merging is given in file palmantimerge on GitHub. Overall logic is as follows: Read entire vCard file and each card, delimited by BEGIN:VCARD and END:VCARD, is put on a hashmap. Each hashmap entry is a list of vCards. Hash key is the N: entry, i.e., the concatentation of lastname and firstname. Once everything is hashed, then walk through hash. Those hash entries, where the list contains just one entry, can be output as is. Where the list contains more than one entry, then these entries would otherwise be merged, and then the N: part is modified by using the ORG: field.

use strict;
my @singleCard = ();    # all info between BEGIN:VCARD and END:VCARD
my ($name) = "";        # N: part, i.e., lastname semicolon firstname
my ($clashes,$line,$org) = (0,"","");
my %allCards = {};      # each entry is list of single cards belonging to same first and lastname, so hash of array of array

while (<>) {
        if (/BEGIN:VCARD/) {
                ($name,@singleCard) = ("", ());
                push @singleCard, $_;
        } elsif (/END:VCARD/) {
                push @singleCard, $_;
                push @{ $allCards{$name} }, [ @singleCard ];
        } else {
                push @singleCard, $_;
                $name = $_ if (/^N:/);
        }
}

for $name (keys %allCards) {
        $clashes = $#{$allCards{$name}};
        for my $sglCrd (@{$allCards{$name}}) {
                if ($clashes == 0) {
                        for $line (@{$sglCrd}) { print $line; }
                } else {
                        $org = "";
                        for $line (@{$sglCrd}) {
                                $org = $1 if ($line =~ /^ORG:([ \-\+\w]+)/);
                        }
                        for $line (@{$sglCrd}) {
                                $line =~ s/;/ \/${org}\/;/ if ($line =~ /^N:/);
                                print $line;
                        }
                }
        }
}

Every lastname is appended with “/organization/” if the combination of firstname and lastname is not unique. For example, two records with Peter Miller in ABC-Corp and XYZ-Corp, will be written as N:Miller /ABC-Corp/;Peter and N:Miller /XYZ-Corp/;Peter.

This way Simple Mobile Tools Contacts will not merge records together which it shouldn’t. Issue #446 for this is on GitHub.

J-Pilot Plugin For SQLite Export

In SQL Datamodel For J-Pilot I described the SQLite datamodel. I wrote a J-Pilot plugin which can export the below entities and write them to an SQLite database file. The direction is one-way: from J-Pilot to SQLite.

  1. Address
  2. Datebook
  3. Memo
  4. To-Do
  5. Expense
  6. Various categories for above entities

Adding more entities is pretty easy. For example, if people need the Calendar Palm database exported, this can be implemented quickly. We use the usual SQLite API with sqlite3_exec(), and sqlite3_prepare(), sqlite3_bind(), sqlite3_step(), and finally sqlite3_finalize().

The general mechanics of a J-Pilot plugin are described by Judd Montgomery, the author of J-Pilot, in this document. I took the Expense/expense.c source code from the Expense plugin as a guide.

The plugin provides the following functionality:

  1. Create new database from scratch, it is called jptables.db
  2. Export above mentioned entities
  3. In debug mode you can use J-Pilot‘s search to search in the SQLite database

If you call jpilot -d then debug-mode is activated.

Installation.

  1. Compile single source code file jpsqlite.c
  2. Copy library (.so file) in plugin directory ($HOME/.jpilot/plugins)
  3. Copy datamodel SQL file jptables.sql in plugin directory

Compilation is with below command:

gcc `pkg-config -cflags-only-I gtk+-2.0` -I <J-Pilot src dir> -s -fPIC -shared jpsqlite.c -o libjpsqlite.so -lsqlite3

For this to work you need the Pilot-Link header files and the J-Pilot (AUR) source code at hand.

Running the plugin: go to the plugins menu by main-menu selection or function key (F7 in my case), then press SQL button. All previous data is completey erased in the database, then all data is written to database within a single transaction.

In debug mode and in debug mode only, the J-Pilot search also searches through all entities in the SQLite database.

The long-term goal is that SQLite is the internal data structure for J-Pilot, thereby abandoning the binary files entirely.

SQL Datamodel For J-Pilot

Currently J-Pilot stores its data in binary form which is compatible with the original Palm data format. See Palm File Format Specification. Reading these binary formats is not simple, see for example pdbrd.c. Portion of the internal structure looks something like this:

typedef struct {        // header of record list
        LocalID nextRecordListID;
        UInt16 numRecords;
        UInt16 firstEntry;
} RecordListType;

typedef struct {        // single element of record list
        LocalID localChunkID;   // offset, start of actual data
        UInt8 attributes;       // 1st nibble: ?/private, 2nd: this is the category
        UInt8 uniqueID[3];
} RecordEntryType;

typedef struct {
        UInt16 renamedCategories;
        char categoryLabels[16][16];
        UInt8 categoryUniqIDs[16];
        UInt16 lastUniqID_pad;  //UInt8 lastUniqID; UInt8 padding;
} AppInfoType;

In 2013 I proposed to use SQLite as internal data format instead, see Possible Enhancements to J-Pilot.

Below datamodel closely follows the field declaration in J-Pilot and Pilot-Link, e.g., /usr/include/pi-datebook.h or utils.h.

Continue reading

J-Pilot Data on Android Phone: Contacts

Keeping J-Pilot data in sync with Android smartphone is quite a challenge. See my post about Google calendar. Copying contact data from J-Pilot to Android in the past went like this:

  1. Export data from J-Pilot in vCard (vcf) format
  2. If required, fiddle with this vcf file with some Perl script
  3. Stopping contact app in Android and stopping WiFi. Deleting all data in Android app.
  4. Manually delete contacts in Google Contacts in batches of ca. 500 records (Added 10-May-2020: Meanwhile there is no longer a limit here, so you can delete all your contacts in one go.)
  5. Import vCard data in Google Contacts
  6. Turn on WiFi on Android, starting contact-app

As one can easily see, this was very cumbersome. To make this even more difficult, now Google has limited the number of imported records in vCard file to 1,000 records per day. If you have more than that, then you are stuck. The limits are documented in quotas. Continue reading

Syncing with J-Pilot via Bluetooth

Probably I am a little bit old-fashioned, but I still use J-Pilot on my PC and my Tungsten T5 and sync them. It mostly serves as a kind of backup so I have my valuable address and datebook data on separate media. So in case my PC gets inaccessible then I have my data on this device at least.

These are the commands I use.

echo 1 > /proc/sys/net/ipv4/ip_forward
/usr/bin/dund --listen --channel 1 --pppd /usr/sbin/pppd call dun
hciconfig hci0 piscan

File /etc/ppp/peers/dun is

debug
115200
noauth
nopersist
local
passive
netmask 255.255.255.0
192.168.178.118:192.168.178.119
ms-dns 192.168.178.1

Continue reading

Possible Enhancements to J-Pilot

Here are some thoughts about possible enhancements for J-Pilot.

  1. Convert pdb’s and pc3’s to SQLite. This makes it easier to analyze data according some criteria, e.g., find how many addresses have the same telephone numbers, how may entries in datebook contain the same substring, etc.
  2. Convert and transform pdb’s and pc3’s to Google GData (shut down or deprecated). The Google id, which is returned after transmitting data, is then possibly stored in pdb/pc3.
  3. Use mmap() instead of all its fread() and malloc()‘s inside pilot-link and J-Pilot.
  4. When J-Pilot searches for strings in the case-insensitive case, then it copies all elements and uses malloc() for each element, see routine jp_strstr(). Instead, just use a home-brewed strstr() which takes care of case.
  5. Provide ncurses interfaces instead of Gtk. See for example calcurse.
  6. When searching for strings and then jumping to the result, it just shows the record, but it does not directly position the cursor to the string where it was found. In particular for memos this is a small nuisance.