Towards web-based delta synchronization for cloud storage systems

Very interesting article.

Some remarkable excerpts:

To isolate performance issues to the JavaScript VM, the authors rebuilt the client side of WebRsync using the Chrome native client support and C++. It’s much faster.

Replacing MD5 with SipHash reduces computation complexity by almost 5x. As a fail-safe mechanism in case of hash collisions, WebRsync+ also uses a lightweight full content hash check. If this check fails then the sync will be re-started using MD5 chunk fingerprinting instead.

The client side of WebR2sync+ is 1700 lines of JavaScript. The server side is based on node.js (about 500 loc) and a set of C processing modules (a further 1000 loc).

the morning paper

Towards web-based delta synchronization for cloud storage systems Xiao et al., FAST’18

If you use Dropbox (or an equivalent service) to synchronise file between your Mac or PC and the cloud, then it uses an efficient delta-sync (rsync) protocol to only upload the parts of a file that have changed. If you use a web interface to synchronise the same files though, the entire file will be uploaded. This situation seems to hold across a wide range of popular services:

Given the universal presence of the web browser, why can’t we have efficient delta syncing for web clients? That’s the question Xiao et al. set out to investigate: they built an rsync implementation for the web, and found out it performed terribly. Having tried everything to improve the performance within the original rsync design parameters, then they resorted to a redesign which moved more of the heavy lifting back to…

View original post 728 more words

Advertisements

Using Odroid as IP Router

I purchased an Odroid-XU4 for ca. 80 EUR including power-supply and case from Pollin. The original manufacturer is hardkernel. I intended to use this small ARM computer as a router and firewall. In the past I had used routers from multiple vendors, e.g., Linksys/Cisco, TP-Link, AVM/FritzBox, Netgear, and so on. There is a rule of thumb with all these devices: Usually you have to reboot them once or twice a month, otherwise they misbehave somehow. At least three of these device went completely catatonic. Now I had enough of this, I also wanted a command line interface to the router, ideally a real Linux system with bash, cron, gcc, etc. Although I already own an Intel NUC and I am very happy with this computer, an Intel NUC is a little bit too expensive to be used as just a router.

I recommend to additionally purchase a RTC backup battery. The Odroid has a realtime clock, but loses all date and time information once powered off. This way the log of the computer is garbled.

Cablesurf Channel Statistics

Cablesurf is a German internet-cable-provider. They deliver Technicolor modem and set-top boxes to the end customers.

My cable modem model is:

HW Revision       1.0            VENDOR           Technicolor
BOOT Revision     2.4.0          SW Revision      STDD.01.05
MODEL             TC7200.20      Software Version STDD.01.05
Serial Number     00997509604426		
Mta Serial Number 00997509604426

Software Build and Revision
Firmware Name         TC7200.20-DD.01.05-150924-F-1FF.bin
Firmware Build Time   11:45:59 Thu Sep 24 2015

Signal/Noise ratio for downstream is:

Channel	Lock     Modulation  Channel   Symbol   Freq  Power      SNR
        Status               ID        Rate
1       Locked   QAM256      145       6952000        4.1 dBmV   40.3 dB
2       Locked   QAM256      146       6952000        4.4 dBmV   40.4 dB
3       Locked   QAM256      147       6952000        4.8 dBmV   40.8 dB
4       Locked   QAM256      148       6952000        5.1 dBmV   40.8 dB
5       Locked   QAM256      149       6952000        5.2 dBmV   40.8 dB
6       Locked   QAM256      150       6952000        5.0 dBmV   40.8 dB
7       Locked   QAM256      151       6952000        4.7 dBmV   39.9 dB
8       Locked   QAM256      152       6952000        4.1 dBmV   40.3 dB

Signal/Noise ratio for upstream is:

Channel	Lock    Modulation   Channel   Symbol   Freq   Power
        Status               ID        Rate
1       Locked  QAM64        1         5120 Ksym/sec   45.5 dBmV
2       Locked  QAM64        2         5120 Ksym/sec   47.0 dBmV
3       Locked  QAM64        3         5120 Ksym/sec   47.0 dBmV
4       Locked  QAM64        4         5120 Ksym/sec   47.5 dBmV

I ordered 120MBit/s, but speed according T-Online speedtest is as follows:

I attribute the drop from the bought speed to peering between Telekom and Cablesurf.

Checking my Unitymedia connection with T-Online speedtest gives:

Surfing the internet with 100 MBit/s

This week I upgraded again, this time from 50 MBit/s to 100 MBit/s. My internet service provider is Unitymedia with whom I had positive experience since 2009, see Unitymedia experience. They phoned me mid of November and asked whether I would be interested in upgrading from 50 to 100, it would cost 2 euros more per month. I said yes. Here we go. I pay 35 euros per month for 100 MBit/s and a telephone flat rate within whole Germany.

Patrick Pichette on Google Fiber

Google CFO, Patrick Pichette, on the evolution of

1. computing power
2. storage capacity
3. network bandwith

He shows the following remarkable slide on the disparity of the latter to the former. It was this disparity which apparently led Google to enter the broadband market with its own infrastructure.

This speech was held in Kansas, 2012. The full speech is in below YouTube video.

Copy Directories with Symbolic Links via ssh

Although probably known in most circles it is worth repeating that scp by itself does not honor symbolic links. To overcome this limitation just combine tar and ssh, i.e., tar on sending side, untar on receiving side:

tar cf - /src/dir | ssh remotehost "cd /dst/dir ; tar xf -"

Usually this is even faster than using scp, as now only big chunks of data are transfered via TCP. Expect an almost twofold performance increase for larger directories which contain a couple of small files.

See also commandlinefu.

Positive Erfahrungen mit Unitymedia

Bisher habe ich mit der Firma Unitymedia ausschließlich positive Erfahrungen gemacht. Sie sind es wert, kurz festgehalten zu werden.

Am 20. August 2009 habe ich bei Unitymedia Internet (20MBit/s) und Telefon bestellt. Am nächsten Tag stand ein Herr Niclas Roth von Computer-Füchse vor der Tür und hat Kabelmodem und Telefon eingerichtet. Alles lief auf Anhieb.

Anfang 2012 habe ich von 20 MBit/s auf 50MBit/s hochgerüstet. Unitymedia schickte mir das hierfür erforderliche neue Kabelmodem zu. Der Einsatz erfolgte klaglos.

Am 13. Juni 2013 fiel Internet und Telefon aus. Am nächsten Tag schickte Unitymedia einen Techniker, der das Problem behob.

Wenn ich an meine Erfahrungen zu ähnlichen Situationen mit der Telekom oder 1&1 denke, dann ist dies ein Unterschied wie Tag und Nacht.