Skip to main content

External Blogs

(Still) working too much on the computer

Anarcat - mer, 06/01/2016 - 10:39

I have been using Workrave to try to force me to step away from the computer regularly to work around Repetitive Strain Injury (RSI) issues that have plagued my life on the computer intermittently in the last decade.

Workrave itself is only marginally efficient at getting me away from the machine: as any warning systems, it suffers from alarm fatigue as you frenetically click the dismiss button every time a Workrave warning pops up. However, it has other uses.

Analyzing data input

In the past, I have used Workrave to document how I work too much on the computer, but never went through more serious processing of the vast data store that Workrave accumulates about mouse movements and keystrokes. Interested in knowing how much my leave from Koumbit affected time spent on the computer, I decided to look into this again.

It turns out I am working as much, if not more, on the computer since I took that "time off":

We can see here that I type a lot on the computer. Normal days range from 10 000 to 60 000 keystrokes, with extremes at around 100 000 keystrokes per day. The average seem to fluctuate around 30 to 40 000 keystrokes per day, but rises sharply around the end of the second quarter of this very year. For those unfamiliar with the underlying technology, one keystroke is roughly one byte I put on the computer. So the average of 40 000 keystrokes is 40 kilobyte (KB) per day on the computer. That means 15 MB over a year or about 150MB (or 100 MiB if you want to be picky about it) over the course of the last decade.

That is a lot of typing.

I originally thought this could have been only because I type more now, as opposed to use more the mouse previously. Unfortunately, Workrave also tracks general "active time" which we can also examine:

Here we see that I work around 4 hours a day continuously on the computer. That is active time: not just login, logout time. In other words, the time where i look away from the computer and think for a while, jot down notes in my paper agenda or otherwise step away from the computer for small breaks is not counted here. Notice how some days go up to 12 hours and how recently the average went up to 7 hours of continuous activity.

So we can clearly see that I basically work more on the computer now than I ever did in the last 7 years. This is a problem - one of the reasons of this time off was to step away from the computer, and it seems I have failed.

Update: it turns out the graph was skewed towards the last samples. I went more easy on the keyboard in the last few days and things have significantly improved:

Another interesting thing we can see is when I switched from using my laptop to using the server as my main workstation, around early 2011, which is about the time marcos was built. Now that marcos has been turned into a home cinema downstairs, I went back to using my laptop as my main computing device, in late 2015. We can also clearly see when I stopped using Koumbit machines near the end of 2015 as well.

Further improvements and struggle for meaning

The details of how the graph was produced are explained at the end of this article.

This is all quite clunky: it doesn't help that the Workrave data structure is not easily parsable and so easily corruptible. It would be best if each data point was on its own separate line, which would be long, granted, but so easier to parse.

Furthermore, I do not like the perl/awk/gnuplot data processing pipeline much. It doesn't allow me to do interesting analysis like averages, means and linear regressions easily. It could be interesting to rewrite the tools in Python to allow better graphs and easier data analysis, using the tools I learned in 2015-09-28-fun-with-batteries.

Finally, this looks only at keystrokes and non-idle activity. It could be more interesting to look at idle/active times and generally the amount of time spent on the computer each day. And while it is interesting to know that I basically write a small book on the computer every day (according to Wikipedia, 120KB is about the size of a small pocket book), it is mostly meaningless if all that stuff is machine-readable code.

Where is, after all, the meaning in all those shell commands and programs we constantly input on our keyboards, in the grand scheme of human existence? Most of those bytes are bound to be destroyed by garbage collection (my shell's history) or catastrophic backup failures.

While the creative works from the 16th century can still be accessed and used by others, the data in some software programs from the 1990s is already inaccessible. - Lawrence Lessig

But is my shell history relevant? Looking back at old posts on this blog, one has to wonder if the battery life of the Thinkpad 380z laptop or how much e-waste I threw away in 2005 will be of any historical value in 20 years, assuming the data survives that long.

How this graph was made

I was happy to find that Workrave has some contrib scripts to do such processing. Unfortunately, those scripts are not shipped with the Debian package, so I requested that to be fixed (#825982). There were also some fixes necessary to make the script work at all: first, there was a syntax error in the Perl script. But then since my data is so old, there was bound to be some data corruption in there: incomplete entries or just plain broken data. I had lines that were all NULL characters, typical of power failures or disk corruptions. So I have made a patch to fix that script (#826021).

But this wasn't enough: while this processes data on the current machine fine, it doesn't deal with multiple machines very well. In the last 7 years of data I could find, I was using 3 different machines: this server (marcos), my laptop (angela) and Koumbit's office servers (koumbit). I ended up modifying the contrib scripts to be able to collate that data meaningfully. First, I copied over the data from Koumbit in a local fake-koumbit directory. Second, I mounted marcos home directory locally with SSHFS:

sshfs marcos

I also made this script to sum up datasets:

#!/usr/bin/perl -w use List::MoreUtils 'pairwise'; $| = 1; my %data = (); while (<>) { my @fields = split; my $date = shift @fields; if (defined($data{$date})) { my @t = pairwise { $a + $b } @{$data{$date}}, @fields; $data{$date} = \@t; } else { $data{$date} = \@fields; } } foreach my $d ( sort keys %data ) { print "$d @{$data{$d}}\n"; }

Then I made a modified version of the Gnuplot script that processes all those files together:

#!/usr/bin/gnuplot set title "Workrave" set ylabel "Keystrokes per day" set timefmt "%Y%m%d%H%M" #set xrange [450000000:*] set format x "%Y-%m-%d" set xtics rotate set xdata time set terminal svg set output "workrave.svg" plot "workrave-angela.dat" using 1:28 title "angela", \ "workrave-marcos.dat" using 1:28 title "marcos", \ "workrave-koumbit.dat" using 1:28 title "koumbit", \ "workrave-sum.dat" using 1:2 smooth sbezier linewidth 3 title "average" #plot "workrave-angela.dat" using 1:28 smooth sbezier title "angela", \ # "workrave-marcos.dat" using 1:28 smooth sbezier title "marcos", \ # "workrave-koumbit.dat" using 1:28 smooth sbezier title "koumbit"

And finally, I made a small shell script to glue this all together:

#!/bin/sh perl workrave-dump > workrave-$(hostname).dat HOME=$HOME/marcos perl workrave-dump > workrave-marcos.dat HOME=$PWD/fake-koumbit perl workrave-dump > workrave-koumbit.dat # remove idle days as they skew the average sed -i '/ 0$/d' workrave-*.dat # per-day granularity sed -i 's/^\(........\)....\? /\1 /' workrave-*.dat # sum up all graphs cat workrave-*.dat | sort | perl > workrave.dat ./gnuplot-workrave-anarcat

I used a different gnuplot script to generate the activity graph:

#!/usr/bin/gnuplot set title "Workrave" set ylabel "Active hours per day" set timefmt "%Y%m%d%H%M" #set xrange [450000000:*] set format x "%Y-%m-%d" set xtics rotate set xdata time set terminal svg set output "workrave.svg" plot "workrave-angela.dat" using 1:($23/3600) title "angela", \ "workrave-marcos.dat" using 1:($23/3600) title "marcos", \ "workrave-koumbit.dat" using 1:($23/3600) title "koumbit", \ "workrave.dat" using 1:($23/3600) title "average" smooth sbezier linewidth 3 #plot "workrave-angela.dat" using 1:28 smooth sbezier title "angela", \ # "workrave-marcos.dat" using 1:28 smooth sbezier title "marcos", \ # "workrave-koumbit.dat" using 1:28 smooth sbezier title "koumbit"
Catégories: External Blogs

My free software activities, May 2016

Anarcat - jeu, 05/19/2016 - 17:49
Debian Long Term Support (LTS)

This is my 6th month working on Debian LTS, started by Raphael Hertzog at Freexian. This is my largest month so far, for which I had requested 20 hours of work.

Xen work

I spent the largest amount of time working on the Xen packages. We had to re-roll the patches because it turned out we originally just imported the package from Ubuntu as-is. This was a mistake because that package forked off the Debian packaging a while ago and included regressions in the packaging itself, not just security fixes.

So I went ahead and rerolled the whole patchset and tested it on Koumbit's test server. Brian May then completed the uploaded, which included about 40 new patches, mostly from Ubuntu.

Frontdesk duties

Next up was the frontdesk duties I had taken this week. This was mostly uneventful, although I had forgotten how to do some of the work and thus ended up doing extensive work on the contributor's documentation. This is especially important since new contributors joined the team! I also did a lot of Debian documentation work in my non-sponsored work below.

The triage work involved chasing around missing DLAs, triaging away OpenJDK-6 (for which, let me remind you, security support has ended in LTS), raised the question of Mediawiki maintenance.

Other LTS work

I also did a bunch of smaller stuff. Of importance, I can note that I uploaded two advisories that were pending from April: NSS and phpMyAdmin. I also reviewed the patches for the ICU update, since I built the one for squeeze (but didn't have time to upload before squeeze hit end-of-life).

I have tried to contribute to the NTP security support but that was way too confusing to me, and I have left it to the package maintainer which seemed to be on top of things, even if things mean complete chaos and confusion in the world of NTP. I somehow thought that situation had improved with the recent investments in ntpsec and ntimed, but unfortunately Debian has not switched to the ntpsec codebase, so it seems that the NTP efforts have diverged in three different projects instead of closing into a single, better codebase.

Future LTS work

This is likely to be my last month of work on LTS until September. I will try to contribute a few hours in June, but July and August will be very busy for me outside of Debian, so it's unlikely that I contribute much to the project during the summer. My backlog included those packages which might be of interest to other LTS contributors:

  • libxml2: no upstream fix, but needs fixing!
  • tiff{,3}: same mess
  • libgd2: maintainer contacted
  • samba regression: mailed bug #821811 to try to revive the effort
  • policykit-1: to be investigated
  • p7zip: same
Other free software work Debian documentation

I wrote an detailed short guide to Debian package development, something I felt was missing from the existing corpus, which seems to be too focus in covering all alternatives. My guide is opinionated: I believe there is a right and wrong way of doing things, or at least, there are best practices, especially when just patching packages. I ended up retroactively publishing that as a blog post - now I can simply tag an item with blog and it shows up in the blog.

(Of course, because of a mis-configuration on my side, I have suffered from long delays publishing to Debian planet, so all the posts dates are off in the Planet RSS feed. This will hopefully be resolved around the time this post is published, but this allowed me to get more familiar with the Planet Venus software, as detailed in that other article.)

Apart from the guide, I have also done extensive research to collate information that allowed me to create workflow graphs of the various Debian repositories, which I have published in the Debian Release section of the Debian wiki. Here is the graph:

It helps me understand how packages flow between different suites and who uploads what where. This emerged after I realized I didn't really understand how "proposed updates" worked. Since we are looking at implementing a similar process for the security queue, I figured it was useful to show what changes would happen, graphically.

I have also published a graph that describes the relations between different software that make up the Debian archive. The idea behind this is also to provide an overview of what happens when you upload a package in the Debian archive, but it is more aimed at Debian developers trying to figure out why things are not working as expected.

The graphs were done with Graphviz, which allowed me to link to various components in the graph easily, which is neat. I also prefered Graphviz over Dia or other tools because it is easier to version and I don't have to bother (too much) about the layout and tweaking the looks. The downside is, of course, that when Graphviz makes the wrong decision, it's actually pretty hard to make it do the right thing, but there are various workarounds that I have found that made the graphs look pretty good.

The source is of course available in git but I feel all this documentation (including the guide) should go in a more official document somewhere. I couldn't quite figure out where. Advice on this would be of course welcome.


I have made yet another plugin for Ikiwiki, called irker, which enables wikis to send notifications to IRC channels, thanks to the simple irker bot. I had trouble with Irker in the past, since it was not quite reliable: it would disappear from channels and not return when we'd send it a notification. Unfortunately, the alternative, the KGB bot is much heavier: each repository needs a server-side, centralized configuration to operate properly.

Irker's design is simpler and more adapted to a simple plugin like this. Let's hope it will work reliably enough for my needs.

I have also suggested improvements to the footnotes styles, since they looked like hell in my Debian guide. It turns out this was an issue with the multimarkdown plugin that doesn't use proper semantic markup to identify footnotes. The proper fix is to enable footnotes in the default Discount plugin, which will require another, separate patch.

Finally, I have done some improvements (I hope!) on the layout of this theme. I made the top header much lighter and transparent to work around an issue where followed anchors would be hidden under the top header. I have also removed the top menu made out of the sidebar plugin because it was cluttering the display too much. Those links are all on the frontpage anyways and I suspect people were not using them so much.

The code is, as before, available in this git repository although you may want to start from the new ikistrap theme that is based on Bootstrap 4 and that may eventually be merged in ikiwiki directly.

DNS diagnostics

Through this interesting overview of various *ping tools, I got found out about the dnsdiag tool which currently allows users to do DNS traces, tampering detection and ping over DNS. In the hope of packaging it into Debian, I have requested clarifications regarding a modification to the DNSpython library the tool uses.

But I went even further and boldly opened a discussion about replacing DNSstuff, the venerable DNS diagnostic tools that is now commercial. It is somewhat surprising that there is no software that has even been publicly released that does those sanity checks for DNS, given how old DNS is.

Incidentally, I have also requested smtpping to be packaged in Debian as well but httping is already packaged.

Link checking

In the process of writing this article, I suddenly remembered that I constantly make mistakes in the various links I post on my site. So I started looking at a link checker, another tool that should be well established but that, surprisingly, is not quite there yet.

I have found this neat software written in Python called LinkChecker. Unfortunately, it is basically broken in Debian, so I had to do a non-maintainer upload to fix that old bug. I managed to force myself to not take over maintainership of this orphaned package but I may end up doing just that if no one steps up the next time I find issues in the package.

One of the problems I had checking links in my blog is that I constantly refer to sites that are hostile to bots, like the Debian bugtracker and MoinMoin wikis. So I published a patch that adds a --no-robots flag to be able to crawl those sites effectively.

I know there is the W3C tool but it's written in Perl, and there's probably zero chance for me to convince those guys to bypass robots exclusion rules, so I am sticking to Linkchecker.

Other Debian packaging work

At my request, Drush has finally been removed from Debian. Hopefully someone else will pick up that work, but since it basically needs to be redone from scratch, there was no sense in keeping it in the next release of Debian. Similarly, Semanticscuttle was removed from Debian as well.

I have uploaded new versions of tuptime, sopel and smokeping. I have also file a Request For Help for Smokeping. I am happy to report there was a quick response and people will be stepping up to help with the maintenance of that venerable monitoring software.

Background radiation

Finally, here's the generic background noise of me running around like a chicken with his head cut off:

Finally, I should mention that I will be less active in the coming months, as I will be heading outside as the summer finally came! I somewhat feel uncomfortable documenting publicly my summer here, as I am more protective of my privacy than I was before on this blog. But we'll see how it goes, maybe you'll hear non-technical articles here again soon!

Catégories: External Blogs

Meals-on-wheel Hacknight

Montreal Python - mar, 05/17/2016 - 23:00

Join us on May 26th and contribute to new management platform of Santropol Roulant. They need our help to get a better tools to manage and deliver food to people with disabilities and difficulties.

This platform will be at the heart of their service and it's entirely based on Django. The project needs volunteers who would like give a hand and to contribute their open platform and pursue it's development.

It's your opportunity to achieves some tasks of the project (, and to get to know what people of Santropol are doing. Please note that the event is targeting people who are used to Django but not necessarily experts with it, don't be afraid and join us :)

People from Santropol will prepare food that will be serve to the volunteers of the event!

This Free software project was also built by Savoir-Faire Linux and volunteers from la Maison du logiciel Libre.

Please note that the event will be bilingual.


Please confirm your participation on our meetup page at


Thursday, May 26th, starting at 6pm


Santropol Roulant 111 Roy East (Sherbrooke metro)

Catégories: External Blogs

Long delays posting Debian Planet Venus

Anarcat - sam, 05/14/2016 - 14:47

For the last few months, it seems that my posts haven't been reaching the Planet Debian aggregator correctly. I timed the last two posts and they both arrived roughly 10 days late in the feed.

SNI issues

At first, I suspected I was a victim of the SNI bug in Planet Venus: since it is still running in Python 2.7 and uses httplib2 (as opposed to, say, Requests), it has trouble with sites running under SNI. In January, there were 9 blogs with that problem on Planet. When this was discussed elsewhere in February, there were now 18, and then 21 reported in March. With everyone enabling (like me) Let's Encrypt on their website, this number is bound to grow.

I was able to reproduce the Debian Planet setup locally to do further tests and ended up sending two (unrelated) patches to the Debian bug tracker against Planet Venus, the software running Debian planet. In my local tests, I found 22 hosts with SNI problems. I also posted some pointers on how the code could be ported over to the more modern Requests and Cachecontrol modules.

Expiry issues

However, some of those feeds were working fine on philp, the host I found was running as the Planet Master. Even more strange, my own website was working fine!

INFO:planet.runner:Feed unchanged

Now that was strange: why was my feed fetched, but noted as unchanged? For that, I found that there was a FAQ question buried down in the PlanetDebian wikipage which explicitly said that Planet obeys Expires headers diligently and will not get new content again if the headers say they did. Skeptical, I looked my own headers and, ta-da! they were way off:

$ curl -v 2>&1 | egrep '< (Expires|Date)' < Date: Sat, 14 May 2016 19:59:28 GMT < Expires: Sat, 28 May 2016 19:59:28 GMT

So I lowered the expires timeout on my RSS feeds to 3 hours:

root@marcos:/etc/apache2# git diff diff --git a/apache2/conf-available/expires.conf b/apache2/conf-available/expires.conf index 214f3dd..a983738 100644 --- a/apache2/conf-available/expires.conf +++ b/apache2/conf-available/expires.conf @@ -3,8 +3,18 @@ # Enable expirations. ExpiresActive On - # Cache all files for 2 weeks after access (A). - ExpiresDefault A1209600 + # Cache all files 12 hours after access + ExpiresDefault "access plus 12 hours" + + # RSS feeds should refresh more often + <FilesMatch \.(rss)$> + ExpiresDefault "modification plus 4 hours" + </FilesMatch> + + # images are *less* likely to change + <FilesMatch "\.(gif|jpg|png|js|css)$"> + ExpiresDefault "access plus 1 month" + </FilesMatch> <FilesMatch \.(php|cgi)$> # Do not allow scripts to be cached unless they explicitly send cache

I also lowered the general cache expiry, except for images, Javascript and CSS.

Planet Venus maintenance

A small last word about all this: I'm surprised to see that Planet Debian is running a 6 year old software that hasn't seen a single official release yet, with local patches on top. It seems that Venus is well designed, I must give them that, but it's a little worrisome to see great software just rotting around like this.

A good "planet" site seems like a resource a lot of FLOSS communities would need: is there another "Planet-like" aggregator out there that is well maintained and more reliable? In Python, preferably.

PlanetPlanet, which Venus was forked from, is out of the question: it is even less maintained than the new fork, which itself seems to have died in 2011.

There is a discussion about the state of Venus on Github which reflects some of the concerns expressed here, as well as on the mailing list. The general consensus seems to be that everyone should switch over to Planet Pluto, which is written in Ruby.

I am not sure which planet Debian sits on - Pluto? Venus? Besides, Pluto is not even a planet anymore...

Mike check!

So this is also a test to see if my posts reach Debian Planet correctly. I suspect no one will ever see this on the top of their feeds, since the posts do get there, but with a 10 days delay and with the original date, so they are "sunk" down. The above expiration fixes won't take effect until the 10 days delay is over... But if you did see this as noise, retroactive apologies in advance for the trouble.

If you are reading this from somewhere else and wish to say hi, don't hesitate, it's always nice to hear from my readers.

Catégories: External Blogs

Notmuch, offlineimap and Sieve setup

Anarcat - jeu, 05/12/2016 - 18:29

I've been using Notmuch since about 2011, switching away from Mutt to deal with the monstrous amount of emails I was, and still am dealing with on the computer. I have contributed a few patches and configs on the Notmuch mailing list, but basically, I have given up on merging patches, and instead have a custom config in Emacs that extend it the way I want. In the last 5 years, Notmuch has progressed significantly, so I haven't found the need to patch it or make sweeping changes.

The huge INBOX of death

The one thing that is problematic with my use of Notmuch is that I end up with a ridiculously large INBOX folder. Before the cleanup I did this morning, I had over 10k emails in there, out of about 200k emails overall.

Since I mostly work from my laptop these days, the Notmuch tags are only on the laptop, and not propagated to the server. This makes accessing the mail spool directly, from webmail or simply through a local client (say Mutt) on the server, really inconvenient, because it has to load a very large spool of mail, which is very slow in Mutt. Even worse, a bunch of mail that was archived in Notmuch shows up in the spool because it's just removed tags in Notmuch: the mails are still in the inbox, even though they are marked as read.

So I was hoping that Notmuch would help me deal with the giant inbox of death problem, but in fact, when I don't use Notmuch, it actually makes the problem worse. Today, I did a bunch of improvements to my setup to fix that.

The first thing I did was to kill procmail, which I was surprised to discover has been dead for over a decade. I switched over to Sieve for filtering, having already switched to Dovecot a while back on the server. I tried to use the conversion tool but it didn't work very well, so I basically rewrote the whole file. Since I was mostly using Notmuch for filtering, there wasn't much left to convert.

Sieve filtering

But this is where things got interesting: Sieve is so simpler to use and more intuitive that I started doing more interesting stuff in bridging the filtering system (Sieve) with the tagging system (Notmuch). Basically, I use Sieve to split large chunks of emails off my main inbox, to try to remove as much spam, bulk email, notifications and mailing lists as possible from the larger flow of emails. Then Notmuch comes in and does some fine-tuning, assigning tags to specific mailing lists or topics, and being generally the awesome search engine that I use on a daily basis.

Dovecot and Postfix configs

For all of this to work, I had to tweak my mail servers to talk sieve. First, I enabled sieve in Dovecot:

--- a/dovecot/conf.d/15-lda.conf +++ b/dovecot/conf.d/15-lda.conf @@ -44,5 +44,5 @@ protocol lda { # Space separated list of plugins to load (default is global mail_plugins). - #mail_plugins = $mail_plugins + mail_plugins = $mail_plugins sieve }

Then I had to switch from procmail to dovecot for local delivery, that was easy, in Postfix's perennial

#mailbox_command = /usr/bin/procmail -a "$EXTENSION" mailbox_command = /usr/lib/dovecot/dovecot-lda -a "$RECIPIENT"

Note that dovecot takes the full recipient as an argument, not just the extension. That's normal. It's clever, it knows that kind of stuff.

One last tweak I did was to enable automatic mailbox creation and subscription, so that the automatic extension filtering (below) can create mailboxes on the fly:

--- a/dovecot/conf.d/15-lda.conf +++ b/dovecot/conf.d/15-lda.conf @@ -37,10 +37,10 @@ #lda_original_recipient_header = # Should saving a mail to a nonexistent mailbox automatically create it? -#lda_mailbox_autocreate = no +lda_mailbox_autocreate = yes # Should automatically created mailboxes be also automatically subscribed? -#lda_mailbox_autosubscribe = no +lda_mailbox_autosubscribe = yes protocol lda { # Space separated list of plugins to load (default is global mail_plugins). Sieve rules

Then I had to create a Sieve ruleset. That thing lives in ~/.dovecot.sieve, since I'm running Dovecot. Your provider may accept an arbitrary ruleset like this, or you may need to go through a web interface, or who knows. I'm assuming you're running Dovecot and have a shell from now on.

The first part of the file is simply to enable a bunch of extensions, as needed:

# Sieve Filters # # require "fileinto"; require "envelope"; require "variables"; require "subaddress"; require "regex"; require "vacation"; require "vnd.dovecot.debug";

Some of those are not used yet, for example I haven't tested the vacation module yet, but I have good hopes that I can use it as a way to announce a special "urgent" mailbox while I'm traveling. The rationale is to have a distinct mailbox for urgent messages that is announced in the autoreply, that hopefully won't be parsable by bots.

Spam filtering

Then I filter spam using this fairly standard expression:

######################################################################## # spam # possible improvement, server-side: # if header :contains "X-Spam-Flag" "YES" { fileinto "junk"; stop; } elsif header :contains "X-Spam-Level" "***" { fileinto "greyspam"; stop; }

This puts stuff into the junk or greyspam folder, based on the severity. I am very aggressive with spam: stuff often ends up in the greyspam folder, which I need to check from time to time, but it beats having too much spam in my inbox.

Mailing lists

Mailing lists are generally put into a lists folder, with some mailing lists getting their own folder:

######################################################################## # lists # converted from procmail if header :contains "subject" "FreshPorts" { fileinto "freshports"; } elsif header :contains "List-Id" "" { fileinto "alternc"; } elsif header :contains "List-Id" "" { fileinto "koumbit"; } elsif header :contains ["to", "cc"] ["", ""] { fileinto "debian"; # Debian BTS } elsif exists "X-Debian-PR-Message" { fileinto "debian"; # default lists fallback } elsif exists "List-Id" { fileinto "lists"; }

The idea here is that I can safely subscribe to lists without polluting my mailbox by default. Further processing is done in Notmuch.

Extension matching

I also use the magic +extension tag on emails. If you send email to, say, then the emails end up in the foo folder. This is done with the help of the following recipe:

######################################################################## # wildcard +extension # if envelope :matches :detail "to" "*" { # Save name in ${name} in all lowercase except for the first letter. # Joe, joe, jOe thus all become 'Joe'. set :lower "name" "${1}"; fileinto "${name}"; #debug_log "filed into mailbox ${name} because of extension"; stop; }

This is actually very effective: any time I register to a service, I try as much as possible to add a +extension that describe the service. Of course, spammers and marketers (it's the same really) are free to drop the extension and I suspect a lot of them do, but it helps with honest providers and this actually sorts a lot of stuff out of my inbox into topically-defined folders.

It is also a security issue: someone could flood my filesystem with tons of mail folders, which would cripple the IMAP server and eat all the inodes, 4 times faster than just sending emails. But I guess I'll cross that bridge when I get there: anyone can flood my address and I have other mechanisms to deal with this.

The trick is to then assign tags to all folders so that they appear in the Notmuch-emacs welcome view:

echo tagging folders for folder in $(ls -ad $HOME/Maildir/${PREFIX}*/ | egrep -v "Maildir/${PREFIX}(feeds.*|Sent.*|INBOX/|INBOX/Sent)\$"); do tag=$(echo $folder | sed 's#/$##;s#^.*/##') notmuch tag +$tag -inbox tag:inbox and not tag:$tag and folder:${PREFIX}$tag done

This is part of my notmuch-tag script that includes a lot more fine-tuned filtering, detailed below.

Automated reports filtering

Another thing I get a lot of is machine-generated "spam". Well, it's not commercial spam, but it's a bunch of Nagios, cron jobs, and god knows what software thinks it's important to send me emails every day. I get a lot less of those these days since I'm off work at Koumbit, but still, those can be useful for others as well:

if anyof (exists "X-Cron-Env", header :contains ["subject"] ["security run output", "monthly run output", "daily run output", "weekly run output", "Debian Package Updates", "Debian package update", "daily mail stats", "Anacron job", "nagios", "changes report", "run output", "[Systraq]", "Undelivered mail", "Postfix SMTP server: errors from", "backupninja", "DenyHosts report", "Debian security status", "apt-listchanges" ], header :contains "Auto-Submitted" "auto-generated", envelope :contains "from" ["nagios@", "logcheck@"]) { fileinto "rapports"; } # imported from procmail elsif header :comparator "i;octet" :contains "Subject" "Cron" { if header :regex :comparator "i;octet" "From" ".*root@" { fileinto "rapports"; } } elsif header :comparator "i;octet" :contains "To" "root@" { if header :regex :comparator "i;octet" "Subject" "\\*\\*\\* SECURITY" { fileinto "rapports"; } } elsif header :contains "Precedence" "bulk" { fileinto "bulk"; } Refiltering emails

Of course, after all this I still had thousands of emails in my inbox, because the sieve filters apply only on new emails. The beauty of Sieve support in Dovecot is that there is a neat sieve-filter command that can reprocess an existing mailbox. That was a lifesaver. To run a specific sieve filter on a mailbox, I simply run:

sieve-filter .dovecot.sieve INBOX 2>&1 | less

Well, this doesn't do anything. To really execute the filters, you need the -e flags, and to write to the INBOX for real, you need the -w flag as well, so the real run looks something more like this:

sieve-filter -e -W -v .dovecot.sieve INBOX > refilter.log 2>&1

The funky output redirects are necessary because this outputs a lot of crap. Also note that, unfortunately, the fake run output differs from the real run and is actually more verbose, which makes it really less useful than it could be.


I also usually archive my mails every year, rotating my mailbox into an Archive.YYYY directory. For example, now all mails from 2015 are archived in a Archive.2015 directory. I used to do this with Mutt tagging and it was a little slow and error-prone. Now, i simply have this Sieve script:

require ["variables","date","fileinto","mailbox", "relational"]; # Extract date info if currentdate :matches "year" "*" { set "year" "${1}"; } if date :value "lt" :originalzone "date" "year" "${year}" { if date :matches "received" "year" "*" { # Archive Dovecot mailing list items by year and month. # Create folder when it does not exist. fileinto :create "Archive.${1}"; } }

I went from 15613 to 1040 emails in my real inbox with this process (including refiltering with the default filters as well).

Notmuch configuration

My Notmuch configuration is a in three parts: I have small settings in ~/.notmuch-config. The gist of it is:

[new] tags=unread;inbox; ignore= #[maildir] # synchronize_flags=true # tentative patch that was refused upstream # #reckless_trash=true [search] exclude_tags=deleted;spam;

I omitted the fairly trivial [user] section for privacy reasons and [database] for declutter.

Then I have a notmuch-tag script symlinked into ~/Maildir/.notmuch/hooks/post-new. It does way too much stuff to describe in details here, but here are a few snippets:

if hostname | grep angela > /dev/null; then PREFIX=Anarcat/ else PREFIX=. fi

This sets a variable that makes the script work on my laptop (angela), where mailboxes are in Maildir/Anarcat/foo or the server, where mailboxes are in Maildir/.foo.

I also have special rules to tag my RSS feeds, which are generated by feed2imap, which is documented shortly below:

echo tagging feeds ( cd $HOME/Maildir/ && for feed in ${PREFIX}feeds.*; do name=$(echo $feed | sed "s#${PREFIX}feeds\\.##") notmuch tag +feeds +$name -inbox folder:$feed and not tag:feeds done )

Another example that would be useful is how to tag mailing lists, for example, this removes the inbox tag and adds the notmuch tags to emails from the notmuch mailing list.

notmuch tag +lists +notmuch -inbox tag:inbox and ""

Finally, I have a bunch of special keybindings in ~/.emacs.d/notmuch-config.el:

;; autocompletion (eval-after-load "notmuch-address" '(progn (notmuch-address-message-insinuate))) ; use fortune for signature, config is in custom (add-hook 'message-setup-hook 'fortune-to-signature) ; don't remember what that is (add-hook 'notmuch-show-hook 'visual-line-mode) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;; keymappings ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; (define-key notmuch-show-mode-map "S" (lambda () "mark message as spam and advance" (interactive) (notmuch-show-tag '("+spam" "-unread")) (notmuch-show-next-open-message-or-pop))) (define-key notmuch-search-mode-map "S" (lambda (&optional beg end) "mark message as spam and advance" (interactive (notmuch-search-interactive-region)) (notmuch-search-tag (list "+spam" "-unread") beg end) (anarcat/notmuch-search-next-message))) (define-key notmuch-show-mode-map "H" (lambda () "mark message as spam and advance" (interactive) (notmuch-show-tag '("-spam")) (notmuch-show-next-open-message-or-pop))) (define-key notmuch-search-mode-map "H" (lambda (&optional beg end) "mark message as spam and advance" (interactive (notmuch-search-interactive-region)) (notmuch-search-tag (list "-spam") beg end) (anarcat/notmuch-search-next-message))) (define-key notmuch-search-mode-map "l" (lambda (&optional beg end) "undelete and advance" (interactive (notmuch-search-interactive-region)) (notmuch-search-tag (list "-unread") beg end) (anarcat/notmuch-search-next-message))) (define-key notmuch-search-mode-map "u" (lambda (&optional beg end) "undelete and advance" (interactive (notmuch-search-interactive-region)) (notmuch-search-tag (list "-deleted") beg end) (anarcat/notmuch-search-next-message))) (define-key notmuch-search-mode-map "d" (lambda (&optional beg end) "delete and advance" (interactive (notmuch-search-interactive-region)) (notmuch-search-tag (list "+deleted" "-unread") beg end) (anarcat/notmuch-search-next-message))) (define-key notmuch-show-mode-map "d" (lambda () "delete current message and advance" (interactive) (notmuch-show-tag '("+deleted" "-unread")) (notmuch-show-next-open-message-or-pop))) ;; (define-key notmuch-show-mode-map "b" (lambda (&optional address) "Bounce the current message." (interactive "sBounce To: ") (notmuch-show-view-raw-message) (message-resend address) (kill-buffer))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;; my custom notmuch functions ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; (defun anarcat/notmuch-search-next-thread () "Skip to next message from region or point This is necessary because notmuch-search-next-thread just starts from point, whereas it seems to me more logical to start from the end of the region." ;; move line before the end of region if there is one (unless (= beg end) (goto-char (- end 1))) (notmuch-search-next-thread)) ;; Linking to notmuch messages from org-mode ;; (require 'org-notmuch nil t) (message "anarcat's custom notmuch config loaded")

This is way too long: in my opinion, a bunch of that stuff should be factored in upstream, but some features have been hard to get in. For example, Notmuch is really hesitant in marking emails as deleted. The community is also very strict about having unit tests for everything, which makes writing new patches a significant challenge for a newcomer, which will often need to be familiar with both Elisp and C. So for now I just have those configs that I carry around.

Emails marked as deleted or spam are processed with the following script named notmuch-purge which I symlink to ~/Maildir/.notmuch/hooks/pre-new:

#!/bin/sh if hostname | grep angela > /dev/null; then PREFIX=Anarcat/ else PREFIX=. fi echo moving tagged spam to the junk folder notmuch search --output=files tag:spam \ and not folder:${PREFIX}junk \ and not folder:${PREFIX}greyspam \ and not folder:Koumbit/INBOX \ and not path:Koumbit/** \ | while read file; do mv "$file" "$HOME/Maildir/${PREFIX}junk/cur" done echo unconditionnally deleting deleted mails notmuch search --output=files tag:deleted | xargs -r rm

Oh, and there's also customization for Notmuch:

;; -*- mode: emacs-lisp; auto-recompile: t; -*- (custom-set-variables ;; from '(fortune-file "/home/anarcat/.mutt/sigs.fortune") '(message-send-hook (quote (notmuch-message-mark-replied))) '(notmuch-address-command "notmuch-address") '(notmuch-always-prompt-for-sender t) '(notmuch-crypto-process-mime t) '(notmuch-fcc-dirs (quote ((".*" . "Koumbit/INBOX.Sent") (".*" . "Anarcat/Sent")))) '(notmuch-hello-tag-list-make-query "tag:unread") '(notmuch-message-headers (quote ("Subject" "To" "Cc" "Bcc" "Date" "Reply-To"))) '(notmuch-saved-searches (quote ((:name "inbox" :query "tag:inbox and not tag:koumbit and not tag:rt") (:name "unread inbox" :query "tag:inbox and tag:unread") (:name "unread" :query "tag:unred") (:name "freshports" :query "tag:freshports and tag:unread") (:name "rapports" :query "tag:rapports and tag:unread") (:name "sent" :query "tag:sent") (:name "drafts" :query "tag:draft")))) '(notmuch-search-line-faces (quote (("deleted" :foreground "red") ("unread" :weight bold) ("flagged" :foreground "blue"))))/ '(notmuch-search-oldest-first nil) '(notmuch-show-all-multipart/alternative-parts nil) '(notmuch-show-all-tags-list t) '(notmuch-show-insert-text/plain-hook (quote (notmuch-wash-convert-inline-patch-to-part notmuch-wash-tidy-citations notmuch-wash-elide-blank-lines notmuch-wash-excerpt-citations))) )

I think that covers it.


So of course the above works well on the server directly, but how do run Notmuch on a remote machine that doesn't have access to the mail spool directly? This is where OfflineIMAP comes in. It allows me to incrementally synchronize a local Maildir folder hierarchy with a a remote IMAP server. I am assuming you already have an IMAP server configured, since you already configured Sieve above.

Note that other synchronization tools exist. The other popular one is isync but I had trouble migrating to it (see courriels for details) so for now I am sticking with OfflineIMAP.

The configuration is fairly simple:

[general] accounts = Anarcat ui = Blinkenlights maxsyncaccounts = 3 [Account Anarcat] localrepository = LocalAnarcat remoterepository = RemoteAnarcat # refresh all mailboxes every 10 minutes autorefresh = 10 # run notmuch after refresh postsynchook = notmuch new # sync only mailboxes that changed quick = -1 ## possible optimisation: ignore mails older than a year #maxage = 365 # local mailbox location [Repository LocalAnarcat] type = Maildir localfolders = ~/Maildir/Anarcat/ # remote IMAP server [Repository RemoteAnarcat] type = IMAP remoteuser = anarcat remotehost = ssl = yes # without this, the cert is not verified (!) sslcacertfile = /etc/ssl/certs/DST_Root_CA_X3.pem # do not sync archives folderfilter = lambda foldername: not'(Sent\.20[01][0-9]\..*)', foldername) and not'(Archive.*)', foldername) # and only subscribed folders subscribedonly = yes # don't reconnect all the time holdconnectionopen = yes # get mails from INBOX immediately, doesn't trigger postsynchook idlefolders = ['INBOX']

Critical parts are:

  • postsynchook: obviously, we want to run notmuch after fetching mail
  • idlefolders: receives emails immediately without waiting for the longer autorefresh delay, which means that most mailboxes don't see new emails until 10 minutes in the worst case. unfortunately, doesn't run the postsynchook so I need to hit G in Emacs to see new mail
  • quick=-1, subscribedonly, holdconnectionopen: makes most runs much, much faster as it skips unchanged or unsubscribed folders and keeps the connection to the server

The other settings should be self-explanatory.

RSS feeds

I gave up on RSS readers, or more precisely, I merged RSS feeds and email. The first time I heard of this, it sounded like a horrible idea, because it means yet more emails! But with proper filtering, it's actually a really nice way to process emails, since it leverages the distributed nature of email.

For this I use a fairly standard feed2imap, although I do not deliver to an IMAP server, but straight to a local Maildir. The configuration looks like this:

--- include-images: true target-refix: &target "maildir:///home/anarcat/Maildir/.feeds." feeds: - name: Planet Debian url: target: [ *target, 'debian-planet' ]

I have obviously more feeds, the above is just and example. This will deliver the feeds as emails in one mailbox per feed, in ~/Maildir/.feeds.debian-planet, in the above example.


You will fail at writing the sieve filters correctly, and mail will (hopefully?) fall through to your regular mailbox. Syslog will tell you things fail, as expected, and details are in your .dovecot.sieve.log file in your home directory.

I also enabled debugging on the Sieve module

--- a/dovecot/conf.d/90-sieve.conf +++ b/dovecot/conf.d/90-sieve.conf @@ -51,6 +51,7 @@ plugin { # deprecated imapflags extension in addition to all extensions were already # enabled by default. #sieve_extensions = +notify +imapflags + sieve_extensions = +vnd.dovecot.debug # Which Sieve language extensions are ONLY available in global scripts. This # can be used to restrict the use of certain Sieve extensions to administrator

This allowed me to use debug_log function in the rulesets to output stuff directly to the logfile.

Further improvements

Of course, this is all done on the commandline, but that is somewhat expected if you are already running Notmuch. Of course, it would be much easier to edit those filters through a GUI. Roundcube has a nice Sieve plugin, and Thunderbird also has such a plugin as well. Since Sieve is a standard, there's a bunch of clients available. All those need you to setup some sort of thing on the server, which I didn't bother doing yet.

And of course, a key improvement would be to have Notmuch synchronize its state better with the mailboxes directly, instead of having the notmuch-purge hack above. Dovecot and Maildir formats support up to 26 flags, and there were discussions about using those flags to synchronize with notmuch tags so that multiple notmuch clients can see the same tags on different machines transparently.

This, however, won't make Notmuch work on my phone or webmail or any other more generic client: for that, Sieve rules are still very useful.

I still don't have webmail setup at all: so to read email, I need an actual client, which is currently my phone, which means I need to have Wifi access to read email. "Internet Cafés" or "this guy's computer" won't work as well, although I can always use ssh to login straight to the server and read mails with Mutt.

I am also considering using X509 client certificates to authenticate to the mail server without a passphrase. This involves configuring Postfix, which seems simple enough. Dovecot's configuration seems a little more involved and less well documented. It seems that both [OfflimeIMAP][] and K-9 mail support client-side certs. OfflineIMAP prompts me for the password so it doesn't get leaked anywhere. I am a little concerned about building yet another CA, but I guess it would not be so hard...

The server side of things needs more documenting, particularly the spam filters. This is currently spread around this wiki, mostly in configuration.

Security considerations

The whole purpose of this was to make it easier to read my mail on other devices. This introduces a new vulnerability: someone may steal that device or compromise it to read my mail, impersonate me on different services and even get a shell on the remote server.

Thanks to the two-factor authentication I setup on the server, I feel a little more confident that just getting the passphrase to the mail account isn't sufficient anymore in leveraging shell access. It also allows me to login with ssh on the server without trusting the machine too much, although that only goes so far... Of course, sudo is then out of the question and I must assume that everything I see is also seen by the attacker, which can also inject keystrokes and do all sorts of nasty things.

Since I also connected my email account on my phone, someone could steal the phone and start impersonating me. The mitigation here is that there is a PIN for the screen lock, and the phone is encrypted. Encryption isn't so great when the passphrase is a PIN, but I'm working on having a better key that is required on reboot, and the phone shuts down after 5 failed attempts. This is documented in my phone setup.

Client-side X509 certificates further mitigates those kind of compromises, as the X509 certificate won't give shell access.

Basically, if the phone is lost, all hell breaks loose: I need to change the email password (or revoke the certificate), as I assume the account is about to be compromised. I do not trust Android security to give me protection indefinitely. In fact, one could argue that the phone is already compromised and putting the password there already enabled a possible state-sponsored attacker to hijack my email address. This is why I have an OpenPGP key on my laptop to authenticate myself for critical operations like code signatures.

The risk of identity theft from the state is, after all, a tautology: the state is the primary owner of identities, some could say by definition. So if a state-sponsored attacker would like to masquerade as me, they could simply issue a passport under my name and join a OpenPGP key signing party, and we'd have other problems to deal with, namely, proper infiltration counter-measures and counter-snitching.

Catégories: External Blogs

Montréal-Python 58: Dramatics Chartreuse

Montreal Python - mar, 05/03/2016 - 23:00

We're close to a month before the next PyCon Conference in Portland, Oregon. We are organizing our 58th meetup at our lovely UQAM. Join us if you would like to feel what the Python community in Montreal is doing.

As usual we are receiving some guests in both languages and they will present you their projects and realizations.

Don't forget to join us after the meetup at the Benelux to celebrate spring in our lovely city.

Flash presentations

Kate Arthur: Kids CODE Jeunesse

Kids Code Jeunesse is dedicated to giving every Canadian child the chance to learn to code and to learn computational thinking. We introduce educators, parents and communities to intuitive teaching tools. We work in classrooms, community centres, host events and give workshops to supporting engaging educational experiences for everyone.

Christophe Reverd: Club Framboise (

Présentation du Club Framboise, la communauté des utilisateurs de Raspberry Pi à Montréal

Main presentations

Ivo Tzvetkov: Neolixir

An ORM for easy modelling and integration of Neo4j graph databases

Vadim Gubergrits: DIY Quantum Computer

An introduction to Quantum Computing with Python.

Pascal Priori: santropol-feast: Savoir faire Linux et des bénévoles accompagnent le Santropol Roulant (

Dans le cadre de la maison du logiciel libre, Savoir faire Linux et des bénévoles accompagnent le Santropol Roulant, un acteur du milieu communautaire montréalais dans la réalisation d'une plateforme de gestion de la base de donnée des clients en Django. En effet, au cœur des activités du Santropol Roulant, il y a le service de popote roulante qui cuisine, prépare et livre plus d’une centaine de repas chauds chaque jour à des personnes en perte d’autonomie. La base de données des clients joue un rôle clé dans la chaîne de services Réalisé en Django, le projet est à la recherche de bénévoles ayant envie de s'engager et contribuer au projet pour poursuivre le développement de la plateforme!

George Peristerakis: How CI is done in Openstack

In George's last talk, there was a lot of questions on the details of integrating code review and continuous integration in Openstack. This talk is a followup on the process and the technology behind implementing CI for Openstack.


UQÀM, Pavillion PK

201, Président-Kennedy avenue

Room PK-1140


Monday, May 9th 2016

  • 6:00pm — Doors open
  • 6:30pm — Presentations start
  • 7:30pm — Break
  • 7:45pm — Second round of presentations
  • 9:00pm — End of the meeting, have a drink with us
We’d like to thank our sponsors for their continued support:
  • UQÀM
  • Bénélux
  • Outbox
  • Savoir-faire Linux
  • Caravan
  • iWeb
Catégories: External Blogs

My free software activities, April 2016

Anarcat - lun, 04/25/2016 - 13:23
Debian Long Term Support (LTS)

This is my 5th month working on Debian LTS, started by Raphael Hertzog at Freexian. This is my largest month so far, in which I worked on completing the Xen and NSS packages updates from last month, but also spent a significant amount of time working on phpMyAdmin and libidn.

Updates to NSS and Xen completed

This basically consisted on following up on the reviews from other security people. I basically continued building up on Brian's work and tested the package on a test server at Koumbit, which seems to have survived well the upgrade. The details are in this post to the debian-lts mailing list.

As for NSS, the package was mostly complete, but I forgot to ship the tests for some reason, so I added them back. I also wrote the release team to see if it was possible to update NSS to the same version in all suites. Unfortunately, this issue is still pending, but I still hope we can find better ways of managing that package in the long term.

IDN and phpMyAdmin

Most of my time this month was spent working on IDN and phpMyAdmin. Unfortunately, it turns out that someone else had worked on the libidn package. This is partly my fault: I forgot to check in the dsa-needed.txt file for assignment before working on the package. But considering how in flux the workflow currently is with the switch between the main security team and the LTS team for the wheezy maintenance, I don't feel too bad. Still, I prepared a package which was a bit more painful than it should have been because of GNUlib. I didn't even know about GNUlib before, oddly enough, and after that experience, I feel that it should just not exist at all anymore. I have filed a bug to remove that dependency at the very least, but I do not clearly see how such a tool is necessary on Debian at this point in time.

But phpMyAdmin, no one had worked on that. And I understand why: even though it's a very popular package, there were quite a few outstanding issues (8!) in Wheezy, with 10-15 patches to be ported. Plus, it's ... well, PHP. And old PHP at that, with parts of it with modern autoloader classes, and other with mixed HTML and PHP content, require (and not require_once) and all sorts of nasty things you still find in the PHP universe. I nevertheless managed to produce a new Debian package for wheezy and test it on Koumbit's test servers. Hopefully, that can land into Wheezy soon.

Long term software support

I am a little worried that we are, both in Jessie and Wheezy sitting in between stable releases for phpMyAdmin, something that is a recurring issue for a lot of packages in Debian stable or LTS. Sometimes, it just happens that the version that happens to be in Debian testing when it is released as stable is just not a supported release upstream. It's the case for phpMyAdmin in Jessie (4.3, whereas 4.0 and 4.4 are supported) and Wheezy (3.4, although it's unclear how long that was supported upstream). But even if the next Debian stable (Stretch), would pick a stable release upstream, there is actually no phpMyAdmin release that has a support window as long as Debian stable (roughly 3 years), let alone as long as Debian LTS (5 years).

This is a similar problem with NSS: upstream is simply not supporting their product in the long term, or at least not in the terms we are used to in the Debian community (ie. only security fixes or regressions). This is, in my opinion, a real concern for the reliability and sustainability of the computing infrastructure we are creating. While some developers are of the opinion that software older than 18 months is too old, here we are shipping hardware and software in space or we have Solaris, which is supported for 16 years! Now that is a serious commitment and something we can rely on. 18 months is really, really, a tiny short time in the history of our civilization. I know computer programmers and engineers like to think of themselves in elitist terms, that they are changing the world every other year. But the truth is that things have not changed much in the last 4 decades where computing has existed, both in terms of security or functionality. Two quotes from my little quotes collection come to mind here:

Software gets slower faster than hardware gets faster. - Wirth's law

The future is already here – it's just not very evenly distributed. - William Gibson

Because of course, the response to my claims that computing is not really advancing is "but look! we have supercomputers in our pockets now!" Yes, of course, we do have those fancy phones and they are super-connected, but they are a threat to our fundamental rights and freedom. And those glittering advances always pale in comparison to what could be done if we wouldn't be wasting our time rewriting the same software over and over again on different, often proprietary, platforms, simply because of the megalomaniac fantasies of egoistic programmers.

It would be great to build something that lasts, for a while. Software that does not need to be updated every 16 months. You'd think that something as basic as a screensaver release could survive what is basically the range of human infancy. Yet it seems we like to run mindlessly in the minefield of software development, one generation following the other without questioning the underlying assumption of infinite growth that permeates our dying civilization.

I have talked about this before of course, but working with the LTS project just unnerves me so bad that I just needed to let out another rant.

(For the record, I really have a lot of respect for JWZ and all the work he has done in the free software world. I frequently refer to his "no-bullshit" backup guide and use Xscreensaver daily. But I do think he was wrong in this case: asking Debian to remove Xscreensaver is just too much. The response from the maintainer was exemplary of how to handle such issues in the future. I restarted Xscreensaver after the stable update, and the message is gone, and things are still all fine. Thanks JWZ and Tormod for keeping things rolling.)

Other free software work

With that in mind, I obviously didn't stop doing other computing work this month. In fact, I did a lot of work to try to generally fix the internet, that endless and not-quite-gratifying hobby so many of us are destroying our bodies to accomplish.

Build tools

I have experimented a bit more with different build tools. I got worried because at some point cowbuilder got orphaned and I figured I could refresh the way I build packages for LTS. I looked into sbuild, but that ended up not bringing much improvements over my current cowbuilder setup (which I really need to document better). I was asked by the new maintainer to open a bug report to make such setups easier by guessing the basepath better, so we'll see how that goes.

I did enjoy the simplicity of gitpkg and discovered cowpoke which made it about 2 times faster to build packages because I could use another server to build larger packages. I also found that gitpkg doesn't use -n by default when calling gzip, which makes it harder to reproduce tarballs when they are themselves built reproducibly, which is the case for Github tarballs (really nice of them). So I filed bug #820842 about that.

It would be pretty awesome if such buildds would be available for Debian Developers to do their daily tasks. It could simply be a machine that would spin up a chroot with cowbuilder or it could even be a full, temporary VM although that would take way more resources than a simple VM with a cowbuilder setup.

In the meantime, I should probably look at whalebuilder as an alternative to cowbuilder. It is a tool that supports building packages within a Docker chroot, which means that packages are built from a clean environment like pbuilder, and using COW optimisations but also without privileges or network access, which is a huge plus especially when you build untrusted packages.

Ikiwiki admonitions

I have done some work to implement Moinmoin-like admonitions in Ikiwiki, something I am quite happy about since it's something I was really missing about Moinmoin. Admonitions bring a really nice way to outline certain blocks with varying severity levels and distinct styles. For example:

Admonitions are great!

This was done with a macro, but basically, since Markdown allows more or less arbitrary HTML, this can also be done with the <div> tag. I like that we don't have a new weird markup here. Yet, I still liked the sub-parser feature of MoinMoin, something that can be implemented in Ikiwiki, but it's a little more clunky. Normally, you'd probably do this with the inline macro and subpages, but it's certainly less intuitive that directly inlined content.


I got a new e-reader! I was hesitant between the Kobo Aura H20 and the Kobo Glo HD, which were the ones available at Bestbuy. So I bought both and figured I would return the other. That was a painful decision! In the end, both machines are pretty nice:

  • Aura H2O
    • Pros:
      • waterproof
      • larger screen (makes it easier to see web pages and better for the eyes)
      • naturally drawn to it
    • Cons:
      • heavier
      • larger (fits in outside pocket though)
      • port cover finicky
      • more expensive (180$) - prices may go down in future
  • Aura Glo HD
    • Pros
      • smaller (fits in inside pocket of both coats)
      • better resolution in theory (can't notice in practice)
      • cheaper (100$)
      • may be less common on the future (larger models more common? just a guess)
    • Cons
      • no SD card
      • smaller screen
      • power button in the middle

... but in the end, I ended up settling down on the Glo, mostly for the price. Heck, I saved around 100$, so for that amount, I could have gotten two machines so that if one breaks I would still have the other. I have ordered a cover for it on Deal Extreme about two weeks ago, and it still hasn't arrived. I suspect it's going to take a few more months to get there, by which point I may have changed e-reader again.

Note that both e-readers needed an update to calibre, so I started working on a calibre backport (#818309) which I will complete soon.

So anyways, I looked into better ways of transferring articles from the web to the e-reader, something which I do quite a bit to avoid spending too much time on the computer. Since the bookmark manager I use (Bookie) is pretty much dead, I started looking at other alternatives. And partly inspired by Framasoft's choice of Wallabag for their bookmarking service (Framabag), I started to look into that software, especially since my friend who currently runs the Bookie instance is thinking of switching to Wallabag as well.

It seems the simplest way to browse articles remotely through a standard protocol is by implementing OPDS support in Wallabag. OPDS is a standard developed in part by the Internet Archive and it allows for browsing book collections and downloading them. Articles and bookmarks would then be presented as individual books that would be accessible from any OPDS-compatible device.

Unfortunately, the Kobo e-readers don't support OPDS out of the box: you need to setup some OPDS-compatible reader like Koreader. And that I found nearly impossible to do: I was able to setup KSM (the "start menu", not to be confused with that KSM), but not Koreader in KSM. Besides, I do not want a separate OS running here on the tablet: I just want to start Koreader every once in a while. KSM just starts another system when you reboot the e-reader, something which is not really convenient on the Kobo.

Basically, I just want to add Koreader as a tile in the home screen on the e-reader. I found the documentation on that topic to be sparse and hard to follow. It is often dispersed across multiple forum threads and involves uploading random binaries, often proprietary, to the e-reader. It had been a long time since I was asked to "register my software" frenetically, and I hadn't missed that one bit. So I decided to stay away from this until that solution and its documentation matures a bit.

Streaming re-established

I have worked a bit on my home radio stream. I simply turned the Liquidsoap stream back online, and did a few tweaks to the documentation that I had built back then. All that experimenting led me to do two NMUs. One was for gmpc-plugins to fix a FTBFS (bug #807735) and to properly kill the shout streamer when completing the playback (bug #820908).

The other was to fix the ezstream manpage (bug #573928), a patch that had been sitting there for 5 years! This was to try to find an easy way to stream random audio (say from a microphone) to the Icecast server, something which is surprisingly difficult, consider how basic that functionality is. I was surprised to see that Darkice just completely fails to start (bug #821040) and I had to fallback to the simplest ices2 software to stream the audio.

I am still having issues with Liquidsoap: it's really unstable! As a server, I would expect it to keep running for days if not years. Unfortunately, there's always something that makes it crash. I had assertion failed (bug #821112) and I keep seeing it crash after 2-3 days fairly reliably, a bug I reported 3 years ago and that is still not fixed (bug #727307).

Switching back the stream to Vorbis (because I was having problems with the commandline mp3 players and ogg123 is much more lightweight) created another set of problems too, this time with the phone. It seems that Android cannot stream Vorbis at all, something that is even worse in Cyanogenmod... I also had to tweak my MPD config to make the Android client be able to load the larger playlists (see dmix buffer is full).

Android apps

So I have also done quite a bit of work again on my phone. I finally was told how to access from Termux from adb shell which is pretty cool because now I can start a screen on my phone and then, when I'm tired of tapping to type, I can just reconnect to it when I plug in a USB cable on my laptop. I sent a pull request to fix the documentation regarding that.

I also tried to see how my SMS and Signal situation could be improved. Right now, I have two different apps to do SMS on my phone: I use both Signal and the SMS client, because I do not have a contract or SIM card in my phone. Both work well independently, but it's somewhat annoying to have to switch between the two.

(In fact, I feel that Signal itself has an issue with how it depends on the network to send encrypted messages: I often have to "ping" people in clear text (and therefore in the other app) so that they connect to their data plan to fetch my "secure" signal messages...)

Anyways, I figured it would be nice if at least there would be a SMS fallback in Signal that would allow me to send regular text messages from signal through Voip.MS. That was dismissed really fast. Moxie even closed the conversation completely, something I had never experienced before, and doesn't feel exactly pleasant. The Voip.MS side was of course not impressed and basically shrugged it off because Signal was not receptive.

I also tried to clear up the Libresignal confusion: there are 3 different projects named "Libresignal", and I am not sure I figured out the right thing, even though the issue is now closed and there's a FAQ that is supposed to make all that clear. Nevertheless, I opened two distinct feature requests to try to steer the conversation into a more productive direction: GCM-less calls and GCM support. But I'm not sure neither will go anywhere.

In the meantime, I am using the official signal client, which I downloaded using gplaycli and which I keep up to date with Aptoide. Even though the reliability of that site is definitely questionable, it seems that once you have a trusted version, it is safe to upgrade, regardless of the source because there is a trust path between you and the developer.

I also filed a few issues with varying levels of success/response from the community:

Random background radiation

And then there's of course the monthly background noise of all the projects I happened to not only stumble on, but also file bugs or patches against:

Catégories: External Blogs

Call for speakers: Montréal-Python 58: Dramatics Chartreuse

Montreal Python - dim, 04/17/2016 - 23:00

Spring has arrived and Montreal Python just got started. We are holding our next event on Monday, May 9th and it is a great chance to catch up and get out of your winter hibernation. It is an opportunity to come present cause we are looking for speakers for talks of 30, 15 or 5 minutes.

For example, if you are using Python to deploy Docker services, doing Big Data or simply having fun at discovering new tricks that make your life easier, we want you on stage :)

We are also looking for people interested for presenting a module of the month or a module that you love !

To submit your talk, write us at


More informations coming soon


Monday, May 9th 2016

We’d like to thank our sponsors for their continued support:

  • UQÀM
  • Bénélux
  • Outbox
  • Savoir-faire Linux
Catégories: External Blogs

MTL NewTech Startup Demos & Networking + PyCon Contest

Montreal Python - lun, 04/11/2016 - 23:00

PyCon is partnering again with MTL NewTech and Montréal-Python this year to bring one lucky Montreal startup to PyCon at Portland, Oregon, to present alongside with Google, Facebook, Stripe, Heroku, Microsoft, Mozilla and many other technology companies.

If you are a startup that meets the requirements below, apply now by filling this form: With the following information: a) size of the team, b) age of the startup c) your use of Python.

Deadline for applications: May 5th 23h59. Announcement of the startups selected: Starting on May 5th. MTL NewTech Demo & announcement of the winner: May 10th Feel free to invite fellow startups

For more details about PyCon and the startup row in general, please head to the PyCon website at

Eligible companies must meet the following criteria:

  • Fewer than 15 employees, including founders

  • Less than two years old

  • Use Python somewhere in your startup: backend, front-end, testing, wherever

  • If selected, please confirm that you will staff your booth in the Expo Hall on your appointed day. We will try accommodate your preferences: Monday or Tuesday

  • No repeats. If you were on startup row in a previous year, please give another a startup a chance this year.


Catégories: External Blogs

My free software activities, march 2016

Anarcat - jeu, 03/31/2016 - 18:32
Debian Long Term Support (LTS)

This is my 4th month working on Debian LTS, started by Raphael Hertzog at Freexian. I spent half of the month away on a vacation so little work was done, especially since I tried to tackle rather large uploads like NSS and Xen. I also worked on the frontdesk shift last week.


That work mainly consisted of figuring out how to best help the security team with the last uploads to the Wheezy release. For those who don't know, Debian 7 Wheezy, or "oldstable", is going to be unsupported by the security team starting end of april, and the Debian 6 Squeeze (the previous LTS) is now unsupported. The PGP signatures on the archived release have started yielding expiration errors which can be ignored but that are really a strong reminder that it is really time to upgrade.

So the LTS team is now working towards backporting a few security issues from squeeze to wheezy, and this is what I focused on during triage work. I have identified the following high priority packages I will work on after I complete my work on the Xen and NSS packages (detailed below):

  • libidn
  • icu
  • phpmyadmin
  • tomcat6
  • optipng
  • srtp
  • dhcpcd
  • python-tornado
Updates to NSS and Xen

I have spent a lot of time testing and building packages for NSS and Xen. To be fair, Brian May did most of the work on the Xen packages, and I merely did some work to test the packages on Koumbit's infrastructure, something which I will continue doing in the next month.

For NSS, wheezy and jessie are in this weird state where patches were provided to the security team all the way back in November yet were never tested. Since then, yet more issues came up and I worked hard to review and port patches for those new security issues to wheezy.

I'll followup on both packages in the following month.

Other free software work Android

TL;DR: there's an even longer version of this with the step-by-step procedures and that I will update as time goes on in my wiki.

I somehow inherited an Android phone recently, on a loan from a friend because the phone broke one too many times and she got a new one from her provider. This phone is a HTC One S "Ville", which is fairly old, but good enough to play with and give me a mobile computing platform to listen to podcasts, play music, access maps and create GPS traces.

I was previously doing this with my N900, but that device is really showing its age: very little development is happening on it, the SDK is closed source and the device itself is fairly big, compared to the "Ville". Plus, the SIM card actually works on the Ville so, even though I do not have an actual contract with a cell phone provider (too expensive, too invasive on my privacy), I can still make emergency phone calls (911)!

Plus, since there is good wifi on the machine, I can use it to connect to the phone system with the built-in SIP client, send text messages through SMS (thanks to SMS support) or Jabber. I have also played around with LibreSignal, the free software replacement for Signal, which uses proprietary google services. Yes, the SMS app also uses GCM, but hopefully that can be fixed. (While I was writing this, another Debian Developer wrote a good review of Signal so I am happy to skip that step. Go read that.)

See also my apps list for a more complete list of the apps I have installed on the phone. I welcome recommendations on cool free software apps I should use!

I have replaced the stock firmware on the phone with Cyanogenmod 12.1, which was a fairly painful experience, partly because of the difficult ambiance on the #cyanogenmod channel on Freenode, where I had extreme experiences: a brave soul helped me through the first flashing process for around 2 hours, nicely holding my hand at every step. Other times, I have seen flames and obtuse comments from people being vulgar, brutal, obnoxious, if not sometimes downright homophobic and sexist from other users. It is clearly a community that needs to fix their attitude.

I have documented everything I could in details in this wiki page, in case others want to resuscitate their old phones, but also because I ended up reinstalling the freaking phone about 4 times, and was getting tired of forgetting how to do it every time.

I am somewhat fascinated by Android: here is the Linux-based device that should save us all from the proprietary Apple nightmares of fenced in gardens and censorship. Yet, public Android downloads are hidden behind license agreements, even though the code itself is free, which has led fellow Debian developers to work on making libre rebuilds of Androids to workaround this insanity. But worse: all phones are basically proprietary devices down to the core. You need custom firmware to be loaded on the machine for it to boot at all, from the bootloader all the way down to the GSM baseband and Wifi drivers. It is a minefield of closed source software, and trying to run free software on there is a bit of a delusion, especially since the baseband has so much power over the phone.

Still, I think it is really interesting to run free software on those machines, and help people that are stuck with cell phones get familiar with software freedom. It seems especially important to me to make Android developers aware of software freedom considering how many apps are available for free yet on which it is not possible to contribute significantly because the source code is not published at all, or published only on the Google Store, instead of the more open and public F-Droid repository which publishes only free software.

So I did contribute. This month, I am happy to announce that I contributed to the following free software projects on Android:

I have also reviewed the literature surrounding Telegram, a popular messaging app rival to Signal and Whatsapp. Oddly enough, my contributions to Wikipedia on that subject were promptly reverted which made me bring up the subject on the page's Talk page. This lead to an interesting response from the article's main editors which at least added that the "its security features have been contested by security researchers and cryptography experts".

Considering the history of Telegram, I would keep people away from it and direct people to use Signal instead, even though Signal has similar metadata issues, mostly because Telegram's lack of response to the security issues that were outlined by fellow security researchers. Both systems suffer from a lack of federation as well, which is a shame in this era of increasing centralization.

I am not sure I will put much more work in developing for Android for now. It seems like a fairly hostile platform to work on, and unless I have specific pain points I want to fix, it feels so much better to work on my own stuff in Debian.

Which brings me to my usual plethora of free software projects I got stuck in this month.

IRC projects

irssi-plugin-otr had a bunch of idiotic bugs lying around, and I had a patch that I hadn't submitted upstream from the Debian package, which needed a rebuild because the irssi version changed, which is a major annoyance. The version in sid is now a snapshot because upstream needs to make a new release but at least it should fix things for my friends running unstable and testing. Hopefully those silly uploads won't be necessary in the future.

That's for the client side. On the server side, I have worked on updating the Atheme-services package to the latest version, which actually failed because the upstream libmowgli is missing release tags, which means the Debian package for it is not up-to-date either. Still, it is nice to have a somewhat newer version, even though it is not the latest and some bugs were fixed.

I have also looked at making atheme reproducible but was surprised at the hostility of the upstream. In the end, it looks like they are still interested in patches, but they will be a little harder to deploy than for Charybdis, so this could take some time. Hopefully I will find time in the coming weeks to test the new atheme services daemon on the IRC network I operate.


I have also re-discovered Syncthing, a file synchronization software. Amazingly, I was having trouble transferring a single file between two phones. I couldn't use Bluetooth (not sure why), the "Wifi sharing" app was available only on one phone (and is proprietary, and has a limit of 4MB files), and everything else requires an account, the cloud, or cabling.

So. Just heading to f-droid, install syncthing, flash a few qr-codes around and voilà: files are copied over! Pretty amazing: the files were actually copied over the local network, using IPv6 link-local addresses, encryption and the DHT. Which is a real geeky way of saying it's completely fast, secure and fast.

Now, I found a few usability issues, so much that I wrote a whole usability story for the developers, which were really appreciative of my review. Some of the issues were already fixed, others were pretty minor. Syncthing has a great community, and it seems like a great project I encourage everyone to get familiar with.

Battery stats

The battery-status project I mentionned previously has been merged with the battery-stats project (yes, the names are almost the same, which is confusing) and so I had to do some work to fix my Python graph script, which was accepted upstream and will be part of Debian officially from now on, which is cool. The previous package was unofficial. I have also noticed that my battery has a significantly than when I wrote the script. Whereas it was basically full back then, it seems now it has lost almost 15% of its capacity in about 6 months. According to the calculations of the script:

this battery will reach end of life (5%) in 935 days, 19:07:58.336480, on 2018-10-23 12:06:07.270290

Which is, to be fair, a good life: it will somewhat work for about three more years.

Playlist, git-annex and MPD in Python

On top of my previously mentioned photos-import script, I have worked on two more small programs. One is called get-playlist and is an extension to git-annex to easily copy to the local git-annex repository all files present in a given M3U playlist. This is useful for me because my phone cannot possibly fit my whole MP3 collection, and I use playlists in GMPC to tag certain files, particularly the Favorites list which is populated by the "star" button in the UI.

I had a lot of fun writing this script. I started using elpy as an IDE in Emacs. (Notice how Emacs got a new webpage, which is a huge improvement was had been basically unchanged since the original version, now almost 20 years old, and probably written by RMS himself.) I wonder how I managed to stay away from Elpy for so long, as it glues together key components of Emacs in an elegant and functional bundle:

  • Company: the "vim-like" completion mode i had been waiting for forever
  • Jedi: context-sensitive autocompletion for Python
  • Flymake: to outline style and syntax errors (unfortunately not the more modern Flycheck)
  • inline documentation...

In short, it's amazing and makes everything so much easier to work with that I wrote another script. The first program wouldn't work very well because some songs in the playlists had been moved, so I made another program, this time to repair playlists which refer to missing files. The script is simply called fix-playlists, and can operate transparently on multiple playlists. It has a bunch of heuristics to find files and uses a MPD server as a directory to search into. It can edit files in place or just act as a filter.

Useful snippets

Writing so many scripts, so often, I figured I needed to stop wasting time always writing the same boilerplate stuff on top of every file, so I started publishing Yasnippet-compatible file snippets, in my snippets repository. For example, this report is based on the humble lts snippet. I also have a base license snippet which slaps the AGPLv3 license on top of a Python file.

But the most interesting snippet, for me, is this simple script snippet which is a basic scaffolding for a commandline script that includes argument processing, logging and filtering of files, something which I was always copy-pasting around.

Other projects

And finally, a list of interesting issues en vrac:

  • there's a new Bootstrap 4 theme for Ikiwiki. It was delivering content over CDNs, which is bad for privacy issues, so I filed an issue which was almost immediately fixed by the author!
  • I found out about BitHub, a tool to make payments with Bitcoins for patches submitted on OWS projects. It wasn't clear to me where to find the demo, but when I did, I quickly filed a PR to fix the documentation. Given the number of open PRs there and the lack of activity, I wonder if the model is working at all...
  • a fellow Debian Developer shared his photos workflow, which was interesting to me because I have my own peculiar script, which I never mentionned here, but which I ended up sharing with the author
Catégories: External Blogs

State of Mapping on the Debian Desktop

Anarcat - dim, 03/13/2016 - 11:19

TL;DR: Mapbox Studio classic is not as good as Tilemill, Mapbox studio is very promising, but you still need to signup for access, and kosmtik seems to be working right now. Use kosmtik and help getting it in Debian.

The requirement

I have been following the development of fascinating tools from the Development seed people, now seemingly all focused on the Mapbox brand. They created what seemed to me a revolutionary desktop tool called Tilemill. Tilemill allowed you to create custom map stylesheets on your desktop to outline specific areas or patterns or "things". My interest was to create outdoor maps - OSM has pretty good biking maps, but no generic outdoor tiles out there (I need stuff for skiing, canoeing, biking)...

Mapbox has Mapbox outdoors but that's a paid plan as well. Oh and there's a place to download Garmin data files especially made for outdoors as well, and that could be loaded in Qmapshack, but they don't have coverage in Canada at all. Besides, I would like to print maps, I know, crazy...

So I have been looking forward to seeing Tilemill packaged in Debian given how annoying it is to maintain Node apps (period). Unfortunately, by the time Debian people figured out all the Node dependencies, the Tilemill project stopped and has been stalled since 2012. It seems the Mapbox people have now been working on other products, and in the meantime, the community, scratching their heads, just switched to other projects.

Overview of alternatives

I wrote this long post to the Debian bugtrackers to try to untangle all of this, but figured this would be interesting to a wider community than the narrow group of people working on Javascript packages, so I figured I would send this on Debian planet.

So Here's a summary of what happened so far, after Tilemill development stopped. Hang on to your tiles boys and girls, there's a lot going on!

Mapbox Studio classic

Mapbox people have released a new product in September 2014 named Mapbox studio classic. The code is still freely available and seems to be a fork of tilemill. Mapbox classic still has releases on github, last one is from November 2015. It looks like Mapbox studio classic has some sort of lock-in, and there are certainly new copyright issues, if only with the bundled fonts, but it could probably be packaged after addressing those issues.

There is an ITP for Mapbox Studio classic as well.

Mapbox Studio

Then there's Mapbox Studio, which is a full rewrite of Mapbox Studio classic. You need to "signup" somehow to get access, even though parts of the code are free, namely the Mapbox GL studio project. It is an interesting project because it aims to make all this stuff happen in a web browser, which means it "should" work everywhere. Unfortunately for us, it means it doesn't work anywhere without a signup form, so that's out for me at least.

There is an [ITP for Mapbox-studio][] yet it is unclear to me what that one means because the source code to Mapbox-studio doesn't seem to be available, as far as i can tell (and the ITP doesn't say either).

That is actually the ITP for Mapbox studio classic.


The Openstreetmap-carto developers have mostly switched to kosmtik instead of Mapbox. Kosmtik is another Node desktop app that seems fairly lightweight and mostly based on plugins. Ross has an ITP for kosmtik. The package is waiting on other node dependencies to be uploaded (yes, again).

The future of Tilemill

And there's still this RFP for tilemill, which should probably be closed now because the project seems dead and plenty of alternatives exist. I wonder if node some dependencies that were packaged for Tilemill actually now need to be removed from Debian, because they have become useless leaf packages... I am leaving the Tilemill RFP open for someone to clean that up.


Oh, and finally one could mention another Mapbox project, Carto, a command line CSS tools that implements some sort of standard CSS language that all those tools end up using to talk to Mapnik, more or less. There are no RFPs for that.

Catégories: External Blogs

Trying out Keybase

Anarcat - jeu, 03/10/2016 - 14:37

I was privileged enough to get access to the alpha preview of Last year, the project raised 11M$ in venture capital and they seem to be attacking the hard problems, so it is a serious project, worth considering for any cryptography hacker like me. This was obviously discussed elsewhere previously but since Keybase has evolved a bit since then, I figured it would be worth adding my grain of salt.

The elite alpha problem

My first problem with Keybase was of course that I wasn't in the elite club of invited people. Somehow, I finally got in but the problem remains for the everyone else.

In my case, it was the Keybase filesystem announcement that brought my attention back to the project, although I have yet to test that itself - yet another invite-only problem (need to email to get access, I guess). Still, 10GB of storage might be interesting for some people, but considering the number of hurdles or "friction", as they say, that needs to be jumped over to get access, I am not sure how much traction this will get for now.

The project has been in alpha for a while now (more than a year) with no announced public beta just yet, which is way slower than similarly challenging projects with less capital. Let's encrypt, as a comparison, was publicly announced in 2014 and by the end of 2015, the public beta was opened.

Private key leakage

My second major concern is that Keybase does worrisome things to private key material. If you tell it to import your regular GPG keys, it will copy them into the keybase keyring, which is already a problem: now you have two places where you store sensitive secrets.

But worse, Keybase offers you to upload your private key material on the server, without clearly and explicitely exposing the potential risks that means to the user. The key, presumably, is encrypted (at least that is what I can parse from the Keybase privacy policy). Yet this is still a concern: a determined attacker (e.g. the government or another malicious entity) could force Keybase to give up the encrypted private key material and install sniffers that would reveal the passphrase that would allow decryption and impersonation of the related identity.

That is a huge deal. This is what caused Hushmail's demise in 2007: they had similar promises about how they had a "secure" email (in this case) service based on OpenPGP, but since they key was stored on the server, they only had to serve a malicious applet to victims and were able to give cleartext data to the US government (12 CDs!) under a Mutual Legal Assistance Treaty with Canada.

In Keybase, When I generate a new key, it asks me if I want to upload the key to their servers, and "Yes" is chosen as a default, so it is clearly a policy that is encouraged. This is presumably to enable browser-based crypto, yet this could be done by storing keys within the browser, not on the server.

The mere possibility that the client can meddle with secret key material in such a way is a problem. Even if uploading secret keys to the server is optional, the fact that it is a possibility widens the attack surface on the client significantly.

GPG has been working on isolating private key operations to a different process (gpg-agent) or smart cards, SSH has privilege seperation. The idea of uploading secret key material online is so beyond current practices that it should bring the project to a full stop already.

My private key material is pretty sensitive. I am a Debian developer: if someone gets access to that key, they can upload arbitrary code for thousands of packages, affecting thousands if not millions of users. That is a huge deal, and I am not ready to give that power to an arbitrary third party that I know little about.

Privacy policy issues

Furthermore, the Keybase privacy policy explicitely states that "Logs of this information may persist for an indefinite period", which include, and I quote:

  • Your computer’s IP address
  • Your preferences and settings (time zone, language, privacy preferences, application preferences, etc.)
  • The URL of the site that referred you to the Service
  • The buttons and controls you clicked on (if any) within the Service
  • How long you used the Service and which parts and features you used
  • Session times and lengths

For a company working to improve user's privacy, this is a pretty bad policy. It exposes Keybase users to undue surveillance, on a lot of private information that is not necessary to operate the service. I haven't actually read the remaining of the policy, as I was already scared as hell and figure I would just log off for now and try not to add any extra personal information in that already growing pile.

Let's mention, while I'm here, that the keybase commandline client has this odd way of working that it forks a daemon in the background all the time, even to run just keybase status. One has to wonder if this thing is included in the surveillance data above, which would make the keybase client an amazing surveillance device in itself. To stop all this, you need:

keybase logout keybase ctl stop

Not quite obvious...

Usability issues

Which brings me to a series of usability issues I have found while working with Keybase. Having worked on usability myself in the Monkeysphere project, I understand how hard those problems can be. But right now, the fact is that keybase is only a commandline client, with some web-based sugar sprinkled on top. I recognize the huge efforts that have been done to make the user experience (UX) as easy as possible through the commandline, but this is definitely not "grand public" material yet.

Even as a OpenPGP hacker, Keybase can get confusing. Since it has its own separate keyring from GPG, things that used to be obvious before are suddenly new things to learn. For example, I originally generated a key for in keybase, thinking it would be a separate thing from my current GPG identity. Well, that ended up creating a new key in my GPG keyring, duplicating my existing identity, with no way for me to tell this was a Keybase copy. This could have ended up on the keyservers and confuse anyone looking for my PGP key.

Revoking keys

So I revoked that key, which, in itself, is not such an obvious process either. I first had to "drop" the key:

keybase pgp drop '010143572d4e438f457a7447c9758804cc7be44c1ee2b7915c3904567d0a3fb5cf590a'

To find that magic number, I had to go on the website and click through the PGP key to end up on the revoke link which told me which magic string to put. The web UI was telling me I could use keybase status to get that information, but I couldn't find that in the output here:

$ keybase status Username: anarcat Logged in: no Device: name: angela ID: 070f6a13f2a66afbb463f49dadfd4518 status: active Keybase keys: unlocked Session: anarcat [salt only] KBFS: status: not running version: log: /home/anarcat/.cache/keybase/keybase.kbfs.log Service: status: running version: 1.0.14-1 log: /home/anarcat/.cache/keybase/keybase.service.log Platform Information: OS: linux Runtime: go1.5.3 Arch: amd64 Client: version: 1.0.14-1 Desktop app: status: not running version: log: /home/anarcat/.cache/keybase/ Config path: /home/anarcat/.config/keybase/config.json Default user: anarcat Other users: command-line client: keybase status [pid: 14562, version: 1.0.14-1]

So keybase drop "deletes" the key from the keybase client (and presumably the server). One problem here is that we have three different verbs to mean the same thing: in some places, the website says "deleted" (in the web graph), the commandline client uses "drop" and everyone else using PGP in the world uses "revoke". So what is it?

Also note that just dropping that key is not enough to eradicate that key: you still need to generate and import a revocation certificate on GPG's side as well, for that key. This makes sense, because you may have imported that key and you may not want to destroy it when you remove it from keybase, but there could be a little better guessing and UX here as well.

Password usage

Then another problem is that, when you signup on the commandline, it asks you to choose a passphrase. It's not clear at all from the UI what that passphrase is for. Is it to protect the private key material? To login the website? Both? It turns out it is actually to login to the website and the Keybase API. It could also be the key to the private key material, but I haven't quite figured that out yet.

Similarly, the "paper devices" are a little confusing to me as well. The registration process is fairly insistent that I need to write a series of secret words down on a piece of paper and put it my wallet. This is based on the idea that you can use that key to recover your account and setup new devices. However, it is not made clear at all how those credentials should be protected or used. What if I lose my wallet? What do I do then? Is it okay if i write it on the back of my laptop instead? I know it sounds like stupid questions to crypto geek, but it's definitely stuff people will do, and education, at all levels, is necessary here.

The existing web of trust

Finally, I found it surprisingly counter-intuitive to sign keys with keybase. I was assuming the whole point of the thing was to expand the web of trust, but it seems their mission lies somewhere else: there is no facility to sign keys directly in keybase. You sign statements on various social sites (Github, Twitter, Reddit...) but doing traditional key exchanges seems to be off limits.

My obvious use case was to sign my existing key material with my newly generated key, so that keybase would become another way of verifying my identity. Well, this is not possible with keybase directly - you still have to go through the regular:

gpg --sign-key DEADBEEF

Which is really unfortunate, because some craaaazy people like me insist on doing in person, actually secure key exchanges sometimes, and the tools to do this right now (mainly gpg, caff and monkeysign) really need a lot of love and help. Keybase could have helped a lot in pushing those initiatives forward and bring new ways of doing secure key exchanges, instead of only lowering the standards on cryptographic identity.

Usual proprietary startup rant

I can't help but finish by noticing that the whole basic problem with Keybase is more fundamental than a few usability issues, which can all be fixed fairly easily.

Keybase.IO is a web-based service that is entirely proprietary. The client is freely downloadable and free software - but the server side is not. This makes us rely on a central point of failure and the goodwill of those operators to not only keep the service running (for free?) indefinitely, but to protect us against all attackers, state-run or otherwise.

This is definitely against basic free software principles and the open architecture of the internet federation, well embodied in the Franklin street statement. Online services should be decentralized and open like the OpenPGP keyservers have always been, period.


I am curious to see where Keybase brings us, as a community. So far, I am concerned about centralization and devaluation of the web of trust as cryptographic system. I do not believe that trusting multiple corporate social networking sites bring much benefit to our security, although it does improve usability significantly and is certainly better than TOFU, so that part of the project is definitely interesting, especially since it allows leveraging classical internet protocols like DNS and HTTPS.

Key exchange is a critical problem that still hasn't properly been resolved in the cryptographic community. Most efforts at building communication tools (say like Signal or Telegram) mostly ignore the problem and expect people to read up strings of numbers or rely on synchronous communications (see the Axolotl rachet for Signal's take on it). Keybase's attempts at fixing this are great, but needs much more work to actually resolve the PKI issues in a significant way.

More fundamentally, the practice of storing keys on the servers should just stop: it is a definite no go for me, and a classic crypto mistake that has bitten way too many people. In my mind, it discredits the whole project. The point of OpenPGP is end to end encryption across devices: storing the keys on a server breaks that apart, and doesn't improve much on the existing HTTPS security systems already in place on the web.

It is especially a problem given the new waves of attacks on cryptography from western governments, from the UK to California... Until those issues are resolved, I can't get myself to recommend Keybase to anyone just yet.


Note to Keybase developers: I have considered filing issues regarding the above, but unfortunately, I was unable to filter through the gigantic github issues list for duplicates, and didn't have time to file detailed reports for everything. I apologize for bypassing those usual conventions.

If you want to try out Keybase to make your own opinion, I do have 4 invites I can give away, but I will do so only if you need to test specific issues. Otherwise, I note that there is a Request For Package for the Debian package to become official, that Debian developers (or Keybase developers!) may be interested in completing.

Catégories: External Blogs

Epic Lameness

Eric Dorland - lun, 09/01/2008 - 17:26 now supports OpenID. Hooray! I'd like to make a comment on a thread about the RTL8187se chip I've got in my new MSI Wind. So I go to sign in with OpenID and instead of signing me in it prompts me to create an account with a name, username and password for the account. Huh? I just want to post to their forum, I don't want to create an account (at least not explicitly, if they want to do it behind the scenes fine). Isn't the point of OpenID to not have to create accounts and particularly not have to create new usernames and passwords to access websites? I'm not impressed.
Catégories: External Blogs

Sentiment Sharing

Eric Dorland - lun, 08/11/2008 - 23:28
Biella, I am from there and I do agree. If I was still living there I would try to form a team and make a bid. Simon even made noises about organizing a bid at DebConfs past. I wish he would :)

But a DebConf in New York would be almost as good.
Catégories: External Blogs
Syndiquer le contenu