The OpenDataPlane (ODP) project is a founding initiative by the Linaro Networking Group to produce an open-source, cross-platform application programming interface (API) for the networking Software Defined Data Plane. Linaro Ltd. more>>
I like the idea of life hacking. more>>
HTTPS is a small island of security in this insecure world, and in this day and age, there is absolutely no reason not to have it on every Web site you host. Up until last year, there was just a single last excuse: purchasing certificates was kind of pricey. more>>
I have worked on the following packages and CVEs:
- tre: CVE-2016-8859
- graphicsmagick: CVE-2016-7448, CVE-2016-7996, CVE-2016-7997, CVE-2016-8682, CVE-2016-8683, CVE-2016-8684
- tar: CVE-2016-6321
I have also helped review work on the following packages:
- imagemagick: reviewed BenH's work to figure out what was done. unfortunately, I forgot to officially take on the package and Roberto started working on it in the meantime. I nevertheless took time to review Roberto's work and outline possible issues with the original patchset suggested
- tiff: reviewed Raphael's work on the hairy TIFFTAG_* issues, all the gory details in this email
The work on ImageMagick and GraphicsMagick was particularly intriguing. Looking at the source of those programs makes me wonder why were are still using them at all: it's a tangled mess of C code that is bound to bring up more and more vulnerabilities, time after time. It seems there's always an "Magick" vulnerability waiting to be fixed out there... I somehow hoped that the fork would bring more stability and reliability, but it seems they are suffering from similar issues because, fundamentally, they haven't rewritten ImageMagick...
It looks this is something that affects all image programs. The review I have done on the tiff suite give me the same shivering sensation as reviewing the "Magick" code. It feels like all image libraries are poorly implemented and then bound to be exploited somehow... Nevertheless, if I had to use a library of the sort in my software, I would stay away from the "Magick" forks and try something like imlib2 first...
Finally, I also did some minor work on the user and developer LTS documentation and some triage work on samba, xen and libass. I also looked at the dreaded CVE-2016-7117 vulnerability in the Linux kernel to verify its impact on wheezy users. I also looked at implementing a --lts flag for dch (see bug #762715).
It was difficult to get back to work after such a long pause, but I am happy I was able to contribute a significant number of hours. It's a bit difficult to find work sometimes in LTS-land, even if there's actually always a lot of work to be done. For example, I used to be one of the people doing frontdesk work, but those duties are now assigned until the end of the year, so it's unlikely I will be doing any of that for the forseable future. Similarly, a lot of packages were assigned when I started looking at the available packages. There was an interesting discussion on the internal mailing list regarding unlocking package ownership, because some people had packages locked for weeks, sometimes months, without significant activity. Hopefully that situation will improve after that discussion.
Another interesting discussion I participated in is the question of whether the LTS team should be waiting for unstable to be fixed before publishing fixes in oldstable. It seems the consensus right now is that it shouldn't be mandatory to fix issues in unstable before we fix security isssues in oldstable and stable. After all, security support for testing and unstable is limited. But I was happy to learn that working on brand new patches is part of our mandate as part of the LTS work. I did work on such a patch for tar which ended up being adopted by the original reporter, although upstream ended up implementing our recommendation in a better way.
It's coincidentally the first time since I start working on LTS that I didn't get the number of requested hours, which means that there are more people working on LTS. That is a good thing, but I am worried it may also mean people are more spread out and less capable of focusing for longer periods of time on more difficult problems. It also means that the team is growing faster than the funding, which is unfortunate: now is a good time as any to remind you to see if you can make your company fund the LTS project if you are still running Debian wheezy.Other free software work
It seems like forever that I did such a report, and while I was on vacation, a lot has happened since the last report.Monkeysign
I have done extensive work on Monkeysign, trying to bring it kicking and screaming in the new world of GnuPG 2.1. This was the objective of the 2.1 release, which collected about two years of work and patches, including arbitrary MUA support (e.g. Thunderbird), config files support, and a release on PyPI. I have had to release about 4 more releases to try and fix the build chain, ship the test suite with the program and have a primitive preferences panel. The 2.2 release also finally features Tor suport!
I am also happy to have moved more documentation to Read the docs, part of which I mentionned in in a previous article. The git repositories and issues were also moved to a Gitlab instance which will hopefully improve the collaboration workflow, although we still have issues in streamlining the merge request workflow.
All in all, I am happy to be working on Monkeysign, but it has been a frustrating experience. In the last years, I have been maintaining the project largely on my own: although there are about 20 contributors in Monkeysign, I have committed over 90% of the commits in the code. New contributors recently showed up, and I hope this will release some pressure on me being the sole maintainer, but I am not sure how viable the project is.Funding free software work
More and more, I wonder how to sustain my contributions to free software. As a previous article has shown, I work a lot on the computer, even when I am not on a full-time job. Monkeysign has been a significant time drain in the last months, and I have done this work on a completely volunteer basis. I wouldn't mind so much except that there is a lot of work I do on a volunteer basis. This means that I sometimes must prioritize paid consulting work, at the expense of those volunteer projects. While most of my paid work usually revolves around free sofware, the benefits of paid work are not always immediately obvious, as the primary objective is to deliver to the customer, and the community as a whole is somewhat of a side-effect.
I have watched with interest joeyh's adventures into crowdfunding which seems to be working pretty well for him. Unfortunately, I cannot claim the incredible (and well-deserved) reputation Joey has, and even if I could, I can't live with 500$ a month.
I would love to hear if people would be interested in funding my work in such a way. I am hesitant in launching a crowdfunding campaign because it is difficult to identify what exactly I am working on from one month to the next. Looking back at earlier reports shows that I am all over the place: one month I'll work on a Perl Wiki (Ikiwiki), the next one I'll be hacking at a multimedia home cinema (Kodi). I can hardly think of how to fund those things short of "just give me money to work on anything I feel like", which I can hardly ask for of anyone. Even worse, it feels like the audience here is either friends or colleagues. It would make little sense for me to seek funding from those people: colleagues have the same funding problems I do, and I don't want to empoverish my friends...
So far I have taken the approach of trying to get funding for work I am doing, bit by bit. For example, I have recently been told that LWN actually pays for contributed articles and have started running articles by them before publishing them here. This is looking good: they will publish an article I wrote about the Omnia router I have recently received. I give them exclusive rights on the article for two weeks, but I otherwise retain full ownership over the article and will publish them after the exclusive period here.
Hopefully, I will be able to find more such projects that pays for the work I do on a day to day basis.Open Street Map editing
I have ramped up my OpenStreetMap contributions, having (temporarily) moved to a different location. There are lots of things to map here: trails, gaz stations and lots of other things are missing from the map. Sometimes the effort looks a bit ridiculous, reminding me of my early days of editing OSM. I have registered to OSM Live, a project to fund OSM editors that, I must admit, doesn't help much with funding my work: with the hundreds of edits I did in October, I received the equivalent of 1.80$CAD in Bitcoins. This may be the lowest hourly salary I have ever received, probably going at a rate of 10¢ per hour!
Still, it's interesting to be able to point people to the project if someone wants to contribute to OSM mappers. But mappers should have no illusions about getting a decent salary from this effort, I am sorry to say.Bounties
I feel this is similar to the "bounty" model used by the Borg project: I claimed around $80USD in that project for what probably amounts to tens of hours of work, yet another salary that would qualify as "poor".
Another example is a feature I would like to implement in Borg: support for protocols other than SSH. There is currently no bounty on this, but a similar feature, S3 support has one of the largest bounties Borg has ever seen: $225USD. And the claimant for the bounty hasn't actually implemented the feature, instead backing up to S3, the patch (to a third-party tool) actually enables support for Amazon Cloud Drive, a completely different API.
Even at $225, I wouldn't be able to complete any of those features and get a decent salary. As well explained by the Snowdrift reviews, bounties just don't work at all... The ludicrous 10% fee charged by Bountysource made sure I would never do business with them ever again anyways.Other work
- fixed a bug in python's setuptools_scm that was a blocker for Monkeysign's use of the package to avoid duplicating version numbers everywhere...
- tried to figure out how activitypub was going to work with existing social networking software (TL;DR: it won't)
- improved the sicherboot README so that others that come after me will better understand how that secure boot system works
- more charybdis IRCd work: SIGABRT on jessie and another segfault, still more issues pending here, even
- arbitrary source URL support for the Sphinx Alabaster theme
- a bunch of issues on unattended-upgrades: raspbian configuration fails and point releases upgrades
There are probably more things I did recently, but I am having difficulty keeping track of the last 5 months of on and off work, so you will forgive that I am not as exhaustive as I usually am.
"Does the world need a new computer mouse?" asks Penclic. "Yes it does!" says the Swedish peripherals developer. Most devices in our lives have undergone extensive changes through the years, notes Penclic, save the unlucky, unglamorous computer mouse. more>>
Wired magazine likes the Vivaldi web browser, calling it a tool for power users just like "500-pound squats are to power lifters". Led by a founder of the Opera browser, Vivaldi Technologies' browser eschews the pared-down base browser plus extensions model for one in which personalization rules. more>>
I've looked at general astronomy programs in the past that are helpful for many tasks you might need to do in your stargazing career. But, several specific jobs are more complicated and require specialized software to make relevant calculations, so here, let's take a look at Nightfall. more>>
In recent years, hardware virtualization has become commonplace in the computing industry and more available to end users. The idea behind it is a noble one. Why invest in allocating more server hardware and not utilize it to its full potential, when instead you can consolidate it all onto one or a few servers and share their resources? more>>
After an amazing edition of our presentation nights at Ubisoft, last month, we follow up with Montréal-Python 60: Gravitational Hipster on Tuesday, November 1st at WeWork.
It is a great opportunity to get a preview of what to expect at PyCon Canada, which is taking place in Toronto from the 12th through to the 15th of November.
For the occasion, we've invited speakers from Montreal who are going to present at Toronto to come to test-drive their talk. This edition Montreal-Python is gonna presented by Immunio.Where
WeWork - 3 Place Ville-Marie, 4th floorWhen
November 1st, 2016Schedule
- 6:00 - Doors Open
- 6:30 - Start of the presentations
- 7:30 - Break
- 7:45 - Part 2 of the presentations
- 9:00 - End of presentations. Drinks afterward.
Pandas is a powerful data analysis library that rivals R among data scientists. It's also gaining popularity in journalism. Pandas can: - Read multiple data formats, including CSV, Excel, SQL, JSON, HTML, Stata, HDF, and others - Reshape and pivot tables in flexible ways - Filter data - Perform quick aggregations on categorical data - Visualize data so you can quickly see what it looks like In this talk, Roberto will show how easy it is to get started and find interesting stories in big, complex datasets.
Serverless architecture is a hot topic: It promises low operational costs, infinite scalability, less management. In reality, it strongly challenges the way you design, implement and deploy your application. Moreover, you have to choose amongst a variety of vendors providing function as a service platform (FaaS) with highly opinionated and different APIs. This talk aims to release the pain by presenting a summary of nice idioms and practises with python examples based on AWS Lambdas.
You finish what you think is the best REST API anyone has ever seen. Proud of your work, you decide to post it on the internet and collect the appraisal of you peers. To your surprise, your API receives comments like "This is not REST! This is simply JSON over HTTP!" or "Oh my god, this is only a level 2 REST API!". It's time to learn how to become a REST elitist and fight back.
Shopify strives to be an ever moving company. With a strong force of hundreds in the R&D team, the challenges to keep delivering and shipping scaled accordingly.
From a manual CLI that only ops could execute just a few years back, to today’s automatic 180+ deploys a day, this talk will highlight the different tools, processes and integrations used throughout the team that fuels today's scale.
Nowadays, everybody uses version control. But until you learn to misuse your version control system, you're missing out on ways to minimize developer productivity and pessimize your workflow. I'll show you 12 time-tested worst practices that will set you down the wrong path from Day One.
We would like to thank our sponsors for their constant support
- Savoir-faire Linux
I've stated for years how much I dislike Ubuntu's Unity interface. Yes, it's become more polished through the years, but it's just not an interface that thinks the same way I do. That's likely because I'm old and inflexible, but nevertheless, I've done everything I could to avoid using Unity, which usually means switching to Xubuntu. more>>
I do a lot of my day-job work on a Windows computer. Part of this involves taking screenshots for training purposes. For years, I've used the built-in "Snipping Tool" that comes with Windows, but I've always hated it. more>>
Dries Buytaert announced today that Nasdaq Corporate Solutions has selected Drupal 8 and will work with Acquia to create its Investor Relations Website Platform. In the words of Angela Byron, a.k.a "Webchick", "This is a big freakin' deal." more>>
Canonical Ltd.'s "Snappy" Ubuntu Core, a stripped-down version of Ubuntu designed for autonomous machines, devices and other internet-connected digital things, has gained significant traction in the chipset/semiconductor market recently. more>>
During the past few years, my BirdCam setup has evolved significantly. As I mention in the UpFront section of this issue, I hope to get the stream transferred to a YouTube Live stream at some point, so I can watch the feathery show on my television. And although watching the birds is the end goal, I'm constantly on a mission to improve the quality and flexibility of my setup. more>>
The metachallenge in today's data-saturated world is turning Big Data into actionable insight. A straight line to faster insights can be found in Netlist, Inc.'s new HybriDIMM Storage Class Memory (SCM), which the company describes as the industry's first standards-based, plug-and-play SCM solution. more>>
Bug reporting is an art form that is too often neglected in software projects. Bug reports allow contributors to participate without deep technical knowledge and at the same time provide a crucial space for developers to be made aware of issues with their software that they could not have foreseen or found themselves, for lack of resources, variety or imagination.Prior art
Unfortunately, there are rarely good guidelines for submitting bug reports. Historically, people have pointed towards How to report bugs effectively or How to ask questions the smart way. While those guides can be useful for motivated people and may seem attractive references for project managers, they suffer from serious issues:
- they are written by technical people, for non-technical people
- as a result, they have a deeply condescending attitude such as calling people "stupid" or various animal names like "mongoose"
- they are also very technical themselves: one starts with a copyright notice and a changelog, the other uses magic words like "Core dumps" and $Id$
- they are too long: sgtatham's is about 3600 words long, esr's is even longer at about 11800 words. those texts will take about 20 to 60 minutes to read by an average reader, according to research
Individual projects have their own guides as well. Linux has the REPORTING_BUGS file that is a much shorter 1200 that can be read under 5 minutes, provided that you can understand the topic at hand. Interestingly, that guide refers to both esr's and sgtatham's guidelines, which means, in the degenerate case where the user hasn't had the "privilege" of reading esr's prose already, they will have an extra hour and a half of reading to do to have honestly followed the guidelines before reporting the bug.
I often find good documentation in the Tails project. Their bug reporting guidelines are easily accessible and quick to read, although they still might be too technical. It could be argued that you need to get technical at some point to get that information out, of course.
In the Monkeysign project, I have started a bug reporting guide that doesn't yet address all those issues. I am considering writing a new guide, but I figured I would look at other people's work and get feedback before writing my own standard.What's the point?
Why have those documents been written? Are people really expected to read them before seeking help? It seems to me unlikely that someone would:
- be motivated enough to do something about a broken part of their computer
- figure out they can do something about it
- read a fifteen thousand words novel about how to report a bug...
- just to finally write a 20-line bug report that has no warranty of support attached to it
And if I would be a paying customer, I wouldn't want to be forced to waste my time reading that prose either: it's your job to help me fix your broken things, not the reverse. As someone doing consulting these days: I totally understand: it's not you, the user, it's us, the developers, that have a problem. We have been socialized through computers, and it makes us weird and obtuse, but that's no excuse, and we need to clean up our act.
Furthermore, it's surprising how often we get (and make!) bug reports that can be difficult to use. The Monkeysign project is very "technical" and I have expected that the bug reports I would get would be well written, with ways to reproduce and so on, but it happened that I received bug reports that were all over the place, didn't have any ways of reproducing or were simply incomplete. Those three bug reports were filed by people that I know to be very technically capable: one is a fellow Debian developer, the second had filed a good bug report 5 days before, and the third one is a contributor that sent good patches before.
In all three cases, they knew what they were doing. Those three people probably read the guidelines mentioned in the past. They may have even read the Monkeysign bug reporting guidelines as well. I can only explain those bug reports by the lack of time: people thought the issue was obvious, that it would get fixed rapidly because, obviously, something is broken.
We need a better way.The takeaway
What are those guides trying to tell us?
- ask questions in the right place
- search for similar questions and issues before reporting the bug
- try to make the developers reproduce the issues
- failing that, try to describe the issue as well as you can
- write clearly, be specific and verbose yet concise
There are obviously contradictions in there, like sgtatham telling us to be verbose and esr tells us to, basically, not be verbose. There is definitely a tension in there, and there are many, many more details about how great bug reports can be if done properly.
I tend towards the side of terseness in our descriptions: people that will know how to be concise will be, people that don't will most likely not learn by reading a 12 000 words novel that, in itself, didn't manage to be parsimonious.
But I am willing to allow for verbosity in bug reports: I prefer too many details instead of missing a key bit of information.Issue trackers
Step 1 is our job: we should send people in the right place, and give them the right tools. Monkeysign used to manage bugs with bugs-everywhere and this turned out to be a terrible idea: you had to understand git and bugs-everywhere to file any bug reports. As a result, there were exactly zero bug reports filed by non-developers during the whole time BE was used, although some bugs were filed in the Debian Bugtracker.
So have a good bug tracker. A mailing list or email address is not a good bug tracker: you lose track of old issues, and it's hard for newcomers to search the archives. It does have the advantage of having a unified interface for the support forum and bug tracking, however.
Redmine, Gitlab, Github and others are all decent-enough bug trackers. The key point is that the issue tracker should be publicly available, and users should be able to register easily to file new issues. You should also be able to mass-edit tickets and users should be able to discover the tracker's features easily. I am sorry to say that the Debian BTS somewhat falls short on those two features.
Step 2 is a shared responsibility: there should be an easy way to search for issues, and we should help the user looking for similar issues. Stackexchange sites do an excellent job at this, by automatically searching for similar questions while you write your question, suggesting similar ones in an attempt to weed out duplicates. Duplicates still happen, but they can then clearly be marked and linked with a distinct mechanism. Most bug trackers do not offer such high level functionality, but should, so I feel the fault lies more on "our" end than at the user's end.Reproducing the environment
Step 3 and 4 are more or less the user's responsibility. We can detail in our documentation how to clearly share the environment where we reproduced the bug, for example, but in the end, the user decides if they want to share that information or not.
In Monkeysign, I have finally implemented joeyh's suggestion of shipping the test suite with the program. I can now tell people to run the test suite in their environment to see if this is a regression that is specific to their environment - so a known bug, in a way - or a novel bug for which I can look at writing a new unit test. I also include way more information about the environment in the --version output, an idea I brought forward in the borg project to ease debugging. That way, people can just send the output of monkeysign --test and monkeysign --version, and I have a very good overview of what is happening on their end. Of course, Monkeysign also supports the usual --verbose and --debug flag that users should enable when reproducing issues.
Another idea is to report bugs directly from the application. We have all seen Firefox or other software have automatic bug reporting tools, but somehow those seem unsatisfactory for a user: we have no feedback of where the report goes, if it's followed up on. It is useful for larger project to get statistical data, but not so useful for users in the short term.
Monkeysign tries to handle exceptions in the code in a graceful way, but could do better. We use a small library to handle exceptions, but that library has since then been improved to directly file bugs against the Github project. This assumes the user is logged into Github, but it is nice to pre-populate bug reports with the relevant information up front.Issue templates
In the meantime, to make sure people provide enough information, I have now moved a lot of the bug reporting guidelines to a separate issue template. That issue template is available through the issue creation form now, although it is not enabled by default, a weird limitation of Gitlab. Issue templates are available in Gitlab and Github.
Issue templates somewhat force users in a straight jacket: there is already something to structure their bug report. Those could be distinct form elements that had to be filled in, but I like the flexibility of the template, and the possibility for users to just escape the formalism and just plead for help in their own way.Issue guidelines
In the end, I opted for a short few paragraphs in the style of the Tails documentation, including a reference to sgtatham, as an optional future reference:
The first aim of a bug report is to tell the developers exactly how to reproduce the failure, so try to reproduce the issue yourself and describe how you did that.
If that is not possible, try to describe what went wrong in detail. Write down the error messages, especially if they have numbers.
Take the necessary time to write clearly and precisely. Say what you mean, and make sure it cannot be misinterpreted.
Include the output of monkeysign --test, monkeysign --version and monkeysign --debug in your bug reports. See the issue template for more details about what to include in bug reports.
If you wish to read more about issues regarding communication in bug reports, you can read How to Report Bugs Effectively, which takes around 20 to 30 minutes.
In the end, there is a fundamental issue with reporting bugs: it assumes our users are literate and capable of writing amazing prose that we will enjoy reading as the last J.K. Rowling novel (if you're into that kind of thing). It's just an unreasonable expectation: some of your users don't even speak the same language as you, let alone read or write it. This makes for challenging collaboration, to say the least. This is where automated reporting makes sense: it doesn't require user's intervention, and the communication is mediated by machines without human intervention and their pesky culture.
But we should, as maintainers, "be liberal in what we accept and conservative in what we send". Be tolerant, and help your users in fixing their issues. It's what you are there for, after all.
And in the end, we all fail the same way. In an attempt to improve the situation on bug reporting guides, I seem to have myself written a 2000 short story that will have taken up a hopefully pleasant 10 minutes of your time at minimum. Hopefully I will have succeeded at being clear, specific, verbose and concise all at once and look forward to your feedback on how to improve our bug reporting culture.