Skip to main content

External Blogs

The future of Python Nights

Montreal Python - Tue, 12/02/2014 - 00:00

After more then a dozen python nights, I'm glad to announce the success of the event.

Categories: External Blogs

bup vs attic silly benchmark

Anarcat - Tue, 11/18/2014 - 00:39

after see attic introduced in a discussion about bup, i figured out i could give it a try. it was answering two of my biggest concerns with bup:

  • backup removal
  • encryption

and seemed to magically out of nowhere and basically do everything i need, with an inline manual on top of it.

disclaimer

Note: this is not a real benchmark! i would probably need to port bup and attic to liw's seivot software to report on this properly (and that would amazing and really interesting, but it's late now). even worse, this was done on a production server with other stuff going on so take results with a grain of salt.

procedure and results

Here's what I did. I setup backups of my ridiculously huge ~/src directory on the external hard drive where I usually make my backups. I ran a clean backup with attic, than redid it, then I ran a similar backup with bup, then redid it. Here are the results:

anarcat@marcos:~$ sudo apt-get install attic # this installed 0.13 on debian jessie amd64 [...] anarcat@marcos:~$ attic init /mnt/attic-test: Initializing repository at "/media/anarcat/calyx/attic-test" Encryption NOT enabled. Use the "--encryption=passphrase|keyfile" to enable encryption. anarcat@marcos:~$ time attic create --stats /mnt/attic-test::src ~/src/ Initializing cache... ------------------------------------------------------------------------------ Archive name: src Archive fingerprint: 7bdcea8a101dc233d7c122e3f69e67e5b03dbb62596d0b70f5b0759d446d9ed0 Start time: Tue Nov 18 00:42:52 2014 End time: Tue Nov 18 00:54:00 2014 Duration: 11 minutes 8.26 seconds Number of files: 283910 Original size Compressed size Deduplicated size This archive: 6.74 GB 4.27 GB 2.99 GB All archives: 6.74 GB 4.27 GB 2.99 GB ------------------------------------------------------------------------------ 311.60user 68.28system 11:08.49elapsed 56%CPU (0avgtext+0avgdata 122824maxresident)k 15279400inputs+6788816outputs (0major+3258848minor)pagefaults 0swaps anarcat@marcos:~$ time attic create --stats /mnt/attic-test::src-2014-11-18 ~/src/ ------------------------------------------------------------------------------ Archive name: src-2014-11-18 Archive fingerprint: be840f1a49b1deb76aea1cb667d812511943cfb7fee67f0dddc57368bd61c4bf Start time: Tue Nov 18 00:05:57 2014 End time: Tue Nov 18 00:06:35 2014 Duration: 38.15 seconds Number of files: 283910 Original size Compressed size Deduplicated size This archive: 6.74 GB 4.27 GB 116.63 kB All archives: 13.47 GB 8.54 GB 3.00 GB ------------------------------------------------------------------------------ 30.60user 4.66system 0:38.38elapsed 91%CPU (0avgtext+0avgdata 104688maxresident)k 18264inputs+258696outputs (0major+36892minor)pagefaults 0swaps anarcat@marcos:~$ sudo apt-get install bup # this installed bup 0.25 anarcat@marcos:~$ free && sync && echo 3 | sudo tee /proc/sys/vm/drop_caches && free # flush caches anarcat@marcos:~$ export BUP_DIR=/mnt/bup-test anarcat@marcos:~$ bup init Dépôt Git vide initialisé dans /mnt/bup-test/ anarcat@marcos:~$ time bup index ~/src Indexing: 345249, done. 56.57user 14.37system 1:45.29elapsed 67%CPU (0avgtext+0avgdata 85236maxresident)k 699920inputs+104624outputs (4major+25970minor)pagefaults 0swaps anarcat@marcos:~$ time bup save -n src ~/src Reading index: 345249, done. bloom: creating from 1 file (200000 objects). bloom: adding 1 file (200000 objects). bloom: creating from 3 files (600000 objects). Saving: 100.00% (6749592/6749592k, 345249/345249 files), done. bloom: adding 1 file (126005 objects). 383.08user 61.37system 10:52.68elapsed 68%CPU (0avgtext+0avgdata 194256maxresident)k 14638104inputs+5944384outputs (50major+299868minor)pagefaults 0swaps anarcat@marcos:attic$ time bup index ~/src Indexing: 345249, done. 56.13user 13.08system 1:38.65elapsed 70%CPU (0avgtext+0avgdata 133848maxresident)k 806144inputs+104824outputs (137major+38463minor)pagefaults 0swaps anarcat@marcos:attic$ time bup save -n src2 ~/src Reading index: 1, done. Saving: 100.00% (0/0k, 1/1 files), done. bloom: adding 1 file (1 object). 0.22user 0.05system 0:00.66elapsed 42%CPU (0avgtext+0avgdata 17088maxresident)k 10088inputs+88outputs (39major+15194minor)pagefaults 0swaps

Disk usage is comparable:

anarcat@marcos:attic$ du -sc /mnt/*attic* 2943532K /mnt/attic-test 2969544K /mnt/bup-test

People are encouraged to try and reproduce those results, which should be fairly trivial.

Observations

Here are interesting things I noted while working with both tools:

  • attic is Python3: i could compile it, with dependencies, by doing apt-get build-dep attic and running setup.py - i could also install it with pip if i needed to (but i didn't)
  • bup is Python 2, and has a scary makefile
  • both have an init command that basically does almost nothing and takes little enough time that i'm ignoring it in the benchmarks
  • attic backups are a single command, bup requires me to know that i first want to index and then save, which is a little confusing
  • bup has nice progress information, especially during save (because when it loaded the index, it knew how much was remaining) - just because of that, bup "feels" faster
  • bup, however, lets me know about its deep internals (like now i know it uses a bloom filter) which is probably barely understandable by most people
  • on the contrary, attic gives me useful information about the size of my backups, including the size of the current increment
  • it is not possible to get that information from bup, even after the fact - you need to du before and after the backup
  • attic modifies the files access times when backing up, while bup is more careful (there's a pull request to fix this in attic, which is how i found out about this)
  • both backup systems seem to produce roughly the same data size from the same input
Summary

attic and bup are about equally fast. bup took 30 seconds less than attic to save the files, but that's not counting the 1m45s it took indexing them, so on the total run time, bup was actually slower. attic is also (almost) two times faster on the second run as well. but this could be within the margin of error of this very quick experiment, so my provisional verdict for now would be that they are about as fast.

bup may be more robust (for example it doesn't modify the atimes), but this has not been extensively tested and is more based with my familiarity with the "conservatism" of the bup team rather than actual tests.

considering all the features promised by attic, it makes for a really serious contender to the already amazing bup.

Next steps

The properly do this, we would need to:

  • include other software (thinking of Zbackup, Burp, ddar, obnam, rdiff-backup and duplicity)
  • bench attic with the noatime patch
  • bench dev attic vs dev bup
  • bench data removal
  • bench encryption
  • test data recovery
  • run multiple backup runs, on different datasets, on a cleaner environment
  • ideally, extend seivot to do all of that

Note that the Burp author already did an impressive comparative benchmark of a bunch of those tools for the burp2 design paper, but it unfortunately doesn't include attic or clear ways to reproduce the results.

Categories: External Blogs

Montréal-Python 50: Monochromatic Necessity

Montreal Python - Mon, 11/17/2014 - 00:00

**UPDATE** Due to a scheluding mix-up our event will be at UQAM

For our 50th edition of our Monthly meetups we wanted to offer you an amazing evening with an amazing lineup. That's why this time we have the chance to be welcome at Notman Housse by our friends from Real Ventures.

We would like to thanks all the sponsors and Datacratic to support us for this special evening.

In the mean time, don't forget to join us on Tuesday for our http://montrealpython.org/en/2014/11/python-night-xiii/

And also don't forget, if you have anything you would like to present or announce, just show up and tell us, we are always happy to give a chance to anyone !

Flash Presentations (5-10 minutes each):

George Peristerakis: How openstack automated the software development process.

(http://enovance.com)

The Openstack system is a collection of open source services that helps you setup a cloud infrastructure. It is also one of the largest and most active code base. I will talk about how openstack automated most of its development process.

David Taylor: Top 10 'Python idioms it took me way too long to figure out

(http://www.prooffreader.com/)

Since I adopted Python after learning programming in Visual Basic, I gradually found out features that, had I known them before, would have saved me from having to reinvent the wheel, like collections.Counter, dict.get(), list.enumerate(), etc.

Slides: http://prooffreaderplus.blogspot.ca/2014/11/top-10-python-idioms-i-wished-id.html

Main Presentations:

Nicolas Kruchten: Data Science and Machine Learning with PyData

(http://datacratic.com/)

Nicolas will walk through a demonstration of ways to explore, visualize and extract insights from data using PyData tools like IPython Notebook, numpy, pandas and scikit-learn.

Slides: http://opensource.datacratic.com/mtlpy50/slides.html

Sources: https://github.com/datacratic/mtlpy50

Video: https://www.youtube.com/watch?v=2lpS6gUwiJQ

Stéphane Guidoin: CKAN plateforme de données ouvertes en python: forces et défis

(http://nordouvert.ca/)

Lancé par l'Open Knowledge Foundation et développé en Python, CKAN a vocation a servir de portail de données ouvertes, notamment pour les gouvernements. CKAN a acquis une position dominante, surtout auprès des gouvernements nationaux. Malgré cette position, le développement de l'outil n'est pas aussi rapide que beaucoup souhaiteraient et plateformes comme Socrata demeurent très présentes, notamment auprès des municipalités, malgré leur prix nettement supérieur. La présentation couvrira l'architecture technologique de CKAN, ce qui lui a permis de devenir une solution dominante, mais aussi les défis de croissance auxquels fait face ce projet.

Cameron Davidson-Pilon: What is PySpark and When Should I Use it?

(http://www.shopify.ca/)

Spark is being called the next generation of Hadoop: it's faster, more accessible and has a large community behind it. PySpark is a Python interface to Spark. In this talk, we'll discuss what MapReduce is, PySparks API, and when to use PySpark versus another tool.

When:

Monday, the December 1st 2014

Where:

UQÀM, Pavillion PK

201, avenue du Président-Kennedy

Room PK-1140

Comment

It's free, just join us !

Schedule:
  • 6:00pm — Doors open
  • 6:30pm — Presentations start
  • 7:30pm — Break
  • 7:45pm — Second round of presentations
  • 9:00pm — One free beer offered at Bénélux just across the street
We’d like to thank our sponsors for their continuous support:
  • Datacratic
  • Real Ventures
  • UQÀM
  • Bénélux
  • w.illi.am/
  • Outbox
  • Savoir-Faire Linux
  • Caravan
  • iWeb
Categories: External Blogs

Project Night XIII

Montreal Python - Tue, 11/11/2014 - 00:00

Welcome to Python Night XIII (hosted by Caravan)

After a long absence, Python Night is back and we are inviting everyone to join us in a night of coding, laughs and all out good time.

What? You don't already know what a Python Night is?

Well, it's a friendly evening where we code, chat about projects (not too loud so people won't code bugs :) ) and help each other. You come, with or without your project and we'll try to something with it.

Already announces projects:

WHEN November 18, 2014 @6pm

WHERE Caravan, 5334 de Gaspé, office #1204 (Montreal)

HOW Just grab your free ticket on: http://python-project-night-13.eventbrite.ca

Bring your computer and your smile, we'll provide beer and pizza !

Categories: External Blogs

Meeting Python and VFX

Montreal Python - Mon, 11/10/2014 - 00:00

On Monday, November the 17th, the NAD Centre will partner up with our friends at [Savoir-Faire Linux to organize a "Python and VFX" meetup. Python is ubiquitous in the world of special effects: from Maya to Softimage and 3D Studio Max, not to mention Blender of course.

The presentations will be held at NAD Centre, 405 rue Ogilvy, 3rd floor, at 6PM.

As you rpobably know, we've already had an event about VFX and Python during MP31:http://montrealpython.org/fr/2012/09/mp31/.

And this one is promissing to be very interesting if you are using any 3D software or if you would like to know more about real case study of useful way of using Python.

Program for the evening:

  • 6:00PM – Welcome
  • 6:30PM – Eric Thivierge, Rigging R&D at Hybride Technologie “Python: Bridging Technologies”
  • 7:00PM – Dave Lajoie, R&D Director at Digital District “Python in a vfx/animation pipeline”
  • 7:45PM – Jordi Riera, Python Software Developer at Savoir-Faire Linux “How to train your Python or how to improve python codes”
  • 8:15PM – Networking
  • 9:00PM – Wrap up

Don't forget to register your place at: http://elite.nad.ca/product/meetup-python-2/

For more informations:

See you there !

Categories: External Blogs

Call for speakers - Montréal-Python 50: Monochromatic Necessity

Montreal Python - Tue, 11/04/2014 - 00:00

For the last 7 years, Montreal-Python has been and still is the most awesome Python community in Montreal, if not the world. We are proud to celebrate our 50th monthly meetup in couple of weeks.

For this momentus occasion, we are looking for the best speakers in town for either a 5, 10, 20, 30 or 45 minutes presentation. If you are cooking up a crazy idea powered by Python or if you have been working on something awesome, this is the perfect opportunity to present it to all of Montreal!

When

Monday, December 1st, 2014 at 6:30pm

Where

To be defined

How

Just send us an email at mtlpyteam@googlegroups.com with the title of your talk, the length, and a short one sentence description.

Categories: External Blogs

Election for MTL:PY this Monday

Montreal Python - Thu, 10/30/2014 - 23:00

It's that time of the year again: the organisation team for the Montréal-Python community is going to proceed to the election of its board. People elected to the board are the legal representatives of the association, but they're also people who contribute their energy and ideas to drive the community.

Anyone who is subscribed to the mailing list can apply to one of these positions:

  • President
  • Vice-president
  • Secretary
  • Treasurer

Please note that other positions could be added if needed.

We'd like to stress that it is not necessary to be a member of the board (or even a member of the permanent organisation team) to organise events under the banner of Montréal-Python; the board is the official organ, but it is you, the members of the community, who do great things with Python in Montreal.

When:

Monday, November 3rd, 8pm

Where:

Ajah offices, 1124 Marie-Anne Est https://goo.gl/maps/xN1ck

How:

Show of hands

Comme see us on Monday! :)

UPDATE The new board as been elected, please welcome your new montreal-python's board for 2014-2015:

  • President: Mathieu Leduc-Hamel
  • Vice-president: Jean-Philippe Caissy
  • Secretary: George Peristerakis
  • Treasurer: Rory geoghegan

Rock on !

Categories: External Blogs

Mercurial: An easy and powerful alternative to git!

Montreal Python - Wed, 10/22/2014 - 23:00

You have probably heard about git and you understand that source control is a good thing. Who made this change? When? Why? How do I go back to before this change happened? Which change broke the code? How do I combine two different streams of the same code? How do I collaborate with others? How do I collaborate with my past self, who knew things that my present self has forgotten?

Mercurial answers all of these questions! Like git and others, Mercurial is a distributed version control system (DVCS). Big players like Python, Mozilla, Facebook and others use it to keep track of their source code. Many hosting services exist for it, such as Mozdev, Google Code, or Bitbucket.

During our workshop, we will introduce the basics of using DVCS and how to configure and use Mercurial to suit your needs. The presentation will be in English, but we encourage questions and discussions in French.

Just bring your laptop, we'll have power and wifi!

Where

Room A-3230
École de Technologie Supérieure
1100 Rue Notre-Dame Ouest
Montréal, QC H3C 1K3 (Canada)
(https://goo.gl/maps/9auBB)

When

Novembre 6th, at 6pm, until 9pm

Subscription

http://mercurial-workshop.eventbrite.ca

Categories: External Blogs

Montréal-Python 49: Kaleidoscopic Lunette

Montreal Python - Tue, 10/14/2014 - 23:00

For the the begining of fall, we are proposing subjects ranging from open data, to SciPy and Django, walking by Docker and finally Go-lang. What a program for one evening.

Before that we'll have Rory presenting the module of the month pyp and Brice Salmon will come present how E-180 is using Python.

Please, also, don't hesitate, if you have anything to announce or present, just come, we are always happy to give you your chance !

As usual, the evening will continue with our friends at the Benelux, it's your chance to meet the Python Community in Montreal.

Flash Presentations (5-10 minutes each):
  • Rory Geoghegan: Module of the month: pyp

  • Patrick Picher: Python and E-180 - Brain dates for learning humans

E-180 est une entreprise sociale montréalaise qui se spécialise dans la création d’outils web et mobile de jumelage, connectant des gens aux intérêts similaires intéressés à apprendre les uns des autres, en personne et un-à-un. Cette courte présentation a pour but de montrer ce que nous faisons ainsi que de balayer notre stack technologique utilisant, entre autres, le langage Python

Main Presentations:
  • Guillaume Pratte: "Les efforts d'importation des données ouvertes de la Ville de Montréal"

The city of Montreal recently authorized the entry of its open data into OpenStreetMap. The challenge is the data is in mutliple formats (ShapeFile, KML, even CSV). This presentation goes through the process of converting the data to a format appropriate for OpenStreetMap using Python.

  • Eleyine Zarour: "Automating microscopy data analysis using Django and Scipy"

Eleyine will be demonstrating how she used Django coupled with Scipy to automate microscopy data analysis tasks during her undergraduate research project in biology.

  • Rory Geoghegan and Luis Rojas: How to get your python services running in docker

  • Alexandre Bourget: Golang for Pythonistas

When:

Monday, the 20th of October 2014

Where:

UQÀM, Pavillion PK

145, avenue du Président-Kennedy

Room PK-1140

Schedule:
  • 6:00pm — Doors open
  • 6:30pm — Presentations start
  • 7:30pm — Break
  • 7:45pm — Second round of presentations
  • 9:00pm — One free beer offered at Bénélux just across the street
We’d like to thank our sponsors for their continuous support:
  • UQÀM
  • Bénélux
  • w.illi.am/
  • Outbox
  • Savoir-Faire Linux
  • Caravan
  • iWeb
Categories: External Blogs

Call for Speakers for MP-49: Kaleidoscopic Lunette

Montreal Python - Tue, 10/07/2014 - 23:00

We are proud to announce our next monthly event, Montréal-Python 49: Kaleidoscopic Lunette, on the 20th of October at 6:30pm at UQAM.

We especially love to hear from new speakers. If you haven't given a talk at Montréal-Python before, a 5 or 10 minute lightning talk would be a great start, but we also have slots for 20 to 40 minutes talks!

If you have an idea for a talk, send us a title, the length of your talk and a short description to mtlpyteam@googlegroups.com.

Categories: External Blogs

Montréal-Python 48: Incorrect jiujitsu

Montreal Python - Tue, 09/09/2014 - 23:00

It's time to go back to school and also for us, pythonistas from Montreal to have again our monthly meetups. We are changing a bit the tradition of having our meetup on the second Monday of the month and we are ready for an amazing edition full of great presenters.

This time, our friends of Shopify will welcome us on Tuesday, the 23th of September, at their amazing offices of the Saint-Laurent street (not so far away from the Benelux). It's an opportunity to meet the best of the python community in Montreal.

Don't forget that the call for speakers of PyCon is ending in less than a week, on September 15th. If you want more details, please have a look at the following page: http://montrealpython.org/2014/08/pycon2015-cfp/.

Flash Presentations (5 minutes each):
  • Rory Geoghegan: L Module of the month: pyp
  • Jordi Gutiérrez Hermoso: doitlive: shell demo magic for the masses Amaze your friends! Type like a hacker! Smash the keyboard and witness l33t h4xx0rz c0adz unfold before your eyes! Great parlour trick, and if you want something more boring, great way to do live shell demos
  • Simon Génier: Pyston, An open-source Python implementation using JIT techniques.
  • Mathieu Leduc-Hamel: Proctor, a swarm of Tor to protect your privacy
Main Presentations:
  • Thibault Cohen: Québec 24/7, le Québec supervisé. Québec 24/7 est un tableau de bord, Open source, basé sur des applications Python, présentant l'état Live du Québec.

  • Jonathan Doyle: Python Laser Lab in the Far North CANDAC RMR lidar (CRL) is a laser radar capable of measuring aerosol, cloud, and thermodynamic properties over Eureka, Nunavut (79.99N, 85.94W) operated remotely using Python. Two high powered Nd:YAG lasers are powered on, diagnostic tests performed, co-aligned to approximately 20 km, and then set to collect data via satellite link. Python is used (almost) exclusively to run the lidar system, including serial drivers, unittest suite with real-world conditions, XML-RPC/GUI or script based interfaces, babysitters, and data-analysis suite. In this talk I review the development and operation of the CRL highlighting the role of Python, as well as the problems we confronted while working in such a remote location.

  • Rory Geoghegan and Luis Rojas: Python and Docker How to get your python services running in docker.

  • Cameron Davidson-Pilon, Data Analyst at Shopify: Lifelines: Survival Analysis in Python While tools like linear regression and logistic regression moved from statistics to machine learning, the field of survival analysis was unfortunately left behind. This talk introduces survival analysis, how it is used, and a pure Python implementation: Lifelines.

When:

Tuesday, the 23th of september 2014

Where:

Shopify Montréal Offices

3575 Boul. Saint-Laurent Suite 110 (1st level) Montreal, Quebec H2X 2T7

Schedule:
  • 6:00pm — Doors open
  • 6:30pm — Presentations start
  • 7:30pm — Break
  • 7:45pm — Second round of presentations
  • 9:00pm — One free beer offered at Bénélux
We’d like to thank our sponsors for their continuous support:
  • UQÀM
  • Bénélux
  • w.illi.am/
  • Outbox
  • Savoir-Faire Linux
  • Caravan
  • iWeb

And we would like to thanks Shopify for their support and to welcome us at their offices !

This event will be held during the week of the free softwares. To get more information about the other events held during the week, head your browser to the following website: http://2014.sqil.info/

Categories: External Blogs

Epic Lameness

Eric Dorland - Mon, 09/01/2008 - 17:26
SF.net now supports OpenID. Hooray! I'd like to make a comment on a thread about the RTL8187se chip I've got in my new MSI Wind. So I go to sign in with OpenID and instead of signing me in it prompts me to create an account with a name, username and password for the account. Huh? I just want to post to their forum, I don't want to create an account (at least not explicitly, if they want to do it behind the scenes fine). Isn't the point of OpenID to not have to create accounts and particularly not have to create new usernames and passwords to access websites? I'm not impressed.
Categories: External Blogs

Sentiment Sharing

Eric Dorland - Mon, 08/11/2008 - 23:28
Biella, I am from there and I do agree. If I was still living there I would try to form a team and make a bid. Simon even made noises about organizing a bid at DebConfs past. I wish he would :)

But a DebConf in New York would be almost as good.
Categories: External Blogs
Syndicate content