!echo Bye-bye Python 2 !

In early 2013, we decided to port our chatbot project Errbot to Python 3. It was for its version 2.0.0beta.

I still remember that vividly, it was crazy at that time. First, absolutely nobody would use Errbot under Python 3. Finding compatible libraries was a gigantic pain not only for the plugin designers but also for ourselves: Check out this extract from the change log...

- xmpp backend has been replaced by sleekxmpp
- flask has been replaced by bottle (sorry flask no py3 support, no future)
- now the IRC backend uses the simpler python/irc package

We had to remove flask for rocket a small alternative and switch over the other 2 main dependencies we had for the 2 chat systems we supported. We barely wiggled through this ... and of course we broke stuff all over (I still have PTSD from the unicode breakages :) ).

2.0.0 stayed in beta for 10 months while users stayed safely on 1.7.1, a version from 2012.

We managed pretty well the next 3 years: the code was developed in Python 3 and translated automatically at install time to Python 2 with a tool called 3to2. The project grew in popularity and the general quality of the code, unit tests and documentation with it.

But something changed progressively, maintaining the Python 2 compatibility became more and more time consuming for not much benefit: simply less users were stuck with it. We could not find any reason for keeping the backward compatibility: all the main distributions have Python 3, the ecosystem moved on (even flask !), most plugins have the dual compatibility.

So today we pulled the plug, we are removing the Python 2 compatibility on master. We will maintain the backward compatibility for the latest version (4.2.x) until the end of the year while the new version will be Python 3.3+ only.

Some takeways

- It is quite impressive that, despite 3to2 making a very good job at converting the source code, a lot of hacks crept in over the years.  Just by browsing the diff, it is easy to see the jump in code readability.

- 3to2 was retrospectively the good choice: It allowed us to switch to the new Python version 3 years ago and today in just one long but very simple PR remove the support for the old one.

- 3 years ago it was definitely a little early to support Python 3 but today it is definitely the right time to drop Python 2.

If you have to port your project today, you are in luck ! You can drop Python 2 right away without all the complexity of maintaining a dual compatibility. Polish your test coverage, 2to3 in-place, fix it until it is done and enjoy the cleanliness and the new features of Python 3 without ever thinking again about Python 2.

Quick hack to plot data on a map in python

I wanted to share a cool code mash up I did to show some stats about Err usage in the world.

The result is here, in blue roughly after a non reliable homebrew datamining from the apache logs:

in blue the "fidelity" of an installation

in red the "development" activity

I applied this tip

As dependencies you gonna need :

matplotlib

GeoIP

Basemap

PyQT4 (but you can change the backend to your favorite toolkit)

Here is the full snippet :

#! /usr/bin/python
import matplotlib
matplotlib.use('Qt4Agg')
from datetime import datetime
import os
DIR = 'data/'

dirList=os.listdir(DIR)

unzip = lambda l:tuple(zip(*l))

all_dates = {}
for fname in dirList:
    if fname.startswith('gootz-access'):
        counts = {}
        d = datetime.strptime(fname.split('.')[1], '%Y-%m-%d')
        with open(DIR+fname) as f:
            data = f.readlines()
            for line in data:
                if line.find('err/version') != -1:
                    addr = line.split('-')[0].strip()
                    if counts.has_key(addr):
                        counts[addr] += 1
                    else:
                        counts[addr] = 1
        all_dates[d] = counts

sorted_dates = sorted(all_dates.keys())

import GeoIP
gi = GeoIP.open('GeoLiteCity.dat',GeoIP.GEOIP_STANDARD)

insts_pos = {}
devs_pos = {}

for date in sorted_dates:
    for ip, count in all_dates[date].iteritems():
        record = gi.record_by_name(ip)
        if record and record['latitude']:
            lon = record['longitude']
            lat = record['latitude']

            toinc = devs_pos if count > 5 else insts_pos
            if toinc.has_key((lon,lat)):
                toinc[(lon,lat)] += 1
            else:
                toinc[(lon,lat)] = 1

from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
import numpy as np
# lon_0 is central longitude of robinson projection.
# resolution = 'c' means use crude resolution coastlines.
m = Basemap(projection='robin',lon_0=0,resolution='c')
#set a background colour
m.drawmapboundary(fill_color='#85A6D9')
# draw coastlines, country boundaries, fill continents.
m.fillcontinents(color='white',lake_color='#85A6D9')
m.drawcoastlines(color='#6D5F47', linewidth=.4)
m.drawcountries(color='#6D5F47', linewidth=.4)
# draw lat/lon grid lines every 30 degrees.
m.drawmeridians(np.arange(-180, 180, 30), color='#bbbbbb')
m.drawparallels(np.arange(-90, 90, 30), color='#bbbbbb')

inst_lngs = [entry[0][0] for entry in insts_pos.iteritems()]
inst_lats = [entry[0][1] for entry in insts_pos.iteritems()]
inst_count = [entry[1] for entry in insts_pos.iteritems()]
inst_x,inst_y = m(inst_lngs,inst_lats)

s_inst_count = [p * p for p in inst_count]
m.scatter(
    inst_x,
    inst_y,
    s=s_inst_count, #size
    c='blue', #color
    marker='o', #symbol
    alpha=0.25, #transparency
    zorder = 2, #plotting order
    )
for population, xpt, ypt in zip(inst_count, inst_x, inst_y):
    label_txt = int(round(population, 0)) #round to 0 dp and display as integer
    plt.text(
        xpt,
        ypt,
        label_txt,
        color = 'blue',
        size='small',
        horizontalalignment='center',
        verticalalignment='center',
        zorder = 3,
        )

devs_lngs = [entry[0][0] for entry in devs_pos.iteritems()]
devs_lats = [entry[0][1] for entry in devs_pos.iteritems()]
devs_count = [entry[1] for entry in devs_pos.iteritems()]
devs_x,devs_y = m(devs_lngs,devs_lats)

s_devs_count = [p * p for p in devs_count]
m.scatter(
    devs_x,
    devs_y,
    s=s_devs_count, #size
    c='red', #color
    marker='o', #symbol
    alpha=0.25, #transparency
    zorder = 4, #plotting order
    )
for population, xpt, ypt in zip(devs_count, devs_x, devs_y):
    label_txt = int(round(population, 0)) #round to 0 dp and display as integer
    plt.text(
        xpt,
        ypt,
        label_txt,
        color = 'red',
        size='small',
        horizontalalignment='center',
        verticalalignment='center',
        zorder = 5,
        )


#add a title and display the map on screen
plt.title('From where Err is used.')
plt.show()

Track git repos in chatrooms with err-gitbot

We just released a new cool plugin for err : err-gitbot

It allows you to follow specific heads of git repos and get notifications in your chat. This is awesome for teams that work with a chatroom and want to follow the current development progress.

Every time somebody commits to the repo for a specifiy project and branch, err will say something like that in the chat :
djmt:
  Branch master:
    4807cc        Emile Raoul     2012-06-06T14:43:45 -- WHITELABEL-4226 improve getAccountLimits
android_client_app:
  Branch master:
    554d9e        Jack Bougnazal  2012-06-06T14:44:12 -- Add string for mobile data

In order to install and configure it, it is really simple, talk to the bot directly and be sure you are one of the authorized admins (BOT_ADMIN list in the general config.py).

To install the plugin just say :

!install err-gitbot

Then to track all the branches of a repo just say with the repo url i.e.:

!follow git://github.com/opdenkamp/xbmc.git

Err will extract smartly the name "xbmc" from it.

if you want to follow a specific branches instead of everything just :

!follow git://github.com/opdenkamp/xbmc.git Eden-pvr Dharma

If you redo a follow this time simply with the symbolic name you can add new branches to track :

!follow xbmc staging

Or you can remove a specific branch with !unfollow :

!unfollow xbmc staging

Or the repo alltogether

!unfollow xbmc 

If at any point you are lost, you can ask the list of currently followed repos / branches with :

!following

Note : If you have several chatrooms defined in your err config, it will only "spam" the first one.

Enjoy and feel free to improve it  : Fork me on github !

Guillaume.

Fresh new plugins for err

A serie of new err plugins arrived :


err-stalkerbot
A plugin that tells you when the bot saw somebody for the last time. For example : !seen bidule

err-calcbot
A smart calculator based on qalculator,  Examples: !calc 5+2 !calc x²+x+1=5 !calc 5 in² = x cm²


err-dictbot
It can give you the definition/synonymous of a word. Example !define fish

err-weatherbot
It can give you the weather at the given location for the next few days. Example:  !weather san francisco, CA

And now err provides a public plugins repository so in order to install a public one just install it by name :
!install err-calcbot

It is uber-simple to create a plugin so feel free to add your own creations and I will add them to the official repository !

"err" a plugin-based XMPP chatbot

Just to let you know we published the first preliminary version of "err", a plugin based XMPP chatbot.


It is based on the long tradition of IRC bots but for a more modern media (Jabber/XMPP).
It has been tested under openfire and hipchat, but it probably runs on any jabber service. It can be used 1on1 or with MUCs (chatrooms).

We love it and the development team uses it daily at mondial telecom with a bunch of custom plugins.

It that can download and start plugins on the fly from git repos / tar.gz urls and developing plugins is quite entertaining so feel free to send me the link to your creations !

The core is there :
https://github.com/gbin/err

A first batch of plugins examples has been published so far, more gonna come as we are cleaning them up.
https://github.com/gbin/err-codebot
https://github.com/gbin/err-elizabot
https://github.com/gbin/err-pollbot
https://github.com/gbin/err-devops_borat

Enjoy !

Python optimizations, third round up !

Thanks to @fijall (a pypy core developer), we gained 2 new implementations !

def htmlGB4():
    l = []
    for c in r:
        if (c>=u'a' and c<=u'z') or (c>=u'A' and c<=u'Z') or (c>=u'0' and c<=u'9'):
            l.append(chr(ord(c)))
        else:
            insert = '&#' + str(ord(c)) + ';'
            l.append(insert)
    return ''.join(l)
assert(html7() == htmlGB4())

from cStringIO import StringIO
 
def htmlGB5():
    s = StringIO()
    for c in r:
        if (c>=u'a' and c<=u'z') or (c>=u'A' and c<=u'Z') or (c>=u'0' and c<=u'9'):
            s.write(chr(ord(c)))
        else:
            insert = '&#' + str(ord(c)) + ';'
            s.write(insert)
    return s.getvalue()

The specialized pypy StringBuilder he passed on looks interesting but for some reason it didn't appear on the build I had on gentoo on PyPy 1.5 I ported the htmlGB5 to cython also. When no data appears it means I didn't ported it. Here is the new comparative graph, I omitted htmlGB3 because it was throwing the graph out of scale.

So pypy shines with the StringIO where it is more then twice as fast as cython where cython is more then twice as fast as python. Thank a lot for the contribution. Enjoy !

Python optimizations continued ...

To give a fair treatment to pypy, I tried 2 c-like implementations similar to the cython one and the pypy 1.5 results are around 30% to 250% slower then standard python interpreter. Feel free to correct me if I did something obviously wrong here. Add those to test.py:

def htmlGB2():
    s = r
    a = array.array('c', itertools.repeat('\0', len(s)*10))
    i = 0
    for c in s:
        if (c>=u'a' and c<=u'z') or (c>=u'A' and c<=u'Z') or (c>=u'0' and c<=u'9'):
            a[i] = chr(ord(c))
            i += 1
        else:
            insert = '&#' + str(ord(c)) + ';'
            for cc in insert:
                a[i] = cc
                i += 1
    return a.tostring()

def htmlGB3():
    s = r
    result = ''
    for c in s:
        if (c>=u'a' and c<=u'z') or (c>=u'A' and c<=u'Z') or (c>=u'0' and c<=u'9'):
            result += chr(ord(c))
        else:
            result +='&#'
            result +=str(ord(c))
            result +=';'
    return result

⚫ python test.py
html7:   2.45566296577
htmlGB:   2.13124704361
htmlGB2:   11.926774025
htmlGB3:   3.90490102768

⚫ pypy-c1.5 test.py 
html7:    2.41169714928
htmlGB:   1.77979898453
htmlGB2:   15.9636659622 <-
htmlGB3:   91.441778183 <-

Python optimizations exercise.

From there I wanted to post cleanly all my findings. I include all my sources, so you can reproduce it. Basically, Pavel wanted to optimize a simple use case. Here is my shot at it: First the test.py, the html7 implementation and a slightly improved one where I don't force python to go back and forth between bytes and unicode + the benchmark timer :

import timeit


WHITELIST2 = set('abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789')
WHITELIST2_UNI = set(u'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789')
r = u'234782384723!#$#@!%$@#%$@#%#$%%%%%%%%%%%@#$%342583492058934028590342853490285902344#$%%*%****%7jkb6h546777776ynkk4b56byhbh5j'*500

def html7():
    s = r
    lstr = str; lord = ord; lWHITELIST2 = WHITELIST2
    return "".join([
        c if c in lWHITELIST2
        else "&#" + lstr(lord(c)) + ";"
        for c in s])

def htmlGB():
    s = r
    lstr = unicode; lord = ord; lWHITELIST2 = WHITELIST2_UNI
    return u''.join([
        c if c in lWHITELIST2
        else u'&#' + lstr(lord(c)) + u';'
        for c in s])
assert(html7() == htmlGB())

t = timeit.Timer(stmt = html7)
print 'html7:' + str(t.timeit(number=100))

t = timeit.Timer(stmt = htmlGB)
print 'htmlGB:' + str(t.timeit(number=100))

It gives me that as timing under python and pypy 1.5 :

⚫ python test.py 
html7:2.48782491684
htmlGB:2.14983606339

⚫ pypy-c1.5 test.py
html7:2.44286298752
htmlGB:1.84629392624

Note: the other implementation with caching & statistical tryouts are too convoluted and too much on the speed over memory tradeoff for this simple task IMHO. Then I wanted to go further, so I made a simple port on cython then a "bare metal" version on cython, and I have been really impressed by it ! In order to compile in cython you need a simple makefile-like setup.py file:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [Extension("encode", ["encode.pyx"])]

setup(
  name = 'Encoding Test',
  cmdclass = {'build_ext': build_ext},
  ext_modules = ext_modules
)

Then here are my implementations : cython is the direct port, cython_alt is the C-like bare metal one. In the file encode.pyx put:

WHITELIST = set(u'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789')

from libc.stdlib cimport malloc, free
from libc.stdio cimport sprintf

def cython(unicode s):
    return u''.join([
    c if c in WHITELIST
    else u'&#' + unicode(< long > c) + u';'
    for c in s])

def cython_alt(unicode s):
    cdef char * buffer = < char * > malloc(len(s) * 10)
    cdef int i = 0
    cdef Py_UCS4 c

    for c in s:
        if (c >= u'a' and c <= u'z') or (c >= u'A' and c <= u'Z') or (c >= u'0' and c <= u'9'):
            buffer[i] = < char > c
            i += 1
        else:
            sprintf(buffer + i,'&#%d;',c)
            while buffer[i]:
                i += 1
    result = < bytes > buffer
    free(buffer)
    return result

And the same test runner test_cython :

import timeit
from encode import cython
from encode import cython_alt


r = u'234782384723!#$#@!%$@#%$@#%#$%%%%%%%%%%%@#$%342583492058934028590342853490285902344#$%%*%****%7jkb6h546777776ynkk4b56byhbh5j'*500

def cython2():
    cython(r)

def cython_alt2():
    cython_alt(r)

assert(cython(r) == cython_alt(r))

t = timeit.Timer(stmt = cython2)
print 'cython:' + str(t.timeit(number=100))

t = timeit.Timer(stmt = cython_alt2)
print 'cython_alt:' + str(t.timeit(number=100))

To compile it just do :

python setup.py build_ext --inplace

The results :

⚫ python test_cython.py 
cython:1.70311307907
cython_alt:0.348756790161

Booya ! :)