Compiling Python

As I take a new (professional) interest in Python, I use it more often. I noticed on previous occasion that the default interpreter’s performance is dismal, and I concluded that the next logical step would be to recompile it to get better performance.

It would not prove easy.

On my OS, Python 2.7.1 is available as a precompile package, and 2.7.2 is the current (at the time of writing, anyway) production version of Python. Downloading Python 2.7.2 is not complicated at all. Building it isn’t either. But somewhat, the results I got were … less than impressive.

Using the n-queens problem I used before, I get the following times:

Size Default Custom
12 8.2s 8.6s
13 46.7s 52.2s
14 267.0s 320.1s
15 1764.4s 2030.9s

Hmm. So invoking ./configure and make doesn’t generate a better executable than the distribution’s default binary. It’s about 15% slower while I expected 5–10% faster.

On certain occasions, Intel’s C compiler, ICC, showed itself capable of generating better code than GCC, and it seemed reasonable to try to compile Python with ICC, but that wasn’t meant to be.

For some reason, Python simply does not build using ICC. The configure script seems to be b0rked somehow and it decides that I’m compiling for Windows… On 64 bits Linux. After some investigations, I found that I’m not the only one with the problem. Solutions found while googling did not work for me, and while discussing the issue on #python, I got all kinds of answers ranging from “your ICC is broken” to “python does not support ICC at all.” The latter being the real problem.

Of course, I could spend some time (well, probably quite a large amount, given the state of my knowledge of Autotools) to fix this and commit it back in the Python repository, but I feel more expert people should do this and integrate the fixes in the trunk.

* *

Clearly, the people at Ubuntu exerted their magic at compiling Python (or, maybe more accurately, I lack the magic to do so efficiently) but I remain curious as to how much faster it would be if one managed to compile it using ICC. Anyone out there willing to help?

* *

I don’t really like to write a blog entry about a failed experiment, but sometimes it’s the right thing to do. Maybe a reader knows a simple fix (either as to why the default ./configure and make fail to build an executable at least as efficient as the distribution binary, or as to why Windows is detected when ICC is used) and using somebody else’s knowledge is the most time-efficient way to tackle the problem.

2 Responses to Compiling Python

  1. thirdhaf says:

    Sorry if it’s a stupid question but could you use PyPy for your current project? I’ve gotten some fairly substantial speedups over CPython.

    • Steven Pigeon says:

      It’s not a stupid question. The answer might be, however : it is easier to convince customers/admins to use a compiled version of the same python they use than to propose an alternative compiler/interpreter. A better reason is not use an alternative compiler (with possible compatibility issue) to develop and the official version on deployment.

      But initially, I was curious to know whether or not I could manage to get speed-ups just by using a custom-built (I have many other examples where I got significant speed-ups). Turns out that I do not know autotools enough to fix 2.7.x build and push the mods back into the repository.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: