2to3 speed

[Vladimir Perić]

Hello,

I'm trying to port SymPy[0] (symbolic mathematics library) to Python 3. The
goal is to use a single code base and then run 2to3 automatically when
building for Python 3. Unfortunately, I've hit a small problem: running 2to3
takes a _long_ time, somewhere around 10 mins. This is a problem for users,
but a particularly big one for me as it makes development last much longer
(the change (Py2) code, run 2to3, test cycle takes a long time). Is there a
way to speed up 2to3 execution? I do realize we've got quite a large code
base. Will running 2to3 with some fixers disabled help with running time
(ie. if I know I've fixed reduce and callable before, I might as well
disable the fixers)? Any other advice to making it run faster?

Thanks a lot!

[0] http://sympy.org/

[Lennart Regebro]

Post by Vladimir Perić
Hello,
I'm trying to port SymPy[0] (symbolic mathematics library) to Python 3. The
goal is to use a single code base and then run 2to3 automatically when
building for Python 3. Unfortunately, I've hit a small problem: running 2to3
takes a _long_ time, somewhere around 10 mins. This is a problem for users,
but a particularly big one for me as it makes development last much longer
(the change (Py2) code, run 2to3, test cycle takes a long time). Is there a
way to speed up 2to3 execution? I do realize we've got quite a large code
base. Will running 2to3 with some fixers disabled help with running time
(ie. if I know I've fixed reduce and callable before, I might as well
disable the fixers)? Any other advice to making it run faster?

The major way to speed up 2to3 during development is to only run it on
the files that has changed, by comparing timestamps. Running it on one
file usually only takes a couple of seconds. More explanation here:
http://python3porting.com/strategies.html#converting-to-python-3-with-2to3

Code examples for how to do this exists in distutils build_2to3
command, if I remember correctly, but since SymPy is already using
distutils I would recommend using the build_2to3 command distutils
has. Or in fact, I'd recommend using Distribute as it's test command
already will run the tests in the build directory and hence
automatically use 2to3 if you have set it up to do that. See
http://packages.python.org/distribute/python3.html for more on that.

You can speed it up a bit more by only including fixers that you are
using, but for a complex software that is likely to be most of them
anyway.

//Lennart

[Benjamin Peterson]

Post by Vladimir Perić
Hello,
I'm trying to port SymPy[0] (symbolic mathematics library) to Python 3. The
goal is to use a single code base and then run 2to3 automatically when
building for Python 3. Unfortunately, I've hit a small problem: running 2to3
takes a _long_ time, somewhere around 10 mins. This is a problem for users,
but a particularly big one for me as it makes development last much longer
(the change (Py2) code, run 2to3, test cycle takes a long time). Is there a
way to speed up 2to3 execution? I do realize we've got quite a large code
base. Will running 2to3 with some fixers disabled help with running time
(ie. if I know I've fixed reduce and callable before, I might as well
disable the fixers)? Any other advice to making it run faster?

What version of 2to3 are you using? 2to3 got a nice performance boost
over the summer which should have made it into Python 3.2.

[Vladimir Perić]

Thanks to both of you! I am currently using tox for testing/development, but
I think I understood from your explanation how to make it all work under
Python 3 as well. I've also downloaded the 3.2 version of it and it's about
40% faster, so that was a great help as well!

I have encountered an interesting issue, though - one of the libraries SymPy
bundles is written to support Python 3 in a single code base (without 2to3).
Unfortunately, some of the fixes 2to3 makes bungle this up. Now, editing the
library is not a good solution, I'd rather keep it as is - is it possible to
tell to 2to3 to skip a part of the code base? And, of course, can I do this
from distutils/distribute?

[Lennart Regebro]

Post by Vladimir Perić
Thanks to both of you! I am currently using tox for testing/development, but
I think I understood from your explanation how to make it all work under
Python 3 as well. I've also downloaded the 3.2 version of it and it's about
40% faster, so that was a great help as well!
I have encountered an interesting issue, though - one of the libraries SymPy
bundles is written to support Python 3 in a single code base (without 2to3).
Unfortunately, some of the fixes 2to3 makes bungle this up. Now, editing the
library is not a good solution, I'd rather keep it as is - is it possible to
tell to 2to3 to skip a part of the code base? And, of course, can I do this
from distutils/distribute?

If those are included in package itself, then there isn't. If they are
separate packages it will work with no changes.