Python 2/3 compatible distutils installation?

[Barry Warsaw]

Here's an interesting porting issue that came up today. I'm not crossposting
this to distutils-sig, but I will if no one here has come up with a solution.

I'm looking at testtools[1], which Jerry Seutter has done a good, initial port
of to Python 3. His merge proposal, along with the diff is here[2]. My
branch which fixes a few of Robert Collins' comments is here[3].

One of the tricky problems is with re-raise syntax, which can't be
compatibly represented syntactically across Python 2 and 3. The six module
has a hack for this, but testtools uses its own trick, which is to provide a
_compat2x.py and a _compat3x.py file containing the proper syntax.

This works well at execution time, because a simple sys.version_info check can
be used to decide which module gets imported; the other is for all intents and
purposes, ignored.

The problem comes when installing the package with distutils. The
byte-compilation of _compat2x.py throws a SyntaxError, of course rightly so.

The question I have is how can we avoid trying to byte compile _compat2x.py
when the package is installed via Python 3? I've tried various distutils
command overrides and hacks (including trying to load a MANIFEST3.in which
exclusions that file), but haven't hit upon the right magic to make it work in
all cases.

Has anybody encountered a similar situation, and if so, how have you worked
around this? Disgusting kludges welcome. :)

Cheers,
-Barry

[1] http://launchpad.net/testtools
[2] https://code.launchpad.net/~jseutter/testtools/py3_again/+merge/105569
[3] https://code.launchpad.net/~barry/testtools/py3

[Chris Jerdonek]

Post by Barry Warsaw
The question I have is how can we avoid trying to byte compile _compat2x.py
when the package is installed via Python 3?  I've tried various distutils
command overrides and hacks (including trying to load a MANIFEST3.in which
exclusions that file), but haven't hit upon the right magic to make it work in
all cases.

I also encountered this issue recently. Can you just avoid importing
anything from your package from within setup.py? It looks like you're
importing your package mostly just to get the __version__ number in
your package's __init__.py (though I also see you use it in the
cmdclass argument to setup(), which might require a different
work-around).

The consequence for me was that I needed to maintain the version
string for my project in two places: both setup.py and my package's
__init__.py (and I added a unit test to check that they were the
same). I rationalized this by telling myself that the script
responsible for installing a project shouldn't need to be able to run
the project: it should be independent.

--Chris

[Barry Warsaw]

Post by Chris Jerdonek
I also encountered this issue recently. Can you just avoid importing
anything from your package from within setup.py?

It's not actually *my* setup.py, but I get what you're saying. In my own
packages, I do try to avoid those types of imports.

Post by Chris Jerdonek
It looks like you're importing your package mostly just to get the
__version__ number in your package's __init__.py (though I also see you use
it in the cmdclass argument to setup(), which might require a different
work-around).

Yep. It seems like the testtools.TestCommand is required.

Post by Chris Jerdonek
The consequence for me was that I needed to maintain the version
string for my project in two places: both setup.py and my package's
__init__.py (and I added a unit test to check that they were the
same). I rationalized this by telling myself that the script
responsible for installing a project shouldn't need to be able to run
the project: it should be independent.

In my own packages, I still keep the version string in my __init__.py, but I
don't import it from setup.py to get at it. Instead, I usually just open the
file and grep its lines for the value. I also have a helper to make this
easier, e.g.

http://bazaar.launchpad.net/~barry/flufl.enum/trunk/view/head:/setup_helpers.py

and its use in my setup.py

http://bazaar.launchpad.net/~barry/flufl.enum/trunk/view/head:/setup.py

Cheers,
-Barry

[Chris Jerdonek]

I also encountered this issue recently.  Can you just avoid importing
anything from your package from within setup.py?

It's not actually *my* setup.py, but I get what you're saying.  In my own
packages, I do try to avoid those types of imports.

Sorry, I meant to say the package that you're helping with.

I realize now that the issue I had might be completely different,
because I was using 2to3 and testtools protects the import with a
version check, etc. So perhaps my suggestion won't help. What causes
the install process to byte-compile the files?

Also, thanks for the idea and pointers re: grabbing the version
number. That idea had crossed my mind, too, but I didn't implement
it.

--Chris

[Barry Warsaw]

What causes the install process to byte-compile the files?

That's the question. :) By which I mean, the files get byte compiled on
installation, and I've tried various things to have it ignore _compat2x.py
when installing in Python 3, but with no luck.

-Barry

[Éric Araujo]

Post by Barry Warsaw

What causes the install process to byte-compile the files?

That's the question. :) By which I mean, the files get byte compiled on
installation, and I've tried various things to have it ignore _compat2x.py
when installing in Python 3, but with no luck.

distutils does that. As far as I know you can either disable
byte-compilation wholesale, or have it for all modules. I don’t think
even a dirty hack like registering the _compat?.py files as package_data
would work, because the byte-compilation function is applied to the
whole build or install dir (but try it anyway, we never know).

See http://bugs.python.org/issue10530 for a distutils2 feature request
about this same use case.

Cheers

[Barry Warsaw]

Post by Éric Araujo
See http://bugs.python.org/issue10530 for a distutils2 feature request
about this same use case.

Thanks, nosied.

-Barry

[Chris Jerdonek]

That's the question. :)  By which I mean, the files get byte compiled on
installation, and I've tried various things to have it ignore _compat2x.py
when installing in Python 3, but with no luck.

Have you thought about storing the two files with an extension other
than .py? You could argue that from Python 3's perspective, for
example, the compat2 file isn't really a Python file. You could write
your setup script so that it copies the correct file into place at
compat.py before calling setup(), and then cleans up afterwards. This
way, the distutils code could proceed normally. Also, you wouldn't
need special version-checking code in the actual package because then
both versions would be importing compat when installed.

--Chris

[Barry Warsaw]

Post by Chris Jerdonek
Have you thought about storing the two files with an extension other
than .py? You could argue that from Python 3's perspective, for
example, the compat2 file isn't really a Python file. You could write
your setup script so that it copies the correct file into place at
compat.py before calling setup(), and then cleans up afterwards. This
way, the distutils code could proceed normally. Also, you wouldn't
need special version-checking code in the actual package because then
both versions would be importing compat when installed.

It's an interesting idea, thanks. I'll point upstream at this thread and see
if they want to explore it more or if they want to go in the direction of
using six (which would eliminate the need to use the _compatX.py files in this
particular case).

-Barry

[Brett Cannon]

Post by Barry Warsaw

Post by Chris Jerdonek
Have you thought about storing the two files with an extension other
than .py? You could argue that from Python 3's perspective, for
example, the compat2 file isn't really a Python file. You could write
your setup script so that it copies the correct file into place at
compat.py before calling setup(), and then cleans up afterwards. This
way, the distutils code could proceed normally. Also, you wouldn't
need special version-checking code in the actual package because then
both versions would be importing compat when installed.

It's an interesting idea, thanks. I'll point upstream at this thread and see
if they want to explore it more or if they want to go in the direction of
using six (which would eliminate the need to use the _compatX.py files in this
particular case).

Your other option is to keep the code in string literals and then do the
proper compile/exec step just like import does. Basically do you want the
hack in distutils or the modules themselves.