A few questions about the psycopg2 porting to Python 3

Discussion:

Daniele Varrazzo

2011-01-11 18:41:17 UTC

Hello,

I've finished the porting of psycopg2 to Python 3. The porting is
based on the Martin v. Löwis patch (thank you *very* much) back in
2008; unfortunately at that time it wasn't promptly merged and the
code base diverged too much: the library has grown a lot of features
since then (and luckily a big fat test suite). Martin's "abstraction
layer" of macros in python.h has been the base for the new porting
anyway. The code is in available in the python3 branch of my psycopg
repos <https://www.develer.com/gitweb/pub?p=users/piro/psycopg2.git;a=shortlog;h=refs/heads/python3>.

Something I've done in a different way is the "adapters" area: Martin
did much of the processing in str, but there is a critical function
provided by the libpq, PQEscapeString
<http://www.postgresql.org/docs/9.0/static/libpq-exec.html#LIBPQ-PQESCAPESTRINGCONN>,
defined char* -> char*, that we use on strings before passing to the
backend: Py3 strings and Py2 unicode must be converted to bytes (in
the connection encoding) to use it: I feel awkward to go back to
unicode after passing through that function and to go back to bytes
again when pushing data to the socket, so I've used mostly bytes in
the Python->Postgres adaptation code path. OTOH doing my way we need
eventually a "format % args" with bytes in both format and args:
because this function is not provided by the Python API, I made my own
"PyBytes_Format" converting the PyString_Format from the Python 2.7
source code. I understand there is no problem in using parts of the
Python (2.7) source code into an LGPL-licensed work: is this right?
The resulting file is in <http://tinyurl.com/4w5oogn> and includes the
Python license as well: I'd like an opinion about whether the result
is "legal" and/or if the license had to be specified some other way.

As emerged from the discussion in this ML back in 2008, there is
somewhere the need for a python function b('literal') that would
evaluate to 'literal' in Py2 and to b'literal' in py3 (we want to keep
compatibility with Python 2.4). Currently there is an encode()
involved in Py3, so it would be great to have the transformation
b('literal') -> b'literal' performed by 2to3 instead. Looking at the
other fixers it seems easy, but I haven't found how to register a
custom fixer for the use of build_py_2to3 in setup.py. Is there any
reference?

There may be a few other points still open: they are more specifically
psycopg-related as they influence how 3rd party adapters should be
written, so they will be probably discussed in the psycopg mailing
list before releasing what is currently in the python3 branch. If you
want to join you are welcome.

Thank you very much.

-- Daniele

Daniele Varrazzo

2011-01-12 17:34:12 UTC

Permalink

On Tue, Jan 11, 2011 at 6:41 PM, Daniele Varrazzo

Post by Daniele Varrazzo
As emerged from the discussion in this ML back in 2008, there is
somewhere the need for a python function b('literal') that would
evaluate to 'literal' in Py2 and to b'literal' in py3 (we want to keep
compatibility with Python 2.4). Currently there is an encode()
involved in Py3, so it would be great to have the transformation
b('literal') -> b'literal' performed by 2to3 instead. Looking at the
other fixers it seems easy, but I haven't found how to register a
custom fixer for the use of build_py_2to3 in setup.py. Is there any
reference?

I got in PM the suggestion to use distribute. My answer is that I
would prefer to avoid an extra dependency to solve this problem.

I've tested with some nasting monkeypatching to have the fix_b
injected. This seems working for instance:

diff --git a/setup.py b/setup.py
index 926169c..836d3e6 100644
--- a/setup.py
+++ b/setup.py
@@ -58,6 +58,16 @@ try:
from distutils.command.build_py import build_py_2to3 as build_py
except ImportError:
from distutils.command.build_py import build_py
+else:
+ # Monkeypatch lib2to3 to make it found our custom fixers
+ import lib2to3.refactor
+ from lib2to3.refactor import get_fixers_from_package
+ def get_fixers_from_package_hacked(pkg_name):
+ rv = get_fixers_from_package(pkg_name)
+ return rv + ['fix_b']
+
+ lib2to3.refactor.get_fixers_from_package =
get_fixers_from_package_hacked
+ sys.path.insert(0, 'scripts')

try:
import configparser

Is there a more proper way to use a custom fixer?

As b() fixer I've written the following:

"""Fixer to change b('string') into b'string'."""
# Author: Daniele Varrazzo

import token
from lib2to3 import fixer_base
from lib2to3.pytree import Leaf

class FixB(fixer_base.BaseFix):

PATTERN = """
power< wrapper='b' trailer< '(' arg=[any] ')' > rest=any* >
"""

def transform(self, node, results):
arg = results['arg']
wrapper = results["wrapper"]
if len(arg) == 1 and arg[0].type == token.STRING:
b = Leaf(token.STRING, 'b' + arg[0].value,
prefix=wrapper.prefix)
node.children = [ b ] + results['rest']

It has been obtained by reverse-engineering the other fixers: is it
written as is meant to be?

Thanks

-- Daniele

Martin v. Löwis

2011-01-12 20:27:16 UTC

Permalink

Post by Daniele Varrazzo
I understand there is no problem in using parts of the
Python (2.7) source code into an LGPL-licensed work: is this right?
The resulting file is in <http://tinyurl.com/4w5oogn> and includes the
Python license as well: I'd like an opinion about whether the result
is "legal" and/or if the license had to be specified some other way.

I can certainly offer an opinion. Not as a PSF director, but as a legal
layman: this sounds all fine to me.

See the source of distutils.util.Mixin2to3. Overriding fixer_names
should do the trick. The default list is computed as

get_fixers_from_package('lib2to3.fixes')

and you'll have to append to this list.

I'm glad this has progress!

Regards,
Martin

Daniele Varrazzo

2011-01-13 11:23:09 UTC

Permalink

Post by Martin v. LÃ¶wis

See the source of distutils.util.Mixin2to3. Overriding fixer_names
should do the trick. The default list is computed as
get_fixers_from_package('lib2to3.fixes')
and you'll have to append to this list.

Yes, configuring the mixin by setting the class variable works
perfectly. As a reference here it is:

try:
from distutils.command.build_py import build_py_2to3 as build_py
except ImportError:
from distutils.command.build_py import build_py
else:
# Configure distutils to run our custom 2to3 fixers as well
from lib2to3.refactor import get_fixers_from_package
build_py.fixer_names = get_fixers_from_package('lib2to3.fixes')
build_py.fixer_names.append('fix_b')

setup(name="yourmodule",
cmdclass={ 'build_py': build_py, },
[...] )

-- Daniele

Lennart Regebro

2011-01-12 21:35:38 UTC

Permalink

On Tue, Jan 11, 2011 at 19:41, Daniele Varrazzo

Post by Daniele Varrazzo
other fixers it seems easy, but I haven't found how to register a
custom fixer for the use of build_py_2to3 in setup.py. Is there any
reference?

Yes: http://packages.python.org/distribute/python3.html#distributing-python-3-modules

setup(
...
use_2to3_fixers = ['your.fixers'],
...
)

Daniele Varrazzo

2011-01-13 11:40:27 UTC

Permalink

Post by Lennart Regebro
On Tue, Jan 11, 2011 at 19:41, Daniele Varrazzo

Post by Daniele Varrazzo
other fixers it seems easy, but I haven't found how to register a
custom fixer for the use of build_py_2to3 in setup.py. Is there any
reference?

Yes: http://packages.python.org/distribute/python3.html#distributing-python-3-modules

Isn't "distribute" a third party library? I would be fine using it if
there was no chance to do what I want with the stdlib; but if some
hacking is enough, even if the stdlib interface is not as polished as
distribute's, I prefer to avoid an extra dependency.

-- Daniele

Lennart Regebro

2011-01-13 11:43:18 UTC

Permalink

On Thu, Jan 13, 2011 at 12:40, Daniele Varrazzo

Post by Daniele Varrazzo
Isn't "distribute" a third party library?

Yes, I answered before I read the part where you say you didn't want
to use it. :-)