Ondrej Certik
2009-07-05 22:22:04 UTC
Received: from localhost.localdomain (HELO mail.python.org) (127.0.0.1)
by albatross.python.org with SMTP; 06 Jul 2009 00:22:05 +0200
X-policyd-weight: using cached result; rate: -8.4
Received: from mail-yx0-f178.google.com (mail-yx0-f178.google.com
[209.85.210.178]) by mail.python.org (Postfix) with ESMTP
for <python-porting-+ZN9ApsXKcEdnm+***@public.gmane.org>; Mon, 6 Jul 2009 00:22:05 +0200 (CEST)
Received: by yxe8 with SMTP id 8so5511951yxe.9
for <python-porting-+ZN9ApsXKcEdnm+***@public.gmane.org>; Sun, 05 Jul 2009 15:22:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
h=domainkey-signature:mime-version:sender:received:date
:x-google-sender-auth:message-id:subject:from:to:content-type
:content-transfer-encoding;
bh=0YuZ1agkX4dnFp6nLRqiosYzJ60efDUcBxvgBJ5k7UY=;
b=Zg6Hv9zsjBHF/qujGlU1gxfdUOQ5vMmTl2CjAcjC1sledWNtqT+VaTRSmypY5uMTRE
HnDGs0B+aQDkaRnYHxUXnq60sA5BYQLnUZxOYjwfGuICCRSGuO8lI4wKhhQ+DTCOJRqt
ysoFjok2rueFCJJXUps8s53vSYrtlwEzi/Ou0=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
h=mime-version:sender:date:x-google-sender-auth:message-id:subject
:from:to:content-type:content-transfer-encoding;
b=x98qAc4bCEx25HWjw8V2cLvCbCZyR+NOscKU9Lw/r3UmFA+dubl36qVFnfcG+LrU25
7G3IhdQmVMGZo66ynvVswlsF32jc3AoZypUPRIW68k+hSscv4/squVtvAoJUWr6Oim92
fnsPzqEt/dAGzDtFpo9e+Ws9cy+Ao2kpgda80=
Received: by 10.90.69.7 with SMTP id r7mr3490795aga.47.1246832524669; Sun, 05
Jul 2009 15:22:04 -0700 (PDT)
X-Google-Sender-Auth: 6832b25d45f39da3
X-BeenThere: python-porting-+ZN9ApsXKcEdnm+***@public.gmane.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion about porting Python code \(mainly 2.x -> 3.x\)"
<python-porting.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-porting>,
<mailto:python-porting-request-+ZN9ApsXKcEdnm+***@public.gmane.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-porting>
List-Post: <mailto:python-porting-+ZN9ApsXKcEdnm+***@public.gmane.org>
List-Help: <mailto:python-porting-request-+ZN9ApsXKcEdnm+***@public.gmane.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-porting>,
<mailto:python-porting-request-+ZN9ApsXKcEdnm+***@public.gmane.org?subject=subscribe>
Sender: python-porting-bounces+gcpp-python-porting=m.gmane.org-+ZN9ApsXKcEdnm+***@public.gmane.org
Errors-To: python-porting-bounces+gcpp-python-porting=m.gmane.org-+ZN9ApsXKcEdnm+***@public.gmane.org
Archived-At: <http://permalink.gmane.org/gmane.comp.python.porting/41>
Hi,
here is a list of bugs that are triggered by running the 2to3 tool on
sympy, together with steps how to reproduce them. I am running 2to3
tool from python3.1. I created a "2to3-bugs" branch on my github
account with sympy and I am going to leave that branch there without
further modificaitons, so that you can easily debug it anytime.
1) fails with unicode:
$ git clone git://github.com/certik/sympy.git
$ cd sympy
$ git checkout -b 2to3-bugs origin/2to3-bugs
$ 2to3 . > p
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
RefactoringTool: Can't parse ./data/IPython/ipythonrc-sympy:
ParseError: bad input: type=1, value='ipythonrc', context=(' ', (25,
8))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/BasicGAtest.py: ParseError: bad input:
type=5, value=' ', context=('', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/conformalgeometryGAtest.py: ParseError:
bad input: type=5, value=' ', context=('', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/headerGAtest.py: ParseError: bad input:
type=0, value='', context=('\n', (26, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/hyperbolicGAtest.py: ParseError: bad
input: type=5, value=' ', context=('', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/reciprocalframeGAtest.py: ParseError:
bad input: type=5, value=' ', context=('', (1, 0))
Traceback (most recent call last):
File "/home/ondrej/ext/Python-3.1/Tools/scripts/2to3", line 6, in <module>
sys.exit(main("lib2to3.fixes"))
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/main.py", line 132, in main
options.processes)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
544, in refactor
items, write, doctests_only)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
207, in refactor
self.refactor_dir(dir_or_file, write, doctests_only)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
225, in refactor_dir
self.refactor_file(fullname, write, doctests_only)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
584, in refactor_file
*args, **kwargs)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
264, in refactor_file
write=write, encoding=encoding)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
363, in processed_file
self.print_output(diff_texts(old_text, new_text, filename))
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/main.py", line 47, in
print_output
print(line)
UnicodeEncodeError: 'ascii' codec can't encode character '\u03b1' in
position 32: ordinal not in range(128)
this was already reported here: http://bugs.python.org/issue5093
it also reports the "RefactoringTool: Can't parse" errors, I don't
know what that means.
2) It doesn't fix all imports:
$ git clone git://github.com/certik/sympy.git
$ cd sympy
$ git checkout -b 2to3-bugs origin/2to3-bugs
$ 2to3 -w . # this is necessary due to the bug 1)
now if you look into the sympy/__init__.py file, you need to apply the
following patch:
-from series import *
-from functions import *
-from ntheory import *
-from concrete import *
-from simplify import *
-from solvers import *
-from matrices import *
-from geometry import *
-from utilities import *
-from integrals import *
+from .series import *
+from .functions import *
+from .ntheory import *
+from .concrete import *
+from .simplify import *
+from .solvers import *
+from .matrices import *
+from .geometry import *
+from .utilities import *
+from .integrals import *
the same about most of other __init__.py files in sympy. I believe
this is a bug in 2to3 tool, that should do it automatically.
3) there is a file sympy/printing/repr.py which is imported in
sympy/printing/__init__.py. For some reason the 2to3 tool produces a
patch:
-from repr import srepr
+from reprlib import srepr
however the correct patch is:
-from repr import srepr
+from .repr import srepr
I think 2to3 tool should first check if a local file is present and if
so, treat this as a local import, not a std library import.
Besides the 3 bugs above, there are lots of small annoyances, like
that sys.version_info has changed, so one has to use tuple() around
it, that .sort() doesn't accept a cmp() like callback anymore, and
similar things, which hopefully should be relatively easy to fix in a
manner that works in all pythons.
Then there is a big problem that for some reason, hash() doesn't work
on our Integer class anymore, but seems to work on some other classes.
After I fixed the above errors by hand, here is a result of our test
suite (and this is only sympy/core, e.g. about 1/10 of all our
tests...):
http://groups.google.com/group/sympy/msg/525e8420ebb2b08d
so it is now clear that it will be a lot of work to port everything.
But I still believe it would be possible to just maintain one
codebase, and run 2to3 tool to produce python3 compatible library.
If anyone could give me some help with the bugs in 2to3 tool, it'd be
awesome, it would simplify our job a lot. Otherwise we would have to
create some specialized scripts to convert the rest of sympy (after
the 2to3 tool run) to be python3 compatible.
Ondrej
by albatross.python.org with SMTP; 06 Jul 2009 00:22:05 +0200
X-policyd-weight: using cached result; rate: -8.4
Received: from mail-yx0-f178.google.com (mail-yx0-f178.google.com
[209.85.210.178]) by mail.python.org (Postfix) with ESMTP
for <python-porting-+ZN9ApsXKcEdnm+***@public.gmane.org>; Mon, 6 Jul 2009 00:22:05 +0200 (CEST)
Received: by yxe8 with SMTP id 8so5511951yxe.9
for <python-porting-+ZN9ApsXKcEdnm+***@public.gmane.org>; Sun, 05 Jul 2009 15:22:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
h=domainkey-signature:mime-version:sender:received:date
:x-google-sender-auth:message-id:subject:from:to:content-type
:content-transfer-encoding;
bh=0YuZ1agkX4dnFp6nLRqiosYzJ60efDUcBxvgBJ5k7UY=;
b=Zg6Hv9zsjBHF/qujGlU1gxfdUOQ5vMmTl2CjAcjC1sledWNtqT+VaTRSmypY5uMTRE
HnDGs0B+aQDkaRnYHxUXnq60sA5BYQLnUZxOYjwfGuICCRSGuO8lI4wKhhQ+DTCOJRqt
ysoFjok2rueFCJJXUps8s53vSYrtlwEzi/Ou0=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
h=mime-version:sender:date:x-google-sender-auth:message-id:subject
:from:to:content-type:content-transfer-encoding;
b=x98qAc4bCEx25HWjw8V2cLvCbCZyR+NOscKU9Lw/r3UmFA+dubl36qVFnfcG+LrU25
7G3IhdQmVMGZo66ynvVswlsF32jc3AoZypUPRIW68k+hSscv4/squVtvAoJUWr6Oim92
fnsPzqEt/dAGzDtFpo9e+Ws9cy+Ao2kpgda80=
Received: by 10.90.69.7 with SMTP id r7mr3490795aga.47.1246832524669; Sun, 05
Jul 2009 15:22:04 -0700 (PDT)
X-Google-Sender-Auth: 6832b25d45f39da3
X-BeenThere: python-porting-+ZN9ApsXKcEdnm+***@public.gmane.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion about porting Python code \(mainly 2.x -> 3.x\)"
<python-porting.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-porting>,
<mailto:python-porting-request-+ZN9ApsXKcEdnm+***@public.gmane.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-porting>
List-Post: <mailto:python-porting-+ZN9ApsXKcEdnm+***@public.gmane.org>
List-Help: <mailto:python-porting-request-+ZN9ApsXKcEdnm+***@public.gmane.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-porting>,
<mailto:python-porting-request-+ZN9ApsXKcEdnm+***@public.gmane.org?subject=subscribe>
Sender: python-porting-bounces+gcpp-python-porting=m.gmane.org-+ZN9ApsXKcEdnm+***@public.gmane.org
Errors-To: python-porting-bounces+gcpp-python-porting=m.gmane.org-+ZN9ApsXKcEdnm+***@public.gmane.org
Archived-At: <http://permalink.gmane.org/gmane.comp.python.porting/41>
Hi,
here is a list of bugs that are triggered by running the 2to3 tool on
sympy, together with steps how to reproduce them. I am running 2to3
tool from python3.1. I created a "2to3-bugs" branch on my github
account with sympy and I am going to leave that branch there without
further modificaitons, so that you can easily debug it anytime.
1) fails with unicode:
$ git clone git://github.com/certik/sympy.git
$ cd sympy
$ git checkout -b 2to3-bugs origin/2to3-bugs
$ 2to3 . > p
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
RefactoringTool: Can't parse ./data/IPython/ipythonrc-sympy:
ParseError: bad input: type=1, value='ipythonrc', context=(' ', (25,
8))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/BasicGAtest.py: ParseError: bad input:
type=5, value=' ', context=('', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/conformalgeometryGAtest.py: ParseError:
bad input: type=5, value=' ', context=('', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/headerGAtest.py: ParseError: bad input:
type=0, value='', context=('\n', (26, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/hyperbolicGAtest.py: ParseError: bad
input: type=5, value=' ', context=('', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/reciprocalframeGAtest.py: ParseError:
bad input: type=5, value=' ', context=('', (1, 0))
Traceback (most recent call last):
File "/home/ondrej/ext/Python-3.1/Tools/scripts/2to3", line 6, in <module>
sys.exit(main("lib2to3.fixes"))
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/main.py", line 132, in main
options.processes)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
544, in refactor
items, write, doctests_only)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
207, in refactor
self.refactor_dir(dir_or_file, write, doctests_only)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
225, in refactor_dir
self.refactor_file(fullname, write, doctests_only)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
584, in refactor_file
*args, **kwargs)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
264, in refactor_file
write=write, encoding=encoding)
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
363, in processed_file
self.print_output(diff_texts(old_text, new_text, filename))
File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/main.py", line 47, in
print_output
print(line)
UnicodeEncodeError: 'ascii' codec can't encode character '\u03b1' in
position 32: ordinal not in range(128)
this was already reported here: http://bugs.python.org/issue5093
it also reports the "RefactoringTool: Can't parse" errors, I don't
know what that means.
2) It doesn't fix all imports:
$ git clone git://github.com/certik/sympy.git
$ cd sympy
$ git checkout -b 2to3-bugs origin/2to3-bugs
$ 2to3 -w . # this is necessary due to the bug 1)
now if you look into the sympy/__init__.py file, you need to apply the
following patch:
-from series import *
-from functions import *
-from ntheory import *
-from concrete import *
-from simplify import *
-from solvers import *
-from matrices import *
-from geometry import *
-from utilities import *
-from integrals import *
+from .series import *
+from .functions import *
+from .ntheory import *
+from .concrete import *
+from .simplify import *
+from .solvers import *
+from .matrices import *
+from .geometry import *
+from .utilities import *
+from .integrals import *
the same about most of other __init__.py files in sympy. I believe
this is a bug in 2to3 tool, that should do it automatically.
3) there is a file sympy/printing/repr.py which is imported in
sympy/printing/__init__.py. For some reason the 2to3 tool produces a
patch:
-from repr import srepr
+from reprlib import srepr
however the correct patch is:
-from repr import srepr
+from .repr import srepr
I think 2to3 tool should first check if a local file is present and if
so, treat this as a local import, not a std library import.
Besides the 3 bugs above, there are lots of small annoyances, like
that sys.version_info has changed, so one has to use tuple() around
it, that .sort() doesn't accept a cmp() like callback anymore, and
similar things, which hopefully should be relatively easy to fix in a
manner that works in all pythons.
Then there is a big problem that for some reason, hash() doesn't work
on our Integer class anymore, but seems to work on some other classes.
After I fixed the above errors by hand, here is a result of our test
suite (and this is only sympy/core, e.g. about 1/10 of all our
tests...):
http://groups.google.com/group/sympy/msg/525e8420ebb2b08d
so it is now clear that it will be a lot of work to port everything.
But I still believe it would be possible to just maintain one
codebase, and run 2to3 tool to produce python3 compatible library.
If anyone could give me some help with the bugs in 2to3 tool, it'd be
awesome, it would simplify our job a lot. Otherwise we would have to
create some specialized scripts to convert the rest of sympy (after
the 2to3 tool run) to be python3 compatible.
Ondrej