Strange behavior of eval() in Python 3.1

[Bo Peng]

Dear all,

This might be the wrong mailinglist to ask, but I bumped into a
strange problem when I ported my code from Python 2.7 to 3.1. More
specifically, my code involves the evaluation of expressions in a lot
of local namespaces (dictionaries). For example, the following code
evaluates expressions in a dictionary dd under Python 2.7:

Python 2.7 (r27:82525, Jul 4 2010, 09:01:59) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

dd = {'a': {1: 0.1, 2: 0.2}}
print(eval('a[1]', {}, dd))

0.1

print(eval("[x for x in a.keys()]", {}, dd))

[1, 2]

print(eval("[a[x] for x in a.keys()]", dd, dd))

[0.1, 0.2]

print(eval("[a[x] for x in a.keys()]", {}, dd))

[0.1, 0.2]

However, the last statement fails under Python 3.1

Python 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

dd = {'a': {1: 0.1, 2: 0.2}}
print(eval('a[1]', {}, dd))

0.1

print(eval("[x for x in a.keys()]", {}, dd))

[1, 2]

print(eval("[a[x] for x in a.keys()]", dd, dd))

[0.1, 0.2]

print(eval("[a[x] for x in a.keys()]", {}, dd))

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1, in <module>
File "<string>", line 1, in <listcomp>
NameError: global name 'a' is not defined
Because a is in clearly accessible from the passed dictionary, I have
no idea why it has to be defined in the global dictionary. Does anyone
know what is going on here? How can I change make it work with Python
3?

Many thanks in advance.
Bo

[Benjamin Peterson]

Post by Bo Peng
Dear all,
This might be the wrong mailinglist to ask, but I bumped into a
strange problem when I ported my code from Python 2.7 to 3.1. More
specifically, my code involves the evaluation of expressions in a lot
of local namespaces (dictionaries). For example, the following code
Python 2.7 (r27:82525, Jul  4 2010, 09:01:59) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

dd = {'a': {1: 0.1, 2: 0.2}}
print(eval('a[1]', {}, dd))

0.1

print(eval("[x for x in a.keys()]", {}, dd))

[1, 2]

print(eval("[a[x] for x in a.keys()]", dd, dd))

[0.1, 0.2]

print(eval("[a[x] for x in a.keys()]", {}, dd))

[0.1, 0.2]
However, the last statement fails under Python 3.1
Python 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

dd = {'a': {1: 0.1, 2: 0.2}}
print(eval('a[1]', {}, dd))

0.1

print(eval("[x for x in a.keys()]", {}, dd))

[1, 2]

print(eval("[a[x] for x in a.keys()]", dd, dd))

[0.1, 0.2]

print(eval("[a[x] for x in a.keys()]", {}, dd))

 File "<stdin>", line 1, in <module>
 File "<string>", line 1, in <module>
 File "<string>", line 1, in <listcomp>
NameError: global name 'a' is not defined
Because a is in clearly accessible from the passed dictionary, I have
no idea why it has to be defined in the global dictionary. Does anyone
know what is going on here? How can I change make it work with Python
3?

List comprehensions are now properly scoped, so having one in the
global scope, like you do in this case, will lookup the names in the
global namespace.

[Bo Peng]

Post by Benjamin Peterson

Post by Bo Peng

dd = {'a': {1: 0.1, 2: 0.2}}
print(eval("[a[x] for x in a.keys()]", {}, dd))

 File "<stdin>", line 1, in <module>
 File "<string>", line 1, in <module>
 File "<string>", line 1, in <listcomp>
NameError: global name 'a' is not defined

List comprehensions are now properly scoped, so having one in the
global scope, like you do in this case, will lookup the names in the
global namespace.

Could you please provide more details (like a link to documentation)
what is going on here? From what I can see, I am evaluating an
expression in a dictionary with 'a' defined (and x is a iterator
variable), why does simuPOP have to look for it in the global
namespace? I mean, if 'a[1]', 'a[2]', '[a[1], a[2]]' are all valid
expressions, why not '[a[x] for x in [1,2]]'? Should not "[a[1],
a[2]]" always return the same result as "[a[x] for x in [1,2]]"? The
fact that

dd = {'a': {1:0.1, 2:0.2}}
dd1 = {'a': {1:1.1, 2:1.2}}
eval('a[1], a[2], list(a.keys())', dd1, dd)

returns 0.1, 0.2, [1, 2] is REALLY puzzling.

Bo

[Bo Peng]

Post by Bo Peng
dd = {'a': {1:0.1, 2:0.2}}
dd1 = {'a': {1:1.1, 2:1.2}}
eval('a[1], a[2], list(a.keys())', dd1, dd)

Sorry, I actually meant:

'a[1], a[2], [a[x] for x in a.keys()]'

Post by Bo Peng
returns 0.1, 0.2, [1, 2] is REALLY puzzling.

Bo

[Georg Brandl]

Post by Bo Peng

Post by Benjamin Peterson

Post by Bo Peng

dd = {'a': {1: 0.1, 2: 0.2}}
print(eval("[a[x] for x in a.keys()]", {}, dd))

File "<stdin>", line 1, in <module>
File "<string>", line 1, in <module>
File "<string>", line 1, in <listcomp>
NameError: global name 'a' is not defined

List comprehensions are now properly scoped, so having one in the
global scope, like you do in this case, will lookup the names in the
global namespace.

Could you please provide more details (like a link to documentation)
what is going on here? From what I can see, I am evaluating an
expression in a dictionary with 'a' defined (and x is a iterator
variable), why does simuPOP have to look for it in the global
namespace? I mean, if 'a[1]', 'a[2]', '[a[1], a[2]]' are all valid
expressions, why not '[a[x] for x in [1,2]]'? Should not "[a[1],
a[2]]" always return the same result as "[a[x] for x in [1,2]]"?

No. As I wrote in my reply, try executing

list(a[x] for x in [1,2])

in Python 2, and you will see that this exhibits the same behavior as
the list comprehension does in Python 3.

The difference between simple list literals and a comprehension is that
the comprehension can bind new names. List comprehensions in Python 2
"leaked" these names into the surrounding scope, which was fixed by
unifying their implementation with generator expressions' in Python 3.

Note that this specific issue only appears because you're using eval().
For static code, your access to "a" outside of the comprehension's scope
will be recognized and a proper closure generated.

Georg

[Bo Peng]

Post by Georg Brandl
Note that this specific issue only appears because you're using eval().
For static code, your access to "a" outside of the comprehension's scope
will be recognized and a proper closure generated.

Thank you very much for your explanation and now I have a real problem
to fix for my Python extension module. More specifically, I call
"eval(expression, module_dict, user_dict)" to evaluate expressions in
user_dict. However, to evaluate expressions such as "[a[x] for x in
a]", I would need "eval(expression, user_dict, user_dict)" to make 'a'
available in globals. But then I do not have access to names in
module_dict, and simple expressions such as 'len(a)' would fail
because I am doing all these from the C/C++ level and there is no
__builtins__ in user_dict.

Happy python-porting, :-)

Bo

[Georg Brandl]

Post by Bo Peng

Post by Georg Brandl
Note that this specific issue only appears because you're using eval().
For static code, your access to "a" outside of the comprehension's scope
will be recognized and a proper closure generated.

Thank you very much for your explanation and now I have a real problem
to fix for my Python extension module. More specifically, I call
"eval(expression, module_dict, user_dict)" to evaluate expressions in
user_dict. However, to evaluate expressions such as "[a[x] for x in
a]", I would need "eval(expression, user_dict, user_dict)" to make 'a'
available in globals. But then I do not have access to names in
module_dict, and simple expressions such as 'len(a)' would fail
because I am doing all these from the C/C++ level and there is no
__builtins__ in user_dict.

Surely you can do

namespace = module_dict.copy()
namespace.update(user_dict)
eval(..., namespace)

?

Georg

[Bo Peng]

Post by Georg Brandl
Surely you can do
namespace = module_dict.copy()
namespace.update(user_dict)
eval(..., namespace)

I know, but this would be really slow because a lot of such
evaluations are needed. I think what I will do is to set __builtins__
to user_dict and use eval(expression, user_dict, user_dict) to
evaluate the expressions. Because my expressions rarely involve
variables in module_dict (except for __builtins__), restricting the
expressions to user_dict might be acceptable. If this solution is not
enough, I can still use

try:
eval(expression, user_dict, user_dict)
except NameError:
get namespace like what you suggested
eval(expression, namespace, user_dict)

to handle these rare situations.

Thank you very much for your help,
Bo

[Jürgen Hamel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, 29 Aug 2010 10:16:35 -0500

Post by Bo Peng
Happy python-porting, :-)

Hi you,
I thought since 2 years about porting and i gave up. Do it like me:
I DO NOT PORTING TO 3.xxx

I hope, that the next 10 years the most distris have a Python 2.7 and then it gives perhaps a Python 4.0, that compatible is to 2.7 and 3.x .

ok, really, I will wait some years, it is to much work for me to port and so I hope really, that they found a solution.

And, to all at this lists, is there an Internet site, that show me the porting status of some important python modules:
pygtk, twisted, GeoIP, aiml, gtksourceview, pil, ... and so on ?

best regards
Jürgen

- --
Cyrus-Computer GmbH Linux Server Support Jürgen Hamel
Cuon - Warenwirtschaft mit Linux http://www.cuon.org
Twitter: cuonOne Jabber: ***@cuonsim2.de

[Georg Brandl]

Post by Bo Peng
However, the last statement fails under Python 3.1
Python 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

dd = {'a': {1: 0.1, 2: 0.2}}
print(eval('a[1]', {}, dd))

0.1

print(eval("[x for x in a.keys()]", {}, dd))

[1, 2]

print(eval("[a[x] for x in a.keys()]", dd, dd))

[0.1, 0.2]

print(eval("[a[x] for x in a.keys()]", {}, dd))

File "<stdin>", line 1, in <module>
File "<string>", line 1, in <module>
File "<string>", line 1, in <listcomp>
NameError: global name 'a' is not defined
Because a is in clearly accessible from the passed dictionary, I have
no idea why it has to be defined in the global dictionary. Does anyone
know what is going on here? How can I change make it work with Python
3?

In Python 3, all comprehensions are executed in their own function scope,
like generator expressions in Python 2. (If you used ``list(a[x] for x
in a.keys())``, it would fail in Python 2 too.)

Therefore, and because the compilation of the expression to eval() can
not generate closures (because it does not know in advance what locals
are available), it has to look up "a" as a global.

Georg