Home » Python » Existence of mutable named tuple in Python?

Existence of mutable named tuple in Python?

Posted by: admin November 30, 2017 Leave a comment

Questions:

Can anyone amend namedtuple or provide an alternative class so that it works for mutable objects?

Primarily for readability, I would like something similar to namedtuple that does this:

from Camelot import namedgroup

Point = namedgroup('Point', ['x', 'y'])
p = Point(0, 0)
p.x = 10

>>> p
Point(x=10, y=0)

>>> p.x *= 10
Point(x=100, y=0)

It must be possible to pickle the resulting object. And per the characteristics of named tuple, the ordering of the output when represented must match the order of the parameter list when constructing the object.

RESPONSE:
Thanks to everyone who submitted suggestions. I believe that the recordclass referred by @intellimath is the best solution (also see here).

recordclass 0.4
Mutable variant of collections.namedtuple, which supports assignments

recordclass is MIT Licensed python library. It implements the type
memoryslots and factory function recordclass in order to create
record-like classes.

memoryslots is tuple-like type, which supports assignment operations.
recordclass is a factory function that create a “mutable” analog of
collection.namedtuple. This library actually is a “proof of concept”
for the problem of “mutable” alternative of namedtuple.

I also ran some tests against all of the suggestions. Not all of these features were requested, so the comparison isn’t really fair. The tests are here just to point out the usability of each class.

# Option 1 (p1): @kennes913
# Option 2 (p2): @MadMan2064
# Option 3 (p3): @intellimath   
# Option 4 (p4): @Roland Smith
# Option 5 (p5): @agomcas
# Option 6 (p6): @Antti Haapala


# TEST:                             p1     p2     p3     p4     p5      p6 
# 1.  Mutation of field values   |  x   |   x  |   x  |   x  |   x  |   x  |
# 2.  String                     |      |   x  |   x  |   x  |      |   x  |
# 3.  Representation             |      |   x  |   x  |   x  |      |   x  |
# 4.  Sizeof                     |  x   |   x  |   x  |   ?  |  ??  |   x  |
# 5.  Access by name of field    |  x   |   x  |   x  |   x  |   x  |   x  |
# 6.  Access by index.           |      |      |   x  |      |      |      |
# 7.  Iterative unpacking.       |      |   x  |   x  |      |      |   x  |
# 8.  Iteration                  |      |   x  |   x  |      |      |   x  |
# 9.  Ordered Dict               |      |      |   x  |      |      |      |
# 10. Inplace replacement        |      |      |   x  |      |      |      |
# 11. Pickle and Unpickle        |      |      |   x  |      |      |      |
# 12. Fields*                    |      |      | yes  |      |  yes |      |
# 13. Slots*                     |  yes |      |      |      |  yes |      |

# *Note that I'm not very familiar with slots and fields, so please excuse 
# my ignorance in reporting their results.  I have included them for completeness.

# Class/Object creation.
p1 = Point1(x=1, y=2)

Point2 = namedgroup("Point2", ["x", "y"])
p2 = Point2(x=1, y=2)

Point3 = recordclass('Point3', 'x y')   # ***
p3 = Point3(x=1, y=2)

p4 = AttrDict()
p4.x = 1
p4.y = 2

p5 = namedlist('Point5', 'x y')

Point6 = namedgroup('Point6', ['x', 'y'])
p6 = Point6(x=1, y=2)

point_objects = [p1, p2, p3, p4, p5, p6]

# 1. Mutation of field values.
for n, p in enumerate(point_objects):
    try:
        p.x *= 10
        p.y += 10
        print('p{0}: {1}, {2}'.format(n + 1, p.x, p.y))
    except Exception as e:
        print('p{0}: Mutation not supported. {1}'.format(n + 1, e))

p1: 10, 12
p2: 10, 12
p3: 10, 12
p4: 10, 12
p5: 10, 12
p6: 10, 12


# 2. String.
for n, p in enumerate(point_objects):
    print('p{0}: {1}'.format(n + 1, p))
p1: <__main__.Point1 instance at 0x10c72dc68>
p2: Point2(x=10, y=12)
p3: Point3(x=10, y=12)
p4: {'y': 12, 'x': 10}
p5: <class '__main__.Point5'>
p6: Point6(x=10, y=12)


# 3. Representation.
[('p{0}'.format(n + 1), p) for n, p in enumerate(point_objects)]

[('p1', <__main__.Point1 instance at 0x10c72dc68>),
 ('p2', Point2(x=10, y=12)),
 ('p3', Point3(x=10, y=12)),
 ('p4', {'x': 10, 'y': 12}),
 ('p5', __main__.Point5),
 ('p6', Point6(x=10, y=12))]


# 4. Sizeof.
for n, p in enumerate(point_objects):
    print("size of p{0}:".format(n + 1), sys.getsizeof(p))

size of p1: 72
size of p2: 64
size of p3: 72
size of p4: 280
size of p5: 904
size of p6: 64


# 5. Access by name of field.
for n, p in enumerate(point_objects):
    print('p{0}: {1}, {2}'.format(n + 1, p.x, p.y))

p1: 10, 12
p2: 10, 12
p3: 10, 12
p4: 10, 12
p5: 10, 12
p6: 10, 12


# 6. Access by index.
for n, p in enumerate(point_objects):
    try:
        print('p{0}: {1}, {2}'.format(n + 1, p[0], p[1]))
    except:
        print('p{0}: Unable to access by index.'.format(n+1))

p1: Unable to access by index.
p2: Unable to access by index.
p3: 10, 12
p4: Unable to access by index.
p5: Unable to access by index.
p6: Unable to access by index.


# 7. Iterative unpacking.
for n, p in enumerate(point_objects):
    try:
        x, y = p
        print('p{0}: {1}, {2}'.format(n + 1, x, y))
    except:
        print('p{0}: Unable to unpack.'.format(n + 1))

p1: Unable to unpack.
p2: 10, 12
p3: 10, 12
p4: y, x
p5: Unable to unpack.
p6: 10, 12


# 8. Iteration
for n, p in enumerate(point_objects):
    try:
        print('p{0}: {1}'.format(n + 1, [v for v in p]))
    except:
        print('p{0}: Unable to iterate.'.format(n + 1))

p1: Unable to iterate.
p2: [10, 12]
p3: [10, 12]
p4: ['y', 'x']
p5: Unable to iterate.
p6: [10, 12]
In [95]:


# 9. Ordered Dict
for n, p in enumerate(point_objects):
    try:
        print('p{0}: {1}'.format(n + 1, p._asdict()))
    except:
        print('p{0}: Unable to create Ordered Dict.'.format(n + 1))

p1: Unable to create Ordered Dict.
p2: Unable to create Ordered Dict.
p3: OrderedDict([('x', 10), ('y', 12)])
p4: Unable to create Ordered Dict.
p5: Unable to create Ordered Dict.
p6: Unable to create Ordered Dict.


# 10. Inplace replacement
for n, p in enumerate(point_objects):
    try:
        p_ = p._replace(x=100, y=200)
        print('p{0}: {1} - {2}'.format(n + 1, 'Success' if p is p_ else 'Failure', p))
    except:
        print('p{0}: Unable to replace inplace.'.format(n + 1))

p1: Unable to replace inplace.
p2: Unable to replace inplace.
p3: Success - Point3(x=100, y=200)
p4: Unable to replace inplace.
p5: Unable to replace inplace.
p6: Unable to replace inplace.


# 11. Pickle and Unpickle.
for n, p in enumerate(point_objects):
    try:
        pickled = pickle.dumps(p)
        unpickled = pickle.loads(pickled)
        if p != unpickled:
            raise ValueError((p, unpickled))
        print('p{0}: {1}'.format(n + 1, 'Pickled successfully', ))
    except Exception as e:
        print('p{0}: {1}; {2}'.format(n + 1, 'Pickle failure', e))

p1: Pickle failure; (<__main__.Point1 instance at 0x10c72dc68>, <__main__.Point1 instance at 0x10ca631b8>)
p2: Pickle failure; (Point2(x=10, y=12), Point2(x=10, y=12))
p3: Pickled successfully
p4: Pickle failure; '__getstate__'
p5: Pickle failure; Can't pickle <class '__main__.Point5'>: it's not found as __main__.Point5
p6: Pickle failure; (Point6(x=10, y=12), Point6(x=10, y=12))


# 12. Fields.
for n, p in enumerate(point_objects):
    try:
        print('p{0}: {1}'.format(n + 1, p._fields))
    except Exception as e:
        print('p{0}: {1}; {2}'.format(n + 1, 'Unable to access fields.', e))

p1: Unable to access fields.; Point1 instance has no attribute '_fields'
p2: Unable to access fields.; 'Point2' object has no attribute '_fields'
p3: ('x', 'y')
p4: Unable to access fields.; '_fields'
p5: ('x', 'y')
p6: Unable to access fields.; 'Point6' object has no attribute '_fields'


# 13. Slots.
for n, p in enumerate(point_objects):
    try:
        print('p{0}: {1}'.format(n + 1, p.__slots__))
    except Exception as e:
        print('p{0}: {1}; {2}'.format(n + 1, 'Unable to access slots', e))

p1: ['x', 'y']
p2: Unable to access slots; 'Point2' object has no attribute '__slots__'
p3: ()
p4: Unable to access slots; '__slots__'
p5: ('x', 'y')
p6: Unable to access slots; 'Point6' object has no attribute '__slots__'
Answers:

There is a mutable alternative to collections.namedtuplerecordclass.

It has the same API and memory footprint as namedtuple and it supports assignments (It should be faster as well). For example:

from recordclass import recordclass

Point = recordclass('Point', 'x y')

>>> p = Point(1, 2)
>>> p
Point(x=1, y=2)
>>> print(p.x, p.y)
1 2
>>> p.x += 2; p.y += 3; print(p)
Point(x=3, y=5)

There is a more complete example (it also includes performance comparisons).

Questions:
Answers:

It seems like the answer to this question is no.

Below is pretty close, but it’s not technically mutable. This is creating a new namedtuple() instance with an updated x value:

Point = namedtuple('Point', ['x', 'y'])
p = Point(0, 0)
p = p._replace(x=10) 

On the other hand, you can create a simple class using __slots__ that should work well for frequently updating class instance attributes:

class Point:
    __slots__ = ['x', 'y']
    def __init__(self, x, y):
        self.x = x
        self.y = y

To add to this answer, I think __slots__ is good use here because it’s memory efficient when you create lots of class instances. The only downside is that you can’t create new class attributes.

Here’s one relevant thread that illustrates the memory efficiency – Dictionary vs Object – which is more efficient and why?

The quoted content in the answer of this thread is a very succinct explanation why __slots__ is more memory efficient – Python slots

Questions:
Answers:

The latest namedlist 1.7 passes all of your tests with both Python 2.7 and Python 3.5 as of Jan 11, 2016. It is a pure python implementation whereas the recordclass is a C extension. Of course, it depends on your requirements whether a C extension is preferred or not.

Your tests (but also see the note below):

from __future__ import print_function
import pickle
import sys
from namedlist import namedlist

Point = namedlist('Point', 'x y')
p = Point(x=1, y=2)

print('1. Mutation of field values')
p.x *= 10
p.y += 10
print('p: {}, {}\n'.format(p.x, p.y))

print('2. String')
print('p: {}\n'.format(p))

print('3. Representation')
print(repr(p), '\n')

print('4. Sizeof')
print('size of p:', sys.getsizeof(p), '\n')

print('5. Access by name of field')
print('p: {}, {}\n'.format(p.x, p.y))

print('6. Access by index')
print('p: {}, {}\n'.format(p[0], p[1]))

print('7. Iterative unpacking')
x, y = p
print('p: {}, {}\n'.format(x, y))

print('8. Iteration')
print('p: {}\n'.format([v for v in p]))

print('9. Ordered Dict')
print('p: {}\n'.format(p._asdict()))

print('10. Inplace replacement (update?)')
p._update(x=100, y=200)
print('p: {}\n'.format(p))

print('11. Pickle and Unpickle')
pickled = pickle.dumps(p)
unpickled = pickle.loads(pickled)
assert p == unpickled
print('Pickled successfully\n')

print('12. Fields\n')
print('p: {}\n'.format(p._fields))

print('13. Slots')
print('p: {}\n'.format(p.__slots__))

Output on Python 2.7

1. Mutation of field values  
p: 10, 12

2. String  
p: Point(x=10, y=12)

3. Representation  
Point(x=10, y=12) 

4. Sizeof  
size of p: 64 

5. Access by name of field  
p: 10, 12

6. Access by index  
p: 10, 12

7. Iterative unpacking  
p: 10, 12

8. Iteration  
p: [10, 12]

9. Ordered Dict  
p: OrderedDict([('x', 10), ('y', 12)])

10. Inplace replacement (update?)  
p: Point(x=100, y=200)

11. Pickle and Unpickle  
Pickled successfully

12. Fields  
p: ('x', 'y')

13. Slots  
p: ('x', 'y')

The only difference with Python 3.5 is that the namedlist has become smaller, the size is 56 (Python 2.7 reports 64).

Note that I have changed your test 10 for in-place replacement. The namedlist has a _replace() method which does a shallow copy, and that makes perfect sense to me because the namedtuple in the standard library behaves the same way. Changing the semantics of the _replace() method would be confusing. In my opinion the _update() method should be used for in-place updates. Or maybe I failed to understand the intent of your test 10?

Questions:
Answers:

The following is a good solution for Python 3: A minimal class using __slots__ and Sequence abstract base class; does not do fancy error detection or such, but it works, and behaves mostly like a mutable tuple (except for typecheck).

from collections import Sequence

class NamedMutableSequence(Sequence):
    __slots__ = ()

    def __init__(self, *a, **kw):
        slots = self.__slots__
        for k in slots:
            setattr(self, k, kw.get(k))

        if a:
            for k, v in zip(slots, a):
                setattr(self, k, v)

    def __str__(self):
        clsname = self.__class__.__name__
        values = ', '.join('%s=%r' % (k, getattr(self, k))
                           for k in self.__slots__)
        return '%s(%s)' % (clsname, values)

    __repr__ = __str__

    def __getitem__(self, item):
        return getattr(self, self.__slots__[item])

    def __setitem__(self, item, value):
        return setattr(self, self.__slots__[item], value)

    def __len__(self):
        return len(self.__slots__)

class Point(NamedMutableSequence):
    __slots__ = ('x', 'y')

Example:

>>> p = Point(0, 0)
>>> p.x = 10
>>> p
Point(x=10, y=0)
>>> p.x *= 10
>>> p
Point(x=100, y=0)

If you want, you can have a method to create the class too (though using an explicit class is more transparent):

def namedgroup(name, members):
    if isinstance(members, str):
        members = members.split()
    members = tuple(members)
    return type(name, (NamedMutableSequence,), {'__slots__': members})

Example:

>>> Point = namedgroup('Point', ['x', 'y'])
>>> Point(6, 42)
Point(x=6, y=42)

In Python 2 you need to adjust it slightly – if you inherit from Sequence, the class will have a __dict__ and the __slots__ will stop from working.

The solution in Python 2 is to not inherit from Sequence, but object. If isinstance(Point, Sequence) == True is desired, you need to register the NamedMutableSequence as a base class to Sequence:

Sequence.register(NamedMutableSequence)

Questions:
Answers:

Let’s implement this with dynamic type creation:

import copy
def namedgroup(typename, fieldnames):

    def init(self, **kwargs): 
        attrs = {k: None for k in self._attrs_}
        for k in kwargs:
            if k in self._attrs_:
                attrs[k] = kwargs[k]
            else:
                raise AttributeError('Invalid Field')
        self.__dict__.update(attrs)

    def getattribute(self, attr):
        if attr.startswith("_") or attr in self._attrs_:
            return object.__getattribute__(self, attr)
        else:
            raise AttributeError('Invalid Field')

    def setattr(self, attr, value):
        if attr in self._attrs_:
            object.__setattr__(self, attr, value)
        else:
            raise AttributeError('Invalid Field')

    def rep(self):
         d = ["{}={}".format(v,self.__dict__[v]) for v in self._attrs_]
         return self._typename_ + '(' + ', '.join(d) + ')'

    def iterate(self):
        for x in self._attrs_:
            yield self.__dict__[x]
        raise StopIteration()

    def setitem(self, *args, **kwargs):
        return self.__dict__.__setitem__(*args, **kwargs)

    def getitem(self, *args, **kwargs):
        return self.__dict__.__getitem__(*args, **kwargs)

    attrs = {"__init__": init,
                "__setattr__": setattr,
                "__getattribute__": getattribute,
                "_attrs_": copy.deepcopy(fieldnames),
                "_typename_": str(typename),
                "__str__": rep,
                "__repr__": rep,
                "__len__": lambda self: len(fieldnames),
                "__iter__": iterate,
                "__setitem__": setitem,
                "__getitem__": getitem,
                }

    return type(typename, (object,), attrs)

This checks the attributes to see if they are valid before allowing the operation to continue.

So is this pickleable? Yes if (and only if) you do the following:

>>> import pickle
>>> Point = namedgroup("Point", ["x", "y"])
>>> p = Point(x=100, y=200)
>>> p2 = pickle.loads(pickle.dumps(p))
>>> p2.x
100
>>> p2.y
200
>>> id(p) != id(p2)
True

The definition has to be in your namespace, and must exist long enough for pickle to find it. So if you define this to be in your package, it should work.

Point = namedgroup("Point", ["x", "y"])

Pickle will fail if you do the following, or make the definition temporary (goes out of scope when the function ends, say):

some_point = namedgroup("Point", ["x", "y"])

And yes, it does preserve the order of the fields listed in the type creation.

Questions:
Answers:

types.SimpleNamespace was introduced in Python 3.3 and supports the requested requirements.

from types import SimpleNamespace
t = SimpleNamespace(foo='bar')
t.ham = 'spam'
print(t)
namespace(foo='bar', ham='spam')
print(t.foo)
'bar'
import pickle
with open('/tmp/pickle', 'wb') as f:
    pickle.dump(t, f)

“`

Questions:
Answers:

Tuples are by definition immutable.

You can however make a dictionary subclass where you can access the attributes with dot-notation;

In [1]: %cpaste
Pasting code; enter '--' alone on the line to stop or use Ctrl-D.
:class AttrDict(dict):
:
:    def __getattr__(self, name):
:        return self[name]
:
:    def __setattr__(self, name, value):
:        self[name] = value
:--

In [2]: test = AttrDict()

In [3]: test.a = 1

In [4]: test.b = True

In [5]: test
Out[5]: {'a': 1, 'b': True}

Questions:
Answers:

If you want similar behavior as namedtuples but mutable try namedlist

Note that in order to be mutable it cannot be a tuple.

Questions:
Answers:

Provided performance is of little importance, one could use a silly hack like:

from collection import namedtuple

Point = namedtuple('Point', 'x y z')
mutable_z = Point(1,2,[3])