Home » Python » Use cases for the 'setdefault' dict method

Use cases for the 'setdefault' dict method

Posted by: admin November 1, 2017 Leave a comment

Questions:

The addition of collections.defaultdict in Python 2.5 greatly reduced the need for dict‘s setdefault method. This question is for our collective education:

  1. What is setdefault still useful for, today in Python 2.6/2.7?
  2. What popular use cases of setdefault were superseded with collections.defaultdict?
Answers:

You could say defaultdict is useful for settings defaults before filling the dict and setdefault is useful for setting defaults while or after filling the dict.

Probably the most common use case: Grouping items (in unsorted data, else use itertools.groupby)

# really verbose
new = {}
for (key, value) in data:
    if key in new:
        new[key].append( value )
    else:
        new[key] = [value]


# easy with setdefault
new = {}
for (key, value) in data:
    group = new.setdefault(key, []) # key might exist already
    group.append( value )


# even simpler with defaultdict 
new = defaultdict(list)
for (key, value) in data:
    new[key].append( value ) # all keys have a default already

Sometimes you want to make sure that specific keys exist after creating a dict. defaultdict doesn’t work in this case, because it only creates keys on explicit access. Think you use something HTTP-ish with many headers — some are optional, but you want defaults for them:

headers = parse_headers( msg ) # parse the message, get a dict
# now add all the optional headers
for headername, defaultvalue in optional_headers:
    headers.setdefault( headername, defaultvalue )

Questions:
Answers:

I commonly use setdefault for keyword argument dicts, such as in this function:

def notify(self, level, *pargs, **kwargs):
    kwargs.setdefault("persist", level >= DANGER)
    self.__defcon.set(level, **kwargs)
    try:
        kwargs.setdefault("name", self.client.player_entity().name)
    except pytibia.PlayerEntityNotFound:
        pass
    return _notify(level, *pargs, **kwargs)

It’s great for tweaking arguments in wrappers around functions that take keyword arguments.

Questions:
Answers:

defaultdict is great when the default value is static, like a new list, but not so much if it’s dynamic.

For example, I need a dictionary to map strings to unique ints. defaultdict(int) will always use 0 for the default value. Likewise, defaultdict(intGen()) always produces 1.

Instead, I used a regular dict:

nextID = intGen()
myDict = {}
for lots of complicated stuff:
    #stuff that generates unpredictable, possibly already seen str
    strID = myDict.setdefault(myStr, nextID())

Note that dict.get(key, nextID()) is insufficient because I need to be able to refer to these values later as well.

intGen is a tiny class I build that automatically increments an int and returns its value:

class intGen:
    def __init__(self):
        self.i = 0

    def __call__(self):
        self.i += 1
    return self.i

If someone has a way to do this with defaultdict I’d love to see it.

Questions:
Answers:

I use setdefault() when I want a default value in an OrderedDict. There isn’t a standard Python collection that does both, but there are ways to implement such a collection.

Questions:
Answers:

As Muhammad said, there are situations in which you only sometimes wish to set a default value. A great example of this is a data structure which is first populated, then queried.

Consider a trie. When adding a word, if a subnode is needed but not present, it must be created to extend the trie. When querying for the presence of a word, a missing subnode indicates that the word is not present and it should not be created.

A defaultdict cannot do this. Instead, a regular dict with the get and setdefault methods must be used.

Questions:
Answers:

Theoretically speaking, setdefault would still be handy if you sometimes want to set a default and sometimes not. In real life, I haven’t come across such a use case.

However, an interesting use case comes up from the standard library (Python 2.6, _threadinglocal.py):

>>> mydata = local()
>>> mydata.__dict__
{'number': 42}
>>> mydata.__dict__.setdefault('widgets', [])
[]
>>> mydata.widgets
[]

I would say that using __dict__.setdefault is a pretty useful case.

Edit: As it happens, this is the only example in the standard library and it is in a comment. So may be it is not enough of a case to justify the existence of setdefault. Still, here is an explanation:

Objects store their attributes in the __dict__ attribute. As it happens, the __dict__ attribute is writeable at any time after the object creation. It is also a dictionary not a defaultdict. It is not sensible for objects in the general case to have __dict__ as a defaultdict because that would make each object having all legal identifiers as attributes. So I can’t foresee any change to Python objects getting rid of __dict__.setdefault, apart from deleting it altogether if it was deemed not useful.

Questions:
Answers:

Here are some examples of setdefault to show its usefulness:

"""
d = {}
# To add a key->value pair, do the following:
d.setdefault(key, []).append(value)

# To retrieve a list of the values for a key
list_of_values = d[key]

# To remove a key->value pair is still easy, if
# you don't mind leaving empty lists behind when
# the last value for a given key is removed:
d[key].remove(value)

# Despite the empty lists, it's still possible to 
# test for the existance of values easily:
if d.has_key(key) and d[key]:
    pass # d has some values for key

# Note: Each value can exist multiple times!
"""
e = {}
print e
e.setdefault('Cars', []).append('Toyota')
print e
e.setdefault('Motorcycles', []).append('Yamaha')
print e
e.setdefault('Airplanes', []).append('Boeing')
print e
e.setdefault('Cars', []).append('Honda')
print e
e.setdefault('Cars', []).append('BMW')
print e
e.setdefault('Cars', []).append('Toyota')
print e

# NOTE: now e['Cars'] == ['Toyota', 'Honda', 'BMW', 'Toyota']
e['Cars'].remove('Toyota')
print e
# NOTE: it's still true that ('Toyota' in e['Cars'])

Questions:
Answers:

A likely drawback of defaultdict over dict (dict.setdefault) is that a defaultdict object creates a new item everytime non existing key is given (eg with print, ==). Also defaultdict class is way less common then dict class (serialization, representation, etc).

P.S. IMO functions (methods) not meant to mutate an object, should not mutate an object.

Questions:
Answers:

I use setdefault frequently when, get this, setting a default (!!!) in a dictionary; somewhat commonly the os.environ dictionary:

# Set the venv dir if it isn't already overridden:
os.environ.setdefault('VENV_DIR', '/my/default/path')

Less succinctly, this looks like this:

# Set the venv dir if it isn't already overridden:
if 'VENV_DIR' not in os.environ:
    os.environ['VENV_DIR'] = '/my/default/path')

It’s worth noting that you can also use the resulting variable:

venv_dir = os.environ.setdefault('VENV_DIR', '/my/default/path')

But that’s less necessary than it was before defaultdicts existed.

Questions:
Answers:

Another use case that I don’t think was mentioned above.
Sometimes you keep a cache dict of objects by their id where primary instance is in the cache and you want to set cache when missing.

return self.objects_by_id.setdefault(obj.id, obj)

That’s useful when you always want to keep a single instance per distinct id no matter how you obtain an obj each time. For example when object attributes get updated in memory and saving to storage is deferred.

Questions:
Answers:

One very important use-case I just stumbled across: dict.setdefault() is great for multi-threaded code when you only want a single canonical object (as opposed to multiple objects that happen to be equal).

For example, the (Int)Flag Enum in Python 3.6.0 has a bug: if multiple threads are competing for a composite (Int)Flag member, there may end up being more than one:

from enum import IntFlag, auto
import threading

class TestFlag(IntFlag):
    one = auto()
    two = auto()
    three = auto()
    four = auto()
    five = auto()
    six = auto()
    seven = auto()
    eight = auto()

    def __eq__(self, other):
        return self is other

    def __hash__(self):
        return hash(self.value)

seen = set()

class cycle_enum(threading.Thread):
    def run(self):
        for i in range(256):
            seen.add(TestFlag(i))

threads = []
for i in range(8):
    threads.append(cycle_enum())

for t in threads:
    t.start()

for t in threads:
    t.join()

len(seen)
# 272  (should be 256)

The solution is to use setdefault() as the last step of saving the computed composite member — if another has already been saved then it is used instead of the new one, guaranteeing unique Enum members.

Questions:
Answers:
[Edit] Very wrong! The setdefault would always trigger long_computation, Python being eager.

Expanding on Tuttle’s answer. For me the best use case is cache mechanism. Instead of:

if x not in memo:
   memo[x]=long_computation(x)
return memo[x]

which consumes 3 lines and 2 or 3 lookups, I would happily write :

return memo.setdefault(x, long_computation(x))

Questions:
Answers:

When the required default value is not always the same, or it’s only desired for specific keys but it’s preferred not to have one for others, one might consider using setdefault:

d = {}
...
# `i` should default to zero
i = d.setdefault(key, 0)
...
# `s` should default to an empty string
s = d.setdefault(key, '')
...

 

d = {}
...
# v should always default to a list
v = d.setdefault(key, [])
...
try:
    # EAFP, but I need the dict to raise a KeyError if the key is not found.
    w = d[k2]
except KeyError:
    ...
...

Questions:
Answers:

I like the answer given here:

http://stupidpythonideas.blogspot.com/2013/08/defaultdict-vs-setdefault.html

In short, the decision (in non-performance-critical apps) should be made on the basis of how you want to handle lookup of empty keys downstream (viz. KeyError versus default value).

Questions:
Answers:

The different use case for setdefault() is when you don’t want to overwrite the value of an already set key. defaultdict overwrites, while setdefault() does not. For nested dictionaries it is more often the case that you want to set a default only if the key is not set yet, because you don’t want to remove the present sub dictionary. This is when you use setdefault().

Example with defaultdict:

>>> from collection import defaultdict()
>>> foo = defaultdict()
>>> foo['a'] = 4
>>> foo['a'] = 2
>>> print(foo)
defaultdict(None, {'a': 2})

setdefault doesn’t overwrite:

>>> bar = dict()
>>> bar.setdefault('a', 4)
>>> bar.setdefault('a', 2)
>>> print(bar)
{'a': 4}