Home » Mysql » imploding a list for use in a python MySQLDB IN clause

imploding a list for use in a python MySQLDB IN clause

Posted by: admin November 1, 2017 Leave a comment

Questions:

I know how to map a list to a string:

foostring = ",".join( map(str, list_of_ids) )

And I know that I can use the following to get that string into an IN clause:

cursor.execute("DELETE FROM foo.bar WHERE baz IN ('%s')" % (foostring))

What I need is to accomplish the same thing SAFELY (avoiding SQL injection) using MySQLDB. In the above example because foostring is not passed as an argument to execute, it is vulnerable. I also have to quote and escape outside of the mysql library.

(There is a related SO question, but the answers listed there either do not work for MySQLDB or are vulnerable to SQL injection.)

Answers:

Use the list_of_ids directly:

format_strings = ','.join(['%s'] * len(list_of_ids))
cursor.execute("DELETE FROM foo.bar WHERE baz IN (%s)" % format_strings,
                tuple(list_of_ids))

That way you avoid having to quote yourself, and avoid all kinds of sql injection.

Note that the data (list_of_ids) is going directly to mysql’s driver, as a parameter (not in the query text) so there is no injection. You can leave any chars you want in the string, no need to remove or quote chars.

Questions:
Answers:

Pain-less MySQLdb execute('...WHERE name1 = %s AND name2 IN (%s)', value1, values2)

def execute(sql, *values):

    assert sql.count('%s') == len(values), (sql, values)
    placeholders = []
    new_values = []
    for value in values:
        if isinstance(value, (list, tuple)):
            placeholders.append(', '.join(['%s'] * len(value)))
            new_values.extend(value)
        else:
            placeholders.append('%s')
            new_values.append(value)
    sql = sql % tuple(placeholders)
    values = tuple(new_values)

    # ... cursor.execute(sql, values)

Questions:
Answers:
list_of_ids = [ 1, 2, 3]
query = "select * from table where x in %s" % str(tuple(list_of_ids))
print query

This could work for some use-cases if you don’t wish to be concerned with the method in which you have to pass arguments to complete the query string and would like to invoke just cursror.execute(query).

Another way could be:

"select * from table where x in (%s)" % ', '.join(str(id) for id in list_of_ids)

Questions:
Answers:

As this person suggested (Executing "SELECT … WHERE … IN …" using MySQLdb), it is faster to use itertools.repeat() to create the list of ‘%s’s than to multiply a [‘%s’] list (and much faster than using map()), especially for long lists.

in_p = ', '.join(itertools.repeat('%s', len(args)))

These timeits were done using Python 2.7.3 with an Intel Core i5 CPU M 540 @ 2.53GHz × 4:

>>> timeit.timeit("repeat('%s', len(short_list))", 'from itertools import repeat; short_list = range(3)')
0.20310497283935547
>>> timeit.timeit("['%s'] * len(short_list)", 'short_list = range(3)')
0.263930082321167
>>> timeit.timeit("list(map(lambda x:'%s', short_list))", 'short_list = range(3)')
0.7543060779571533


>>> timeit.timeit("repeat('%s', len(long_list))", 'from itertools import repeat; long_list = range(1000)')
0.20342397689819336
>>> timeit.timeit("['%s'] * len(long_list)", 'long_list = range(1000)')
4.700995922088623
>>> timeit.timeit("list(map(lambda x:'%s', long_list))", 'long_list = range(1000)')
100.05319118499756