Home » Python » asynchronous programming in python

asynchronous programming in python

Posted by: admin December 8, 2017 Leave a comment


Is there a generic notion of asynchronous programming in python? Could I assign a callback to a function, execute it and return to the main program flow immediately, no matter how long the execution of that function would take?


Take a look here:

Asynchronous Programming in Python

An Introduction to Asynchronous Programming and Twisted

Worth checking out:

asyncio (previously Tulip) has been checked into the Python default branch


What you describe (the main program flow resuming immediately while another function executes) is not what’s normally called “asynchronous” (AKA “event-driven”) programming, but rather “multitasking” (AKA “multithreading” or “multiprocessing”). You can get what you described with the standard library modules threading and multiprocessing (the latter allows actual concurrent execution on multi-core machines).

Asynchronous (event-driven) programming is supported in the standard Python library in the asyncore and asynchat modules, which are very oriented to networking tasks (indeed they internally use the select module, which, on Windows, only supports sockets — though on Unixy OSs it can also support any file descriptor).

For a more general (though also mostly networking oriented, but not limited to that) support for asynchronous (event-driven) programming, check out the twisted third-party package.


Good news everyone!

Python 3.4 would include brand new ambitious asynchronous programming implementation!

It is currently called tulip and already has an active following.

As described in PEP 3153: Asynchronous IO support and PEP 3156: Asynchronous IO Support Rebooted:

People who want to write asynchronous code in Python right now have a few options:

  • asyncore and asynchat;
  • something bespoke, most likely based on the select module;
  • using a third party library, such as Twisted or gevent.

Unfortunately, each of these options has its downsides, which this PEP tries to address.

Despite having been part of the Python standard library for a long time, the asyncore module suffers from fundamental flaws following from an inflexible API that does not stand up to the expectations of a modern asynchronous networking module.

Moreover, its approach is too simplistic to provide developers with all the tools they need in order to fully exploit the potential of asynchronous networking.

The most popular solution right now used in production involves the use of third party libraries. These often provide satisfactory solutions, but there is a lack of compatibility between these libraries, which tends to make codebases very tightly coupled to the library they use.

This current lack of portability between different asynchronous IO libraries causes a lot of duplicated effort for third party library developers. A sufficiently powerful abstraction could mean that asynchronous code gets written once, but used everywhere.

Here is the brief overview of it’s abilities.


The other respondents are pointing you to Twisted, which is a great and very comprehensive framework but in my opinion it has a very un-pythonic design. Also, AFAICT, you have to use the Twisted main loop, which may be a problem for you if you’re already using something else that provides its own loop.

Here is a contrived example that would demonstrate using the threading module:

from threading import Thread

def background_stuff():
  while True:
    print "I am doing some stuff"

t = Thread(target=background_stuff)

# Continue doing some other stuff now

However, in pretty much every useful case, you will want to communicate between threads. You should look into synchronization primitives, and become familiar with the concept of concurrency and the related issues.

The threading module provides many such primitives for you to use, if you know how to use them.


You may well want to checkout the Twisted library for Python. They provide many useful tools.

  1. A little primer
  2. Defer and Related stuff

You may see my Python Asynchronous Programming tool: http://www.ideawu.com/blog/2010/08/delegate-in-pythonpython-asynchronous-programming.html

import time, random, sys
from delegate import *

def proc(a):
    return str(a)

def proc_callback(handle, args=None):
    ret = d.end(handle)

d = Delegate()
d.init(2) # number of workers

handle = d.begin(proc, '12345', proc_callback, 'test')