Is there a generic notion of asynchronous programming in python? Could I assign a callback to a function, execute it and return to the main program flow immediately, no matter how long the execution of that function would take?
Take a look here:
Worth checking out:
What you describe (the main program flow resuming immediately while another function executes) is not what’s normally called “asynchronous” (AKA “event-driven”) programming, but rather “multitasking” (AKA “multithreading” or “multiprocessing”). You can get what you described with the standard library modules
multiprocessing (the latter allows actual concurrent execution on multi-core machines).
Asynchronous (event-driven) programming is supported in the standard Python library in the
asynchat modules, which are very oriented to networking tasks (indeed they internally use the
select module, which, on Windows, only supports sockets — though on Unixy OSs it can also support any file descriptor).
For a more general (though also mostly networking oriented, but not limited to that) support for asynchronous (event-driven) programming, check out the twisted third-party package.
Good news everyone!
Python 3.4 would include brand new ambitious asynchronous programming implementation!
People who want to write asynchronous code in Python right now have a few options:
- asyncore and asynchat;
- something bespoke, most likely based on the select module;
- using a third party library, such as Twisted or gevent.
Unfortunately, each of these options has its downsides, which this PEP tries to address.
Despite having been part of the Python standard library for a long time, the asyncore module suffers from fundamental flaws following from an inflexible API that does not stand up to the expectations of a modern asynchronous networking module.
Moreover, its approach is too simplistic to provide developers with all the tools they need in order to fully exploit the potential of asynchronous networking.
The most popular solution right now used in production involves the use of third party libraries. These often provide satisfactory solutions, but there is a lack of compatibility between these libraries, which tends to make codebases very tightly coupled to the library they use.
This current lack of portability between different asynchronous IO libraries causes a lot of duplicated effort for third party library developers. A sufficiently powerful abstraction could mean that asynchronous code gets written once, but used everywhere.
Here is the brief overview of it’s abilities.
The other respondents are pointing you to Twisted, which is a great and very comprehensive framework but in my opinion it has a very un-pythonic design. Also, AFAICT, you have to use the Twisted main loop, which may be a problem for you if you’re already using something else that provides its own loop.
Here is a contrived example that would demonstrate using the
from threading import Thread def background_stuff(): while True: print "I am doing some stuff" t = Thread(target=background_stuff) t.start() # Continue doing some other stuff now
However, in pretty much every useful case, you will want to communicate between threads. You should look into synchronization primitives, and become familiar with the concept of concurrency and the related issues.
threading module provides many such primitives for you to use, if you know how to use them.
You may well want to checkout the Twisted library for Python. They provide many useful tools.
You may see my Python Asynchronous Programming tool: http://www.ideawu.com/blog/2010/08/delegate-in-pythonpython-asynchronous-programming.html
import time, random, sys from delegate import * def proc(a): time.sleep(random.random()) return str(a) def proc_callback(handle, args=None): ret = d.end(handle) d = Delegate() d.init(2) # number of workers handle = d.begin(proc, '12345', proc_callback, 'test') sys.stdin.readline() d.free()