Asynchronous programming is a software paradigm which involves scheduling many small tasks that are invoked when events occurs. It is also known as event-driven programming. It is an alternative (although it can also be a complement) to both multi-threading and multiprocessing. Asynchronous programming is well suited to tasks which are IO bound and not CPU bound. It is well suited for IO bound applications because it allows other tasks to occurs while one task is blocked, waiting on some external process to complete. Because control is only given up explicitly with the
await keyword, you do not have to worry about common multi-threading issues such as data contention. It is not well suited to CPU bound applications because it does not make use of multiple cores/CPUs.
Two core parts of Pythons asynchronous capabilities a provided through the
async keywords. The rest of the functionality is largely supplied by the
asyncio library. The
asyncio library provides event loops and functions to create, run and await tasks. Event loops are the “runners” of asynchronous functions (these functions are officially called coroutines). They keep track of all the coroutines which are currently blocked waiting for an event, and continue these coroutines from where they left off once the event occurs.
When you wait for an event with the
await keyword, Python can save the state of the function (i.e. the value of all the local variables, and the point of execution), and return to the active event loop. In the active event loop, the application can respond to other events while it is waiting. Once the specific event you waited on occurs, Python restores the state of the function and returns execution to that exact point is was saved at (they are very similar to Python generators).
promises became popular. It occurred because the only way to perform asynchronous programming was to provide callbacks (lambda functions). These nested within each other, broke the flow of the code, and severely hindered the readability of the software.
However smart are flexible asynchronous programming may be, synchronous programming is still the bread-and-butter of the Python language. Unfortunately, the two don’t mix that well (you can’t
await a synchronous function — and forgetting to await an asynchronous function will just return a
coroutine object). You can think of synchronous Python and asynchronous Python as two separate programming styles, and most of your libraries have to be specifically designed to work with the style you are using.
What Is A Coroutine?
A coroutine is a Python function that has the keyword
await before the
Calling a coroutine normally won’t actually do what you expect!
It would be wrong to say that nothing at all happens. Instead of calling the function,
my_coroutine() creates and returns a
coroutine object. This
coroutine object can be waited on with:
But please remember,
await can only be called within a asynchronous function. So in reality, the call would have to look something like this:
So now you a probably thinking, since the parent function, and the parent’s parent function, and the parent’s parent’s parent function all have to defined with
async to be able to use
await…where does it stop? What if my
main() is not
async? And even if that was, how would I call it? This is where the
asyncio library comes into play.
So actually, I lied, you can actually call an
async function from a non-
async function, but you have to use
asyncio to do so. The simplest way is to use
asyncio.run(), which takes a coroutine, runs it in a new event loop, and then returns.
asyncio.run()is only available is Python v3.7 or higher.
If you forget to await all coroutines, Python will print the warning:
Before Python v3.5
Before Python v3.5, the
async keyword is not available. You can however use a decorator to define a coroutine:
And instead of using
await to call the above coroutine, you would use the
yield from syntax:
Calling Async Code From Sync
Invariably, at some point you will want to call asynchronous code from a synchronous function. What you can’t do is:
However, remember that we can always pass control over to the event loop from synchronous code. The easiest way to do this is with
Creating A Worker Model
Below is a Python snippet showing a worker/job application using asynchronous programming. 10 jobs are created. 3 workers are created which will process these 10 jobs. Each worker is started as a task with
asyncio.create_task(). The jobs are fed to the workers via a
asyncio.Queue. Each worker
awaits a job on the queue, processes the job, and then waits for another one. Once all of the jobs are processed, the workers are terminated and the application exits.
Will produce the following output:
Make sure that you terminate all the tasks before terminating the application. If you terminate while a task is still waiting on a queue you will get the following warning:
What Are Awaitables?
An Python object is called awaitable if it can be used in an
await expression, i.e. works in the line
await <object>. The three main types of awaitable objects are:
Tasks are one way you can schedule multiple coroutines to run concurrently. Tasks can be created with
The following example shows how tasks will be scheduled to run immediately, and not just when they are awaited:
This work is licensed under a Creative Commons Attribution 4.0 International License .
- September 2019 Updates
- Python Classes And Object Orientated Design
- Parsing Command-Line Arguments In Python
- Python Sets
- A Tutorial On geopandas
- programming languages
- event loops
- asynchronous programming
- IO bound
- CPU bound