Question

15

How to run asynchronous tasks in Python GObject Introspection apps

rated 0 times [ 15] [ 0] / answers: 1 / hits: 16330 / 3 Years ago, wed, august 18, 2021, 9:47:30

I'm writing a Python + GObject app that needs to read a non-trivial amount of data from disk upon start. The data is read synchronously and it takes about 10 seconds to finish the read operation, during which time the loading of the UI is delayed.

I'd like to run the task asynchronously, and get a notification when it's ready, without blocking the UI, more or less like:

def take_ages():

    read_a_huge_file_from_disk()



def on_finished_long_task():

    print "Finished!"



run_long_task(task=take_ages, callback=on_finished_long_task)

load_the_UI_without_blocking_on_long_task()

I've used GTask in the past for this sort of thing, but I'm concerned that its code hasn't been touched in 3 years, let alone been ported to GObject Introspection. Most importantly, it's no longer available in Ubuntu 12.04. So I'm looking for an easy way to run tasks asynchronously, either in a standard Python way or in a GObject/GTK+ standard way.

Edit: here's some code with an example of what I'm trying to do. I've tried python-defer as suggested in the comments, but I could not manage to run the long task asynchronously and let the UI load without having to wait for it to finish. Browse the test code.

Is there an easy and widely used way of running asynchronous tasks and get notified when they're finished?

Answers

Only authorized users can answer the question. Please sign in first, or register a free account.

horicgly

Add To Favorites

Follow

Total Points: 36

Total Questions: 126

Total Answers: 104

Location: Iceland

Member since Thu, Dec 1, 2022

1 Year ago

answered 3 Years ago ipentainer · Accepted Answer

Your problem is a very common one, therefore there are tons of solutions (sheds, queues with multiprocessing or threading, worker pools, ...)

Since it is so common, there is also a python build-in solution (in 3.2, but backported here: http://pypi.python.org/pypi/futures) called concurrent.futures. 'Futures' are available in many languages, therefore python calls them the same. Here are the typical calls (and here is your full example, however, the db part is replaced by sleep, see below why).

from concurrent import futures

executor = futures.ProcessPoolExecutor(max_workers=1)

#executor = futures.ThreadPoolExecutor(max_workers=1)

future = executor.submit(slow_load)

future.add_done_callback(self.on_complete)

Now to your problem, which is much more complicated than your simple example suggests. In general you have threads or processes to solve this, but here is why your example is so complicated:

Most Python implementations have a GIL, which makes threads not fully utilize multicores. So: do not use threads with python!

The objects you want to return in slow_load from the DB are not pickelable, which means that they can not simply be passed between processes. So: no multiprocessing with softwarecenter results!

The library you call (softwarecenter.db) is not threadsafe (seems to include gtk or similar), therefore calling these methods in a thread results in strange behaviour (in my test, everything from 'it works' over 'core dump' to simple quitting without results). So: no threads with softwarecenter.

Every asynchronous callback in gtk should not do anything except sheduling a callback which will be called in the glib mainloop. So: no print, no gtk state changes, except adding a callback!

Gtk and alike does not work with threads out of the box. You need to do threads_init, and if you call a gtk or alike method, you have to protect that method (in earlier versions this was gtk.gdk.threads_enter(), gtk.gdk.threads_leave(). see for example gstreamer: http://pygstdocs.berlios.de/pygst-tutorial/playbin.html).

I can give you the following suggestion:

Rewrite your slow_load to return pickelable results and use futures with processes.

Switch from softwarecenter to python-apt or similar (you probably don't like that). But since your employed by Canonical, you could ask the softwarecenter developers directly to add documention to their software (e.g. stating that it is not thread safe) and even better, making softwarecenter threadsafe.

As a note: the solutions given by the others (Gio.io_scheduler_push_job, async_call) do work with time.sleep but not with softwarecenter.db. This is, because it all boils down to threads or processes and threads to not work with gtk and softwarecenter.