How To Run Python Code Concurrently In C# Using Python.NET

In this article you’ll learn how to create a multithreaded C# application that interoperates with Python using the Python.NET library.

The GIL

The GIL (Global Interpreter Lock) is a lock mechanism that limits execution at any given time on the Python interpreter to a single thread.

If you’re unfamiliar with multithreading locks, here’s how it works: when a thread starts executing its code, the Python interpreter locks other threads, preventing them from running. Once the thread finishes its execution, the interpreter unlocks the next thread, allowing it to run. This cycle repeats, with only one thread’s code executing at a time.

When threads are created directly within Python, the interpreter automatically handles the GIL for switching threads. However, you’ll need to control the GIL explicitly if you want to run Python code from a multithreaded C# application using the Python.NET library.

Enabling Multithreading With Python.NET In C# Code

You can enable multithreading in Python.NET by using the following code:

PythonEngine.Initialize(); // Initialization has to come before BeginAllowThreads
PythonEngine.BeginAllowThreads();

PythonEngine.BeginAllowThreads() frees the lock that the main thread currently holds (the one that calls PythonEngine.Initialize()).

If you call PythonEngine.BeginAllowThreads(), all Python interpreter calls need to use the GIL, as shown in the following code snippet:

using (Py.GIL()) {
    // Python code runs here
    PythonEngine.RunSimpleString("print('Some python code just executed!')");
}

In the preceding example, Py.GIL() acquires the lock, runs the Python code, and finally releases the lock for other C# threads to execute Python code.

The Py.GIL() block is only meant for use with multithreaded applications, not single-threaded ones. Consider the following example:

static void Main(string[] args) {
    Runtime.PythonDLL = "/usr/lib/x86_64-linux-gnu/libpython3.10.so"; // The path to access the Python interpreter
    PythonEngine.Initialize();

    using(Py.GIL()) {
        PythonEngine.RunSimpleString("print('Python code has executed!')");
    }

    PythonEngine.Shutdown() // Required if PythonEngine.BeginAllowThreads() is not called
}

Since this program is only using a single thread, the GIL is unnecessary, so you can rewrite it as:

static void Main(string[] args) {
    Runtime.PythonDLL = "/usr/lib/x86_64-linux-gnu/libpython3.10.so"; // The path to access the Python interpreter
    PythonEngine.Initialize();

    PythonEngine.RunSimpleString("print('Python code has executed!')");

    PythonEngine.Shutdown(); // Required if PythonEngine.BeginAllowThreads() is not called
}

And it will still work.

This is not the case if you call PythonEngine.BeginAllowThreads(), as the last presented code snippet would fail execution, and the Py.GIL() block would still be necessary, as in the first example.

A Practical Example Of Multithreading With Python.NET In C#

Let’s consider the following multithreaded C# code:

static void Main(string[] args) {
    Runtime.PythonDLL = "/usr/lib/x86_64-linux-gnu/libpython3.10.so";
    PythonEngine.Initialize();
    PythonEngine.BeginAllowThreads();

    Thread thread = new Thread(new ThreadStart(SecondThread));
    thread.Start();

    using(Py.GIL()) {
        PythonEngine.RunSimpleString("print('''First thread's Python code was executed!''')");
    }
}

static void SecondThread() {
    using(Py.GIL()){
        PythonEngine.RunSimpleString("print('''Second thread's Python code was executed!''')");
    }
}

The output would be either

First thread’s Python code was executed!

Second thread’s Python code was executed!

Or

Second thread’s Python code was executed!

First thread’s Python code was executed!

Depending on which thread acquires the GIL first.

If the code did not use PythonEngine.BeginAllowThreads(), the output would then be:

First thread’s Python code was executed!

And the program would stay in a deadlock state where it can’t finish its execution.

That’s because the main thread never releases the lock for other threads to run (with the PythonEngine.BeginAllowThreads() call), blocking the second thread’s Py.GIL() enclosed code from ever running (the code locks at the Py.GIL() method call).

Note that all calls to the interpreter must be within a Py.GIL() call when using a multithreaded environment; this includes automatic type conversions (as in ToPython() calls), script imports, and others.

How Many Py.GIL Calls Should I Use?

A single Py.GIL() call can encompass as many Python interpreter calls as needed, based on the programmer’s discretion.

You can have a single call that encapsulates all of your program or multiple ones that separate the interpreter calls in multiple steps.

Using multiple Py.GIL() calls with a small amount of interpreter calls inside each enables smoother concurrency between the threads.

Running Native Python Threads From C#

If C# code calls Python code that spawns new threads, those threads are managed by the Python interpreter. However, if the C# code never releases the GIL (by calling PythonEngine.BeginAllowThreads()), the Python threads can only run when control returns to the interpreter (via additional calls to it, necessarily contained within Py.GIL() blocks had you called PythonEngine.BeginAllowThreads()). Releasing the GIL in C# allows the native Python threads to execute concurrently with C# code, even outside interpreter calls.