:mod:`!multiprocessing` --- Process-based parallelism

=====================================================

.. module:: multiprocessing

:synopsis: Process-based parallelism.

**Source code:** :source:`Lib/multiprocessing/`

--------------

.. include:: ../includes/wasm-mobile-notavail.rst

Introduction

------------

:mod:`!multiprocessing` is a package that supports spawning processes using an

API similar to the :mod:`threading` module. The :mod:`!multiprocessing` package

offers both local and remote concurrency, effectively side-stepping the

:term:`Global Interpreter Lock <global interpreter lock>` by using

subprocesses instead of threads. Due

to this, the :mod:`!multiprocessing` module allows the programmer to fully

leverage multiple processors on a given machine. It runs on both POSIX and

Windows.

The :mod:`!multiprocessing` module also introduces the

:class:`~multiprocessing.pool.Pool` object which offers a convenient means of

parallelizing the execution of a function across multiple input values,

distributing the input data across processes (data parallelism). The following

example demonstrates the common practice of defining such functions in a module

so that child processes can successfully import that module. This basic example

of data parallelism using :class:`~multiprocessing.pool.Pool`, ::

from multiprocessing import Pool

def f(x):

return x*x

if __name__ == '__main__':

with Pool(5) as p:

print(p.map(f, [1, 2, 3]))

will print to standard output ::

[1, 4, 9]

The :mod:`!multiprocessing` module also introduces APIs which do not have

analogs in the :mod:`threading` module, like the ability to :meth:`terminate

<Process.terminate>`, :meth:`interrupt <Process.interrupt>` or :meth:`kill

<Process.kill>` a running process.

.. seealso::

:class:`concurrent.futures.ProcessPoolExecutor` offers a higher level interface

to push tasks to a background process without blocking execution of the

calling process. Compared to using the :class:`~multiprocessing.pool.Pool`

interface directly, the :mod:`concurrent.futures` API more readily allows

the submission of work to the underlying process pool to be separated from

waiting for the results.

The :class:`Process` class

^^^^^^^^^^^^^^^^^^^^^^^^^^

In :mod:`!multiprocessing`, processes are spawned by creating a :class:`Process`

object and then calling its :meth:`~Process.start` method. :class:`Process`

follows the API of :class:`threading.Thread`. A trivial example of a

multiprocess program is ::

from multiprocessing import Process

def f(name):

print('hello', name)

if __name__ == '__main__':

p = Process(target=f, args=('bob',))

p.start()

p.join()

To show the individual process IDs involved, here is an expanded example::

from multiprocessing import Process

import os

def info(title):

print(title)

print('module name:', __name__)

print('parent process:', os.getppid())

print('process id:', os.getpid())

def f(name):

info('function f')

print('hello', name)

if __name__ == '__main__':

info('main line')

p = Process(target=f, args=('bob',))

p.start()

p.join()

For an explanation of why the ``if __name__ == '__main__'`` part is

necessary, see :ref:`multiprocessing-programming`.

The arguments to :class:`Process` usually need to be picklable so they can be

passed to the child process. If you tried typing the above example directly

into a REPL it could lead to an :exc:`AttributeError` in the child process

trying to locate the *f* function in the ``__main__`` module.

.. _multiprocessing-start-methods:

Contexts and start methods

^^^^^^^^^^^^^^^^^^^^^^^^^^

Depending on the platform, :mod:`!multiprocessing` supports three ways

to start a process. These *start methods* are

.. _multiprocessing-start-method-spawn:

*spawn*

The parent process starts a fresh Python interpreter process. The

child process will only inherit those resources necessary to run

the process object's :meth:`~Process.run` method. In particular,

unnecessary file descriptors and handles from the parent process

will not be inherited. Starting a process using this method is

rather slow compared to using *fork* or *forkserver*.

Available on POSIX and Windows platforms. The default on Windows and macOS.

.. _multiprocessing-start-method-fork:

*fork*

The parent process uses :func:`os.fork` to fork the Python

interpreter. The child process, when it begins, is effectively

identical to the parent process. All resources of the parent are

inherited by the child process. Note that safely forking a

multithreaded process is problematic.

Available on POSIX systems.

.. versionchanged:: 3.14

This is no longer the default start method on any platform.

Code that requires *fork* must explicitly specify that via

:func:`get_context` or :func:`set_start_method`.

.. versionchanged:: 3.12

If Python is able to detect that your process has multiple threads, the

:func:`os.fork` function that this start method calls internally will

raise a :exc:`DeprecationWarning`. Use a different start method.

See the :func:`os.fork` documentation for further explanation.

.. _multiprocessing-start-method-forkserver:

*forkserver*

When the program starts and selects the *forkserver* start method,

a server process is spawned. From then on, whenever a new process

is needed, the parent process connects to the server and requests

that it fork a new process. The fork server process is single threaded

unless system libraries or preloaded imports spawn threads as a

side-effect so it is generally safe for it to use :func:`os.fork`.

No unnecessary resources are inherited.

Available on POSIX platforms which support passing file descriptors over

Unix pipes such as Linux. The default on those.

.. versionchanged:: 3.14

This became the default start method on POSIX platforms.

.. versionchanged:: 3.4

*spawn* added on all POSIX platforms, and *forkserver* added for

some POSIX platforms.

Child processes no longer inherit all of the parents inheritable

handles on Windows.

.. versionchanged:: 3.8

On macOS, the *spawn* start method is now the default. The *fork* start

method should be considered unsafe as it can lead to crashes of the

subprocess as macOS system libraries may start threads. See :issue:`33725`.

.. versionchanged:: 3.14

On POSIX platforms the default start method was changed from *fork* to

*forkserver* to retain the performance but avoid common multithreaded

process incompatibilities. See :gh:`84559`.

On POSIX using the *spawn* or *forkserver* start methods will also

start a *resource tracker* process which tracks the unlinked named

system resources (such as named semaphores or

:class:`~multiprocessing.shared_memory.SharedMemory` objects) created

by processes of the program. When all processes

have exited the resource tracker unlinks any remaining tracked object.

Usually there should be none, but if a process was killed by a signal

there may be some "leaked" resources. (Neither leaked semaphores nor shared

memory segments will be automatically unlinked until the next reboot. This is

problematic for both objects because the system allows only a limited number of

named semaphores, and shared memory segments occupy some space in the main

memory.)

To select a start method you use the :func:`set_start_method` in

the ``if __name__ == '__main__'`` clause of the main module. For

example::

import multiprocessing as mp

def foo(q):

q.put('hello')

if __name__ == '__main__':

mp.set_start_method('spawn')

q = mp.Queue()

p = mp.Process(target=foo, args=(q,))

p.start()

print(q.get())

p.join()

:func:`set_start_method` should not be used more than once in the

program.

Alternatively, you can use :func:`get_context` to obtain a context

object. Context objects have the same API as the multiprocessing

module, and allow one to use multiple start methods in the same

program. ::

import multiprocessing as mp

def foo(q):

q.put('hello')

if __name__ == '__main__':

ctx = mp.get_context('spawn')

q = ctx.Queue()

p = ctx.Process(target=foo, args=(q,))

p.start()

print(q.get())

p.join()

Note that objects related to one context may not be compatible with

processes for a different context. In particular, locks created using

the *fork* context cannot be passed to processes started using the

*spawn* or *forkserver* start methods.

Libraries using :mod:`!multiprocessing` or

:class:`~concurrent.futures.ProcessPoolExecutor` should be designed to allow

their users to provide their own multiprocessing context. Using a specific

context of your own within a library can lead to incompatibilities with the

rest of the library user's application. Always document if your library

requires a specific start method.

.. warning::

The ``'spawn'`` and ``'forkserver'`` start methods generally cannot

be used with "frozen" executables (i.e., binaries produced by

packages like **PyInstaller** and **cx_Freeze**) on POSIX systems.

The ``'fork'`` start method may work if code does not use threads.

Exchanging objects between processes

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

:mod:`!multiprocessing` supports two types of communication channel between

processes:

**Queues**

The :class:`Queue` class is a near clone of :class:`queue.Queue`. For

example::

from multiprocessing import Process, Queue

def f(q):

q.put([42, None, 'hello'])

if __name__ == '__main__':

q = Queue()

p = Process(target=f, args=(q,))

p.start()

print(q.get()) # prints "[42, None, 'hello']"

p.join()

Queues are thread and process safe.

Any object put into a :mod:`!multiprocessing` queue will be serialized.

**Pipes**

The :func:`Pipe` function returns a pair of connection objects connected by a

pipe which by default is duplex (two-way). For example::

from multiprocessing import Process, Pipe

def f(conn):

conn.send([42, None, 'hello'])

conn.close()

if __name__ == '__main__':

parent_conn, child_conn = Pipe()

p = Process(target=f, args=(child_conn,))

p.start()

print(parent_conn.recv()) # prints "[42, None, 'hello']"

p.join()

The two connection objects returned by :func:`Pipe` represent the two ends of

the pipe. Each connection object has :meth:`~Connection.send` and

:meth:`~Connection.recv` methods (among others). Note that data in a pipe

may become corrupted if two processes (or threads) try to read from or write

to the *same* end of the pipe at the same time. Of course there is no risk

of corruption from processes using different ends of the pipe at the same

time.

The :meth:`~Connection.send` method serializes the object and

:meth:`~Connection.recv` re-creates the object.

Synchronization between processes

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

:mod:`!multiprocessing` contains equivalents of all the synchronization

primitives from :mod:`threading`. For instance one can use a lock to ensure

that only one process prints to standard output at a time::

from multiprocessing import Process, Lock

def f(l, i):

l.acquire()

try:

print('hello world', i)

finally:

l.release()

if __name__ == '__main__':

lock = Lock()

for num in range(10):

Process(target=f, args=(lock, num)).start()

Without using the lock output from the different processes is liable to get all

mixed up.

Sharing state between processes

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

As mentioned above, when doing concurrent programming it is usually best to

avoid using shared state as far as possible. This is particularly true when

using multiple processes.

However, if you really do need to use some shared data then

:mod:`!multiprocessing` provides a couple of ways of doing so.

**Shared memory**

Data can be stored in a shared memory map using :class:`Value` or

:class:`Array`. For example, the following code ::

from multiprocessing import Process, Value, Array

def f(n, a):

n.value = 3.1415927

for i in range(len(a)):

a[i] = -a[i]

if __name__ == '__main__':

num = Value('d', 0.0)

arr = Array('i', range(10))

p = Process(target=f, args=(num, arr))

p.start()

p.join()

print(num.value)

print(arr[:])

will print ::

3.1415927

[0, -1, -2, -3, -4, -5, -6, -7, -8, -9]

The ``'d'`` and ``'i'`` arguments used when creating ``num`` and ``arr`` are

typecodes of the kind used by the :mod:`array` module: ``'d'`` indicates a

double precision float and ``'i'`` indicates a signed integer. These shared

objects will be process and thread-safe.

For more flexibility in using shared memory one can use the

:mod:`multiprocessing.sharedctypes` module which supports the creation of

arbitrary ctypes objects allocated from shared memory.

**Server process**

A manager object returned by :func:`Manager` controls a server process which

holds Python objects and allows other processes to manipulate them using

proxies.

A manager returned by :func:`Manager` will support types

:class:`list`, :class:`dict`, :class:`set`, :class:`~managers.Namespace`, :class:`Lock`,

:class:`RLock`, :class:`Semaphore`, :class:`BoundedSemaphore`,

:class:`Condition`, :class:`Event`, :class:`Barrier`,

:class:`Queue`, :class:`Value` and :class:`Array`. For example, ::

from multiprocessing import Process, Manager

def f(d, l, s):

d[1] = '1'

d['2'] = 2

d[0.25] = None

l.reverse()

s.add('a')

s.add('b')

if __name__ == '__main__':

with Manager() as manager:

d = manager.dict()

l = manager.list(range(10))

s = manager.set()

p = Process(target=f, args=(d, l, s))

p.start()

p.join()

print(d)

print(l)

print(s)

will print ::

{0.25: None, 1: '1', '2': 2}

[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

{'a', 'b'}

Server process managers are more flexible than using shared memory objects

because they can be made to support arbitrary object types. Also, a single

manager can be shared by processes on different computers over a network.

They are, however, slower than using shared memory.

Using a pool of workers

^^^^^^^^^^^^^^^^^^^^^^^

The :class:`~multiprocessing.pool.Pool` class represents a pool of worker

processes. It has methods which allows tasks to be offloaded to the worker

processes in a few different ways.

For example::

from multiprocessing import Pool, TimeoutError

import time

import os

def f(x):

return x*x

if __name__ == '__main__':

# start 4 worker processes

with Pool(processes=4) as pool:

# print "[0, 1, 4,..., 81]"

print(pool.map(f, range(10)))

# print same numbers in arbitrary order

for i in pool.imap_unordered(f, range(10)):

print(i)

# evaluate "f(20)" asynchronously

res = pool.apply_async(f, (20,)) # runs in *only* one process

print(res.get(timeout=1)) # prints "400"

# evaluate "os.getpid()" asynchronously

res = pool.apply_async(os.getpid, ()) # runs in *only* one process

print(res.get(timeout=1)) # prints the PID of that process

# launching multiple evaluations asynchronously *may* use more processes

multiple_results = [pool.apply_async(os.getpid, ()) for i in range(4)]

print([res.get(timeout=1) for res in multiple_results])

# make a single worker sleep for 10 seconds

res = pool.apply_async(time.sleep, (10,))

try:

print(res.get(timeout=1))

except TimeoutError:

print("We lacked patience and got a multiprocessing.TimeoutError")

print("For the moment, the pool remains available for more work")

# exiting the 'with'-block has stopped the pool

print("Now the pool is closed and no longer available")

Note that the methods of a pool should only ever be used by the

process which created it.

.. note::

Functionality within this package requires that the ``__main__`` module be

importable by the children. This is covered in :ref:`multiprocessing-programming`

however it is worth pointing out here. This means that some examples, such

as the :class:`multiprocessing.pool.Pool` examples will not work in the

interactive interpreter. For example::

>>> from multiprocessing import Pool

>>> p = Pool(5)

>>> def f(x):

... return x*x

...

>>> with p:

... p.map(f, [1,2,3])

Process PoolWorker-1:

Process PoolWorker-2:

Process PoolWorker-3:

Traceback (most recent call last):

Traceback (most recent call last):

Traceback (most recent call last):

AttributeError: Can't get attribute 'f' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>

AttributeError: Can't get attribute 'f' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>

AttributeError: Can't get attribute 'f' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>

(If you try this it will actually output three full tracebacks

interleaved in a semi-random fashion, and then you may have to

stop the parent process somehow.)

Reference

---------

The :mod:`!multiprocessing` package mostly replicates the API of the

:mod:`threading` module.

.. _global-start-method:

Global start method

^^^^^^^^^^^^^^^^^^^

Python supports several ways to create and initialize a process.

The global start method sets the default mechanism for creating a process.

Several multiprocessing functions and methods that may also instantiate

certain objects will implicitly set the global start method to the system's default,

if it hasn’t been set already. The global start method can only be set once.

If you need to change the start method from the system default, you must

proactively set the global start method before calling functions or methods,

or creating these objects.

:class:`Process` and exceptions

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. class:: Process(group=None, target=None, name=None, args=(), kwargs={}, \

*, daemon=None)

Process objects represent activity that is run in a separate process. The

:class:`Process` class has equivalents of all the methods of

:class:`threading.Thread`.

The constructor should always be called with keyword arguments. *group*

should always be ``None``; it exists solely for compatibility with

:class:`threading.Thread`. *target* is the callable object to be invoked by

the :meth:`run` method. It defaults to ``None``, meaning nothing is

called. *name* is the process name (see :attr:`name` for more details).

*args* is the argument tuple for the target invocation. *kwargs* is a

dictionary of keyword arguments for the target invocation. If provided,

the keyword-only *daemon* argument sets the process :attr:`daemon` flag

to ``True`` or ``False``. If ``None`` (the default), this flag will be

inherited from the creating process.

By default, no arguments are passed to *target*. The *args* argument,

which defaults to ``()``, can be used to specify a list or tuple of the arguments

to pass to *target*.

If a subclass overrides the constructor, it must make sure it invokes the

base class constructor (``super().__init__()``) before doing anything else

to the process.

.. note::

In general, all arguments to :class:`Process` must be picklable. This is

frequently observed when trying to create a :class:`Process` or use a

:class:`concurrent.futures.ProcessPoolExecutor` from a REPL with a

locally defined *target* function.

Passing a callable object defined in the current REPL session causes the

child process to die via an uncaught :exc:`AttributeError` exception when

starting as *target* must have been defined within an importable module

in order to be loaded during unpickling.

Example of this uncatchable error from the child::

>>> import multiprocessing as mp

>>> def knigit():

... print("Ni!")

...

>>> process = mp.Process(target=knigit)

>>> process.start()

>>> Traceback (most recent call last):

File ".../multiprocessing/spawn.py", line ..., in spawn_main

File ".../multiprocessing/spawn.py", line ..., in _main

AttributeError: module '__main__' has no attribute 'knigit'

>>> process

<SpawnProcess name='SpawnProcess-1' pid=379473 parent=378707 stopped exitcode=1>

See :ref:`multiprocessing-programming-spawn`. While this restriction is

not true if using the ``"fork"`` start method, as of Python ``3.14`` that

is no longer the default on any platform. See

:ref:`multiprocessing-start-methods`.

See also :gh:`132898`.

.. versionchanged:: 3.3

Added the *daemon* parameter.

.. method:: run()

Method representing the process's activity.

You may override this method in a subclass. The standard :meth:`run`

method invokes the callable object passed to the object's constructor as

the target argument, if any, with sequential and keyword arguments taken

from the *args* and *kwargs* arguments, respectively.

Using a list or tuple as the *args* argument passed to :class:`Process`

achieves the same effect.

Example::

>>> from multiprocessing import Process

>>> p = Process(target=print, args=[1])

>>> p.run()

1

>>> p = Process(target=print, args=(1,))

>>> p.run()

1

.. method:: start()

Start the process's activity.

This must be called at most once per process object. It arranges for the

object's :meth:`run` method to be invoked in a separate process.

.. method:: join([timeout])

If the optional argument *timeout* is ``None`` (the default), the method

blocks until the process whose :meth:`join` method is called terminates.

If *timeout* is a positive number, it blocks at most *timeout* seconds.

Note that the method returns ``None`` if its process terminates or if the

method times out. Check the process's :attr:`exitcode` to determine if

it terminated.

A process can be joined many times.

A process cannot join itself because this would cause a deadlock. It is

an error to attempt to join a process before it has been started.

.. attribute:: name

The process's name. The name is a string used for identification purposes

only. It has no semantics. Multiple processes may be given the same

name.

The initial name is set by the constructor. If no explicit name is

provided to the constructor, a name of the form

'Process-N\ :sub:`1`:N\ :sub:`2`:...:N\ :sub:`k`' is constructed, where

each N\ :sub:`k` is the N-th child of its parent.

.. method:: is_alive

Return whether the process is alive.

Roughly, a process object is alive from the moment the :meth:`start`

method returns until the child process terminates.

.. attribute:: daemon

The process's daemon flag, a Boolean value. This must be set before

:meth:`start` is called.

The initial value is inherited from the creating process.

When a process exits, it attempts to terminate all of its daemonic child

processes.

Note that a daemonic process is not allowed to create child processes.

Otherwise a daemonic process would leave its children orphaned if it gets

terminated when its parent process exits. Additionally, these are **not**

Unix daemons or services, they are normal processes that will be

terminated (and not joined) if non-daemonic processes have exited.

In addition to the :class:`threading.Thread` API, :class:`Process` objects

also support the following attributes and methods:

.. attribute:: pid

Return the process ID. Before the process is spawned, this will be

``None``.

.. attribute:: exitcode

The child's exit code. This will be ``None`` if the process has not yet

terminated.

If the child's :meth:`run` method returned normally, the exit code

will be 0. If it terminated via :func:`sys.exit` with an integer

argument *N*, the exit code will be *N*.

If the child terminated due to an exception not caught within

:meth:`run`, the exit code will be 1. If it was terminated by

signal *N*, the exit code will be the negative value *-N*.

.. attribute:: authkey

The process's authentication key (a byte string).

When :mod:`!multiprocessing` is initialized the main process is assigned a

random string using :func:`os.urandom`.

When a :class:`Process` object is created, it will inherit the

authentication key of its parent process, although this may be changed by

setting :attr:`authkey` to another byte string.

See :ref:`multiprocessing-auth-keys`.

.. attribute:: sentinel

A numeric handle of a system object which will become "ready" when

the process ends.

You can use this value if you want to wait on several events at

once using :func:`multiprocessing.connection.wait`. Otherwise

calling :meth:`join` is simpler.

On Windows, this is an OS handle usable with the ``WaitForSingleObject``

and ``WaitForMultipleObjects`` family of API calls. On POSIX, this is

a file descriptor usable with primitives from the :mod:`select` module.

.. versionadded:: 3.3

.. method:: interrupt()

Terminate the process. Works on POSIX using the :py:const:`~signal.SIGINT` signal.

Behavior on Windows is undefined.

By default, this terminates the child process by raising :exc:`KeyboardInterrupt`.

This behavior can be altered by setting the respective signal handler in the child

process :func:`signal.signal` for :py:const:`~signal.SIGINT`.

Note: if the child process catches and discards :exc:`KeyboardInterrupt`, the

process will not be terminated.

Note: the default behavior will also set :attr:`exitcode` to ``1`` as if an

uncaught exception was raised in the child process. To have a different

:attr:`exitcode` you may simply catch :exc:`KeyboardInterrupt` and call

``exit(your_code)``.

.. versionadded:: 3.14

.. method:: terminate()

Terminate the process. On POSIX this is done using the :py:const:`~signal.SIGTERM` signal;

on Windows :c:func:`!TerminateProcess` is used. Note that exit handlers and

finally clauses, etc., will not be executed.

Note that descendant processes of the process will *not* be terminated --

they will simply become orphaned.

.. warning::

If this method is used when the associated process is using a pipe or

queue then the pipe or queue is liable to become corrupted and may

become unusable by other process. Similarly, if the process has

acquired a lock or semaphore etc. then terminating it is liable to

cause other processes to deadlock.

.. method:: kill()

Same as :meth:`terminate` but using the ``SIGKILL`` signal on POSIX.

.. versionadded:: 3.7

.. method:: close()

Close the :class:`Process` object, releasing all resources associated

with it. :exc:`ValueError` is raised if the underlying process

is still running. Once :meth:`close` returns successfully, most

other methods and attributes of the :class:`Process` object will

raise :exc:`ValueError`.

.. versionadded:: 3.7

Note that the :meth:`start`, :meth:`join`, :meth:`is_alive`,

:meth:`terminate` and :attr:`exitcode` methods should only be called by

the process that created the process object.

Example usage of some of the methods of :class:`Process`:

.. doctest::

>>> import multiprocessing, time, signal

>>> mp_context = multiprocessing.get_context('spawn')

>>> p = mp_context.Process(target=time.sleep, args=(1000,))

>>> print(p, p.is_alive())

<...Process ... initial> False

>>> p.start()

>>> print(p, p.is_alive())

<...Process ... started> True

>>> p.terminate()

>>> time.sleep(0.1)

>>> print(p, p.is_alive())

<...Process ... stopped exitcode=-SIGTERM> False

>>> p.exitcode == -signal.SIGTERM

True

.. exception:: ProcessError

The base class of all :mod:`!multiprocessing` exceptions.

.. exception:: BufferTooShort

Exception raised by :meth:`Connection.recv_bytes_into` when the supplied

buffer object is too small for the message read.

If ``e`` is an instance of :exc:`BufferTooShort` then ``e.args[0]`` will give

the message as a byte string.

.. exception:: AuthenticationError

Raised when there is an authentication error.

.. exception:: TimeoutError

Raised by methods with a timeout when the timeout expires.

Pipes and Queues

^^^^^^^^^^^^^^^^

When using multiple processes, one generally uses message passing for

communication between processes and avoids having to use any synchronization

primitives like locks.

For passing messages one can use :func:`Pipe` (for a connection between two

processes) or a queue (which allows multiple producers and consumers).

The :class:`Queue`, :class:`SimpleQueue` and :class:`JoinableQueue` types

are multi-producer, multi-consumer :abbr:`FIFO (first-in, first-out)`

queues modelled on the :class:`queue.Queue` class in the

standard library. They differ in that :class:`Queue` lacks the

:meth:`~queue.Queue.task_done` and :meth:`~queue.Queue.join` methods introduced

into Python 2.5's :class:`queue.Queue` class.

If you use :class:`JoinableQueue` then you **must** call

:meth:`JoinableQueue.task_done` for each task removed from the queue or else the

semaphore used to count the number of unfinished tasks may eventually overflow,

raising an exception.

One difference from other Python queue implementations, is that :mod:`!multiprocessing`

queues serializes all objects that are put into them using :mod:`pickle`.

The object returned by the get method is a re-created object that does not share

memory with the original object.

Note that one can also create a shared queue by using a manager object -- see

:ref:`multiprocessing-managers`.

.. note::

:mod:`!multiprocessing` uses the usual :exc:`queue.Empty` and

:exc:`queue.Full` exceptions to signal a timeout. They are not available in

the :mod:`!multiprocessing` namespace so you need to import them from

:mod:`queue`.

.. note::

When an object is put on a queue, the object is pickled and a

background thread later flushes the pickled data to an underlying

pipe. This has some consequences which are a little surprising,

but should not cause any practical difficulties -- if they really

bother you then you can instead use a queue created with a

:ref:`manager <multiprocessing-managers>`.

(1) After putting an object on an empty queue there may be an

infinitesimal delay before the queue's :meth:`~Queue.empty`

method returns :const:`False` and :meth:`~Queue.get_nowait` can

return without raising :exc:`queue.Empty`.

(2) If multiple processes are enqueuing objects, it is possible for

the objects to be received at the other end out-of-order.

However, objects enqueued by the same process will always be in

the expected order with respect to each other.

.. warning::

If a process is killed using :meth:`Process.terminate` or :func:`os.kill`

while it is trying to use a :class:`Queue`, then the data in the queue is

likely to become corrupted. This may cause any other process to get an

exception when it tries to use the queue later on.

.. warning::

As mentioned above, if a child process has put items on a queue (and it has

not used :meth:`JoinableQueue.cancel_join_thread

<multiprocessing.Queue.cancel_join_thread>`), then that process will

not terminate until all buffered items have been flushed to the pipe.

This means that if you try joining that process you may get a deadlock unless

you are sure that all items which have been put on the queue have been

consumed. Similarly, if the child process is non-daemonic then the parent

process may hang on exit when it tries to join all its non-daemonic children.

Note that a queue created using a manager does not have this issue. See

:ref:`multiprocessing-programming`.

For an example of the usage of queues for interprocess communication see

:ref:`multiprocessing-examples`.

.. function:: Pipe(duplex=True)

Returns a pair ``(conn1, conn2)`` of

:class:`~multiprocessing.connection.Connection` objects representing the

ends of a pipe.

If *duplex* is ``True`` (the default) then the pipe is bidirectional. If

*duplex* is ``False`` then the pipe is unidirectional: ``conn1`` can only be

used for receiving messages and ``conn2`` can only be used for sending

messages.

The :meth:`~multiprocessing.Connection.send` method serializes the object using

:mod:`pickle` and the :meth:`~multiprocessing.Connection.recv` re-creates the object.

.. class:: Queue([maxsize])

Returns a process shared queue implemented using a pipe and a few

locks/semaphores. When a process first puts an item on the queue a feeder

thread is started which transfers objects from a buffer into the pipe.

Instantiating this class may set the global start method. See

:ref:`global-start-method` for more details.

The usual :exc:`queue.Empty` and :exc:`queue.Full` exceptions from the

standard library's :mod:`queue` module are raised to signal timeouts.

:class:`Queue` implements all the methods of :class:`queue.Queue` except for

:meth:`~queue.Queue.task_done`, :meth:`~queue.Queue.join`, and

:meth:`~queue.Queue.shutdown`.

.. method:: qsize()

Return the approximate size of the queue. Because of

multithreading/multiprocessing semantics, this number is not reliable.

Note that this may raise :exc:`NotImplementedError` on platforms like

macOS where ``sem_getvalue()`` is not implemented.

.. method:: empty()

Return ``True`` if the queue is empty, ``False`` otherwise. Because of

multithreading/multiprocessing semantics, this is not reliable.

May raise an :exc:`OSError` on closed queues. (not guaranteed)

.. method:: full()

Return ``True`` if the queue is full, ``False`` otherwise. Because of

multithreading/multiprocessing semantics, this is not reliable.

.. method:: put(obj[, block[, timeout]])

Put obj into the queue. If the optional argument *block* is ``True``

(the default) and *timeout* is ``None`` (the default), block if necessary until

a free slot is available. If *timeout* is a positive number, it blocks at

most *timeout* seconds and raises the :exc:`queue.Full` exception if no

free slot was available within that time. Otherwise (*block* is

``False``), put an item on the queue if a free slot is immediately

available, else raise the :exc:`queue.Full` exception (*timeout* is

ignored in that case).

.. versionchanged:: 3.8

If the queue is closed, :exc:`ValueError` is raised instead of

:exc:`AssertionError`.

.. method:: put_nowait(obj)

Equivalent to ``put(obj, False)``.

.. method:: get([block[, timeout]])

Remove and return an item from the queue. If optional args *block* is

``True`` (the default) and *timeout* is ``None`` (the default), block if

necessary until an item is available. If *timeout* is a positive number,

it blocks at most *timeout* seconds and raises the :exc:`queue.Empty`

exception if no item was available within that time. Otherwise (block is

``False``), return an item if one is immediately available, else raise the

:exc:`queue.Empty` exception (*timeout* is ignored in that case).

.. versionchanged:: 3.8

If the queue is closed, :exc:`ValueError` is raised instead of

:exc:`OSError`.

.. method:: get_nowait()

Equivalent to ``get(False)``.

:class:`multiprocessing.Queue` has a few additional methods not found in

:class:`queue.Queue`. These methods are usually unnecessary for most

code:

.. method:: close()