****************************

What's New In Python 3.3

****************************

.. Rules for maintenance:

* Anyone can add text to this document. Do not spend very much time

on the wording of your changes, because your text will probably

get rewritten to some degree.

* The maintainer will go through Misc/NEWS periodically and add

changes; it's therefore more important to add your changes to

Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness

is the purpose of Misc/NEWS. Some changes I consider too small

or esoteric to include. If such a change is added to the text,

I'll just remove it. (This is another reason you shouldn't spend

too much time on writing your addition.)

* If you want to draw your new text to the attention of the

maintainer, add 'XXX' to the beginning of the paragraph or

section.

* It's OK to just add a fragmentary note about a change. For

example: "XXX Describe the transmogrify() function added to the

socket module." The maintainer will research the change and

write the necessary text.

* You can comment out your additions if you like, but it's not

necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is

sufficient; the e-mail address isn't necessary.

* It's helpful to add the bug/patch number as a comment:

XXX Describe the transmogrify() function added to the socket

module.

(Contributed by P.Y. Developer in :issue:`12345`.)

This saves the maintainer the effort of going through the Mercurial log

when researching a change.

This article explains the new features in Python 3.3, compared to 3.2.

Python 3.3 was released on September 29, 2012. For full details,

see the `changelog <https://docs.python.org/3.3/whatsnew/changelog.html>`_.

.. seealso::

:pep:`398` - Python 3.3 Release Schedule

Summary -- Release highlights

=============================

.. This section singles out the most important changes in Python 3.3.

Brevity is key.

New syntax features:

* New ``yield from`` expression for :ref:`generator delegation <pep-380>`.

* The ``u'unicode'`` syntax is accepted again for :class:`str` objects.

New library modules:

* :mod:`faulthandler` (helps debugging low-level crashes)

* :mod:`ipaddress` (high-level objects representing IP addresses and masks)

* :mod:`lzma` (compress data using the XZ / LZMA algorithm)

* :mod:`unittest.mock` (replace parts of your system under test with mock objects)

* :mod:`venv` (Python :ref:`virtual environments <pep-405>`, as in the

popular ``virtualenv`` package)

New built-in features:

* Reworked :ref:`I/O exception hierarchy <pep-3151>`.

Implementation improvements:

* Rewritten :ref:`import machinery <importlib>` based on :mod:`importlib`.

* More compact :ref:`unicode strings <pep-393>`.

* More compact :ref:`attribute dictionaries <pep-412>`.

Significantly Improved Library Modules:

* C Accelerator for the :ref:`decimal <new-decimal>` module.

* Better unicode handling in the :ref:`email <new-email>` module

(:term:`provisional <provisional package>`).

Security improvements:

* Hash randomization is switched on by default.

Please read on for a comprehensive list of user-facing changes.

.. _pep-405:

PEP 405: Virtual Environments

=============================

Virtual environments help create separate Python setups while sharing a

system-wide base install, for ease of maintenance. Virtual environments

have their own set of private site packages (i.e. locally installed

libraries), and are optionally segregated from the system-wide site

packages. Their concept and implementation are inspired by the popular

``virtualenv`` third-party package, but benefit from tighter integration

with the interpreter core.

This PEP adds the :mod:`venv` module for programmatic access, and the

``pyvenv`` script for command-line access and

administration. The Python interpreter checks for a ``pyvenv.cfg``,

file whose existence signals the base of a virtual environment's directory

tree.

.. seealso::

:pep:`405` - Python Virtual Environments

PEP written by Carl Meyer; implementation by Carl Meyer and Vinay Sajip

PEP 420: Implicit Namespace Packages

====================================

Native support for package directories that don't require ``__init__.py``

marker files and can automatically span multiple path segments (inspired by

various third party approaches to namespace packages, as described in

:pep:`420`)

.. seealso::

:pep:`420` - Implicit Namespace Packages

PEP written by Eric V. Smith; implementation by Eric V. Smith

and Barry Warsaw

.. _pep-3118-update:

PEP 3118: New memoryview implementation and buffer protocol documentation

=========================================================================

The implementation of :pep:`3118` has been significantly improved.

The new memoryview implementation comprehensively fixes all ownership and

lifetime issues of dynamically allocated fields in the Py_buffer struct

that led to multiple crash reports. Additionally, several functions that

crashed or returned incorrect results for non-contiguous or multi-dimensional

input have been fixed.

The memoryview object now has a PEP-3118 compliant getbufferproc()

that checks the consumer's request type. Many new features have been

added, most of them work in full generality for non-contiguous arrays

and arrays with suboffsets.

The documentation has been updated, clearly spelling out responsibilities

for both exporters and consumers. Buffer request flags are grouped into

basic and compound flags. The memory layout of non-contiguous and

multi-dimensional NumPy-style arrays is explained.

Features

--------

* All native single character format specifiers in struct module syntax

(optionally prefixed with '@') are now supported.

* With some restrictions, the cast() method allows changing of format and

shape of C-contiguous arrays.

* Multi-dimensional list representations are supported for any array type.

* Multi-dimensional comparisons are supported for any array type.

* One-dimensional memoryviews of hashable (read-only) types with formats B,

b or c are now hashable. (Contributed by Antoine Pitrou in :issue:`13411`.)

* Arbitrary slicing of any 1-D arrays type is supported. For example, it

is now possible to reverse a memoryview in *O*\ (1) by using a negative step.

API changes

-----------

* The maximum number of dimensions is officially limited to 64.

* The representation of empty shape, strides and suboffsets is now

an empty tuple instead of ``None``.

* Accessing a memoryview element with format 'B' (unsigned bytes)

now returns an integer (in accordance with the struct module syntax).

For returning a bytes object the view must be cast to 'c' first.

* memoryview comparisons now use the logical structure of the operands

and compare all array elements by value. All format strings in struct

module syntax are supported. Views with unrecognised format strings

are still permitted, but will always compare as unequal, regardless

of view contents.

* For further changes see `Build and C API Changes`_ and `Porting C code`_.

(Contributed by Stefan Krah in :issue:`10181`.)

.. seealso::

:pep:`3118` - Revising the Buffer Protocol

.. _pep-393:

PEP 393: Flexible String Representation

=======================================

The Unicode string type is changed to support multiple internal

representations, depending on the character with the largest Unicode ordinal

(1, 2, or 4 bytes) in the represented string. This allows a space-efficient

representation in common cases, but gives access to full UCS-4 on all

systems. For compatibility with existing APIs, several representations may

exist in parallel; over time, this compatibility should be phased out.

On the Python side, there should be no downside to this change.

On the C API side, :pep:`393` is fully backward compatible. The legacy API

should remain available at least five years. Applications using the legacy

API will not fully benefit of the memory reduction, or - worse - may use

a bit more memory, because Python may have to maintain two versions of each

string (in the legacy format and in the new efficient storage).

Functionality

-------------

Changes introduced by :pep:`393` are the following:

* Python now always supports the full range of Unicode code points, including

non-BMP ones (i.e. from ``U+0000`` to ``U+10FFFF``). The distinction between

narrow and wide builds no longer exists and Python now behaves like a wide

build, even under Windows.

* With the death of narrow builds, the problems specific to narrow builds have

also been fixed, for example:

* :func:`len` now always returns 1 for non-BMP characters,

so ``len('\U0010FFFF') == 1``;

* surrogate pairs are not recombined in string literals,

so ``'\uDBFF\uDFFF' != '\U0010FFFF'``;

* indexing or slicing non-BMP characters returns the expected value,

so ``'\U0010FFFF'[0]`` now returns ``'\U0010FFFF'`` and not ``'\uDBFF'``;

* all other functions in the standard library now correctly handle

non-BMP code points.

* The value of :data:`sys.maxunicode` is now always ``1114111`` (``0x10FFFF``

in hexadecimal). The :c:func:`!PyUnicode_GetMax` function still returns

either ``0xFFFF`` or ``0x10FFFF`` for backward compatibility, and it should

not be used with the new Unicode API (see :issue:`13054`).

* The :file:`./configure` flag ``--with-wide-unicode`` has been removed.

Performance and resource usage

------------------------------

The storage of Unicode strings now depends on the highest code point in the string:

* pure ASCII and Latin1 strings (``U+0000-U+00FF``) use 1 byte per code point;

* BMP strings (``U+0000-U+FFFF``) use 2 bytes per code point;

* non-BMP strings (``U+10000-U+10FFFF``) use 4 bytes per code point.

The net effect is that for most applications, memory usage of string

storage should decrease significantly - especially compared to former

wide unicode builds - as, in many cases, strings will be pure ASCII

even in international contexts (because many strings store non-human

language data, such as XML fragments, HTTP headers, JSON-encoded data,

etc.). We also hope that it will, for the same reasons, increase CPU

cache efficiency on non-trivial applications. The memory usage of

Python 3.3 is two to three times smaller than Python 3.2, and a little

bit better than Python 2.7, on a Django benchmark (see the PEP for

details).

.. seealso::

:pep:`393` - Flexible String Representation

PEP written by Martin von Löwis; implementation by Torsten Becker

and Martin von Löwis.

.. _pep-397:

PEP 397: Python Launcher for Windows

====================================

The Python 3.3 Windows installer now includes a ``py`` launcher application

that can be used to launch Python applications in a version independent

fashion.

This launcher is invoked implicitly when double-clicking ``*.py`` files.

If only a single Python version is installed on the system, that version

will be used to run the file. If multiple versions are installed, the most

recent version is used by default, but this can be overridden by including

a Unix-style "shebang line" in the Python script.

The launcher can also be used explicitly from the command line as the ``py``

application. Running ``py`` follows the same version selection rules as

implicitly launching scripts, but a more specific version can be selected

by passing appropriate arguments (such as ``-3`` to request Python 3 when

Python 2 is also installed, or ``-2.6`` to specifically request an earlier

Python version when a more recent version is installed).

In addition to the launcher, the Windows installer now includes an

option to add the newly installed Python to the system PATH. (Contributed

by Brian Curtin in :issue:`3561`.)

.. seealso::

:pep:`397` - Python Launcher for Windows

PEP written by Mark Hammond and Martin v. Löwis; implementation by

Vinay Sajip.

Launcher documentation: :ref:`launcher`

Installer PATH modification: :ref:`windows-path-mod`

.. _pep-3151:

PEP 3151: Reworking the OS and IO exception hierarchy

=====================================================

The hierarchy of exceptions raised by operating system errors is now both

simplified and finer-grained.

You don't have to worry anymore about choosing the appropriate exception

type between :exc:`OSError`, :exc:`IOError`, :exc:`EnvironmentError`,

:exc:`WindowsError`, :exc:`!mmap.error`, :exc:`socket.error` or

:exc:`select.error`. All these exception types are now only one:

:exc:`OSError`. The other names are kept as aliases for compatibility

reasons.

Also, it is now easier to catch a specific error condition. Instead of

inspecting the ``errno`` attribute (or ``args[0]``) for a particular

constant from the :mod:`errno` module, you can catch the adequate

:exc:`OSError` subclass. The available subclasses are the following:

* :exc:`BlockingIOError`

* :exc:`ChildProcessError`

* :exc:`ConnectionError`

* :exc:`FileExistsError`

* :exc:`FileNotFoundError`

* :exc:`InterruptedError`

* :exc:`IsADirectoryError`

* :exc:`NotADirectoryError`

* :exc:`PermissionError`

* :exc:`ProcessLookupError`

* :exc:`TimeoutError`

And the :exc:`ConnectionError` itself has finer-grained subclasses:

* :exc:`BrokenPipeError`

* :exc:`ConnectionAbortedError`

* :exc:`ConnectionRefusedError`

* :exc:`ConnectionResetError`

Thanks to the new exceptions, common usages of the :mod:`errno` can now be

avoided. For example, the following code written for Python 3.2::

from errno import ENOENT, EACCES, EPERM

try:

with open("document.txt") as f:

content = f.read()

except IOError as err:

if err.errno == ENOENT:

print("document.txt file is missing")

elif err.errno in (EACCES, EPERM):

print("You are not allowed to read document.txt")

else:

raise

can now be written without the :mod:`errno` import and without manual

inspection of exception attributes::

try:

with open("document.txt") as f:

content = f.read()

except FileNotFoundError:

print("document.txt file is missing")

except PermissionError:

print("You are not allowed to read document.txt")

.. seealso::

:pep:`3151` - Reworking the OS and IO Exception Hierarchy

PEP written and implemented by Antoine Pitrou

.. index::

single: yield; yield from (in What's New)

.. _pep-380:

PEP 380: Syntax for Delegating to a Subgenerator

================================================

PEP 380 adds the ``yield from`` expression, allowing a :term:`generator` to

delegate

part of its operations to another generator. This allows a section of code

containing :keyword:`yield` to be factored out and placed in another generator.

Additionally, the subgenerator is allowed to return with a value, and the

value is made available to the delegating generator.

While designed primarily for use in delegating to a subgenerator, the ``yield

from`` expression actually allows delegation to arbitrary subiterators.

For simple iterators, ``yield from iterable`` is essentially just a shortened

form of ``for item in iterable: yield item``::

>>> def g(x):

... yield from range(x, 0, -1)

... yield from range(x)

...

>>> list(g(5))

[5, 4, 3, 2, 1, 0, 1, 2, 3, 4]

However, unlike an ordinary loop, ``yield from`` allows subgenerators to

receive sent and thrown values directly from the calling scope, and

return a final value to the outer generator::

>>> def accumulate():

... tally = 0

... while 1:

... next = yield

... if next is None:

... return tally

... tally += next

...

>>> def gather_tallies(tallies):

... while 1:

... tally = yield from accumulate()

... tallies.append(tally)

...

>>> tallies = []

>>> acc = gather_tallies(tallies)

>>> next(acc) # Ensure the accumulator is ready to accept values

>>> for i in range(4):

... acc.send(i)

...

>>> acc.send(None) # Finish the first tally

>>> for i in range(5):

... acc.send(i)

...

>>> acc.send(None) # Finish the second tally

>>> tallies

[6, 10]

The main principle driving this change is to allow even generators that are

designed to be used with the ``send`` and ``throw`` methods to be split into

multiple subgenerators as easily as a single large function can be split into

multiple subfunctions.

.. seealso::

:pep:`380` - Syntax for Delegating to a Subgenerator

PEP written by Greg Ewing; implementation by Greg Ewing, integrated into

3.3 by Renaud Blanch, Ryan Kelly and Nick Coghlan; documentation by

Zbigniew Jędrzejewski-Szmek and Nick Coghlan

PEP 409: Suppressing exception context

======================================

PEP 409 introduces new syntax that allows the display of the chained

exception context to be disabled. This allows cleaner error messages in

applications that convert between exception types::

>>> class D:

... def __init__(self, extra):

... self._extra_attributes = extra

... def __getattr__(self, attr):

... try:

... return self._extra_attributes[attr]

... except KeyError:

... raise AttributeError(attr) from None

...

>>> D({}).x

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

File "<stdin>", line 8, in __getattr__

AttributeError: x

Without the ``from None`` suffix to suppress the cause, the original

exception would be displayed by default::

>>> class C:

... def __init__(self, extra):

... self._extra_attributes = extra

... def __getattr__(self, attr):

... try:

... return self._extra_attributes[attr]

... except KeyError:

... raise AttributeError(attr)

...

>>> C({}).x

Traceback (most recent call last):

File "<stdin>", line 6, in __getattr__

KeyError: 'x'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

File "<stdin>", line 8, in __getattr__

AttributeError: x

No debugging capability is lost, as the original exception context remains

available if needed (for example, if an intervening library has incorrectly

suppressed valuable underlying details)::

>>> try:

... D({}).x

... except AttributeError as exc:

... print(repr(exc.__context__))

...

KeyError('x',)

.. seealso::

:pep:`409` - Suppressing exception context

PEP written by Ethan Furman; implemented by Ethan Furman and Nick

Coghlan.

PEP 414: Explicit Unicode literals

======================================

To ease the transition from Python 2 for Unicode aware Python applications

that make heavy use of Unicode literals, Python 3.3 once again supports the

"``u``" prefix for string literals. This prefix has no semantic significance

in Python 3, it is provided solely to reduce the number of purely mechanical

changes in migrating to Python 3, making it easier for developers to focus on

the more significant semantic changes (such as the stricter default

separation of binary and text data).

.. seealso::

:pep:`414` - Explicit Unicode literals

PEP written by Armin Ronacher.

PEP 3155: Qualified name for classes and functions

==================================================

Functions and class objects have a new :attr:`~definition.__qualname__`

attribute representing

the "path" from the module top-level to their definition. For global functions

and classes, this is the same as :attr:`~definition.__name__`.

For other functions and classes,

it provides better information about where they were actually defined, and

how they might be accessible from the global scope.

Example with (non-bound) methods::

>>> class C:

... def meth(self):

... pass

...

>>> C.meth.__name__

'meth'

>>> C.meth.__qualname__

'C.meth'

Example with nested classes::

>>> class C:

... class D:

... def meth(self):

... pass

...

>>> C.D.__name__

'D'

>>> C.D.__qualname__

'C.D'

>>> C.D.meth.__name__

'meth'

>>> C.D.meth.__qualname__

'C.D.meth'

Example with nested functions::

>>> def outer():

... def inner():

... pass

... return inner

...

>>> outer().__name__

'inner'

>>> outer().__qualname__

'outer.<locals>.inner'

The string representation of those objects is also changed to include the

new, more precise information::

>>> str(C.D)

"<class '__main__.C.D'>"

>>> str(C.D.meth)

'<function C.D.meth at 0x7f46b9fe31e0>'

.. seealso::

:pep:`3155` - Qualified name for classes and functions

PEP written and implemented by Antoine Pitrou.

.. _pep-412:

PEP 412: Key-Sharing Dictionary

===============================

Dictionaries used for the storage of objects' attributes are now able to

share part of their internal storage between each other (namely, the part

which stores the keys and their respective hashes). This reduces the memory

consumption of programs creating many instances of non-builtin types.

.. seealso::

:pep:`412` - Key-Sharing Dictionary

PEP written and implemented by Mark Shannon.

PEP 362: Function Signature Object

==================================

A new function :func:`inspect.signature` makes introspection of python

callables easy and straightforward. A broad range of callables is supported:

python functions, decorated or not, classes, and :func:`functools.partial`

objects. New classes :class:`inspect.Signature`, :class:`inspect.Parameter`

and :class:`inspect.BoundArguments` hold information about the call signatures,

such as, annotations, default values, parameters kinds, and bound arguments,

which considerably simplifies writing decorators and any code that validates

or amends calling signatures or arguments.

.. seealso::

:pep:`362`: - Function Signature Object

PEP written by Brett Cannon, Yury Selivanov, Larry Hastings, Jiwon Seo;

implemented by Yury Selivanov.

PEP 421: Adding sys.implementation

==================================

A new attribute on the :mod:`sys` module exposes details specific to the

implementation of the currently running interpreter. The initial set of

attributes on :data:`sys.implementation` are ``name``, ``version``,

``hexversion``, and ``cache_tag``.

The intention of ``sys.implementation`` is to consolidate into one namespace

the implementation-specific data used by the standard library. This allows

different Python implementations to share a single standard library code base

much more easily. In its initial state, ``sys.implementation`` holds only a

small portion of the implementation-specific data. Over time that ratio will

shift in order to make the standard library more portable.

One example of improved standard library portability is ``cache_tag``. As of

Python 3.3, ``sys.implementation.cache_tag`` is used by :mod:`importlib` to

support :pep:`3147` compliance. Any Python implementation that uses

``importlib`` for its built-in import system may use ``cache_tag`` to control

the caching behavior for modules.

SimpleNamespace

---------------

The implementation of ``sys.implementation`` also introduces a new type to

Python: :class:`types.SimpleNamespace`. In contrast to a mapping-based

namespace, like :class:`dict`, ``SimpleNamespace`` is attribute-based, like

:class:`object`. However, unlike ``object``, ``SimpleNamespace`` instances

are writable. This means that you can add, remove, and modify the namespace

through normal attribute access.

.. seealso::

:pep:`421` - Adding sys.implementation

PEP written and implemented by Eric Snow.

.. _importlib:

Using importlib as the Implementation of Import

===============================================

:issue:`2377` - Replace __import__ w/ importlib.__import__

:issue:`13959` - Re-implement parts of :mod:`!imp` in pure Python

:issue:`14605` - Make import machinery explicit

:issue:`14646` - Require loaders set __loader__ and __package__

The :func:`__import__` function is now powered by :func:`importlib.__import__`.

This work leads to the completion of "phase 2" of :pep:`302`. There are

multiple benefits to this change. First, it has allowed for more of the

machinery powering import to be exposed instead of being implicit and hidden

within the C code. It also provides a single implementation for all Python VMs

supporting Python 3.3 to use, helping to end any VM-specific deviations in

import semantics. And finally it eases the maintenance of import, allowing for

future growth to occur.

For the common user, there should be no visible change in semantics. For

those whose code currently manipulates import or calls import

programmatically, the code changes that might possibly be required are covered

in the `Porting Python code`_ section of this document.

New APIs

--------

One of the large benefits of this work is the exposure of what goes into

making the import statement work. That means the various importers that were

once implicit are now fully exposed as part of the :mod:`importlib` package.

The abstract base classes defined in :mod:`importlib.abc` have been expanded

to properly delineate between :term:`meta path finders <meta path finder>`

and :term:`path entry finders <path entry finder>` by introducing

:class:`importlib.abc.MetaPathFinder` and

:class:`importlib.abc.PathEntryFinder`, respectively. The old ABC of

:class:`!importlib.abc.Finder` is now only provided for backwards-compatibility

and does not enforce any method requirements.

In terms of finders, :class:`importlib.machinery.FileFinder` exposes the

mechanism used to search for source and bytecode files of a module. Previously

this class was an implicit member of :data:`sys.path_hooks`.

For loaders, the new abstract base class :class:`importlib.abc.FileLoader` helps

write a loader that uses the file system as the storage mechanism for a module's

code. The loader for source files

(:class:`importlib.machinery.SourceFileLoader`), sourceless bytecode files

(:class:`importlib.machinery.SourcelessFileLoader`), and extension modules

(:class:`importlib.machinery.ExtensionFileLoader`) are now available for

direct use.

:exc:`ImportError` now has ``name`` and ``path`` attributes which are set when

there is relevant data to provide. The message for failed imports will also

provide the full name of the module now instead of just the tail end of the

module's name.

The :func:`importlib.invalidate_caches` function will now call the method with

the same name on all finders cached in :data:`sys.path_importer_cache` to help

clean up any stored state as necessary.

Visible Changes

---------------

For potential required changes to code, see the `Porting Python code`_

section.

Beyond the expanse of what :mod:`importlib` now exposes, there are other

visible changes to import. The biggest is that :data:`sys.meta_path` and

:data:`sys.path_hooks` now store all of the meta path finders and path entry

hooks used by import. Previously the finders were implicit and hidden within

the C code of import instead of being directly exposed. This means that one can

now easily remove or change the order of the various finders to fit one's needs.

Another change is that all modules have a ``__loader__`` attribute, storing the

loader used to create the module. :pep:`302` has been updated to make this

attribute mandatory for loaders to implement, so in the future once 3rd-party

loaders have been updated people will be able to rely on the existence of the

attribute. Until such time, though, import is setting the module post-load.

Loaders are also now expected to set the ``__package__`` attribute from

:pep:`366`. Once again, import itself is already setting this on all loaders

from :mod:`importlib` and import itself is setting the attribute post-load.

``None`` is now inserted into :data:`sys.path_importer_cache` when no finder

can be found on :data:`sys.path_hooks`. Since :class:`!imp.NullImporter` is not

directly exposed on :data:`sys.path_hooks` it could no longer be relied upon to

always be available to use as a value representing no finder found.

All other changes relate to semantic changes which should be taken into

consideration when updating code for Python 3.3, and thus should be read about

in the `Porting Python code`_ section of this document.

(Implementation by Brett Cannon)

Other Language Changes

======================

Some smaller changes made to the core Python language are:

* Added support for Unicode name aliases and named sequences.

Both :func:`unicodedata.lookup` and ``'\N{...}'`` now resolve name aliases,

and :func:`unicodedata.lookup` resolves named sequences too.

(Contributed by Ezio Melotti in :issue:`12753`.)

* Unicode database updated to UCD version 6.1.0

* Equality comparisons on :func:`range` objects now return a result reflecting

the equality of the underlying sequences generated by those range objects.

(:issue:`13201`)

* The ``count()``, ``find()``, ``rfind()``, ``index()`` and ``rindex()``

methods of :class:`bytes` and :class:`bytearray` objects now accept an

integer between 0 and 255 as their first argument.

(Contributed by Petri Lehtinen in :issue:`12170`.)

* The ``rjust()``, ``ljust()``, and ``center()`` methods of :class:`bytes`

and :class:`bytearray` now accept a :class:`bytearray` for the ``fill``

argument. (Contributed by Petri Lehtinen in :issue:`12380`.)

* New methods have been added to :class:`list` and :class:`bytearray`:

``copy()`` and ``clear()`` (:issue:`10516`). Consequently,

:class:`~collections.abc.MutableSequence` now also defines a

:meth:`!clear` method (:issue:`11388`).

* Raw bytes literals can now be written ``rb"..."`` as well as ``br"..."``.

(Contributed by Antoine Pitrou in :issue:`13748`.)

* :meth:`dict.setdefault` now does only one lookup for the given key, making

it atomic when used with built-in types.

(Contributed by Filip Gruszczyński in :issue:`13521`.)

* The error messages produced when a function call does not match the function

signature have been significantly improved.

(Contributed by Benjamin Peterson.)

A Finer-Grained Import Lock

===========================

Previous versions of CPython have always relied on a global import lock.

This led to unexpected annoyances, such as deadlocks when importing a module

would trigger code execution in a different thread as a side-effect.

Clumsy workarounds were sometimes employed, such as the

:c:func:`PyImport_ImportModuleNoBlock` C API function.

In Python 3.3, importing a module takes a per-module lock. This correctly

serializes importation of a given module from multiple threads (preventing

the exposure of incompletely initialized modules), while eliminating the

aforementioned annoyances.

(Contributed by Antoine Pitrou in :issue:`9260`.)

Builtin functions and types

===========================

* :func:`open` gets a new *opener* parameter: the underlying file descriptor

for the file object is then obtained by calling *opener* with (*file*,

*flags*). It can be used to use custom flags like :const:`os.O_CLOEXEC` for

example. The ``'x'`` mode was added: open for exclusive creation, failing if

the file already exists.

* :func:`print`: added the *flush* keyword argument. If the *flush* keyword

argument is true, the stream is forcibly flushed.

* :func:`hash`: hash randomization is enabled by default, see

:meth:`object.__hash__` and :envvar:`PYTHONHASHSEED`.

* The :class:`str` type gets a new :meth:`~str.casefold` method: return a

casefolded copy of the string, casefolded strings may be used for caseless

matching. For example, ``'ß'.casefold()`` returns ``'ss'``.

* The sequence documentation has been substantially rewritten to better

explain the binary/text sequence distinction and to provide specific

documentation sections for the individual builtin sequence types

(:issue:`4966`).

New Modules

===========

faulthandler

------------

This new debug module :mod:`faulthandler` contains functions to dump Python tracebacks explicitly,

on a fault (a crash like a segmentation fault), after a timeout, or on a user

signal. Call :func:`faulthandler.enable` to install fault handlers for the

:const:`~signal.SIGSEGV`, :const:`~signal.SIGFPE`, :const:`~signal.SIGABRT`,

:const:`~signal.SIGBUS`, and :const:`~signal.SIGILL` signals.

You can also enable them at startup by setting the :envvar:`PYTHONFAULTHANDLER`

environment variable or by using :option:`-X` ``faulthandler`` command line option.

Example of a segmentation fault on Linux:

.. code-block:: shell-session

$ python -q -X faulthandler

>>> import ctypes

>>> ctypes.string_at(0)

Fatal Python error: Segmentation fault

Current thread 0x00007fb899f39700:

File "/home/python/cpython/Lib/ctypes/__init__.py", line 486 in string_at

File "<stdin>", line 1 in <module>

Segmentation fault

ipaddress

---------

The new :mod:`ipaddress` module provides tools for creating and manipulating

objects representing IPv4 and IPv6 addresses, networks and interfaces (i.e.

an IP address associated with a specific IP subnet).

(Contributed by Google and Peter Moody in :pep:`3144`.)

lzma

----

The newly added :mod:`lzma` module provides data compression and decompression

using the LZMA algorithm, including support for the ``.xz`` and ``.lzma``

file formats.

(Contributed by Nadeem Vawda and Per Øyvind Karlsen in :issue:`6715`.)

Improved Modules

================

abc

---

Improved support for abstract base classes containing descriptors composed with

abstract methods. The recommended approach to declaring abstract descriptors is

now to provide :attr:`!__isabstractmethod__` as a dynamically updated

property. The built-in descriptors have been updated accordingly.

* :deco:`abc.abstractproperty` has been deprecated, use :deco:`property`

with :deco:`abc.abstractmethod` instead.

* :deco:`abc.abstractclassmethod` has been deprecated, use

:deco:`classmethod` with :deco:`abc.abstractmethod` instead.

* :deco:`abc.abstractstaticmethod` has been deprecated, use

:deco:`staticmethod` with :deco:`abc.abstractmethod` instead.

(Contributed by Darren Dale in :issue:`11610`.)

:meth:`abc.ABCMeta.register` now returns the registered subclass, which means

it can now be used as a class decorator (:issue:`10868`).

array

-----

The :mod:`array` module supports the :c:expr:`long long` type using ``q`` and

``Q`` type codes.

(Contributed by Oren Tirosh and Hirokazu Yamamoto in :issue:`1172711`.)

base64

------

ASCII-only Unicode strings are now accepted by the decoding functions of the

:mod:`base64` modern interface. For example, ``base64.b64decode('YWJj')``

returns ``b'abc'``. (Contributed by Catalin Iacob in :issue:`13641`.)

binascii

--------

In addition to the binary objects they normally accept, the ``a2b_`` functions

now all also accept ASCII-only strings as input. (Contributed by Antoine

Pitrou in :issue:`13637`.)

bz2

---

The :mod:`bz2` module has been rewritten from scratch. In the process, several

new features have been added:

* New :func:`bz2.open` function: open a bzip2-compressed file in binary or

text mode.

* :class:`bz2.BZ2File` can now read from and write to arbitrary file-like

objects, by means of its constructor's *fileobj* argument.

(Contributed by Nadeem Vawda in :issue:`5863`.)

* :class:`bz2.BZ2File` and :func:`bz2.decompress` can now decompress

multi-stream inputs (such as those produced by the :program:`pbzip2` tool).

:class:`bz2.BZ2File` can now also be used to create this type of file, using

the ``'a'`` (append) mode.

(Contributed by Nir Aides in :issue:`1625`.)

* :class:`bz2.BZ2File` now implements all of the :class:`io.BufferedIOBase` API,

except for the :meth:`!detach` and :meth:`!truncate` methods.

codecs

------

The :mod:`~encodings.mbcs` codec has been rewritten to handle correctly

``replace`` and ``ignore`` error handlers on all Windows versions. The

:mod:`~encodings.mbcs` codec now supports all error handlers, instead of only

``replace`` to encode and ``ignore`` to decode.

A new Windows-only codec has been added: ``cp65001`` (:issue:`13216`). It is the

Windows code page 65001 (Windows UTF-8, ``CP_UTF8``). For example, it is used

by ``sys.stdout`` if the console output code page is set to cp65001 (e.g., using

``chcp 65001`` command).

Multibyte CJK decoders now resynchronize faster. They only ignore the first

byte of an invalid byte sequence. For example, ``b'\xff\n'.decode('gb2312',

'replace')`` now returns a ``\n`` after the replacement character.