Datagram Transport Layer Security for Python

PyDTLS brings Datagram Transport Layer Security (DTLS - RFC 6347: http://tools.ietf.org/html/rfc6347) to the Python environment. In a nutshell, DTLS brings security (encryption, server authentication, user authentication, and message authentication) to UDP datagram payloads in a manner equivalent to what SSL/TLS does for TCP stream content.

DTLS is now very easy to use in Python. If you're familiar with the ssl module in Python's standard library, you already know how. All it takes is passing a datagram/UDP socket to the wrap_socket function instead of a stream/TCP socket. Here's how one sets up the client side of a connection:

import ssl
from socket import socket, AF_INET, SOCK_DGRAM
from dtls import do_patch
do_patch()
sock = ssl.wrap_socket(socket(AF_INET, SOCK_DGRAM))
sock.connect(('foo.bar.com', 1234))
sock.send('Hi there')

As of version 1.2.0, PyDTLS supports DTLS version 1.2 in addition to version 1.0. This version also introduces forward secrecy using elliptic curve cryptography and more fine-grained configuration options.

Installation

To install from PyPI, on any supported platform enter:

pip install Dtls

Design Goals

The primary design goal of PyDTLS is broad availability. It has therefore been built to be widely compatible with the following:

• Operating systems: apart from the Python standard library, PyDTLS relies on the OpenSSL library only. OpenSSL is widely ported, and in fact the Python standard library's ssl module also uses it.
• Python runtime environments: PyDTLS is a package consisting of pure Python modules only. It should therefore be portable to many interpreters and runtime environments. It interfaces with OpenSSL strictly through the standard library's ctypes foreign function library.
• The Python standard library: the standard library's ssl module is Python's de facto interface to SSL/TLS. PyDTLS aims to be compatible with the full public interface presented by this module. The ssl module ought to behave identically with respect to all of its functions and options when used in conjunction with PyDTLS and datagram sockets as when used without PyDTLS and stream sockets.
• Connection-based protocols: as outlined below, layering security on top of datagram sockets requires introducing certain connection constructs normally absent from datagram sockets. These constructs have been added in such a way as to be compatible with code that expects to interoperate with connection-oriented stream sockets. For example, code that expects to go through server-side bind/listen/accept connection establishment should be reusable with PyDTLS sockets.

Distributions

PyDTLS requires version 1.0.0 or higher of the OpenSSL library. Earlier versions are reported not to offer stable DTLS support. Since packaged distributions of this version of OpenSSL are available for many popular operating systems, OpenSSL-1.0.0 is an installation requirement before PyDTLS functionality can be called. On Ubuntu 12.04 LTS, for example, the Python interpreter links with libcrypto.so.1.0.0 and libssl.so.1.0.0, and so use of PyDTLS requires no further installation steps.

In comparison, installation of OpenSSL on Microsoft Windows operating systems is inconvenient. For this reason, source distributions of PyDTLS are available that include OpenSSL dll's for 32-bit and 64-bit Windows. All dll's have been linked with the Visual Studio 2008 version of the Microsoft C runtime library, msvcr90.dll, the version used by CPython 2.7. Installation of Microsoft redistributable runtime packages should therefore not be required on machines with CPython 2.7. The version of OpenSSL distributed with PyDTLS 0.1.0 is 1.0.1c. The version distributed with PyDTLS 1.2.0 is commit 248cf959672041f38f4d80a4a09ee01d8ab04fe8 (branch OpenSSL_1_0_2-stable, 1.0.2l-dev, containing a desirable fix to DTLSv1_listen not present in 1.0.2k, the stable version at the time of PyDTLS 1.2.0 release).

The OpenSSL version used by PyDTLS can be determined from the values of sslconnection's DTLS_OPENSSL_VERSION_NUMBER, DTLS_OPENSSL_VERSION, and DTLS_OPENSSL_VERSION_INFO. These variables are available through the ssl module also if do_patch has been called (see below). Note that the OpenSSL version used by PyDTLS may differ from the one used by the ssl module.

Interfaces

PyDTLS' top-level package, dtls, provides DTLS support through the SSLConnection class of its sslconnection module. SSLConnection is in-line documented, and dtls/test/echo_seq.py demonstrates how to take a simple echo server through a listen/accept/echo/shutdown sequence using this class. The corresponding client side can look like the snippet at the top of this document, followed by a call to the unwrap method for shutdown (or a call to the SSLConnection shutdown method, if an instance of this class is used for the client side also). Note that the dtls package does not depend on the standard library's ssl module, and SSLConnection can therefore be used in environments where ssl is unavailable or incompatible.

It is expected that with the ssl module being an established, familiar interface to TLS, it will be the preferred module through which to access DTLS. To do so, one must call the dtls package's do_patch function before passing sockets of type SOCK_DGRAM to either ssl's wrap_socket function, or ssl's SSLSocket constructor.

It should be noted that once do_patch is called, dtls will raise exceptions of type ssl.SSLError instead of its default dtls.err.SSLError. This allows users' error handling code paths to remain identical when interfacing with ssl across stream and datagram sockets.

Connection Handling

The DTLS protocol implies a connection as an association between two network peers where the overall association state is characterized by the handshake status of each peer endpoint (see RFC 6347). The OpenSSL library records this handshake status in "SSL" type instances (a.k.a. struct ssl_st). Datagrams can be securely sent and received by referring to a unique "SSL" instance after handshaking has been completed with this instance and its network peer. A connection is implied in that traffic may be directed to or received from only that network peer with whose "SSL" instance the handshake has been completed. The fact that the underlying network protocol, UDP in most cases, is itself connectionless is immaterial.

Further, in order to prevent denial-of-service attacks on UDP DTLS servers, clients must undergo a cookie exchange phase early in the handshaking protocol, and before server-side resources are committed to a particular client (see section 4.2.1 of RFC 6347). The cookie exchange proves to the server that a client can indeed receive IP traffic at the source IP address with which its handshake-initiating ClientHello datagram is marked.

PyDTLS implements this connection establishment through the connect method on the client side, and the accept method on the server side. The latter returns a new dtls.SSLConnection or ssl.SSLSocket object (depending on which interface is used, see above), which is "connected" to its peer. In addition to the read and write methods (at both interface levels), SSLSocket's send and recv methods can be used; use of sendto and recvfrom on connected sockets is prohibited by ssl. accept returns peer address information, as expected. Note that when using the ssl interface to dtls, listen must be called before calling accept.

Demultiplexing

At the network io layer, only datagrams from its connected peer must be passed to a SSLConnection or SSLSocket object (unless the object is unconnected on the server-side, in which case it can be in listening mode, the initial server-side socket whose role it is to listen for incoming client connection requests).

The initial server-side listening socket is not useful for performing this datagram routing function. This is because it must remain unconnected and ready to receive additional connection requests from new, unknown clients.

The function of passing incoming datagrams to the proper connection is performed by the dtls.demux package. SSLConnection requests a new connection from the demux when a handshake has cleared the cookie exchange phase. An efficient implementation of this request is provided by the osnet module of the demux package: it creates a new socket that is bound to the same network interface and port as the listening socket, but connected to the peer. UDP stacks such as the one included with Linux route incoming datagrams to such a connected socket in preference to an unconnected socket bound to the same port.

Unfortunately such is not the behavior on Microsoft Windows. Windows UDP routes datagrams to whichever currently existing socket bound to the particular port the earliest (and whether or not that socket is unconnected, or connected to the datagram's peer, or a different peer). Other sockets bound to the same port will not receive traffic, if and until they become the earliest bound socket because another socket is closed.

The demux package therefore provides and automatically selects the module router on Windows platforms. This module also creates a new socket when receiving a new connection request; but instead of binding this socket to the same port as the listening socket, it binds to a new ephemeral port. router then forwards datagrams originating from the peer for which a connection was requested to the corresponding socket.

For efficiency's sake, no forwarding is performed on outgoing traffic. Instead, SSLConnection directs outgoing traffic from the original listening socket, using sendto. At the OpenSSL level this requires separate read and write datagram BIO's for an "SSL" instance, one in "connected" state and one in "peer set" state, respectively, and associated with two separate network sockets.

From the perspective of a PyDTLS user, this selection of and difference between demux implementations should be transparent, with the possible exception of performance deviation. This transparency does however have some limits: for example, when router is in use, the accept methods can return None. This happens when SSLConnection detects that the demux has forwarded a datagram to a known connection instead of initiating a connection to a new peer through accept. Returning None in this case is important whenever non-blocking sockets or sockets with timeouts are used, since another socket might now be readable as a result of the forwarded datagram. accept must return so that the application can iterate on its asynchronous select loop.

Shutdown and Unwrapping

PyDTLS implements the SSL/TLS shutdown protocol as it has been adapted for DTLS. SSLConnection's shutdown and SSLSocket's unwrap invoke this protocol. As is the case with DTLS handshaking in general, applications must be prepared to use the get_timeout and handle_timeout methods in addition to re-invoking shutdown or unwrap when sockets become readable and an exception carried SSL_ERROR_WANT_READ. (See more on asynchronous IO in the Testing section.)

SSLConnection's shutdown and SSLSocket's unwrap return a (possibly new) socket that can be used for unsecured communication with the peer, as set forth by the ssl module. The demux infrastructure remains in use for this communication until the returned socket is cleaned up. Note that when the router demux is in use, the object returned will be one derived from socket.socket. This is because the send and recv paths must still be directed to two different OS sockets. In addition, the right thing happens if secured communication is resumed over such a socket by passing it to ssl.wrap_socket or the SSLConnection constructor. If osnet is used, an actual socket.socket instance is returned.

Framework Compatibility

PyDTLS sockets have been tested under the following usage modes:

• Using blocking sockets and sockets with timeouts in multi-threaded UDP servers
• Using non-blocking sockets, and in conjunction with the asynchronous socket handler, asyncore
• Using blocking sockets, and in conjunction with the network server framework SocketServer - ThreadingTCPServer (this works because of PyDTLS's emulation of connection-related calls)

Multi-thread Support

Using multiple threads with OpenSSL requires implementing a locking callback. PyDTLS does implement this, and therefore multi-threaded programming with PyDTLS is safe in any environment. However, being a pure Python library, these callbacks do carry some overhead. The ssl module implements an equivalent locking callback in its C extension module. Not requiring interpreter re-entry, this approach can be expected to perform better. PyDTLS therefore queries OpenSSL as to whether a locking callback is already in place, and does not overwrite it if there is. Loading ssl can therefore improve performance, even when only the sslconnection interface is used.

Note that loading order does not matter: to obtain the performance benefit, ssl can be loaded before or after the dtls package. This is because ssl does not do an equivalent existing locking callback check, and will simply overwrite the PyDTLS callback if it has already been installed. But ssl should not be loaded while dtls operation is already in progress, when some locks may be in their acquired state.

Also note that this performance enhancement is available only on platforms where PyDTLS loads the same OpenSSL shared object as ssl. On Ubuntu 12.04, for example, this is the case, but on Microsoft Windows it is not.

Testing

A simple echo server is available to be executed from the project root directory with python -m dtls.test.echo_seq. The echo server is reachable using the code snippet at the top of this document, using port 28000 at "localhost".

Unit test suites can be executed from the project root directory with python -m dtls.test.unit [-v] and python -m dtls.test.unit_wrapper (for the client and server wrappers)

Almost all of the Python standard library's ssl unit tests from the module test_ssl.py have been ported to dtls.test.unit.py. All tests have been adjusted to operate with datagram sockets. On Linux, each test is executed four times, varying the address family among IPv4 and IPv6 and the demux among osnet and router. On Windows, where osnet is unavailable, each test is run twice, once with IPv4 and once with IPv6.

The unit test suite includes tests for each of the above-mentioned compatible frameworks. The class AsyncoreEchoServer serves as an example of how to use non-blocking datagram sockets and implement the resulting timeout detection requirements. DTLS in general and OpenSSL in particular require being called back when used with non-blocking sockets (or sockets with timeout option) after DTLS timeouts expire to handle packet loss using re-transmission during a handshake. Handshaking may occur during any read or write operation, even after an initial handshake completes successfully, in case renegotiation is requested by a peer.

Running with the -v switch executes all unit tests in verbose mode.

dtls/test/test_perf.py implements an interactive performance test suite that compares the raw throughput of TCP, UDP, SSL, and DTLS. It can be executed locally through the loopback interface, or between remote clients and servers. In the latter case, test jobs are sent to remote connected clients whenever a suite run is initiated through the interactive interface. Run test_perf.py -h for more information.

It should be noted that comparing the performance of protocols that don't offer congestion control (UDP and DTLS) with those that do (TCP and SSL) is a difficult undertaking. Raw throughput even across gigabit network links can be expected to suffer without congestion control and peers that generate data as fast as possible without throttling (as this test does): the link's throughput will drop significantly as it enters congestion collapse. Similarly, loopback is an imperfect test interface since it rarely drops packets, and never duplicates or reorders them (thus negating the relative performance benefits of DTLS over SSL). Nevertheless, some useful insights can be gained by observing the operation of test_perf.py, including software stack behavior in the presence of some amount of packet loss.

Logging

The dtls package and its sub-packages log various occurrences, primarily events that can aid debugging. Especially router emits many messages when the logging level is set to at least logging.DEBUG. dtls/test/echo_seq.py activates this logging level during its operation.

Currently Supported Platforms

At the time of initial release, PyDTLS 0.1.0 has been tested on Ubuntu 12.04.1 LTS 32-bit and 64-bit, as well as Microsoft Windows 7 32-bit and 64-bit, using CPython 2.7.3. Patches with additional platform ports are welcome.

As of release 1.2.0, PyDTLS is tested on Ubuntu 16.04 LTS as well as Microsoft Windows 10, using CPython 2.7.13.