pydtls/README.txt

============================================
Datagram Transport Layer Security for Python
============================================

PyDTLS brings Datagram Transport Layer Security (DTLS - RFC 6347:
http://tools.ietf.org/html/rfc6347) to the Python environment. In a
nutshell, DTLS brings security (encryption, server authentication,
user authentication, and message authentication) to UDP datagram
payloads in a manner equivalent to what SSL/TLS does for TCP stream
content.

DTLS is now very easy to use in Python. If you're familiar with the
ssl module in Python's standard library, you already know how. All it
takes is passing a datagram/UDP socket to the *wrap_socket* function
instead of a stream/TCP socket. Here's how one sets up the client side
of a connection::

    import ssl
    from socket import socket, AF_INET, SOCK_DGRAM
    from dtls import do_patch
    do_patch()
    sock = ssl.wrap_socket(socket(AF_INET, SOCK_DGRAM))
    sock.connect(('foo.bar.com', 1234))
    sock.send('Hi there')

Design Goals
============

The primary design goal of PyDTLS is broad availability. It has therefore
been built to be widely compatible with the following:

    * Operating systems: apart from the Python standard library, PyDTLS
      relies on the OpenSSL library only. OpenSSL is widely ported, and
      in fact the Python standard library's *ssl* module also uses it.
    * Python runtime environments: PyDTLS is a package consisting of
      pure Python modules only. It should therefore be portable to many
      interpreters and runtime environments. It interfaces with OpenSSL
      strictly through the standard library's *ctypes* foreign function
      library.
    * The Python standard library: the standard library's *ssl* module is
      Python's de facto interface to SSL/TLS. PyDTLS aims to be compatible
      with the full public interface presented by this module. The ssl
      module ought to behave identically with respect to all of its
      functions and options when used in conjunction with PyDTLS and
      datagram sockets as when used without PyDTLS and stream sockets.
    * Connection-based protocols: as outlined below, layering security
      on top of datagram sockets requires introducing certain
      connection constructs normally absent from datagram
      sockets. These constructs have been added in such a way as to be
      compatible with code that expects to interoperate with
      connection-oriented stream sockets. For example, code that
      expects to go through server-side bind/listen/accept connection
      establishment should be reusable with PyDTLS sockets.

Distributions
=============

PyDTLS requires version 1.0.0 or higher of the OpenSSL
library. Earlier versions are reported not to offer stable DTLS
support. Since packaged distributions of this version of OpenSSL are
available for many popular operating systems, OpenSSL-1.0.0 is an
installation requirement before PyDTLS functionality can be called.
On Ubuntu 12.04 LTS, for example, the Python interpreter links with
libcrypto.so.1.0.0 and libssl.so.1.0.0, and so use of PyDTLS requires
no further installation steps.

In comparison, installation of OpenSSL on Microsoft Windows operating
systems is inconvenient. For this reason, source distributions of
PyDTLS are available that include OpenSSL dll's for 32-bit and 64-bit
Windows. For 32-bit Windows, a version built with the MinGW toolchain
is also available. Its archive includes stripped as well as
non-stripped dll's. The latter can be debugged with gdb on
Windows. All dll's have been linked with the Visual Studio 2008
version of the Microsoft C runtime library, msvcr90.dll, the version
used by CPython 2.7. Installation of Microsoft redistributable runtime
packages should therefore not be required on machines with CPython
2.7. The version of OpenSSL distributed with PyDTLS 0.1.0 is 1.0.1c.

The OpenSSL version used by PyDTLS can be determined from the values
of *sslconnection's* DTLS_OPENSSL_VERSION_NUMBER,
DTLS_OPENSSL_VERSION, and DTLS_OPENSSL_VERSION_INFO. These variables
are available through the *ssl* module also if *do_patch* has been
called (see below). Note that the OpenSSL version used by PyDTLS may
differ from the one used by the *ssl* module.

Interfaces
==========

PyDTLS' top-level package, *dtls*, provides DTLS support through the
**SSLConnection** class of its *sslconnection*
module. **SSLConnection** is in-line documented, and
dtls/test/echo_seq.py demonstrates how to take a simple echo server
through a listen/accept/echo/shutdown sequence using this class. The
corresponding client side can look like the snippet at the top of this
document, followed by a call to the *unwrap* method for shutdown (or a
call to the **SSLConnection** *shutdown* method, if an instance of
this class is used for the client side also). Note that the *dtls*
package does not depend on the standard library's *ssl* module, and
**SSLConnection** can therefore be used in environments where *ssl* is
unavailable or incompatible.

It is expected that with the *ssl* module being an established, familiar
interface to TLS, it will be the preferred module through which to
access DTLS. To do so, one must call the *dtls* package's *do_patch*
function before passing sockets of type SOCK_DGRAM to either *ssl's*
*wrap_socket* function, or *ssl's* **SSLSocket** constructor.

It should be noted that once *do_patch* is called, *dtls* will raise
exceptions of type **ssl.SSLError** instead of its default
**dtls.err.SSLError**. This allows users' error handling code paths to
remain identical when interfacing with *ssl* across stream and
datagram sockets.

Connection Handling
===================

The DTLS protocol implies a connection as an association between two
network peers where the overall association state is characterized by the
handshake status of each peer endpoint (see RFC 6347). The OpenSSL library
records this handshake status in "SSL" type instances (a.k.a. struct
ssl_st). Datagrams can be securely sent and received by referring to a
unique "SSL" instance after handshaking has been completed with this
instance and its network peer. A connection is implied in that traffic
may be directed to or received from only that network peer with whose
"SSL" instance the handshake has been completed. The fact that the
underlying network protocol, UDP in most cases, is itself connectionless
is immaterial.

Further, in order to prevent denial-of-service attacks on UDP DTLS
servers, clients must undergo a cookie exchange phase early in the
handshaking protocol, and before server-side resources are committed to
a particular client (see section 4.2.1 of RFC 6347). The cookie exchange
proves to the server that a client can indeed receive IP traffic at
the source IP address with which its handshake-initiating ClientHello
datagram is marked.

PyDTLS implements this connection establishment through the *connect*
method on the client side, and the *accept* method on the server side.
The latter returns a new **dtls.SSLConnection** or **ssl.SSLSocket**
object (depending on which interface is used, see above), which is
"connected" to its peer. In addition to the *read* and *write* methods
(at both interface levels), **SSLSocket's** *send* and *recv* methods
can be used; use of *sendto* and *recvfrom* on connected sockets is
prohibited by *ssl*. *accept* returns peer address information, as
expected. Note that when using the *ssl* interface to *dtls*, *listen*
must be called before calling *accept*.

Demultiplexing
==============

At the network io layer, only datagrams from its connected peer must be
passed to a **SSLConnection** or **SSLSocket** object (unless the object
is unconnected on the server-side, in which case in can be in listening
mode, the initial server-side socket whose role it is to listen for
incoming client connection requests).

The initial server-side listening socket is not useful for performing this
datagram routing function. This is because it must remain unconnected and
ready to receive additional connection requests from new, unknown clients.

The function of passing incoming datagrams to the proper connection is
performed by the *dtls.demux* package. **SSLConnection** requests a new
connection from the demux when a handshake has cleared the cookie
exchange phase. An efficient implementation of this request is provided
by the *osnet* module of the demux package: it creates a new socket that
is bound to the same network interface and port as the listening socket,
but connected to the peer. UDP stacks such as the one included with Linux
route incoming datagrams to such a connected socket in preference to an
unconnected socket bound to the same port.

Unfortunately such is not the behavior on Microsoft Windows. Windows
UDP routes datagrams to whichever currently existing socket bound to
the particular port the earliest (and whether or not that socket is
unconnected, or connected to the datagram's peer, or a different
peer). Other sockets bound to the same port will not receive traffic,
if and until they become the earliest bound socket because another
socket is closed.

The demux package therefore provides and automatically selects the module
*router* on Windows platforms. This module also creates a new socket when
receiving a new connection request; but instead of binding this socket
to the same port as the listening socket, it binds to a new ephemeral
port. *router* then forwards datagrams originating from the peer for which
a connection was requested to the corresponding socket.

For efficiency's sake, no forwarding is performed on outgoing traffic.
Instead, **SSLConnection** directs outgoing traffic from the original
listening socket, using *sendto*. At the OpenSSL level this requires
separate read and write datagram BIO's for an "SSL" instance, one in
"connected" state and one in "peer set" state, respectively, and
associated with two separate network sockets.

From the perspective of a PyDTLS user, this selection of and
difference between demux implementations should be transparent, with
the possible exception of performance deviation. This transparency
does however have some limits: for example, when *router* is in use,
the *accept* methods can return *None*. This happens when
**SSLConnection** detects that the demux has forwarded a datagram to a
known connection instead of initiating a connection to a new peer
through *accept*.  Returning *None* in this case is important whenever
non-blocking sockets or sockets with timeouts are used, since another
socket might now be readable as a result of the forwarded
datagram. *accept* must return so that the application can iterate on
its asynchronous *select* loop.

Shutdown and Unwrapping
=======================

PyDTLS implements the SSL/TLS shutdown protocol as it has been adapted
for DTLS. **SSLConnection's** *shutdown* and **SSLSocket's** *unwrap*
invoke this protocol. As is the case with DTLS handshaking in general,
applications must be prepared to use the *get_timeout* and
*handle_timeout* methods in addition to re-invoking *shutdown* or
*unwrap* when sockets become readable and an exception carried
SSL_ERROR_WANT_READ. (See more on asynchronous IO in the Testing section.)

**SSLConnection's** *shutdown* and **SSLSocket's** *unwrap* return a
(possibly new) socket that can be used for unsecured communication
with the peer, as set forth by the *ssl* module. The demux
infrastructure remains in use for this communication until the
returned socket is cleaned up.  Note that when the *router* demux is
in use, the object returned will be one derived from
*socket.socket*. This is because the send and recv paths must still be
directed to two different OS sockets. In addition, the right thing
happens if secured communication is resumed over such a socket by
passing it to *ssl.wrap_socket* or the **SSLConnection**
constructor. If *osnet* is used, an actual *socket.socket* instance is
returned.

Framework Compatibility
=======================

PyDTLS sockets have been tested under the following usage modes:

    * Using blocking sockets and sockets with timeouts in
      multi-threaded UDP servers
    * Using non-blocking sockets, and in conjunction with the
      asynchronous socket handler, asyncore
    * Using blocking sockets, and in conjunction with the network
      server framework SocketServer - ThreadingTCPServer (this works
      because of PyDTLS's emulation of connection-related calls)

Multi-thread Support
====================

Using multiple threads with OpenSSL requires implementing a locking
callback. PyDTLS does implement this, and therefore multi-threaded
programming with PyDTLS is safe in any environment. However, being a
pure Python library, these callbacks do carry some overhead. The *ssl*
module implements an equivalent locking callback in its C extension
module. Not requiring interpreter re-entry, this approach can be
expected to perform better. PyDTLS therefore queries OpenSSL as to
whether a locking callback is already in place, and does not overwrite
it if there is. Loading *ssl* can therefore improve performance, even
when only the *sslconnection* interface is used.

Note that loading order does not matter: to obtain the performance
benefit, *ssl* can be loaded before or after the dtls package. This is
because *ssl* does not do an equivalent existing locking callback
check, and will simply overwrite the PyDTLS callback if it has already
been installed. But *ssl* should not be loaded while *dtls* operation
is already in progress, when some locks may be in their acquired
state.

Also note that this performance enhancement is available only on
platforms where PyDTLS loads the same OpenSSL shared object as
*ssl*. On Ubuntu 12.04, for example, this is the case, but on
Microsoft Windows it is not.

Testing
=======

Almost all of the Python standard library's *ssl* unit tests from the
module *test_ssl.py* have been ported to *dtls.test.unit.py*. All tests
have been adjusted to operate with datagram sockets. On Linux, each
test is executed four times, varying the address family among IPv4 and
IPv6 and the demux among *osnet* and *router*. On Windows, where
*osnet* is unavailable, each test is run twice, once with IPv4 and once
with IPv6.

The unit test suite includes tests for each of the above-mentioned
compatible frameworks. The class **AsyncoreEchoServer** serves as an
example of how to use non-blocking datagram sockets and implement the
resulting timeout detection requirements. DTLS in general and OpenSSL
in particular require being called back when used with non-blocking
sockets (or sockets with timeout option) after DTLS timeouts expire to
handle packet loss using re-transmission during a
handshake. Handshaking may occur during any read or write operation,
even after an initial handshake completes successfully, in case
renegotiation is requested by a peer.

Running with the -v switch executes all unit tests in verbose mode.

dtls/test/test_perf.py implements an interactive performance test
suite that compares the raw throughput of TCP, UDP, SSL, and DTLS.
It can be executed locally through the loopback interface, or between
remote clients and servers. In the latter case, test jobs are sent to
remote connected clients whenever a suite run is initiated through the
interactive interface. Run test_perf.py -h for more information.

It should be noted that comparing the performance of protocols that
don't offer congestion control (UDP and DTLS) with those that do (TCP
and SSL) is a difficult undertaking. Raw throughput even across
gigabit network links can be expected to suffer without congestion
control and peers that generate data as fast as possible without
throttling (as this test does): the link's throughput will drop
significantly as it enters congestion collapse. Similarly, loopback is
an imperfect test interface since it rarely drops packets, and never
duplicates or reorders them (thus negating the relative performance
benefits of DTLS over SSL). Nevertheless, some useful insights can be
gained by observing the operation of test_perf.py, including software
stack behavior in the presence of some amount of packet loss.

Logging
=======

The *dtls* package and its sub-packages log various occurrences,
primarily events that can aid debugging. Especially *router* emits many
messages when the logging level is set to at least *logging.DEBUG*.
dtls/test/echo_seq.py activates this logging level during its operation.

Currently Supported Platforms
=============================

At the time of initial release, PyDTLS 0.1.0 has been tested on Ubuntu
12.04.1 LTS 32-bit and 64-bit, as well as Microsoft Windows 7 32-bit
and 64-bit, using CPython 2.7.3. Patches with additional platform
ports are welcome.