This patch allows the module test_perf.py to be run on Windows. It
contains fixes primarily accounting for differences in the
implementation of the multiprocessing module among platforms.
This patch completes the README.txt placeholder file. The first version of
external (as opposed to in-line) documentation is complete.
The unused file README.md that was created by the GitHub repository
establishment procedure is deleted.
This project is now licensed under the Apache license. Individual files
now have a license reference header.
The Apache 2.0 license text is copied to the file LICENSE. The file
NOTICE, referred to in the license text, has been added. A placeholder
README.txt has been added.
These three new files are integrated into the distribution/installation
machinery, and are placed into the package directory upon installation.
The new module setup.py handles creation of package distributions in
compressed formats, and performs installation of said distributions
after decompression. Distribution strategy for PyDTLS is in the form
of source distributions only.
Building the OpenSSL library on Windows is non-trivial. Requirements
include linking against the same version of the C runtime that is used
by the Python interpreter, or else Microsoft redistributable
installation may also need to be performed. For the convenience of
Windows users, the distribution procedure includes creation of
prebuilt versions of the OpenSSL library for both 32- and 64-bit
versions of Windows. Despite containing binaries, these are still
source distributions that are installed with "python setup.py
install."
A version of OpenSSL compiled with the MinGW toolchain is included.
cygcrypto-1.0.0.dll and cygssl-1.0.0.dll contain symbols and can be
debugged with gdb. All MinGW OpenSSL dll's link with msvcr90.dll.
In order to avoid making installation a requirement after cloning, the
package initializer looks for prebuilts of the currently executing
platform and copies them into the dtls package directory before
continuing module initialization.
A distribution containing prebuilts for all platforms as well as those
containing no prebuilts (appropriate for Linux) can be generated by
issuing "python setup.py sdist --prebuilts."
When running in remote server mode (-s), the value of the -s option
specifies which port remote clients must connect to for the job
transmission protocol. The servers for individual suite runs, however,
would be placed at dynamically assigned ports, with port numbers
transmitted to clients through job parameters.
This does not work well when servers sit behind firewalls with restricted
UDP port ranges. This patch introduces the -p option, allowing ports for
server runs to remain fixed at the given port (which, of course, must
not conflict with the port number supplied with -s).
The new module test_perf.py can be used to characterize protocol
performance over a particular network link. Two stream protocols
(TCP and SSL) and two datagram protocols (UDP and DTLS) are available
for relative comparison.
The module will run servers in its process, and will spawn clients either
into separate processes, or, depending on command line options, will
expect one or more remote clients to connect to it. In the latter case,
jobs will be sent to such clients via a shared queue whenever the user
selects a test suite.
Stress testing under packet loss conditions revealed that that the
OpenSSL library's compression feature needed to be explicitly disabled
for DTLS: it evidently operates at the stream layer as opposed to the
datagram layer, and packet loss would result in corruption among the
packets that were successfully received, authenticated, and decrypted.
Several performance improvements are included in this patch.
Thread locking callbacks into the interpreter impose considerable overhead.
The standard library's ssl module registers its own thread locking and id
callback functions. Registration of the latter can be detected by a call to
the OpenSSL library. In this case, do not set the Python callback function
and therefore keep using the more efficient ssl module functions.
On a 64-bit OS, pointer return values needed to be marked as c_void_p instead
of a user-defined type, which would result in the transfer of 32 bits only.
In order to still return an instance of the user-defined type to the caller,
imported functions are now marked with the return type, and the return
value is converted to that type by a new error checking function used only
with imported functions that create and return user-defined types.
On 64-bit Linux, the long type becomes 8 bytes, whereas the int type remains
4 bytes. The various sockaddr_* fields therefore needed to be changed from
long to int, as did the type signatures of the packed string to array
conversion functions.
On an Ubuntu server installation, it was found that the name "localhost"
does not resolve to an ipv6 address. A name search has therefore been added
to the unit test driver, along with an ip number fallback.
Tested on Ubuntu Server 12.04.1 LTS 64-bit.
Regression tested on Ubuntu 12.04.1 LTS 32-bit.
This change introduces a demux that uses the kernel's network stack for UDP
datagram-to-socket assignment based on packet source address (as opposed to the
forwarding strategy of the routing demux). The osnet demux is used by default
on non-Windows platforms. When possible, use of the osnet demux is preferred
over the routing demux, since it can be expected to perform better.
The unit test suite has been extended to run all tests first with the demux
selected by default for the current platform, and then with the routing demux,
if the latter differs from the former. Tests were already being run twice, first
with IPv4 and then with IPv6, and thus we now run each test four times on
Linux, twice on Windows.
All unit tests pass with both demux types.
With this change all unit tests pass on the Linux platform (tested on
Ubuntu 12.04.1 LTS). demux/__init__.py has been adjusted temporarily so as
to load the routing demux on Linux until the osnet demux is ready.
Testing on Linux exposed an issue where comparison of the ssl object value from
cookie callbacks failed to compare equal to the value stored in the
SSLConnection callback object. This was because the callback function signature
of c_void_p for this parameter produced a 64-bit value if the 32nd bit was set
(as opposed to producing a negative integer 32-bit value). Changing the
signature from c_void_p to c_int for this parameter fixes the issue.
Prior to this change, the unit tests' echo servers' connections would sometimes
linger beyond server termination. A timeout mechanism is now implemented, which
will terminate a connection and clean up its resources when the timeout is
reached. For the purpose of unit testing, test echo servers now assert that
all connections have been terminated when the servers are closed.
For threaded echo servers, the timeout mechanism involves the use of sockets
with timeouts instead of blocking sockets. This required an implementation with
the proper handling of timeout sockets at the sslconnection level.
The new module tlock provides the locking callback function the OpenSSL library
requires for the case where multiple threads enter it concurrently. tlock is
now automatically initialized by sslconnection's module startup code, and does
nothing if the executing Python environment does not provide threading.
Note that this does introduce some overhead. For example, during the transfer
of CERTFILE by test_socketserver, about 300 locking callbacks are received.
The unit test suite was previously being run with IPv4 protocol addresses only.
With this change, we run the entire test suite twice: first with v4 addresses,
and then with v6 addresses, for all client and server-side sockets.
A patch implementation is provided, which augments and alters the Python
standard library's ssl module to support passing of datagram sockets, in which
case this package's DTLS protocol support will be activated. The ssl module's
interface is intended to operate identically regardless of whether the DTLS
protocol or another protocol is chosen.
The following features of the ssl module are explicitly supported with
datagram sockets:
* socket wrapping, unwrapping, and re-wrapping
* threaded UDP servers
* asynchronous UDP servers (asyncore integration)
* socket servers (SocketServer integration)
The following modules have been added:
* dtls.patch: standard library module patching code and substitution
functions and methods
* unit.py: this is a port of the standard library's testing module
test_ssl.py for datagram sockets; all tests pass at this time;
a couple of inapplicable tests have been dropped; a few other
tests have been added
Also note that the err module's exception raising mechanism has been
augmented so as to raise exceptions of type ssl.SSLError (as opposed to
dtls.err.SSLError) when instructed to do so through activation of the patching
mechanism. This allows code written against the standard library module's
interface to remain unchanged. In some cases, types derived from
ssl.SSLError are raised.
This change introduces the implementation of the SSLConnection methods
getpeercert and cipher. The following has been added:
* dtls.util: utility elements shared by other modules in this package
* dtls.x509: a module for X509-certificate-related functionality,
including formatting a certificate into a Python
dictionary as prescribed by the Python standard
library's ssl module; functionality for testing with
PEM-encoded certificates in the file system is included
* yahoo-cert.pem: the current certificate of www.yahoo.com: this is a good
testing certificate, since it contains the subject
alternate name extension
Other notable changes:
* sslconnection: private attributes are now preceded by "_"
* openssl: null-ness in opaque FuncParam-derived return values is
now properly detected and an exception is raised as
expected
This initial commit for the PyDTLS package includes the following functionality:
* DTLS cookie exchange, using secure hmac cookies
* A platform-independent routing UDP demultiplexer
* SSL handshaking over UDP using the DTLS protocol
* Datagram exchange using the DTLS protocol
* SSL shutdown over UDP
The package is structured as follows:
* dtls: top-level package
* dtls.demux: demultiplexer package; automatically loads a
demultiplexer appropriate for the currently executing
platform
* dtls.demux.router: a routing demux for platforms whose network stacks
cannot assign incoming UDP packets to sockets based
on the sockets' connection information
* dtls.demux.osnet: a demux that uses the operating system's UDP packet
routing functionality
* dtls.err: package-wide error handling and error definitions
* dtls.sslconnection: a client and server-side connection class for
UDP network connections secured with the DTLS protocol
* dtls.openssl: a ctypes-based wrapper for the OpenSSL library
* dtls.test: test scripts, utilities, and unit tests
The following binaries are provided:
* libeay32.dll: cryptographic portion of the OpenSSL library
* ssleay32.dll: protocol portion of the OpenSSL library (depends on former)
* cygcrypto-1.0.0.dll: as libeay32.dll, but with debugging symbols
* cygssl-1.0.0.dll: as ssleay32.dll, but with debugging symbols
All binaries have been built with the MinGW tool chain, targeted for msvcr90.
The unstripped dll's can be debugged on Windows with gdb. Cygwin is not used.
Jayson Vantuyl
Ray Brown
rbit