Do you need help on a specific subject? Use the contact form (Request a blog entry) on the right hand side.

2017-02-07

Swift and OpenSSL, part 5: Server side

With parts 1 - 4, I would expect people to be able to work their way through. However for the server side there is one more thing to discuss if it is necessary to host multiple domain on a single server on a single port.

But first things first, setting up a server using openSSL is almost simpler than a client.

The only things we need to do is calling SSL_accept after the accept POSIC accept call returns a socket for a new connection.

    let result = tipAccept(onSocket: acceptSocket, timeout: timeout, addressHandler: addressHandler)
    
    switch result {
    case .closed: return .closed
    case let .error(msg): return .error(message: msg)
    case .timeout: return .timeout

    case let .accepted(receiveSocket, clientIp):

(PS: I am using code excerpts from SecureSockets which uses SwifterSockets as a wrapper for the POSIX calls)

Of course we also need to set up an SSL_CTX (i.e. context) for the server to use. Lets do that first:

    guard let context = SSL_CTX_new(TLS_server_method()) else { ... }
    
    SSL_CTX_set_options(context, (UInt(SSL_OP_NO_SSLv2) + UInt(SSL_OP_NO_SSLv3) + UInt(SSL_OP_ALL)))
    
    if SSL_CTX_use_certificate_file(context, encodedFileC.path, encodedFileC.encoding) != 1 {
        return .error(message: ...)
    }
        
    if SSL_CTX_use_PrivateKey_file(context, encodedFileK.path, encodedFileK.encoding) != 1 {
        return .error(message: "...")
    }
    
The above code snippet is raked together from different files, it is just intended to show the calls involved in setting up a context.

If the server only wants to serve clients with a specific certificate, then we also have to setup those certificate paths:
        
    var isDirectory: ObjCBool = false

    if FileManager.default.fileExists(atPath: clientCertPath, isDirectory: &isDirectory) {
                        
        if isDirectory.boolValue {
                
            if SSL_CTX_load_verify_locations(context, nil, clientCertPath) != 1 {
                    
                return .error(message: "...")
            }
                
        } else {
                
            if SSL_CTX_load_verify_locations(context, clientCertPath, nil) != 1 {
                    
                return .error(message: "...")
            }
        }
    }
    SSL_CTX_set_verify(context, SSL_VERIFY_PEER + SSL_VERIFY_FAIL_IF_NO_PEER_CERT, nil)

Now the context is ready for use and we can start accepting SSL connections. Keep in mind that the SSL handshake is initiated by the client.

        SSL_ACCEPT_LOOP: while true {
            
            
            // ===================================
            // Try to establish the SSL connection
            // ===================================
                        
            let result = ssl.accept()
            switch result {
                
                
            // On success, return the new SSL structure
            case .completed:
                
                return .accepted(ssl: ssl, socket: receiveSocket, clientIp: clientIp)
                
                
            // Exit if the connection closed (i.e. there is no secure connection)
            case .zeroReturn: return .closed
                
                
            // Only waiting for a read or write is acceptable, everything else is an error
            case .wantRead:
                
                let selres = waitForSelect(socket: acceptSocket, timeout: timeoutTime, forRead: true, forWrite: false)
                
                switch selres {
                case .timeout: return .timeout
                case .closed: return .closed
                case let .error(message): return .error(message: "...")
                case .ready: break
                }
                
                
            // Only waiting for a read or write is acceptable, everything else is an error
            case .wantWrite:
                
                let selres = waitForSelect(socket: acceptSocket, timeout: timeoutTime, forRead: false, forWrite: true)
                
                switch selres {
                case .timeout: return .timeout
                case .closed: return .closed
                case let .error(message): return .error(message: "...")
                case .ready: break
                }
                
                
            // All of these are error's
            case .wantConnect, .wantAccept, .wantX509Lookup, .wantAsync, .wantAsyncJob, .syscall, .ssl, .bios_errno, .errorMessage, .undocumentedSslError, .undocumentedSslFunctionResult:
                return .error(message: "...")
            }
        }

This accept loop shows something important (that I now realise I did forget to mention in the previous parts... sorry... will update those parts later).

Any SSL_xxx call can result in multiple read/write operations on the POSIX level. Hence for the accept we also need to watch for POSIX read/write events and allow these to occur.

Other than that, accepting is easier than connecting since we don't have to verify the connection manually. The client does that.

To me, this makes sense. A server should be able to work autonomously. But a client can have user interactions where a user might want to accept a certificate that the verification process rejected.

For a server to allow an operator to accept a rejected client certificate is unworkable imo.

So far, everything is fine. The connection is accepted, a session is established and the read/write operations can proceed.

Yes, but... those pesky domains...

In the early days, each server had one domain. But it did not take long to figure out that it is much more cost effective for a server to host multiple domains. And since the domain owners may be unwilling to share a certificate, it must be possible to assign a certificate to each domain.

But the CTX can only handle a single certificate/key combination.

And thus SNI (Server Name Indication) was invented.

SNI takes a little more work. Not only from the server, also from the client. In a normal HTTP request, the client will send the domain name. Unfortunately the HTTP request is something that is done AFTER a secure connection is established. Hence that domain name cannot be used to find out which certificate is needed to start the secure connection.

Thus a client needs to tell the SSL layer (i.e. before the HTTP request is made) which domain it wants to access. Double work...

The client does so by setting up its session. In SecureSockets this is done by this function:

    public func setTlsextHostname(_ name: UnsafePointer<Int8>) {
        SSL_ctrl(optr, SSL_CTRL_SET_TLSEXT_HOSTNAME, Int(TLSEXT_NAMETYPE_host_name), UnsafeMutableRawPointer(mutating: name))
    }

On the server side we need to do some more work.

The SSL handshake that occurs after an SSL_accept call, but before the SSL_accept is finished must have a way to allow the server application to select a certificate.

In OpenSSL 1.1.0 this is done by a callback function. When setting up the context for an accept we need to set the callback destination. Unfortunately this makes a little C-glue code necessary. In C they have implemented some trickery whereby a void(*)() ptr needs to be specified as the callback, but the callback signature itself is Int32(*)(OpaquePointer, Int32*, Void*).

I have not found a way to map those two onto each other in Swift. Hence a little C code:

void sslCtxSetTlsExtServernameCallback(SSL_CTX *ctx, int (*cb)(const SSL *ssl, int *num, void *arg), void *arg) {
    SSL_CTX_set_tlsext_servername_arg(ctx, arg);
    SSL_CTX_set_tlsext_servername_callback(ctx, cb);
}
As can be seen in the installation instructions of SecureSockets, I put this code in some openSSL files. That way I did not need to create any additional libraries, but simply compiled openSSL with this extension in it. I choose the ssl.h and ssl_lib.c files for this. But I think it is rather irrelevant in which files it is placed.

In the callback itself, we need to maintain a list of domain contexts. Each domain has its own context, and that context has the certificate & private key set for that domain. When the callback is activated, it hands us the session as a pointer. From that session we retrieved the asked-for domain name. And then we look for the proper domain context with the right certificate. Once we find the proper domain context, we pull a switcheroo and switch the context of the session to the context of the domain. Bam! done.

In SecureSockets code the installation of the callback and the callback itself are found in the class Ctx:

Installing the callback (calls out to the C-glue code):

    sslCtxSetTlsExtServernameCallback(optr, sni_callback, UnsafeMutableRawPointer(Unmanaged.passUnretained(self).toOpaque()))

And the callback itself

    private let sni_callback: @convention(c) (_ ssl: OpaquePointer?, _ num: UnsafeMutablePointer<Int32>?, _ arg: UnsafeMutableRawPointer?) -> Int32 = {
        
        (ssl_ptr, _, arg) -> Int32 in
        
        
        // Get the reference to 'self'
        
        let ourself = Unmanaged<Ctx>.fromOpaque(arg!).takeUnretainedValue()
        
        
        // Get the String with the host name from the SSL session
        
        guard let hostname = SSL_get_servername(ssl_ptr, TLSEXT_NAMETYPE_host_name) else { return SSL_TLSEXT_ERR_NOACK }
        
        
        // Check if the current certificate contains the hostname
        
        if let ctx_ptr = SSL_get_SSL_CTX(ssl_ptr) {
            
            if let x509_ptr = SSL_CTX_get0_certificate(ctx_ptr) {
                
                if X509_check_host(x509_ptr, hostname, 0, 0, nil) == 1 {
                    
                    return SSL_TLSEXT_ERR_OK
                }
            }
        }
        
        
        // Check if there is another CXT with a certificate containing the hostname
        
        var foundCtx: Ctx?
        for testCtx in ourself.domainCtxs {
            if testCtx.x509?.checkHost(hostname) ?? false {
                foundCtx = testCtx
                break
            }
        }
        guard let newCtx = foundCtx else  { return SSL_TLSEXT_ERR_NOACK }
        
        
        // Set the new CTX to the current SSL session
        
        if SSL_set_SSL_CTX(ssl_ptr, newCtx.optr) == nil {
            // The new ctx did not have a certificate (found by source code inspection of ssl_lib.c)
            // This should be impossible since that would have caused this CTX to be rejected
            return SSL_TLSEXT_ERR_NOACK
        }
        
        
        return SSL_TLSEXT_ERR_OK
    }
}

Well, that about wraps things up.

Only one thing left to do ... for you that is... ;-) see below...

Happy coding...

Did this help?, then please help out a small independent.
If you decide that you want to make a small donation, you can do so by clicking this
link: a cup of coffee ($2) or use the popup on the right hand side for different amounts.
Payments will be processed by PayPal, receiver will be sales at balancingrock dot nl
Bitcoins will be gladly accepted at: 1GacSREBxPy1yskLMc9de2nofNv2SNdwqH

We don't get the world we wish for... we get the world we pay for.

2017-02-02

Swift and OpenSSL, part 4: Client details

In the previous post we looked at the big picture of how a SSL connection is created from a client to a SSL-Server.

In this post I want to take a look at some of the details involved.

The first detail is an answer to the question: What is a "ssl-session"?

SSL Session

I am sure that the people that work on openSSL can give a very good answer. But for us API users we can take comfort in the fact that we don't need to know.

That is, an SSL session represents a connection between a SSL-client and a SSL-server. But other than that, we don't need to know how it is represented.

The openSSL team has made this very clear: All sessions are referred to by an OpaquePointer.

We create a session by calling SSL_new and get back an OpaquePointer. All use of the internal structure that hides behind the pointer can be done though API calls.

Usually those API calls take multiple parameters of which the first is the OpaquePointer of the session we want to use.

For example the openSSL read. It is defined as:

int SSL_read(SSL *ssl, void *buf, int num);
Which appears in Swift as:

SSL_read(ssl: OpaquePointer!, buf: UnsafeMutableRawPointer!, num: Int32) -> Int32

The first parameter is the OpaquePointer to the session. The second parameter a pointer to the buffer in which the data should be written and the third parameter is the size of the buffer. Just like the POSIX read call it returns an integer that contains either the number of bytes read, or an error indicator.

The OpaquePointer approach is typical for openSSL 1.1.0. It is used everywhere to avoid exposing the internals of the data structures.

As far as I see, all OpaquePointer's are force unwrapped. This means that they can be nil. In fact, using nil is rather common in C and (to my knowledge) none of the openSSL operations will crash if we pass nil.

Again I glossed over something: how do we create a session?

Well, we use SSL_new ... right, but what parameters do we need?

That brings us to point 2,

SSL Context

To create a session (client side) we provide a context.

SSL *SSL_new(SSL_CTX *ctx);
Which appears in Swift as:

SSL_new(ctx: OpaquePointer!) -> OpaquePointer!

Again we see the usage of OpaquePointer to hide the internal details. Which is good, because now we now we don't need to know :-)

Still we do need to know what a context is used for.

A context is used to store data that a session needs to operate. All kinds of data, most noticeably the certificates! (or at least the path to the certificates).

So before we create the session, we need to set up the context in which that session will operate.

It is worth mentioning that the context used to create a session is copied by that session. Thus if we need to change things afterwards, we need to do that on the session, and not on the context.

This allows us to store a context and keep it handy whenever we need to create a new session.

I strongly suggest you lookup what data can be set in the context, or alternatively know that you can set all parameters necessary for a session in the original context - so you know where to look.

Oh, about those OpaquePointer's...

OpaquePointer's

They hide their structure, but those structures take up memory. And memory must be allocated and deallocated.

You'll see where this is going... yup, reference counters. OpenSSL has them.

When a new session (or context or ... whatever) is created the ref-count is set to 1. When we are done with it, we need to decrease the ref-count. This is done by "free" calls. Each internal openSSL structure has an associated "free" call to decrement the ref-count. Once the ref-count reaches zero, the memory is deallocated.

The SSL_new has a corresponding SSL_free, the SSL_CTX_new has a SSL_CTX_free etc.

When data is retrieved (copied) from another data structure, sometimes openSSL will allocate a new structure for us. And sometimes it won't. It is always a good idea to check the documentation. But there are a couple of calls where this has been made explicit in the name of the call.

These can be recognized by their xxx_get0_xxx and xxx_get1_xxx signature.

The 'get0' returns a structure that should NOT be freed. While the 'get1' returns a structure that MUST be freed.

Now the last of the details that I glossed over, but do want to mention

Certificate Verification

To verify if a client connection to a server is secure, it first should be checked that the server presented a certificate. If that did not happen, there can be no secure connection. OpenSSL does not implicitly check this, we need to do that ourselves.

And then of course the results of the verification must be checked. If that fails, that does not necessary mean that the connection should not be established.

As an example: If we browse to a website that uses self-signed certificated, the verification will fail, but we can still decide to accept that certificate.

In the same way, when a certificate verification fails, it is often a good idea to ask the user what he wants. Does he want to continue or not?

Next in the series: a quick look at the server side.

Happy coding...

Did this help?, then please help out a small independent.
If you decide that you want to make a small donation, you can do so by clicking this
link: a cup of coffee ($2) or use the popup on the right hand side for different amounts.
Payments will be processed by PayPal, receiver will be sales at balancingrock dot nl
Bitcoins will be gladly accepted at: 1GacSREBxPy1yskLMc9de2nofNv2SNdwqH

We don't get the world we wish for... we get the world we pay for.

2017-02-01

Swift and OpenSSL, part 3: Connecting to a SSL Server

In this part I will talk about the high-level view of creating a secure connection to a server. I will use the methods and wrappers from SecureSockets as these are easier to read and follow. (SecureSockets can be downloaded from github)

If you are new to socket programming it is probably best to first read up on socket programming. I always refer to beej's guide on network programming as a very fine resource for this.

It turns out that the way we need to use OpenSSL is very similar to normal socket programming.

For the POSIX "connect" call, there is a parallel "SSL_connect".

For the POSIX "accept" call there is a parallel "SSL_accept". For "read" there is "SSL_read" and for "write" there is "SSL_write".

The main difference is that instead of working on sockets, the openSSL calls work on "sessions".

The way this is implemented is that we first have to set up the socket and then we call the session operations.

An example, suppose we want to connect to a server:

First we set up the socket:

    var socket: Int32
    switch connectToTipServer(atAddress: address, atPort: port) {
    case let .error(message): return .error(message: message)
    case let .success(s): socket = s
    }

I'll hope you pardon me the usage of my own SwifterSockets framework (which can be downloaded for free from github)

The connectToTipServer returns a socket on success. From this moment on, the socket is connected to the server but no data has been transferred. The server has accepted the connection and is now waiting for the client to start transmitting something. In this case -of course- the server wants to start a SSL-handshake. But it waits for the client to initiate this.

At the client side we continue the sequence by creating a session:

    guard let ssl = Ssl(context: ctx) else {
        return .error(message: "Failed to create Ssl,\n\n\(errPrintErrors())")
    }

Then we assign the socket to the session:


    switch ssl.setFd(socket) {
    case let .error(message): return .error(message: "Failed to set socket to ssl,\n\(message)")
    case .success: break
    }

And then we tell the client to connect securely to the server:


    switch ssl.connect(socket: socket, timeout: timeoutTime) {
    case .timeout: return .timeout
    case let .error(message): return .error(message: "Failed to connect via SSL,\n\(message)")
    case .closed: return .error(message: "Connection unexpectedly closed")
    case .ready: break
    }

Before we start transferring data back and forth, we need to check if the connection was established securely.

This is a two step approach: first we need to know if the server did send a certificate, and secondly we need to know if the certificate is valid.


    guard let x509 = ssl.getPeerCertificate() else {
        return .error(message: "Verification failed, no certificate received")
    }
        
    switch ssl.getVerifyResult() {
    case let .error(message): return .error(message: "Verification failed,\n\(message)")  
    case .success: break
    }

Once this sequence completes, the "ssl" session is ready to transfer data back and forth.

Conceptually that is all there is to a secure client / server connection. But I did gloss over some details as you noticed.

Still the big picture on the client side is complete.

Just in case you want to know how the data transfers look like:

Writing: let res = SSL_write(optr, buf, num)

Reading: let res = SSL_read(optr, buf, num)

These are almost "drop in" replacements to the POSIX read and write. Except that they operate on the session instead of the socket.

Once the session can be closed, use SSL_shutdown(optr). It is possible to simply terminate the connection, but let's play nice ;-)

The next post will address some of the details that I glossed over today.

2017-02-07: One more thing...

Managing and maintaining a SSL connection means that the SSL layer will start read & write actions on its own. When using the POSIX select call to wait for events to happen on a connect, write, read etc, be sure to monitor for all events. For example monitoring only read events for a SSL_read will most likely fail at some point because the SSL layer wants to write.

Happy coding...

Did this help?, then please help out a small independent.
If you decide that you want to make a small donation, you can do so by clicking this
link: a cup of coffee ($2) or use the popup on the right hand side for different amounts.
Payments will be processed by PayPal, receiver will be sales at balancingrock dot nl
Bitcoins will be gladly accepted at: 1GacSREBxPy1yskLMc9de2nofNv2SNdwqH

We don't get the world we wish for... we get the world we pay for.