Do you need help on a specific subject? Use the contact form (Request a blog entry) on the right hand side.

2017-01-31

Swift and OpenSSL Part 2: Facing the libraries

OpenSSL is written in plain old C. Which means that we can use it from Swift... right?

Yes, but...

Macro's

While we can reference the header files and link the libraries, openSSL uses macro's in their headers to make some functions more palatable. Unfortunately Swift does not understand all macro's.

If you are not familiar with macro's, macro's are definitions that "replace text with text". But have some extra functionality like parameter passing built in.

This is an example of a macro:

#define SYSerr(f,r)  ERR_PUT_error(ERR_LIB_SYS,(f),(r),OPENSSL_FILE,OPENSSL_LINE)

In C-code when we use the function: SYSerr(f,r)

in reality the function that is called is: ERR_PUT_error(ERR_LIB_SYS,(f),(r),OPENSSL_FILE,OPENSSL_LINE)

During compilation the C pre-processor will replace the first function definition with the second. Before even the real compiler is started.

While Swift does have a pre-processor for C-headers, it is used to translate C-calls to their Swift equivalents. The pre-processor does not have the same capability of the C-preprocessor.

To get back to the example, in Swift the function SYSerr(f,r) does not exist because the pre-processor does not translate it.

But the ERR_PUT_error(ERR_LIB_SYS,(f),(r),OPENSSL_FILE,OPENSSL_LINE) does exist. The pre-processor can handle it just fine.

This means that if we find some code that tells us to use a function (like SYSerr) we can solve this by finding where the macro is defined and then to use the function the macro refers to. So don't despair when auto-complete does not know of a function you just found on the internet, simply do a search on the header files and see how the macro is defined.

OPENSSL_free()

OPENSSL_free(ptr) is one of those macro's. As an example you could try to find out what the proper replacement is. Its a bit tricky as a compile option must be taken into account.

I will give the answer here, but you should try it yourself.

The answer is to replace OPENSSL_free(ptr) with CRYPTO_free(ptr, "", 0)

Pointers

Per documentation, String's are automatically bridged to UnsafePointer<Int8> and UnsafePointer<UInt8>  That is very convenient as in many places we can simply pass a string where the API expects a "char *" or "const char*"

It is good to know that (even though the documentation does not say so) the pointer that is created by the bridge is in fact pointing to a C-string representation of the String, i.e. it is null-terminated.

Some openSSL APIs need a string in an UnsafeMutableRawPointer. I find it convenient to wrap such APIs in a wrapper that receive the string as an input parameter:

func someSslFunctionWrapper(_ str: UnsafePointer<Int8>) {
    let ptr = UnsafeMutableRawPointer(mutating: str)
    // work with ptr, but only inside this function!
}

someSslFunctionWrapper("string-input")

While 'ptr' does point to a null-terminated string, keep in mind that it does point to a copy of the string that was passed into the function. If the API tries to change the 'str' then this approach would not work. I have not encountered such situations yet.

Callbacks

Some openSSL operations require a callback operation. Unfortunately some of the callbacks are passed as

void (*fp) (void)

and are cast in a define statement to the proper signature just before they are passed to in the SSL library. I found this to be a problem as I was not able to find/write Swift trickery to create these.

The way I solved this is by introducing a c-wrapper that takes a Swift function of the proper signature and bridges this to the required signature.

Example:

SSL_CTX_set_tlsext_servername_callback is not available directly, looking for this in the include files gives us:

#define SSL_CTX_set_tlsext_servername_callback(ctx, cb) \
SSL_CTX_callback_ctrl(ctx,SSL_CTRL_SET_TLSEXT_SERVERNAME_CB,(void (*)(void))cb)

Its a macro, thus we need to use SSL_CTX_callback_ctrl

However this function takes a generic function pointer with the signature "() -> Void" (as can also be seen in Xcode auto-complete). Obviously (from the example C-code) that is not what is needed. We ned the proper signature of the callback before we can write the callback.

Tracing the implementation of SSL_CTX_callback_ctrl gives us:

    case SSL_CTRL_SET_TLSEXT_SERVERNAME_CB:
        ctx->tlsext_servername_callback = (int (*)(SSL *, int *, void *))fp;
        break;

Hence the signature of the callback is really:

int (*)(SSL *, int *, void *)

But we cannot use the function with this signature to call SSL_CTX_callback_ctrl. That seems to leave us in a bind. The solution is to create our own C-wrapper function.

In this case our wrapper takes this signature as an input:

void sslCtxSetTlsExtServernameCallback(SSL_CTX *ctx, int (*cb)(SSL *, int *, void *)) {
    SSL_CTX_set_tlsext_servername_callback(ctx, cb);
}

Note that since the wrapper is C code it does makes sense to use the macro again.

It can take some searching in the openSSL sources before the appropriate signature is found. But it is not all that difficult.

Documentation warnings

Depending on how "deep" we need to go with the openSSL libraries the compiler can start to generate 'Documentation warnings'. These are due to "errors" in the comments in the openSSL headers. I found such cases in 'bn.h' and 'ec.h'. If you want to get rid of then, simply accept the xcode suggestions or remove the offending lines entirely. Its only documentation after all...

OpenSSL Documentation.

Sadly the openSSL documentation at the website is not what an API user would like it to be. But do not give up too easily. Sometimes the documentation is there, just not properly linked. When you run into a link error, try again. Sometimes a link from a different page will work.

I have found that a link produced by a search did not work, but a link from an earlier version did. This is a bit difficult to describe; but as you search on the openSSL site you will soon discover that you want to click those links that contain the version number of openSSL that we use. e.g. links that contain "1.1.0". However some of those links are wrong. In that case it can help to click a search result with version "1.0.2" and when that page is found, then click the "1.1.0" version link on that page. Often that will do the trick.

Happy coding...

Did this help?, then please help out a small independent.
If you decide that you want to make a small donation, you can do so by clicking this
link: a cup of coffee ($2) or use the popup on the right hand side for different amounts.
Payments will be processed by PayPal, receiver will be sales at balancingrock dot nl
Bitcoins will be gladly accepted at: 1GacSREBxPy1yskLMc9de2nofNv2SNdwqH

We don't get the world we wish for... we get the world we pay for.

2017-01-30

Swift and OpenSSL Part 1: Setting up from scratch

Secure Sockets: You know you should, you even want to, but a first look at the openssl.org...  Oh sh!t... that looks like a ton of work..., let me do a google and see where I can simply download something .... Ughhh, where are the easy to read articles... where... never mind... some other day perhaps...

Well, that day has arrived :-)

This is the first part in a short series of posts on how to use OpenSSL from Swift. In this first part we have a look at how to install OpenSSL on our Mac so we can to link it with our software.

In case you are wondering: Is there not an easy way to do this, like downloading a library or so?

Yes, that is possible, but also very strongly discouraged! The reason to use a security product like OpenSSL is that we want our customers to be safe. That safety starts when we download OpenSSL from the source, verify the checksums, and compile it on our own machines. If we are not prepared to do that, then we can just as well forget about security because it would be all cosmetics. So forget MacPorts or Homebrew, do this yourself. You owe it to your customers.

The download link for openssl is: https://www.openssl.org/source/

This series will use the openSSL version 1.1.0 series. Simply right-click the openssl-1.1.0c.tar.gz file and select "save-as" to download it to your downloads folder. (The c release was the most current version at the time of writing)

Use the save-as option because we want the openssl-1.1.0c.tar.gz file.

Also download the sha256 checksum.

After the download finishes, open up a terminal window and cd to the download folder. Calculate the sha256 checksum of the gz file with:

> shasum -a 256 openssl-1.1.0c.tar.gz

The next line should display the checksum. Compare that with the downloaded checksum, they should be equal of course. (Open a text editor and put the two checksums below each other, that way it is easy to verify)

Now unpack the gz and tar file to obtain the openssl-1.1.0c folder. A singe double click should do the trick.

Btw: You may not know it, but there are significant changes between version 1.0.2 and 1.1.0. This series is written for 1.1.0 and will not work with 1.0.2.

You can copy the folder to any other place, but I just leave it in downloads for the time being.

Next we should build the libraries and include files.

The OpenSSL 1.1.0 installer needs PERL 5.10 or later. My system has 5.18, so yours will probably be up to date as wel (I do not remember upgrading to a new version...)

Just type:

> perl -v

in a terminal window.

The installation instructions on https://wiki.openssl.org/index.php/Compilation_and_Installation are a little confusing, but we are in luck because the process is very simple. In the INSTALL file in the openssl-1.1.0c directory we find the proper installation instructions for Unix.

By default openssl will be installed in /usr/local. You could check if there is already something there, from an earlier installation. If there is, you will need to decide to override or choose a different location. For changes to the default location, see the INSTALL document.

Note that the wrapper I will be talking about in coming posts will assume that openSSL is installed in the default location.

First we run config:

Note: Do this while the terminal prompt is in the openssl-1.1.0 directory!

> ./config

Messages start flashing over the screen, but it is over rather quick.
I did not encounter any visible issues.

Next is:

> make

This take a little longer. Seemingly a lot of stuff is being compiled...

When it stops (and again no visible problems) it is time to test:

> make test

A lot of tests are executed, while some may be skipped. As long as the result shows:


All tests successful.
Files=89, Tests=477, 44 wallclock secs ( 0.37 usr  0.16 sys + 30.58 cusr  7.34 csys = 38.45 CPU)
Result: PASS

(Note that the number of files and tests may differ)

you are in for the next step:

> sudo make install

Again a lot of text scrolls over the screen, but at the end of it, its ready. (Note that this step took by far the most time)

Note that if you forget 'sudo' you will get a lot of errors. Also note that the INSTALL document does not mention sudo.

For this series, we will be talking about coding. Not tool usage. Hence we do not need to adjust PATH variables or anything.


Happy coding...

Did this help?, then please help out a small independent.
If you decide that you want to make a small donation, you can do so by clicking this
link: a cup of coffee ($2) or use the popup on the right hand side for different amounts.
Payments will be processed by PayPal, receiver will be sales at balancingrock dot nl
Bitcoins will be gladly accepted at: 1GacSREBxPy1yskLMc9de2nofNv2SNdwqH

We don't get the world we wish for... we get the world we pay for.

2017-01-29

Migrating a Xcode project to a Package | Swift Package Manager

SwifterSockets lives on github. You can find it here. Before version 0.9.8 it was a "plain" Xcode project. But since the Swift Package Manager is now out (included in Swift 3), I wanted to change that to an SPM package.

Important: The following is applicable only to frameworks (or frameworks to be). Not applications.

If you want to convert your project, be sure to have a backup - we will destroy the current Xcode project! (If your project is on github too, that should not be any problem...)

The first thing to realize is that SPM needs a git repository. I already had a repository but the content should change rather dramatically. This is how I did that:

Using the terminal, I create a directory with the same name as the project:

    $ mkdir SwifterSockets

In that directory I created a new SPM pakage:

    $ cd SwifterSockets
    $ swift package init

In that directory we now find the following files:

    .gitignore
    Package.swift
    Sources
    Tests

This is the content of the current location of the Xcode project:

    .git
    .gitignore
    SwifterSockets
    SwifterSockets.xcodeproj
    build

Next I merged the content of these two directories. Note that there was already a .gitignore present. That was overwritten with the new version.

    .git
    .gitignore
    Package.swift
    Sources
    SwifterSockets
    SwifterSockets.xcodeproj
    Tests
    build

Next I copied all the source code files that made up SwifterSockets to the Sources directory.

Then came the slightly nerving step: delete the old Xcode project and associated stuff:

    $ rm -rf SwifterSockets
    $ rm -rf SwifterSockets.xcodeproj
    $ rm -rf build

Poof... all gone... this is what remains:

    .git
    .gitignore
    Package.swift
    Sources
    Tests


(Remember, all the source code I need is still there, in the Sources directory)

At this point we can try a build:

    $ swift build

If the sources are complete, this should produce no errors. But your milage may vary if there were dependencies on other packages/libraries etc.

Since editing in vi is not everybody's cup of tea, lets create a new Xcode project:

    $ swift package generate-xcodeproj

The directory content is now:

    .git
    .gitignore
    Package.swift
    Sources
    SwifterSockets.xcodeproj
    Tests

When we open this xcode project, the sources are all there and we can edit/compile it all. Remember though that compiling in Xcode is not the same as compiling at the command line. That is a separate compilation and is an extra step. However for frameworks we don't often have to go there.

Four things left:
1) Make sure that in the new xcode project the Build Setting - Packaging - Defines Module is set to 'Yes'.
2) Update the .gitignore settings. Mine looks as follows:

    .DS_Store
    /.build
    /Packages
    xcuserdata
    build

3) Put everything in the repository:

    $ git add .
    $ git commit -m "Migration to SPM"
    $ git tag 10.0.0
    $ git push

The last bit is of course optional. And the version number is up to you.

The fourth step is using the fresh release.

That can be done as a package in other packages that are build with the SPM. Or it can be done as a Modular Framework.

Using it as a SPM Package should be clear, otherwise you may want to look up the SPM user manual.

Using it as a modular framework is done by creating the framework in the Xcode project and then importing it into another project as either Embedded Binaries (dynamic framework) or Linked Frameworks and Libaries (static framework).

If you create a framework, make sure the "Defines Module" setting in the "Build Settings" (subsection "Packaging") is set to 'yes' as this greatly simplifies the import in another project.

Happy coding...

Did this help?, then please help out a small independent.
If you decide that you want to make a small donation, you can do so by clicking this
link: a cup of coffee ($2) or use the popup on the right hand side for different amounts.
Payments will be processed by PayPal, receiver will be sales at balancingrock dot nl
Bitcoins will be gladly accepted at: 1GacSREBxPy1yskLMc9de2nofNv2SNdwqH

We don't get the world we wish for... we get the world we pay for.

2017-01-28

Swift frameworks and modules

I have been busy creating a SSL library that sits on top of SwifterSockets.

And that uses OpenSSL to do the secure work.

Since the Swift Package Manager (SPM) was included in Swift 3, why not use it and make some nice and clean packages of SwifterSockets and the SSL variant: SecureSockets?

Should not be too difficult ... right?

Now difficult and difficult are two different things ... No it is not difficult to make packages. But the marriage of SPM and Xcode is not too great. Kind of sucks really...

But the good news is it can be done.

The SPM lives at the command line. Outside of Xcode projects. While it is possible to use the same code in both the SPM packages and Xcode projects, there are some caveats.

Before we dive into this, it is first important to know about frameworks and modules.

Modules are an invention to skip on configuration.

When we use frameworks without modules between projects, it will be necessary to configure bridging headers and search paths. Modules can be used to skip all that.

To create a Modular Framework all we need to do is to generate a module.modulemap inside a framework. And that is not too difficult, since Xcode will do it for us if we set the "Build Setting" "Defines Module" (in subsection "Packaging") to 'Yes'. Of course the target should be a framework, it makes no sense for app's (or so I would think...).

With Modular Frameworks we no longer have to create bridging headers or configure search paths. Simply add them as "Embedded Binaries" (for dynamic libraries).

In the next blog I will show you how I changed SwifterSockets from an Xcode project to a SMP Package.

Happy coding...

Did this help?, then please help out a small independent.
If you decide that you want to make a small donation, you can do so by clicking this
link: a cup of coffee ($2) or use the popup on the right hand side for different amounts.
Payments will be processed by PayPal, receiver will be sales at balancingrock dot nl
Bitcoins will be gladly accepted at: 1GacSREBxPy1yskLMc9de2nofNv2SNdwqH

We don't get the world we wish for... we get the world we pay for.

2017-01-07

Cancelling a defer statement | Swift Pattern

I love the defer statement. It prevents so many error that could occur because one of the exits in a module may forget to close a file or release a resource etc.

I also do a fair amount of low-level programming and thus I get to write a lot of code that exits all over the place in case of an error, and has only one exit if everything went fine.

In order to use the defer statement, it need to release the resources on all the error exits, but not on the success exit.

So I would like to cancel a defer statement ... in a sense.

Of course that is not possible... is it?

No.

But there is a simple pattern that solves this problem neatly. It is so simple that it does not even need explaining:

func openFile(kind: String) -> UnsafeMutablePointer<FILE>? {
    
    var file = fopen("MyFile.txt", "r")
    
    var closeFile = true
    defer { if closeFile { fclose(file) } }
    
    if fileIsNotValid(file) { return nil }

    if fileDoesNotContainData(file, kind) { return nil }
    
    if fileDataOutOfDate(file) { return nil }
    
    // Prevent the deferred handler from closing the file
    closeFile = false

    return file
}

From Jay on the swift-evolution forum I received the following alternative:

func openFile(kind: String) -> UnsafeMutablePointer<FILE>? {
    
    var file = fopen("MyFile.txt", "r")
    
    var closeFile: (() -> Void)? = { fclose(file) }
    defer { closeFile?() }
    
    if fileIsNotValid(file) { return nil }

    if fileDoesNotContainsData(file, kind) { return nil }
    
    if fileDataOutOfDate(file) { return nil }
    
    // Prevent the deferred handler from closing the file
    closeFile = nil
    
    return file

}

Happy coding...

Did this help?, then please help out a small independent.
If you decide that you want to make a small donation, you can do so by clicking this
link: a cup of coffee ($2) or use the popup on the right hand side for different amounts.
Payments will be processed by PayPal, receiver will be sales at balancingrock dot nl
Bitcoins will be gladly accepted at: 1GacSREBxPy1yskLMc9de2nofNv2SNdwqH

We don't get the world we wish for... we get the world we pay for.

2017-01-05

Conditional compilation in Swift

There are many 'hits' when I google for 'swift conditional compilation'. But most of them are old and do not give a full example of just how to do this.

So here it is:

Btw, this works for Xcode 8.2.1

The first thing to know is that the simple approach only tests for the existence of definitions, not on the value of a definition.

A definition is simply a "word" that is known or not-known at compile time, by the compiler. Probably for this reason Apple calls them "Active Compiler Condition".

Active Compiler Conditions are defined in Xcode:



Here you can see that I defined an Active Compiler Condition with the name "USE_OPEN_SSL".

The active compiler condition can be used in the code like this:

        public func startReceiverLoop() {
            
            let queue = receiverQueue ?? DispatchQueue(label: "Receiver queue", qos: receiverQueueQoS, attributes: [], autoreleaseFrequency: DispatchQueue.AutoreleaseFrequency.inherit, target: nil)
            
            queue.async {
                
                [weak self] in
                
                switch self?.type {
                
                case nil: fatalError("No receiver type specified")
                
                case let .socket(sock)?:
                    
                    SwifterSockets.receiveLoop(
                        socket: sock,
                        bufferSize: self?.receiverBufferSize ?? 20 * 1024,
                        duration: self?.receiverLoopDuration ?? 5,
                        receiver: self)
                    
                
                case let .ssl(ssl, _)?:
                    
                    #if USE_OPEN_SSL
                        
                        SwifterSockets.Ssl.receiveLoop(
                            ssl: ssl,
                            bufferSize: self?.receiverBufferSize ?? 20 * 1024,
                            duration: self?.receiverLoopDuration ?? 5,
                            receiver: self)
                    #else
                        break
                    #endif
                }
            }

        }

If the ACC USE_OPEN_SSL definition is present, the compiler will compile the code as:

                case let .ssl(ssl, _)?:
                                            
                        SwifterSockets.Ssl.receiveLoop(
                            ssl: ssl,
                            bufferSize: self?.receiverBufferSize ?? 20 * 1024,
                            duration: self?.receiverLoopDuration ?? 5,
                            receiver: self)
                }

If however the definition is absent, the compiler will compile the code as:

                case let .ssl(ssl, _)?:
                    
                        break
                }

Btw, this is indeed a preview of an upcoming release of SwifterSockets that will support secure connections. See the links on the right hand side for a link to the current edition of SwifterSockets on github.

Happy coding...

Did this help?, then please help out a small independent.
If you decide that you want to make a small donation, you can do so by clicking this
link: a cup of coffee ($2) or use the popup on the right hand side for different amounts.
Payments will be processed by PayPal, receiver will be sales at balancingrock dot nl
Bitcoins will be gladly accepted at: 1GacSREBxPy1yskLMc9de2nofNv2SNdwqH

We don't get the world we wish for... we get the world we pay for.