Do you need help on a specific subject? Use the contact form (Request a blog entry) on the right hand side.

2015-06-30

Code sample: Multi staged file reading with NSStream from Swift

When reading from file it is not always necessary to read the whole file in one go. Sometimes it is more advantageous to read only as much data as necessary. The exact amount however is often not known in advance, but must be read from the file itself. Usually in the form of a file header. Since reading data from file is a time consuming activity, it is also advantageous if this reading can be done in the background -as part of the runloop- without blocking other processes.

The following solution implements the above: it provides a class that will read data from a file in a multi-staged operation: it first read a file header, and then reads as much data from the file as needed. It also has a hook to process the data that was read (though that might not always be possible). Further, it provides a callback to an object that is called when the necessary data has been read (and possibly the data processing has completed).

The class is used as follows:

class MyDataModel: ReadyListener {
    
    func setupFromFile(file: NSURL) {
        
        let fr = MyFileReader(listener: self)
        
        fr.startReadingFromFile(file.path!) // Callback can happen from now on, probably should'nt be called from within an "init"
    }
    
    
    // Callback from ReadyListener
    
    func fileIsRead(aFile: MyFileReader) {
        if aFile.isValid {
            // .. update data model
        }
    }

}

The instantiation and start of the MyFileReader are separated because after the startReadingFromFile is called callbacks to fileIsRead can happen. If startReadingFromFile is called from within an init, it could theoretically happen that the callback comes before the init is complete. By separating the init and start of reading the object hierarchy can be build before any callbacks occur. (Note that since the callback in this implementation only happens from within a streaming delegate call, the callback can only happen at the end of a runloop cycle. Thus calling from within an init should be ok, its just not good practise to open any code up to such potential problems.)

The implementation of MyFileReader is as follows:

protocol ReadyListener {
    func fileIsRead(aFile: MyFileReader)
}

class MyFileReader: NSObject, NSStreamDelegate {
    
    var isValid: Bool = false
    var error: String = ""
    private let _listener: ReadyListener
    private var _filepath: String = ""
    
    
    /// The listener will not receive any callbacks until after the 'startReadingFromFile' methods is called.
    
    init(listener: ReadyListener) {
        _listener = listener
        super.init()
    }
    
    
    /// Starts reading from the given filepath.
    /// If false is returned, the error member will contain information about the error.
    /// If true is returned the listener will receive a callback when file operations are completed.
    
    func startReadingFromFile(filepath: String) -> Bool {
        
        _filepath = filepath
        
        let stream: NSInputStream! = NSInputStream(fileAtPath: filepath)
        
        if stream == nil {
            error = "Could not open stream for \(filepath)"
            return false
        }
        
        stream.delegate = self
        
        stream.scheduleInRunLoop(NSRunLoop.currentRunLoop(), forMode: NSDefaultRunLoopMode)
        
        stream.open()
        
        return true
    }
    
    
    // MARK: - Stream delegate protocol with associated local variables
    
    private var _streamBuffer: Array<UInt8> = Array(count: 1024, repeatedValue: 0)
    private var _nofUnusedStreamBufferBytes: Int = 0
    private var _waitingForStreamData: Bool = true
    
    func stream(aStream: NSStream, handleEvent eventCode: NSStreamEvent) {
        
        switch eventCode {
            
        case NSStreamEvent.ErrorOccurred:
            
            var message = "An error occured while reading \(_filepath)"
            if let theError = aStream.streamError {
                message = message + " with code = \(theError.code)"
            }
            error = message
            
            closeStream(aStream)
            
            return
            
            
        case NSStreamEvent.EndEncountered:
            
            // Check if all data was read
            
            if _waitingForStreamData  {
                error = "Premature end of file reached for \(_filepath)"
                isValid = false
            }
            
            closeStream(aStream)
            
            return
            
            
        case NSStreamEvent.HasBytesAvailable:
            
            // First initially, then guaranteed by processStreamBuffer, the streamBuffer is empty. Fill the streamBuffer and then call processStreamBuffer (again). When processStreamBuffer returns either all data is read and the stream is terminated, or the streamBuffer is again empty and can be filled by the next HasBytesAvailable event.
            
            var nofBytesRead = (aStream as! NSInputStream).read(&_streamBuffer, maxLength: _streamBuffer.count)
            
            
            // Check for errors during the read
            
            if nofBytesRead < 0 {
                error = "Streaming read failed for \(_filepath)"
                closeStream(aStream)
                return
            }
            
            
            // Remember the number of bytes read
            
            if nofBytesRead == 0 {
                _nofUnusedStreamBufferBytes = _streamBuffer.count
            } else {
                _nofUnusedStreamBufferBytes = nofBytesRead
            }
            
            
            // Process (all of!) the bytes and close the stream if sufficient data has been read.
            
            if processStreamBuffer() { closeStream(aStream) }
            
            
        default: break
        }
    }
    
    private func closeStream(aStream: NSStream) {
        
        aStream.close()
        aStream.removeFromRunLoop(NSRunLoop.currentRunLoop(), forMode: NSDefaultRunLoopMode)
        
        _listener.fileIsRead(self)
        _waitingForStreamData = false
        
        // Null the internal variables that are no longer used to be able to reclaim memory
        
        _streamBuffer = []
        _headerBytes = []
        // _dataBytes = [] do this only when the processing of the data means that this buffer is no longer needed!
    }
    
    // Byte buffers for the header and data
    
    private var _headerBytes: Array<UInt8> = Array(count: 10, repeatedValue: 0)
    private var _headerNofBytes: Int = 0
    private var _headerIsComplete: Bool = false
    private var _size: Int = 0
    private var _dataBytes: Array<UInt8>?
    private var _dataNofBytes: Int = 0
    
    
    // Process the data from the streambuffer, returns true when stream reading can stop, false when not.
    
    private func processStreamBuffer() -> Bool {
        
        let CONTINUE_STREAM_READING       = false
        let STREAM_READING_COMPLETED      = true
        
        
        // The stream buffer is always freshly filled when this method is called
        
        var firstUnusedStreamBufferByte: Int = 0
        
        
        // If not yet complete, read the header
        
        if !_headerIsComplete {
            
            
            // Copy from the streamBuffer to the headerBytes until all header bytes are received.
            
            while _headerNofBytes < _headerBytes.count {
                
                // Copy 1 byte
                
                _headerBytes[_headerNofBytes++] = _streamBuffer[firstUnusedStreamBufferByte++]
                _nofUnusedStreamBufferBytes-- // Housekeeping
                
                
                // If the header is not complete, loop around to copy the next byte, or if exhausted, wait for the next bytes available event.
                
                if (_nofUnusedStreamBufferBytes == 0) && (_headerNofBytes < _headerBytes.count) {
                    return CONTINUE_STREAM_READING
                }
            }
            
            _headerIsComplete = true
            
            // Process the header, check for validity and find out how many bytes must be read.
            
            let sizeOffset = 0 // start at the first byte, adjust as necessary
            
            _size = Int(_headerBytes[sizeOffset])
            _size = _size << 8
            _size = _size | Int(_headerBytes[sizeOffset + 1])
            _size = _size << 8
            _size = _size | Int(_headerBytes[sizeOffset + 2])
            _size = _size << 8
            _size = _size | Int(_headerBytes[sizeOffset + 3])
            
            // Follow up by size validity check....
            
        }
        
        
        // More data available? (Most likely, but we must be sure because we need at least 1 byte for the next bit)
        
        if (_nofUnusedStreamBufferBytes == 0) { return CONTINUE_STREAM_READING }
        
        
        // Create storage for the data bytes if not yet done
        
        if _dataBytes == nil { _dataBytes = Array(count: _size, repeatedValue: 0) }
        
        
        // Copy as many of the remaining bytes as possible, but no more than needed
        
        let neededBytes = _dataBytes!.count - _dataNofBytes
        
        if neededBytes <= _nofUnusedStreamBufferBytes {
            
            // All bytes are available, copy what is needed to the data buffer
            
            _dataBytes![_dataNofBytes ..< _dataBytes!.count] =
                _streamBuffer[firstUnusedStreamBufferByte ..< firstUnusedStreamBufferByte + neededBytes]
            
            _dataNofBytes = _dataBytes!.count
            
        } else {
            
            // All data is needed, add it to the data buffer
            
            _dataBytes![_dataNofBytes ..< _dataNofBytes + _nofUnusedStreamBufferBytes] =
                _streamBuffer[firstUnusedStreamBufferByte ..< firstUnusedStreamBufferByte + _nofUnusedStreamBufferBytes]
            
            _dataNofBytes += _nofUnusedStreamBufferBytes
        }
        
        
        // Check if all data has been read
        
        if _dataNofBytes != _dataBytes!.count { return CONTINUE_STREAM_READING }
        
        
        // All data is read
        
        processData()
        
        return STREAM_READING_COMPLETED
    }
    
    private func processData() {
        // ....
    }

}

I hope the code is self explanatory, just note that MyFileReader inherits from NSObject and that super.init() *must* be called.

Also note that after the stream is closed, the byte buffers are explicitly freed. Thus allowing the OS to reclaim their memory. This allows continued existence of the MyFileReader object without incurring the penalty of keeping the -now- unused buffers allocated. Especially important when dealing with a lot of files.

Happy coding...

Did this help?, then please help out a small independent.
If you decide that you want to make a small donation, you can do so by clicking this
link: a cup of coffee ($2) or use the popup on the right hand side for different amounts.
Payments will be processed by PayPal, receiver will be sales at balancingrock dot nl
Bitcoins will be gladly accepted at: 1GacSREBxPy1yskLMc9de2nofNv2SNdwqH

We don't get the world we wish for... we get the world we pay for.

2015-06-25

Swift Gotcha: Surprising behaviour for empty array slices in Swift

Today I stumbled over some surprising behaviour of the array slice operations.
I wanted to assign one slice to another but of course had to check what would happen for zero slices. Turns out zero slices work fine if you do them right, like this:

var arr1 = [1, 2, 3, 4]
var arr2 = [5, 6, 7]

arr1[1 ..< 1] = arr2[1 ..< 1]


println(arr1) // Prints "[1, 2, 3, 4]"

However if you do them wrong, things grind to halt. And not in the nicest way.
This is wrong:

var arr1 = [1, 2, 3, 4]
var arr2 = [5, 6, 7]

arr1[1 ... 0] = arr2[1 ... 0]

println(arr1)  // Prints nothing in playground

I did not expect there to be contextual difference between the two, but apparently there is.

Happy coding...

Did this help?, then please help out a small independent.
If you decide that you want to make a small donation, you can do so by clicking this
link: a cup of coffee ($2) or use the popup on the right hand side for different amounts.
Payments will be processed by PayPal, receiver will be sales at balancingrock dot nl
Bitcoins will be gladly accepted at: 1GacSREBxPy1yskLMc9de2nofNv2SNdwqH

We don't get the world we wish for... we get the world we pay for.

2015-06-22

Code Sample: FSEvents and Swift

FSEvents allow an application to process file system events.

My recent App needs to listen for changes in a folder. If a file is removed from the folder, it also needs to be removed from the datamodel.

Unfortunately the interface for FSEvents is at CF level, thus quite low level C, and it needs the dreaded CFunctionPointer. Swift 1.x is not capable of creating a CFunctionPointer, thus a pure Swift implementation for FSEvent handling needs to wait for Swift 2 later this year.

However, my App cannot wait, so I had to work out how to do this in a mix of Swift and Objective-C. It is a dual use solution: Swift calls Objective-C to create the FSEventStreamRef and to process the events (in the callback). And the callback in Objective-C calls Swift to do the final processing.

My Swift code looks as follows:

    func registerForFileSystemEvents() {
        
        // TODO: In Swift 2 replace this code with pure Swift implementation
                
        // Build array with unique folder paths
        var strings = Array<String>()
        for file in files {
            if strings.filter({ $0 == file.stringByDeletingLastPathComponent }).count == 0 {
                strings.append(file.stringByDeletingLastPathComponent)
            }
        }
        if strings.isEmpty { return }
        
        
        // Register for FSEvents

        registerForFsEventStream(strings, self, "dataModelFsEventCallback")
    }
    
    @objc func dataModelFsEventCallback() {
        updateStatus()
        removeRemovedLines()
    }

Luckily, in my data model I do not need to know which event was returned. All events are handled in the same way, by updating the status of all files. Once the status has been updated I can remove those files which are no longer present.

The Swift callback that is called from Objective-C must be marked with @objc.

In a supporting Objective-C header file (included in the bridging header!) the following is defined:

#ifndef c_support_h
#define c_support_h

void registerForFsEventStream(NSArray *arr, id obj, SEL sel);

#endif

And in the corresponding implementation:

#import <Foundation/Foundation.h>
#import "c_support.h"

FSEventStreamRef fsEventStreamRef = NULL;
id callbackObject = NULL;
SEL callbackMethod;

void fsEventStreamCallback(ConstFSEventStreamRef streamRef, void *clientCallBackInfo, size_t numEvents, void *eventPaths, const FSEventStreamEventFlags eventFlags[], const FSEventStreamEventId eventIds[]) {
    
    if (callbackObject != NULL) {
        
        // Supress the "leak" warning.
        #pragma clang diagnostic push
        #pragma clang diagnostic ignored "-Warc-performSelector-leaks"

        [callbackObject performSelector:callbackMethod];
        
        // Restore the old diagnostics
        #pragma clang diagnostic pop
    }
}

void registerForFsEventStream(NSArray *arr, id obj, SEL sel) {

    if (fsEventStreamRef) {
        FSEventStreamStop(fsEventStreamRef);
        FSEventStreamUnscheduleFromRunLoop(fsEventStreamRef, CFRunLoopGetCurrent(), kCFRunLoopDefaultMode);
        FSEventStreamInvalidate(fsEventStreamRef);
        FSEventStreamRelease(fsEventStreamRef);
        fsEventStreamRef = NULL;
    }
    
    callbackObject = obj;
    callbackMethod = sel;
    
    fsEventStreamRef = FSEventStreamCreate(
                                           kCFAllocatorDefault,
                                           fsEventStreamCallback,
                                           NULL,
                                           (__bridge CFArrayRef)(arr),
                                           kFSEventStreamEventIdSinceNow,
                                           1, // delay 1 second between event and callback
                                           kFSEventStreamCreateFlagIgnoreSelf);
    
    FSEventStreamScheduleWithRunLoop(fsEventStreamRef, CFRunLoopGetCurrent(), kCFRunLoopDefaultMode);
    FSEventStreamStart(fsEventStreamRef);
}

I think the implementation speaks for itself. It can be held very simple because the FSEvents flags are pretty useless. For example when a file is removed, it is put in the trash, hence two events are received: first a "rename" event and then a ".DS_store" event. But for a real file renaming there are also two events: first the "rename" event with the old file name and then a second "rename" event for the new filename. Unfortunately without any means to associate the two. The effect is that it becomes impossible to differentiate between a "real" rename event and a remove event.

PS: I know about the sequential event id's of a "real" rename event. However I have not found any guarantee from Apple which states that these id's will always be sequential. It is therefore possible that they won't be sequential and hence unusable.

The only solution I see is to treat any FSEvent as a signal for rescanning a folder and treat any possible rename simply as a removed file. This is harder on the user but easier on the programmer: there is no need for parameters on the call-back to Swift ;-)

Another thing I noticed is that fsEventStreamCallback will not be called for folders that are under the protection of the sandbox. Hence you need to make sure that the user allows the folder to be scanned for FSEvents. (My App does this by asking the user to open the directory in a file-open dialogue)

2015-07-20: To persist file access right between session, see this post

Happy coding...

Did this help?, then please help out a small independent.
If you decide that you want to make a small donation, you can do so by clicking this
link: a cup of coffee ($2) or use the popup on the right hand side for different amounts.
Payments will be processed by PayPal, receiver will be sales at balancingrock dot nl
Bitcoins will be gladly accepted at: 1GacSREBxPy1yskLMc9de2nofNv2SNdwqH

We don't get the world we wish for... we get the world we pay for.

2015-06-20

SwifterJSON version 0.9.1

A quick note to mention that I updated my JSON framework from version 0.9.0 to 0.9.1.
As can be seen from the small increment, these are mostly cosmetic changes. Functionally nothing has changed.
All changes make the interface more consistent, and make the framework easier to adopt in other projects.

Link to github: SwifterJSON

Happy coding...

Did this help?, then please help out a small independent.
If you decide that you want to make a small donation, you can do so by clicking this
link: a cup of coffee ($2) or use the popup on the right hand side for different amounts.
Payments will be processed by PayPal, receiver will be sales at balancingrock dot nl
Bitcoins will be gladly accepted at: 1GacSREBxPy1yskLMc9de2nofNv2SNdwqH

We don't get the world we wish for... we get the world we pay for.

2015-06-19

Adding Apple help to an OS-X application

So this is not Swift related, but it is something many of us will encounter eventually: How to add Apple Help support to an application

I spend a full day trying to get this to work, and I failed. The document from Apple has errors in it, and there is precious little to find elsewhere on the inet. Though I did find one interesting blog entry on this here: Apple help in 2015. Unfortunately the hints and tips provided there also failed to work for me.

I did in fact give up after that. I took the alternative approach: connect the help menu to the AppDelegate and handle the help by opening up a html file that is located in the resources. Like this:

    @IBAction func displayHelp(sender: AnyObject?) {
        
        let url = NSBundle.mainBundle().URLForResource("help", withExtension: "html")
        
        NSWorkspace.sharedWorkspace().openURL(url!)

    }

Of course that worked... but it irked me. Why would the "normal" approach not work?

What I wanted was the most simple implementation that I could think of, no multi language support, no icon, no special features, no index file etc. Just a simple title with a few topics. I figured that when I can get that to work, the rest is a matter of legwork.

And then I had some luck... well, at least it finally started to make some sense after I read up on the Core Foundation Keys. With a little experimenting I finally had my first success.

Here is how it works for me:

First create the content, I created one main file (MyAppHelp.html) and a couple of topic files (Topic1.html, Topic2.html etc). The MyAppHelp.html linked to the Topic files in the usual way (<a href...)

In the MyAppHelp.html file I did not include anything special, no meta tags. Specifically no "AppleTitle" tag and no "Robots" tag. Just plain html content with links to the topic pages.

Next I created the folder hierarchy for the help book on my desktop and placed the html files in the Resources folder.


We need one more file, a property list. I simply copied a Info.plist from another app's help bundle and modified its content as follows:


The key's InfoDictionary version, Bundle OS Type code, Bundle creator OS Type code and HPDBookType should be exactly as above.

The HPDBookAccessPath should be the name of the top level html file. I think that if you put it in a subdirectory (e.g. English/MyAppHelp.html) the name of the subdirectory should be included, but in my example here just the name is sufficient.

The HPDBookTitle is important, it is used in the Info.plist of your application to identify the help book.

The Info.plist just created should reside at the Contents level of the help book hierarchy. The complete help book looks as follows:


Now rename the help book to create a bundle: MyAppHelp becomes MyAppHelp.help. When we do this, the icon changes to the Apple helpbook icon, and we can no longer simply open the folder. Remember that with Right-Click -> Show Package Contents we can still see what is inside.

Note: Even though the icon changed, this does not mean that we can double-click the help book to open it. That won't work. But it will work from within the application...

Now switch to the project in xcode. In Xcode select "Add Files" and choose the help book just created. Make sure that the files and folder structure is copied:


Almost done...
Now edit the Info.plist of the project to include the following two key's:


Forget about the "Help Book File" key, we don't need that.
It should be noted that the Help Book Identifier must have the same value as the HPDBookTitle from the help book Info.plist file. The Help Book directory name key is the name of the help book file that we added to the project.

That is it, launch the application and select "Help -> MyApp Help" from the menu. Voila!

PS: A neat trick is to create an alias into the Resources folder of the help book and place that somewhere at the top level of the project. Use that alias to quickly access the html files for updating etc. Also, most HTML editors will be unable to open a bundle, but they can follow an alias. Unfortunately Xcode will not see the modifications you make to the help book, but git will. So if you use git, the changes to the help book will be put in the repository, even if Xcode does not show the "M" behind the help book.

Happy coding...

Did this help?, then please help out a small independent.
If you decide that you want to make a small donation, you can do so by clicking this
link: a cup of coffee ($2) or use the popup on the right hand side for different amounts.
Payments will be processed by PayPal, receiver will be sales at balancingrock dot nl
Bitcoins will be gladly accepted at: 1GacSREBxPy1yskLMc9de2nofNv2SNdwqH

We don't get the world we wish for... we get the world we pay for.