Building an FTP and HTTP Server

When we moved in to our new office last year we wanted to connect our motion camera to hubot so that we could pull images from it into our Campfire chat on demand, because it’s cool.

We used the camera from the café at our old office which has support for sending periodic snapshots to a server via SMTP and FTP. It was originally configured to send the image files over the public Internet via FTP to a server outside the office network, these files were then served by an HTTP server configured to read files from the same directory.

The problem we encountered with this setup was that the FTP server would write to the same file which the HTTP server was reading from and there was no coordination between the processes. When you requested the image while the FTP server was mid-write, the image you’d receive in response would be in a partially written state and appear corrupt.

To avoid this both processes would have had to; share a file system that supported file locking, and been built with support for a locking mechanism such as lockf. Alternatively if their shared file system supported mandatory locks, only one of the processes would have to support and use file locking - the other process attempting to use the file would have its I/O blocked waiting for the mandatory lock to be released, dependent on the type of operation it was attempting with the file. For an example of a file locking API specific to OS X see Damien’s post which demonstrates File Coordination.

Another possible solution to prevent simultaneous reading and writing to the same file, that is to ensure the HTTP server could only ever read complete files from its directory, would have been to configure the FTP server to write new files to a separate directory on the same file system first and then add a link to the original directory pointing to the new file.

Rather than continuing to run the service on the public Internet and having to either, adapt the servers to coordinate their file access or workaround a lack of coordination, I decided to build an application which would run both an FTP and HTTP server in the same process, sharing an atomic in-memory file system. I had already built an Objective-C HTTP server a while ago so I just needed to build the FTP support. I started with RFC 959 and subsequent updates and was pointed to http://cr.yp.to/ftp.html by my friend Mike Abdullah. I added support for FTP commands incrementally using a combination of Transmit and the FTP camera itself to test out functionality.

The FTP server is bare bones and only supports a few commands, but it’s interoperable enough for what we need. The HTTP server responds to requests by reading files from the file system and includes a small dynamic resource which matches a path with a “/latest” suffix and redirects to the location of the latest file in that file system directory (this is currently determined by sorting the directory contents by name numerically rather than by date).

The project is camera_server. Retrieving the images from it over our local network is crazy fast, and this time they are always atomic copies. The first version of the file system suffered from poor performance which limited its practical usage to less than a thousand or so images, my next post will cover how I tuned its performance to support 250K images.