• We only deal with text files as input. We sort file lines using string comparison.
• Lines have a “reasonable” maximum length and consistent line endings.

We will be using C# here but it wouldn’t be much different in Java.

Split the big file in small sorted chunks

The first step is quite easy: we open a stream reader on the big file and keep reading until we have enough data in our buffer to fill a file chunk. When the buffer is full, we sort the data in memory and write it on disk on a temp file. We continue until the whole input file is processed. As a result, we have K chunk files on disk. The SplitInSortedChunks method below implements this algorithm. It returns the chunk file paths collection as a result.

IEnumerable<string> SplitInSortedChunks(string filepath, long chunkSize)
{
    var buffer = new List<string>();
    var size = 0L;

    using (var reader = new StreamReader(filepath))
    for (string line = reader.ReadLine(); line != null; line = reader.ReadLine())
    {
        if (size + line.Length + 2 >= chunkSize)
        {
            size = 0L;
            yield return FlushBuffer(buffer);
        }

        size += line.Length + 2;
        buffer.Add(line);
    }

    if (buffer.Any())
    {
        yield return FlushBuffer(buffer);
    }
}

The magic number 2 used to increment the current size is the supposed length of the line ending used in the input file. It’s probably more reliable here to use Environment.NewLine.Length or have this value as parameter.

The FlushBuffer method implementation is quite simple:

string FlushBuffer(List<string> buffer)
{
    buffer.Sort(StringComparer.Ordinal);
    var chunkFilePath = Path.GetTempFileName();
    File.WriteAllLines(chunkFilePath, buffer);
    buffer.Clear();
    return chunkFilePath;
}

K-way merge

Now that we have K small files filled with sorted data, we are going to merge them in order to build our final result file. To perform the merge, we need first to open K file streams, one for each chunk file. Then, we have to read one line per file and take the smallest line each time. We repeat this operation until we finish reading all the chunks. In order to avoid doing a lot of string comparisons at each iteration, we keep the current line values in an ordered structure.

The algorithm is quite straightforward. Still we made an interesting mistake in our first implementation that I am going to share with you here.

A buggy implementation

Our K-way merge function takes the chunk file paths and the desired result file path as parameters. In order to save space on disk, we can use the input file path as output path; but this is not mandatory.

private static void KwayMerge(IEnumerable<string> chunkFilePaths, string resultFilePath)
{
    var chunkReaders = chunkFilePaths
        .Select(path => new StreamReader(path))
        .Where(chunkReader => !chunkReader.EndOfStream)
        .ToList();

    // BUG: SortedDictionary won't cut it here
    var sortedDict = new SortedDictionary<string, TextReader>();
    chunkReaders.ForEach(chunkReader => sortedDict.Add(chunkReader.ReadLine(), chunkReader));

    using (var resultWriter = new StreamWriter(resultFilePath, false))       
    while (sortedDict.Any())
    {
        var line = sortedDict.Keys.First();
        var chunkReader = sortedDict[line];
        sortedDict.Remove(line);

        resultWriter.WriteLine(line);

        var nextLine = chunkReader.ReadLine();
        if (nextLine != null)
        {
            sortedDict.Add(nextLine, chunkReader);
        }
        else
        {
            chunkReader.Dispose();
        }
    }
}

This implementation worked fine until we tested it with files with duplicate lines. It crashed complaining about duplicate keys in the SortedDictionary. Ouch! Notice that we would’ve had the same problem with SortedList or SortedSet. We need a sorted multiset structure here to fix the bug.

A better implementation

In order to fix the problem we have with duplicate lines, we are going to use a priority queue (based on a max heap) as a sorted multiset. We have already discussed the implementation of priority queue and heap data structures in a previous post (the implementation is also on github and available as nuget).

void KwayMerge(IEnumerable<string> chunkFilePaths, string resultFilePath)
{
    var chunkReaders = chunkFilePaths
        .Select(path => new StreamReader(path))
        .Where(chunkReader => !chunkReader.EndOfStream)
        .ToList();

    var queue = new PriorityQueue<string,TextReader>((x,y) => -string.CompareOrdinal(x,y));
    chunkReaders.ForEach(chunkReader => queue.Enqueue(chunkReader.ReadLine(), chunkReader));
    
    using (var resultWriter = new StreamWriter(resultFilePath, false))
    while (queue.Count > 0)
    {
        var smallest = queue.Dequeue();
        var line = smallest.Key;
        var chunkReader = smallest.Value;

        resultWriter.WriteLine(line);

        var nextLine = chunkReader.ReadLine();
        if (nextLine != null)
        {
            queue.Enqueue(nextLine, chunkReader);
        }
        else
        {
            chunkReader.Dispose();
        }
    }
}

Conclusion

Now that we have our functions to split the input file in chunks and to do the k-way merge, we simply need to chain them in order to sort.

void SortFile(string filepath, string resultFilePath, long chunkSize)
{
    var chunkFilePaths = SplitInSortedChunks(filepath, chunkSize);
    KwayMerge(chunkFilePaths, resultFilePath);
}

That’s it for this code kata! If you have comments or suggestions on how to improve this algorithm don’t hesitate to comment below.

Before we start

If you want to test and hack the code snippets presented here you’ll need to create three files: editor.js, editor.css and editor.html. The first two files editor.js and editor.css are empty for now. Here is how your editor.html would look like:

<!doctype html>
<html>
 <head>
  <link rel='stylesheet' type='text/css' href='editor.css'>
  <script src='http://cdnjs.cloudflare.com/ajax/libs/jquery/1.7/jquery.min.js'></script>
  <script src='http://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.2.1/underscore-min.js'></script>
  <script src='http://cdnjs.cloudflare.com/ajax/libs/backbone.js/0.5.3/backbone-min.js'></script>
  <script src='editor.js'></script>
 </head>
 <body>
  <div id='page' style='width:2000px;height:2000px;'></div>
 </body>
</html>

I’m using cdnjs here, an alternative free CDN for Javascript brought by cloudflare. It has many libs you won’t find at Google or Microsoft CDNs.

Some of the code samples are also hosted on jsfiddle for convenience. A link is provided in each section to the corresponding jsfiddle code snippet. You can view, test and fork them right there.

Now that we are set up, time to code!

Models

Let’s start by defining a model for a simple shape: the Shape class.

var Shape = Backbone.Model.extend({
    defaults: { x:50, y:50, width:150, height:150, color:'black' },
    setTopLeft: function(x,y) {
        this.set({ x:x, y:y });
    },
    setDim: function(w,h) {
        this.set({ width:w, height:h });
    },
});

Our Shape class extends the Backbone.Model class. The extend method takes a hash as argument in order to configure the model. In our case, we have three properties: defaults, setTopLeft and setDim:

• defaults is a special property that backbone uses to define a set of default properties/values in the model. So by default here, a Shape instance will have the properties x, y, width, height and color defined and set to the provided default values. Notice that these model properties are encapsulated by backbone and instead of getting/setting them directly, we will use the get and set methods that Shape inherits from the backbone model. The encapsulation allows backbone to control modification of the model properties and fire change event on set method calls. This allows registering listeners for model changes.
• setTopLeft and setDim are two helper methods that use the backbone’s set method in order to, respectively, set the shape’s top left corner and dimension.

Now that our model class is defined, here is an example of how we would instantiate it and bind change events to its properties (jsfiddle):

var shape = new Shape();

shape.bind('change', function() { alert('changed!'); });
shape.bind('change:width', function() { alert('width changed! ' + shape.get('width')); });

shape.set({ width: 170 });
shape.setTopLeft(100, 100);

In line 3 we are registering for any property change in the model. In line 4 we are only listening only to property ‘width’ changes. By setting the shape’s width line 6, both callbacks will be invoked. On the other hand, setting the top left corner coordinates using ‘setTopLeft’ in line 7 will only trigger the ‘change’ event.

Binding page elements to model changes

In the last section, we were able to define models and listen to their change events. Now, we will try to do something useful with these events. Let’s first define a div element in our html page:

<div class='shape' />

and let’s tie our model changes to it as follows (we’ll be using jQuery for DOM manipulation):

shape.bind('change', function() {
    $('.shape').css({ left:       shape.get('x'),
                      top:        shape.get('y'),
                      width:      shape.get('width'),
                      height:     shape.get('height'),
                      background: shape.get('color') });
});

That was easy! Now we can modify the model and observe the dom changing in reaction to model changes. Just open firebug or your favorite browser’s javascript console and try something like:

shape.setTopLeft(10, 10);
shape.setDim(500, 500);

and you will see the page automatically updating itself and the shape automatically adapting to the new position/size. The interesting bit here is that we are no longer manipulating the DOM directly. A “piece of code” is listening to model changes and is updating the page on model changes automatically. This snippet is also on jsfiddle.

Time to use user’s input now in order to mutate the model and thus, indirectly, modify the page.

Basic user input handling

In this section, we will enable the user to drag the shape’s element around. In order to do that, we will listen to mouse events mousedown, mouseup and mousemove and update the model accordingly. Here is an example of how we can achieve this (jsfiddle):

var dragging = false;

$('.shape').mousedown(function (e) {
    dragging = true;
    shape.set({ color: 'gray' });
});

$('#page').mouseup(function () {
    dragging = false;
    shape.set({ color: 'black'});
});

$('#page').mousemove(function(e) {
    if(dragging) {
        shape.setTopLeft(e.pageX, e.pageY);
    }
});

As a side note, notice that we are listening to mousemove and mouseup events on the parent page element and not on the shape’s div since we want the div to follow (resp. stop following) the mouse position even when it gets off (resp. is up outside of) the shape’s div borders.

What is worth considering here is the simplicity of the jQuery callbacks handling the user input. We are not modifying or even querying the DOM in this code. We are just listening to the user input — mouse events here– and translating it to model changes. The page magically updates itself through the model change listeners.

To sum up, we have separated the code handling user input and mutating the model, and the code updating the view in reaction to model change events. Looks like we implemented a model-view-controller here .

This is a good step to separate concerns and reduce the callback spaghetti one ends up with when dealing with pages with more than a couple of jQuery callbacks. However, we can do better here as our controller and view code is still scattered across free anonymous functions here and there. We’ll work on this by defining proper view classes later.

Model Collections

Before going further, we have to introduce a special kind of model used in backbone: collections. Collections are simply a container that helps maintain an ordered list of model objects. In addition it comes with built in events for common collection operations like add and remove.

We will use a model collection in order to organize our previously defined Shape objects into a Document:

var Document = Backbone.Collection.extend({ model: Shape });

That’s it! We can now instantiate the Document class and listen to add and remove events as follows:

var document = new Document();

document.bind('add', function(model) { alert('added'); });
document.bind('remove', function(model) { alert('removed'); });

document.add(shape); // fires add event
document.remove(shape); // fires remove event

Views

A Backbone.js view is usually associated with a model (or a model collection). The view is responsible for two things:

1. Rendering the model into a DOM element. It listens to model changes and updates the page accordingly.
2. Handling the events of this DOM element and updating the model.

In theory, Backbone’s view is actually playing both MVC’s view and MVC’s controller roles as it is handling the user input (DOM events) and updating the model, and also listening to model events and updating the visual part. This doesn’t have a big impact in practice as you would have different methods for each operation kind.

Shape View

In this section we will go through the code of the view of our Shape model. The view manages an html element that represents the shape itself and also ‘control’ elements that decorate the shape and that will allow the user to drag, resize, delete and change the shape’s color.

var ShapeView = Backbone.View.extend({
    initialize: function() {
        this.model.bind('change', this.updateView, this);
    },
    render: function() {
        $('#page').append(this.el);
        $(this.el)
            .html('<div class="shape"/>'
                  + '<div class="control delete hide"/>'
                  + '<div class="control change-color hide"/>'
                  + '<div class="control resize hide"/>')
            .css({ position: 'absolute', padding: '10px' });
        this.updateView();
        return this;
    },
    updateView: function() {
        $(this.el).css({
            left:       this.model.get('x'),
            top:        this.model.get('y'),
            width:      this.model.get('width') - 10,
            height:     this.model.get('height') - 10 });
        this.$('.shape').css({ background: this.model.get('color') });
    },
    events: {
        'mousemove'               : 'mousemove',
        'mouseup'                 : 'mouseup',
        'mouseenter .shape'       : 'hoveringStart',
        'mouseleave'              : 'hoveringEnd',
        'mousedown .shape'        : 'draggingStart',
        'mousedown .resize'       : 'resizingStart',
        'mousedown .change-color' : 'changeColor',
        'mousedown .delete'       : 'deleting',
    },
    hoveringStart: function () {
        this.$('.control').removeClass('hide');
    },
    hoveringEnd: function () {
        this.$('.control').addClass('hide');
    },
    draggingStart: function (e) {
        this.dragging = true;
        this.initialX = e.pageX - this.model.get('x');
        this.initialY = e.pageY - this.model.get('y');
        return false; // prevents default behavior
    },
    resizingStart: function() {
        this.resizing = true;
        return false; // prevents default behavior
    },
    changeColor: function() {
        this.model.set({ color: prompt('Enter color value', this.model.get('color')) });
    },
    deleting: function() {
        this.remove();
    },
    mouseup: function () {
        this.dragging = this.resizing = false;
    },
    mousemove: function(e) {
        if (this.dragging) {
            this.model.setTopLeft(e.pageX - this.initialX, e.pageY - this.initialY);
        } else if (this.resizing) {
            this.model.setDim(e.pageX - this.model.get('x'), e.pageY - this.model.get('y'));
        }
    }
});

Don’t be afraid of the length of the code, it’s pretty simple. Here are the most important bits:

• intialize is a special function that is executed on view creation. This is where you usually wire your view to the model by listening to events. In our case, the view registers itself for change events.
• render is also a special function that is executed right after initializing the view. Here, the html element representing the view is initialized and “pushed” –i.e. added– to the DOM. The view’s html element is held into the inherited backbone property el. Line 6, we add el to the page, line 7 we set its html with the shape and controls elements, and finally, line 13 we update the view with the model properties.
• The events hash (line 24) is an important part in our view configuration. It maps events to handler methods. The format is { ‘event selector’ : ‘handler’ }. For example, the pair { ‘mousedown .shape’ : ‘draggingStart’ } means that a mousedown event on an element with class shape will trigger the method draggingStart. The events hash defines how the user input is handled and thus defines the ‘controller’ side of our backbone view.

We have a small technical problem here though. As we did in the previous section Basic user input handling, for a better user experience we should be listening to mousemove and mouseup events on the parent page element and not on the shape’s div itself. The current code works but resizing might a bit choppy if the user moves the mouse too fast. The work around is easy though. It is implemented in the jsfiddle snippet.

Document View

Now we need to add a bit more structure to the shape views by using the document model introduced earlier. The document has also a view that manages all the shape views as follows:

var DocumentView =  Backbone.View.extend({
    id: 'page',
    views: {},
    initialize: function() {
        this.collection.bind('add', this.added, this);
        this.collection.bind('remove', this.removed, this);
    },
    render: function() {
        return this;
    },
    added: function(m) {
        this.views[m.cid] = new ShapeView({
            model: m,
            id:'view_' + m.cid
        }).render();
    },
    removed: function(m) {
        this.views[m.cid].remove();
        delete this.views[m.cid];
    }
});

The id property indicates the identifier of the DOM element the view is tied to. Backbone will use this value to set the el property accordingly. Since we are using an existing element in the html page, there is nothing to do in the render method.

In the intialize method, the view registers itself for two built in model collection events: add and remove. On the add event, we create and render the view corresponding to the added shape model. We maintain a set of these views in the property views. When the remove event is fired, the shape removed from the document its view is fetched in the views set and removed from the page.

Here is the corresponding jsfiddle snippet.

Conclusion

The source of tutorial is available on github and on jsfiddle. You can also have a look at the online demo.

In this tutorial, we’ve been through some of the aspects of Backbone.js, mainly the MVC and event driven principles it puts in place. As I mentioned in the introduction, we didn’t dive into routing and server communication. Backbone.js has, for instance, built-in support for CRUD operations. I’ll try to cover some of these features in a future post.

Do you have questions or suggestions? Do not hesitate, please put a comment below!

While working on this problem, I had the idea of the coding exercise I’m sharing here: a utility for converting back and forth base 10 numerals to strings of base X equivalent –just like itoa and atoi in C (except that atoi doesn’t take a base parameter). In order to make things a bit more interesting, I decided to learn myself some Scala and translate the code to functional style. Both implementations are detailed here and available online under the terms of the Apache version 2 license.

At the end, I didn’t resist the temptation to do a C# vs. Scala performance benchmark. I also coded and added a Java implementation to the tests in order to distinguish the performance change due to switching from the CLR to the JVM, and the one introduced by the translation to functional style and to the Scala runtime.

First, Iterative style

I started by implementing the C# version. It takes less than 30 lines of code:

public static int Decode(string str, int baze)
{
    int result = 0;
    int place = 1;
    int length = str.Length;
    for (int i = 0; i < length; ++i)
    {
        result += Value(str[length - 1 - i]) * place;
        place *= baze;
    }
    return result;
}

public static string Encode(int val, int baze)
{
    var buffer = new char[32];
    int place = 0;
    int q = val;
    do
    {
        buffer[place++] = Symbol(q % baze);
        q /= baze;
    }
    while (q > 0);
    Array.Reverse(buffer, 0, place);
    return new string(buffer, 0, place);
}

Symbol and Value methods convert respectively a numeral to its symbol and vice versa. They are omitted here for the sake of clarity. An example implementation that handles bases from base 2 to base 64 is provided in the C# online sample.

If you can afford having unsafe code in your project you can replace line 16 with something like:

char* buffer = stackalloc char[32];

The string buffer is then allocated on the stack instead of the heap. This is meant to boost performance as it avoids garbage collection for the buffer. The performance of the unsafe version is detailed in the benchmark results below.

I’m not going to list the Java code here since it is very similar to the C# one and it is also available in the online gist.

Enter Scala

I must say that I have little knowledge of functional programming at the time of this writing. Still, having some Java and Ruby background, getting started with Scala was easy and fun. After grasping the bare minimum of the Scala basics, I got back to my C# code and started translating it to the functional language. After all, all I had to do is to translate two methods consisting basically of one plain old iterative loop each, to shiny tail-recursive functions, right?

A list based implementation

Most functional languages have built-in immutable singly linked lists as fundamental data-structure. Prepending, getting the first element and getting the tail of the list (i.e. a sublist with all elements but head) are O(1) operations. In addition, since the list is immutable, it can be shared, manipulated and have multiple pointers at different positions without worrying about data-races. Scala makes no exception and provides the class List as a built-in immutable structure with the :: operator and head and tail functions (obviously, there is no append operation since elements are not meant to be added at the end of the list).

Based on this finding, my first attempt to translate the code to Scala uses lists as underlying data structures for both functions.

def encode(i: Int, baze: Int): String = {
  def process(q: Int, baze: Int, list: List[Char]): List[Char] = {
    if (q > 0) process(q / baze, baze, symbol(q % baze) :: list)
    else list
  }
  if (i == 0) symbol(0).toString
  else process(i, baze, Nil).mkString
}

def decode(str: String, baze: Int): Int = {
  def process(acc: Int, place: Int, lst: List[Char]): Int = {
    if (!lst.isEmpty) process(acc + value(lst.head) * place, place * baze, lst.tail) 
    else acc
  }
  process(0, 1, str.reverse.toList)
}

In the encode function, a list is used as buffer where symbols are prepended. The list is converted to a string object when all symbols are computed. In the decode function, the input string is reversed and mapped to a List where symbols are popped one by one until the result is computed. Note that the string has to be reversed first since we cannot traverse the list backward from tail to head (remember, it’s a singly linked list).

The translation was simple to do and I find the resulting Scala code concise and quite elegant. However, the performance of this initial implementation –discussed in more details below– was too slow.

A more pragmatic implementation

Lessons learned from the first version, I refactored my Scala code as follows:

def encode(i: Int, baze: Int): String = {
  def process(q: Int, place: Int, baze: Int, builder: StringBuilder): StringBuilder = {
    if (q > 0) process(q / baze, place + 1, baze, builder += symbol(q % baze))
    else builder
  }
  if (i == 0) symbol(0).toString
  else process(i, 0, baze, new StringBuilder(32)).reverse.toString
}

def decode(str: String, baze: Int): Int = {
  def process(acc: Int, place: Int, str: String, index: Int): Int = {
    if (index >= 0) process(acc + value(str.charAt(index)) * place, place * baze, str, index - 1)
    else acc
  }
  process(0, 1, str, str.length - 1)
}

The decode function now processes directly the input string, without converting it to a list. An additional index variable is introduced in order to keep track of the current symbol. This is less elegant than the list based version, but way more efficient.

The encode function is basically the same as the list based one, except that the list was replaced by a StringBuilder.

Stopwatch in hand, I was finally ready to start benchmarking!

Lies, Damn Lies and Benchmarks

The results presented here are not meant to compare the performance of Scala and C# in general. They are specific to the context of the stated problem and the provided implementations. This benchmark, like most benchmarks, have to be taken with a grain of salt. Here are some reasons why:

• I only measured raw speed. Raw speed is nice to have but it does not imply scalability.
• The benchmarked applications are stateless and purely sequential. Results may vary considerably with more complex and/or multi-threaded code.
• I’ve basically started learning Scala a few hours before writing this benchmark. Albeit simple, the Scala code probably needs some obvious performance tuning that I’m not aware of.
• I didn’t have a decent server at hand to run the benchmark on. I performed this benchmark on my laptop: an Intel Core i7-720QM Processor (6M Cache, 1.60 GHz) with 4Gb RAM running a 64bit Windows 7.
• The benchmark consists in calling the encode and decode operations 10M times each with input values 2³² -1 and 23527 (a random value I picked for the test) using bases 2, 10 and 64. A more rigorous benchmark should have a wider range of input values.

Despite these reservations, I still find the results of this benchmark worth considering and sharing. They at least show how C#, Java and Scala behave in this kind of contexts.

Results

No need to keep you more in suspense, here are the results:

Execution times are expressed in nanoseconds. The percentages denote the relative performance to the C# safe version taken as a reference. You can see that on average, compared to the C# implementation:

• The C# unsafe encode is 21% faster.
• The Scala list-based implementation is terribly slow: 101% slower for encode, 414% slower for decode.
• Scala code is 18% faster for encode and 46% faster for decode.
• Java is the fastest: 31% faster for encode and 51% faster for decode.

Discussion

It is quite surprising to see Java and Scala largely outperforming the C# –both safe and unsafe versions– on Windows. The JVM is clearly outperforming the CLR on optimizing and JIT-ting code on the fly. I must confess that I didn’t expect this one .

Using lists in the Scala code is an overkill, especially for the decode function. The delay introduced by reversing the input string and mapping it to a list is not necessary for processing it. However, the cause of the overhead in the list-based encode is a bit less obvious to me. I didn’t expect that using a buffer can outperform the singly linked list to that extent. It’s probably the buffer’s faster appending operation and/or its better locality of reference that made the difference. I’m curious to dig further and see what went wrong here. But this will be probably the topic of another blog post.

You have an opinion or comment about this article? Please drop a line here. Your feedback is much appreciated!

My lisp program crashed after hours of computation right before delivering the result due to a missing method runtime error. This would’ve NEVER happened in a statically typed language. For instance, this would’ve never happened in C++ using the latest Microsoft C++ compiler. With all the static analysis it performs, these errors are prevented. You are protected by the compiler!

Well, I don’t know how good is the MS compiler compared to other C++ compilers. Still, we’ll show how ridiculously easy it is to fool it and have the same missing method crash.

We will be using Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 16.00.30319.01 for 80×86, with “enable all warnings” and “treat warnings as errors” flags activated. We will also activate the “Enable Code analysis for C/C++ on Build” option in the “Code Analysis” project properties section and we will use “Microsoft All Rules” as a rule set.
In the code samples below, we will stick to ISO C++ with no third party libraries usage. Not even the standard C++ library.

Case 1 : Dispatch table corruption

Every C++ developer has run into this problem at least once in his life. An object gets deleted but the program keeps, somehow, a reference to it. Everything goes fine until you try to call a virtual method on this object : Boom! You’re most likely going to have a memory protection error followed by a crash. Of course, this will not happen during development or QA testing. This will happen in production when your application is most needed .

Here is a snippet that illustrates the problem.

struct Base
{
    virtual void Do() =0;
};

struct Derived : public Base
{
    virtual void Do() {}
};

void main()
{
    Derived *p = new Derived();
    delete p;
    p->Do();
}

Derived class overrides and implements an abstract pure virtual method defined in class Base. In main(), we create a Derived object on the heap; we delete it and then we try calling the virtual method. The compiler compiles this program with no single warning. Static analysis detects nothing either. However, at runtime, it’s a different story. The dispatch table used for method resolution at runtime is not valid anymore after the object is delete. Calling the virtual method will amount to accessing a memory space that we’ve just disposed; hence a runtime error.

Memory corruption is unfortunately quite common in C++ and the compiler just can’t anything about it. This is why C++ developers use tools like valgrind and purify. They also use smart pointers and/or disciplined memory usage patterns to help containing these problems. You just can’t rely on the compiler.

Case 2 : Virtual method call from constructor

Memory corruption is not mandatory to have our missing method runtime error. Let’s have a look at the following snippet:

struct Base
{
    Base() { Do(); }
    void Do() { ReallyDo(); }
    virtual void ReallyDo() =0;
};

struct Derived : public Base
{
    virtual void ReallyDo() {}
};

void main()
{
    Derived d;
}

This program will crash complaining about a “pure virtual function call”. To understand what’s happening here, we must recall that objects are constructed top down in C++: from ancestors down to children classes. To construct the object d in main(), Base is built first. Base constructor is called, it calls method Do() that calls ReallyDo(). At this point, Derived is not built yet and thus the virtual table doesn’t contain yet the address to the correct virtual method. Therefore, the call to ReallyDo() fails.

To sum up,  the program is apparently calling an abstract (not implemented) method and, again, the compiler and the static analysis engine didn’t see it coming.

But wait, does this error really happen in real life? We’ve only provided toy programs to reproduce it after all. Well, the answer is Yes, it does happen even in widely used and heavily tested applications. Here is a screenshot of a crash I had recently using Adobe Acrobat Reader.

Conclusion

The compiler cannot help you write better code or avoid bugs. This is an urban legend that, unfortunately, some developers and managers keep propagating. Probably because of the false sense of security that a "Build succeeded " compiler message gives. If you are a manager and you want your team to write better quality code, try to invest a bit more on testing, code reviews, setting up best practices etc. You are Not protected by the compiler.

Binary Heap

The heap tree is “threaded”: the tree nodes are stored in an array list provided in the constructor (line 6). For each element at index i: its left child is at index 2i+1, its right child is at 2i+2 and its parent is at (i−1)/2. These relations are respectively implemented in RightChild (line 80), LeftChild  (line 82) and Parent (line 78) methods.

If the provided list is not empty, it will be “heapified”. This is done using the Heapify method (line 40) and is similar to STL’s make_heap operation in C++.

New elements can be inserted to the heap using the Insert method (line 32). A new element will be first put at the end of the tree, then pulled up until the tree satisfies the heap property again. This operation is implemented in the HeapUp method (line 48).

The root element of the heap can be removed from the tree using the PopRoot method (line 16). The last element of the tree is then put at the root and pushed down until the tree satisfies the heap invariant. This is implemented using the HeapDown method (line 60).

public class Heap<T>
{
  private readonly IList<T> _list;
  private readonly IComparer<T> _comparer;

  public Heap(IList<T> list, int count, IComparer<T> comparer)
  {
    _comparer = comparer;
    _list = list;
    Count = count;
    Heapify();
  }

  public int Count { get; private set; }

  public T PopRoot()
  {
    if (Count == 0) throw new InvalidOperationException("Empty heap.");
    var root = _list[0];
    SwapCells(0, Count - 1);
    Count--;
    HeapDown(0);
    return root;
  }

  public T PeekRoot()
  {
    if (Count == 0) throw new InvalidOperationException("Empty heap.");
    return _list[0];
  }

  public void Insert(T e)
  {
    if (Count >= _list.Count) _list.Add(e);
    else _list[Count] = e;
    Count++;
    HeapUp(Count - 1);
  }

  private void Heapify()
  {
    for (int i = Parent(Count - 1); i >= 0; i--)
    {
      HeapDown(i);
    }
  }

  private void HeapUp(int i)
  {
    T elt = _list[i];
    while (true)
    {
      int parent = Parent(i);
      if (parent < 0 || _comparer.Compare(_list[parent], elt) > 0) break;
      SwapCells(i, parent);
      i = parent;
    }
  }

  private void HeapDown(int i)
  {
    while (true)
    {
      int lchild = LeftChild(i);
      if (lchild < 0) break;
      int rchild = RightChild(i);

      int child = rchild < 0
        ? lchild
        : _comparer.Compare(_list[lchild], _list[rchild]) > 0 ? lchild : rchild;

      if (_comparer.Compare(_list[child], _list[i]) < 0) break;
      SwapCells(i, child);
      i = child;
    }
  }

  private int Parent(int i) { return i <= 0 ? -1 : SafeIndex((i - 1) / 2); }

  private int RightChild(int i) { return SafeIndex(2 * i + 2); }

  private int LeftChild(int i) { return SafeIndex(2 * i + 1); }

  private int SafeIndex(int i) { return i < Count ? i : -1; }

  private void SwapCells(int i, int j)
  {
    T temp = _list[i];
    _list[i] = _list[j];
    _list[j] = temp;
  }
}

Quick Complexity Analysis

The complexity of HeapUp and HeapDown are bound by the depth of the (complete) tree. Hence, they have both a O(log(N)) complexity (where N is the number of elements in the heap). As a consequence, Insert and PopRoot have both the same complexity: O(log(N)).

PeekRoot doesn’t imply any structural changes and has a O(1) complexity.

The Heapify method has O(N) complexity; a detailed analysis is presented here. Therefore, constructing a heap with a non empty list of size count will cost O(count).

Heap Sort

The heap sort is quite straightforward given the binary heap above. We just build a heap with the input array, this is O(N), and then pop the root element from the heap until it is empty, this is N times O(log(N)). The presented heap sort here is done in-place, with O(1) space complexity and with O(N log(N)) worst case time complexity.

public class HeapSort<T>
{
  public static void Sort(IList<T> list, IComparer<T> comparer)
  {
    var heap = new Heap<T>(list, list.Count, comparer);
    while (heap.Count > 0)
    {
      heap.PopRoot();
    }
  }
}

Priority Queue

In fact the heap is a priority queue . We just need a wrapper (delegator) object to create the heap and the heap’s list. The code is self-explanatory.

public class PriorityQueue<T>
{
  private readonly Heap<T> _heap;

  public PriorityQueue(IComparer<T> comparer)
  {
    _heap = new Heap<T>(new List<T>(), 0, comparer);
  }

  public int Size { get { return _heap.Count; } }

  public T Top() { return _heap.PeekRoot(); }

  public void Push(T e) { _heap.Insert(e); }

  public T Pop() { return _heap.PopRoot(); }
}

Recursive Merge Sort

       public static T[] Recursive(T[] array, IComparer<T> comparer)
        {
            Recursive(array, 0, array.Length, comparer);
            return array;
        }

        private static void Recursive(T[] array, int start, int end, IComparer<T> comparer)
        {
            if (end - start <= 1) return;
            int middle = start + (end - start) / 2;

            Recursive(array, start, middle, comparer);
            Recursive(array, middle, end, comparer);
            Merge(array, start, middle, end, comparer);
        }

Iterative Merge Sort

        public static T[] Iterative(T[] array, IComparer<T> comparer)
        {
            for (int i = 1; i <= array.Length / 2 + 1; i *= 2)
            {
                for (int j = i; j < array.Length; j += 2 * i)
                {
                    Merge(array, j - i, j, Math.Min(j + i, array.Length), comparer);
                }
            }

            return array;
        }

Merge Function

The merge method below is used for both methods: recursive and iterative. It merges the two provided sub-arrays T[start, middle) and T[middle, end). The result of the merge cannot stored in the input array, it needs to be stored in a separate temporary array. This takes (end-start) memory space and will have a worst case space complexity O(n) where n is the size of the input array.

        private static void Merge(T[] array, int start, int middle, int end, IComparer<T> comparer)
        {
            T[] merge = new T[end-start];
            int l = 0, r = 0, i = 0;
            while (l < middle – start && r < end – middle)
            {
                merge[i++] = comparer.Compare(array[start + l], array[middle + r]) < 0
                    ? array[start + l++]
                    : array[middle + r++];
            }

            while (r < end – middle) merge[i++] = array[middle + r++];

            while (l < middle – start) merge[i++] = array[start + l++];

            Array.Copy(merge, 0, array, start, merge.Length);
        }

Conclusion

As opposed to other in-place sorting algorithms, merge sort needs O(n) space to perform the merging step. On the other hand, it is a stable sort and it can be easily modified to implement external sorting for big data sets that do not fit in RAM.

Chirper’s front-end is written in C# / Asp.net MVC 2.0. The back-end is based on Cassandra using the Aquiles library. The source code is freely available under the terms of the Apache License, Version 2.0.

Installation

Before going into Chirper’s design details in the next sections, you might want to play a bit with it first. For this, you will need to: (1) configure the database store with Chirper’s schema, and (2) setup Chirper in your favorite webserver that supports Asp.net.

Cassandra configuration

Simply edit Cassandra’s storage-conf.xml file and add a Chirper keyspace as follows:

<Keyspaces>
 <Keyspace Name='Chirper'>
  <ColumnFamily Name='Users' CompareWith='UTF8Type'/>
  <ColumnFamily Name='Tweets' CompareWith='UTF8Type'/>
  <ColumnFamily Name='Following' CompareWith='UTF8Type'/>
  <ColumnFamily Name='Followers' CompareWith='UTF8Type'/>
  <ColumnFamily Name='TimeLine' CompareWith='UTF8Type'/>
  <ColumnFamily Name='UserLine' CompareWith='UTF8Type'/>
...
 </Keyspace>
</Keyspaces>

We will go through the data schema in more details below. If you don’t have already a Cassandra node at hand don’t worry, it’s easy to set up. Here is how I did.

Installing the webapp

Just download the Chirper’s latest binary and unzip it somewhere on your hard drive. From your webserver configuration console create a new webapp based on the unzipped Chirper folder. The exact webapp creation steps depend on your webserver. Here are the instructions for IIS7.

The alternative would be to check-out the source code and run (or debug) Chirper from visual studio, using the development web server.

Design

Front-end

Chirper is still work in progress. However, it has already the most important features: tweeting, following other users, displaying timeline and userline, dispalying followers list etc. The front-end is really simple, nothing much to say here. Here are the main components:

• Two MVC controllers are handling all the user actions: one controller for authentication and registration (similar to the one presented here) and another controller for everything else. The views are kept very simple too.
• The model consists essentially of two classes: User and Tweet. Follower and Following are just users with a time-stamp.
• A repository class abstracts all the web application logic from the back-end operations. This is really helpful in case we might want to mock the db store for testing.

Back-end

Chirper’s schema is really straightforward and very similar to the one implemented in Twissandra. Basically, we have two main families: Users and Tweets. Four other families are introduced to join theses main families in order to implement timelines and followers lists.

Users family

All Chirper users are recorded in this family. Rows are keyed by user name. Each column name/value represents a user’s property: Name, DisplayName, Location, Password etc.

Tweets family

This family holds all Chirper tweets (or chirps ).  The rows are keyed by tweet id (a UUID). As for users, the columns represent the tweet properties.

Timeline and Userline families

A timeline (resp. userline) is simply a set of time-stamp/tweet id pairs held in rows keyed by user ids. The timestamps correspond to when the tweet was tweeted. In the userline family, the tweets are those tweeted by the user  used as row key. In the timeline family, the tweets are tweeted by the followed users.

Following and Followers families

The following (resp. followers) family joins users with their following (resp. followers) users.

Conclusion

It was surprisingly easy to interface a .NET webapp with Cassandra. I had to write a few helper functions around the Aquiles connector but it did the job very well. Chirper’s schema is NOT designed to scale. It was kept very simple for the purpose of demonstration. I am currently looking for some help on Chirper’s web design, if you feel like contributing, please drop a comment here. It would also be nice to host a running instance of Chirper on the web (as Twissandra and Retwis do). If you can help on this please contact me.

The bad news is multi-threading is hard to deal with. Really hard. What is surprising is that even some thick reference books covering the subject are mixing up concepts and adding more to the reader’s confusion. I’ve run into such an example recently while having a look at the multi-threading chapter of  C# 3.0 Cookbook, 3rd Edition –then I discovered a similar example in More Effective C#: 50 Specific Ways to Improve Your C#. We will go into the details in this post.

C# 3.0 Cookbook, 3rd Edition presents a collection of multi-threading anti-patterns, each followed by a discussion and a refactoring solution. We’ll have a look at the case presented in page 722 (a similar code is also in  More Effective C#: 50 Specific Ways to Improve Your C# page 86). We’ll see that the problem presentation and the proposed solution are both fishy and misleading.

The phantom deadlock

This is how the problem is introduced in the book:

public class DeadLock
{
  public void Method1()
  {
    lock(this)
    {
      // Do something.
    }
  }
}

public class AnotherCls
{
  public void DoSomething()
  {
    DeadLock deadLock = new DeadLock();
    lock(deadLock)
    {
      Thread thread = new Thread(deadLock.Method1) ;
      thread.Start();
      // Do some time-consuming task here.
    }
  }
}

[...] When Method1 is called, it locks the current DeadLock object. Unfortunately, any object that has access to the DeadLock class may also lock it. This is shown in AnotherCls.DoSomething(). [...]

Indeed, exposing the lock object to clients, by locking on this, can be error prone. But, contrary to what the name of the class Deadlock might suggest, there is no deadlock in this code! All what we have here is that the started thread in line 19 and the current thread have mutually exclusive sections: the lock blocks respectively in Method1 line 5 and in method DoSomething line 17. The lock object used is the one instantiated line 16.

A deadlock is “a situation where in two or more competing actions are each waiting for the other to finish, and thus neither ever does“. It is not the case here. As it is presented, this pattern doesn’t imply any deadlocks.

Wrong problem – wrong solution

Unfortunately, things get worse with the refactoring the book suggests:

[...] The DeadLock class can be rewritten, as follows to fix this problem:

public class DeadLock
{
  private object syncObj = new object( );
  public void Method1( )
  {
    lock(syncObj)
    {
      // Do something.
    }
  }
}
public class AnotherCls
{
  private object deadLockSyncObj = new object( );
  public void DoSomething( )
  {
    DeadLock deadLock = new DeadLock( );
    lock(deadLockSyncObj)
    {
      Thread thread = new Thread(deadLock.Method1) ;
      thread. Start( );
      // Do some time-consuming task here.
    }
  }
}

[...] Now in the DeadLock class, you are locking on the internal syncObj, while the DoSomething method locks on the DeadLock class instance. This resolves the deadlock condition [...]

The mutually exclusive execution condition is relaxed by using two different lock objects. Wow! This is impressive. In order to remove a “deadlock” between two threads you just remove synchronization between them and you’re done! Notice that the functional behavior of the code is deeply changed here. Now we have the two threads authorized to modify concurrently the same instance, created in line 17, at will. We have serious data race problems with the proposed solution.

Conclusion

Again, multi-threading is hard and error-prone.  One should avoid threads as much as possible. If not possible, try to avoid shared state between threads. Also, reading some formal documents to recall the fundamentals can probably be helpful. If you are still in trouble, it’s time for you to switch to a hyped functional language!

Optparse Script Template
Usage: optparse_script_template.rb [OPTIONS]
Example
optparse_script_template.rb -e val1 -i 1000 -f 2.35 -l foo,bar,baz

    -e, --enumeration ENUM           val1 : Value1 description.
                                     val2 : Value2 description.
    -i, --integer INT                An integer.
    -f, --float [FLOAT]              A float.
    -l, --list e1,e2,e3              A list of things.

    -h, --help                       Show this help message.

The code below is really straightforward. You can see that:

• Line 11: we’re using a common Ruby idiom. The script doesn’t get run when loaded from another file (or from irb).
• Line 17: the script name is not hard-coded,  File.basename($0), more robust, is used instead.
• Line 33: an optional float value is specified using square brackets: [FLOAT].
• Line 44: the destructive version of the parsing method parse! is used. This helps detecting unknown options in line 48.

class OptparseScriptTemplate
  def initialize(options)
    @enum, @int, @float, @list = options[:enum], options[:int], options[:float], options[:list]
  end

  def execute()
    puts("enum=#{@enum}|int=#{@int}|float=#{@float || '<value not set>'}|@list=#{@list}")
  end
end

if __FILE__ == $0
  require 'optparse'

  options = {}

  ARGV.options do |opts|
    script_name = File.basename($0)
    opts.banner = 'Optparse Script Template'
    opts.define_head("Usage: #{script_name} [OPTIONS]",
                     'Example', "#{script_name} -e val1 -i 1000 -f 2.35 -l foo,bar,baz")

    opts.separator('')

    # enumeration
    opts.on('-e', '--enumeration ENUM', [:val1, :val2],
            'val1 : Value1 description.',
            'val2 : Value2 description.')  { |v| options[:enum] = v }

    # integer
    opts.on('-i', '--integer INT', Integer, "An integer." )  { |v| options[:int] = v }

    # float (value optional)
    opts.on('-f', '--float [FLOAT]', Float, "A float." )  { |v| options[:float] = v }

    # list
    opts.on('-l', '--list e1,e2,e3', Array, "A list of things." )  { |v| options[:list] = v }

    opts.separator('')

    opts.on_tail('-h', '--help', 'Show this help message.') do
      puts(opts)
      exit(0)
    end
    opts.parse!()
  end

  begin
    raise "Unknown option(s) #{ARGV.join(', ')}" if ARGV.any?()
    raise "Enum option is missing" if !options.key?(:enum)
    raise "Integer option is missing" if !options.key?(:int)
    raise "Float option is missing" if !options.key?(:float)
    raise "List option is missing" if !options.key?(:list)
  rescue Exception => ex
    puts "#{ex.message()}. Please use -h or --h for usage."
    exit(1)
  end

  begin
    OptparseScriptTemplate.new(options).execute()
  rescue Exception => ex
    puts("Error while executing script: #{ex.message}")
    puts("Trace: #{ex.backtrace().join($/)}")
  end
end

Before we begin, if you are not familiar with HectorSharp, here is a gentle introduction. Note also that we will be using .NET framework 3.5 and Asp.NET MVC 1.0 in the code fragments presented here.

Interface

In our application, account management and authentication is handled by a dedicated MVC controller: AccountController. As you would expect, this controller will handle actions like: LogOn, LogOff, Register, ChangePassword etc. The actual work is not implemented in the controller methods. It is instead delegated to a separate authentication service. This separation of concerns is important as it will make the code cleaner and easier to change. Moreover, it eases controller testing as we will be able to inject a mocked disconnected authentication services into the controller.

Let’s start by specifying the contract of our authentication service:

using System.Web.Security;

public interface IAuthentificationService
{
  bool SignIn(string userName, string password, bool createPersistentCookie);

  void SignOut();

  MembershipCreateStatus SignUp(string userName, string password, string email);

  bool ChangePassword(string userName, string oldPassword, string newPassword);
}

This interface allows the basic operations needed to back up our controller actions, let’s keep it simple for the moment. We’ll refactor the code as new requirements and needs emerge during development iterations. Notice that we are reusing the MembershipCreateStatus enumeration from System.Web.Security.

Implementation

Now, time to implement the interface using Cassandra as a store for user information. Basically, each time a new user sings up we will insert his info into Cassandra, and each time a user want’s to sign in, we will look for his info in Cassandra. Before writing code, let’s have a look at the schema we will be using. For the purpose of demonstration, we will simplify things.  We will use a single column family, keyed by user names, to hold the user information: name, password and email. Here is an example of what a column family row would look like:

Our column family is defined in the storage-conf.xml as follows:

<Keyspace Name="Chripper">
  <ColumnFamily Name="Users" CompareWith="UTF8Type" />
</Keyspace>

Based on this schema, the authentication service implementation is straightforward. We will use FormsAuthentication for maintaining the authentication cookie and focus on the membership management.

using System.Web.Security;
using HectorSharp;

public class AuthentificationService : IAuthentificationService
{
  private static readonly ColumnPath UsersPasswordPath =
    new ColumnPath("Users", null, "Password");
  private static readonly ColumnPath UsersNamePath =
    new ColumnPath("Users", null, "Name");
  private static readonly ColumnPath UsersEmailPath =
    new ColumnPath("Users", null, "Email");
  private readonly ICassandraClient client;

  public AuthentificationService(ICassandraClient cassandraClient)
  {
    client = cassandraClient;
  }

  private IKeyspace Keyspace { get { return client.GetKeyspace("Chirpper"); } }

  public bool SignIn(string userName, string password, bool createPersistentCookie)
  {
    if (!ValidateUser(userName, password))
    {
      return false;
    }

    FormsAuthentication.SetAuthCookie(userName, createPersistentCookie);
    return true;
  }

  public void SignOut()
  {
    FormsAuthentication.SignOut();
  }

  public MembershipCreateStatus SignUp(string userName, string password, string email)
  {
    Column nameColumn;
    if (Keyspace.TryGetColumn(userName, UsersNamePath, out nameColumn))
    {
      return MembershipCreateStatus.DuplicateUserName;
    }

    Keyspace.Insert(userName, UsersNamePath, userName);
    Keyspace.Insert(userName, UsersPasswordPath, password);
    Keyspace.Insert(userName, UsersEmailPath, email);
    return MembershipCreateStatus.Success;
  }

  public bool ChangePassword(string userName, string oldPassword, string newPassword)
  {
    if (!ValidateUser(userName, oldPassword))
    {
      return false;
    }

    Keyspace.Insert(userName, UsersPasswordPath, newPassword);
    return true;
  }

  private bool ValidateUser(string userName, string password)
  {
    Column passwordColumn;
    if (Keyspace.TryGetColumn(userName, UsersPasswordPath, out passwordColumn))
    {
      return string.Equals(password, passwordColumn.Value);
    }

      return false;
    }
}

Conclusion

We implemented an authentication service using Cassandra as a store for user information. This implementation is a first iteration and chances are that we would like to have a richer schema (enable searching users by email, add a password question etc.) and to add more functionality to the service (control password strength, hash or crypt stored passwords etc.). We might also want to refactor the code that queries and updates the Cassandra store to a separate class. This class will act as a mediator between Cassandra and our application code (aka a repository).

Epilogue

If you are curious on how the MVC controller would consume our authentication service, here is a sample implementation:

using System;
using System.Security.Principal;
using System.Web.Mvc;
using System.Web.Security;

[HandleError]
public class AccountController : Controller
{
  private readonly AccountInputValidator validator;

  public AccountController()
    : this(null)
  {
  }

  public AccountController(IAuthentificationService authentificationService)
  {
    AuthentificationService = authentificationService
      ?? new AuthentificationService(CassandraClients.Make());
    validator = new AccountInputValidator(ModelState);
  }

  private IAuthentificationService AuthentificationService { get; set; }

  public ActionResult LogOn()
  {
    return View();
  }

  [AcceptVerbs(HttpVerbs.Post)]
  public ActionResult LogOn(string userName, string password, bool rememberMe, string returnUrl)
  {
    if (!validator.ValidateLogOn(userName, password))
    {
      return View();
    }

    if (!AuthentificationService.SignIn(userName, password, rememberMe))
    {
      ModelState.AddModelError("_FORM", "The username or password provided is incorrect.");
    }

    if (!string.IsNullOrEmpty(returnUrl))
    {
      return Redirect(returnUrl);
    }

    return RedirectToAction("Index", "Home");
  }

  public ActionResult LogOff()
  {
    AuthentificationService.SignOut();
    return RedirectToAction("Index", "Home");
  }

  public ActionResult Register()
  {
    return View();
  }

  [AcceptVerbs(HttpVerbs.Post)]
  public ActionResult Register(string userName, string email, string password, string confirmPassword)
  {
    if (validator.ValidateRegistration(userName, email, password, confirmPassword))
    {
      var status = AuthentificationService.SignUp(userName, password, email);
      if (status == MembershipCreateStatus.Success)
      {
        return RedirectToAction("Index", "Home");
      }

      ModelState.AddModelError("_FORM", AuthentificationStatus.ToString(status));
    }

    // If we got this far, something failed, redisplay form
    return View();
  }

  [Authorize]
  public ActionResult ChangePassword()
  {
    return View();
  }

  [Authorize]
  [AcceptVerbs(HttpVerbs.Post)]
  public ActionResult ChangePassword(string currentPassword, string newPassword, string confirmPassword)
  {
    if (!validator.ValidateChangePassword(currentPassword, newPassword, confirmPassword))
    {
      return View();
    }

    try
    {
      if (AuthentificationService.ChangePassword(User.Identity.Name, currentPassword, newPassword))
      {
        return View("ChangePasswordSuccess");
      }

      ModelState.AddModelError("_FORM", "The current password is incorrect or the new password is invalid.");
      return View();
    }
    catch
    {
      ModelState.AddModelError("_FORM", "The current password is incorrect or the new password is invalid.");
      return View();
    }
  }

  protected override void OnActionExecuting(ActionExecutingContext filterContext)
  {
    if (filterContext.HttpContext.User.Identity is WindowsIdentity)
    {
      throw new InvalidOperationException("Windows authentication is not supported.");
    }
  }
}