Sipping from the twitter stream

I’ve been playing around with connecting to twitter’s streaming API, and displaying a live stream of tweets returned.
To do this, I was originally using DotNetOpenAuth, along with the HttpClient, which worked fine for the sample stream, but would return authentication errors for the filtered stream. I looked at the HTML message in fiddler2, and the oauth parameters weren’t ordered, which twitter requires. Instead, I’m using TwitterDoodle, which uses HttpClient.
The C#5/.NET4.5 async/await code is quite elegant – it’s not CPU intensive so it’s fine running on the UI thread, without blocking. My first instinct prior to C#5 would have been to use RX for this, but now if I’m doing something simple I’d stick to async/await, only using RX if doing something more complex like buffering or batching.

    private async void ProcessTweets()
      using (var t = new TwitterConnector())
        var response = await t.GetSampleFirehoseConnection();
        var res = await response.Content.ReadAsStreamAsync();
        using (var streamReader = new StreamReader(resEncoding.UTF8))
          // streamReader.EndOfStream can block, so instead check for null
          while (!cts.IsCancellationRequested)
            var r = await streamReader.ReadLineAsync();
            if (r == null) { return; }

    private void ProcessTweetText(string r)
      if (!string.IsNullOrEmpty(r))
        var tweetJToken = JsonConvert.DeserializeObject<dynamic>(r);
        var tweetObj = tweetJToken["text"];
        if (tweetObj != null)
          var tweetText = tweetObj.ToString();

The equivalent F# async code obviously looks quite similar, with the added goodness of Type Providers. Time dependent, I am planning to do some more analysis of tweets which would be a good fit for F#.

type tweetProvider = JsonProvider<"SampleTweet.json"SampleList=true>

type MainWindowViewModel() =
  inherit ViewModelBase()
  let items = new ObservableCollection<string>()
  member x.Items
    with get () = items
  member x.ProcessTweet tweet =
    let tweetParsed = tweetProvider.Parse(tweet)
    match tweetParsed.Text with
    | Some(v->  x.Items.Add v
    | None -> ()
  member x.ProcessTweets =
    let a = async {
      use t = new TwitterConnector()
      let! response = t.GetSampleFirehoseConnection() |> Async.AwaitTask
      let! result = response.Content.ReadAsStreamAsync() |> Async.AwaitTask
      use streamReader = new StreamReader(resultEncoding.UTF8)
      let rec processTweetsAsync (s:StreamReader=
        async {
          let! r = s.ReadLineAsync() |> Async.AwaitTask
          // streamReader.EndOfStream can block, so instead check for null
          if r <> null then
            x.ProcessTweet r
            return! processTweetsAsync s
      do! processTweetsAsync streamReader
    a |> Async.StartImmediate
  member x.GoCommand = 
    new RelayCommand ((fun canExecute -> true), 
      (fun action -> x.ProcessTweets))

Picking with DirectX 11 and Bullet Physics

I updated the falling cubes demo to separate out the rendering and physics into separate WinRT components – see the WinRT subfolder under . The performance of this wasn’t noticeably different when consumed by a WP8 DirectX application, but I then switched over to a “DirectX with XAML application”, so that the application was hosted via C#, and consumed the WinRT components via a DrawingSurfaceBackgroundGrid, and performance fell off a cliff.

I created the C# host as I intended to put as much game logic as I could into an F# portable library, and the only way to do this is to have a C# hosting application assembly as WP8 apps, unlike Windows Store apps, don’t allow WinRT(P) components to be created in anything other than C++.

Instead, of fighting the performance issues, I decided that it would be better to implement the game logic purely in C++ – see the WinRT subfolder under . This replaces most uses of the C++/CX extensions with standard C++ types, and implements picking, loosely based off

It gives a childish sense of satisfaction to do something as simple as destroy a wall of bricks so that the wall collapses.

Once I get time, the next step is to make the wall collapse more realistically by adding spring constraints between adjacent blocks, and to remove those springs once they are stretched beyond some elastic limit.

Bullet Physics demo DirectX C++ app on Windows Phone 8

I’ve finally gotten around to updating my original Windows Store Bullet Physics demo ( to build for the Windows 8 final release.

While I was at it, I also got it working for Windows Phone 8 – code is on github There weren’t many changes. In building Bullet for ARM, I got around lots of parameter alignment issues by defining BT_USE_DOUBLE_PRECISION. This was the quickest thing to do, and may have performance implications but the performance running on my Nokia Lumia 920 seems encouraging. It wouldn’t have been a much bigger change to instead see if ARM is defined.

ViewGene now has mapping

The newly-released version of my genealogy/family tree Windows 8 app now has Bing Maps integration, which allows the migration paths of all ancestors to be plotted.

I wanted to see visually how my ancestors moved, after researching how my ancestors moved from villages in the 17th century into towns, and then larger towns.

I’d love to see anyone else’s migration paths!

The below is just a made up sample – I’ll share my map after sanitising the data.


Try it in the Windows Store.

Option pricing in F# using the Explicit Finite Difference method

It’s been a little while since I’ve coded any F#, so I’ve done a little coding kata to polish off the rust.

I’ve converted the explicit finite difference option pricing example from (Paul Wilmott Introduces Quantitative Finance).


I followed similar steps as those I performed in my previous blog post on the binomial tree option pricing; I converted the VBA code into very similar looking imperative F# code, before refactoring out the loops into recursive calls.

It was a useful exercise as converting the algorithm made me actually focus on the mechanics of how it worked, much more than simply translating VBA code into imperative F#. It’d be interesting to hear in the comments whether people find it easy to read.

The code is available


This blog has been quiet for a while, but I’ve been busy working on a Windows Store genealogy application, ViewGene.

This is quite a niche application, but it was written to cope with a couple of use cases which are missing from other family tree websites and applications – it’s useful to have it open in conjunction with your favourite family tree software or website.

The first problem is that many of the family tree websites, when showing the whole tree (pedigree chart) of ancestors, compress the tree to make the best use of screen real estate. This is fine to save paper, but it does make it very difficult, when navigating the tree, to know what the generation is of any ancestor on the screen.


ViewGene ensures that ancestors of any generation all appear at the same vertical level – this makes it easy to see which branches of the tree need more investigation. This view wouldn’t necessarily work well when printed out, but touch-based pan and zoom do mean that it’s easy to zoom out to see the ‘shape’ of the tree, and to zoom in to see the individual details, meaning that it works for interactively investigating the data.

The second use-case is to validate and see the lifelines of the ancestors. Many of the family tree websites allow free-form entry of dates. This means that it is easy to mistype and enter a date of the wrong century, or to enter semantically incorrect dates such as a child born before the father. The Timeline Fan chart allows for some of these problems to be easily seen. The chart also allows the user to see, for example, which ancestors were alive on a given census year.


The current release includes very limited validation, to check that dates match allowable GEDCOM formats, but does not check for semantically incorrect dates. Also, there is very basic date inference for missing dates. If I devote any more time to the application, I’ll want to improve both of these points – it should be possible to validate that children were conceived whilst both parents were alive, and that siblings can’t have been born in the same gestation period etc.

Windows Store application observations

The chart functionality is quite generic, and would work well on any touch-screen platform (such as iOS), much of the development effort was spent in building an application following Windows Store idioms.

The first was to ensure that the application works when snapped (though this could do with further improvement). The second was to use semantic zoom, which definitely makes it easier to navigate through a large number of individuals in the GEDCOM file). Adding a very large number of individuals to the GridView results in Invalid Quota exceptions being thrown. Even if this weren’t the case, there would be a usability issue in navigating through so many individuals. To fix this, if there are a large number of individuals in the GEDCOM file the application adds a further level of navigation through the first letter of the surname.

It would be quite easy to add images to the application (such as pictures of ancestors, or scans or pdfs of documentation), but it is quite tricky in a Windows Store application – the GEDCOM file contains paths to images, but it is not possible from a Windows Store application to open files other than via the open file dialog. It may be possible to work around that by instead presenting an open folder dialog (which would allow access to all files and sub-folders within that folder) and then allowing the user to choose the GEDCOM file from a list, but that definitely feels clunky.

The Pedigree Chart is implemented as an ItemsControl, populated with either Boxes or Lines, with an ItemsTemplateSelector to distinguish between them. It works well, and there are no obvious performance issues, but the ItemsControl does crash if too many items are added, so the number of ancestors are limited to 1000.

The timeline fan is implemented as C++/CX WinRT component, deriving from SurfaceImageSource, with all drawing being performed in Direct2D. If I get time to invest in this, I want to switch out to a VirtualSurfaceImageSource, so that the amount of information presented is customised to the zoom level on the screen.

Non Windows Store issues

There are a few issues not related to a Windows Store application.

The application currently does not have editing – the GEDCOM format is too flaky to be a reliable interchange format; it does not round-trip without the possibility of corrupting the user’s information. Ideally, the application would allow connecting to a website to obtain the user’s family tree information, but does not provide a user-accessible API. does, and if I get time I intend to investigate that.

It is also trickier than it needs to be to implement mapping – it looks fairly easy to integrate with Bing maps, but the problem is that location data in the GEDCOM file doesn’t necessarily match up with the required data for plotting points on the map. It is possible to ask the data to enter that information, but then the application should maintain it. This isn’t too much of a problem, but it should be maintained per GEDCOM file, and to allow the user to re-import updated GEDCOM files would rely on GEDCOM merging.

The GEDCOM parser is written in C#, mostly as a GoF state machine, with some functional elements. When I originally started coding this up, I intended to start by creating a clunky procedural looking parser (which maintains the state with many flags), then show how an OO GoF pattern was nicer, then show how a F# implementation is nicer still. If I get time I still intend to create a F# version of the parser.

Speaking of F#, the layout logic for the timeline fan is implemented in a functional way, but would be nicer in F# – the problem is that it then needs to communicate with the WinRT component to do the drawing. F# can’t directly interop with WinRT, so it would be necessary to create a C# wrapper or consumer of the F# library just for this – and it seems overkill.

Overall, developing a Windows Store app is fairly easy if you have Silverlight or WPF skills. There are quite a number of features I want to add to the application, but how much time I’m going to dedicate to it depends on feedback or download numbers of the current version of the app.

Dalmatinska pašticada

Pašticada is a beef pot roast, requiring many hours of preparation, so is usually served at Christmas or Easter.

This recipe quite closely follows the recipe here.

The beef cut in Croatian recipes is usually described as top round, but butchers in the UK who were shown the pictures from Croatian recipe books say that the cut of beef to get is Silverside:



Carefully trim off the white fat that the butchers leave on. Spike the beef all around with a knife, and insert pieces of garlic and pancetta:



Marinate in a bowl of red wine vinegar. Cover in a fridge, and leave to marinate for at least overnight, and ideally 24 hours, turning occasionally.

After marinating, dry off the beef, and fry off in a mixture of lard and olive oil until brown on each side. Remove from the heat, and fry off a kilo of finely chopped onions and some pancetta in the same oil. Once softened, return the beef along with chopped carrots, celery, parsley, a handful of dried prunes, a whole bulb of garlic, a few cloves, and a few peppercorns.

After half an hour, add some beef stock, half a bottle or so of full bodied red wine, a spoon of fruit jam, and some tomato concentrate. Leave to cook on a low heat for two to three hours.

Once the meat is cooked, remove from the pan and slice:


Remove the juice, and put through a colander and then push through a fine sieve.


You don’t want to liquidise the mixture, as it will end up thicker and bittier than desired, instead of a really rich gravy. This is what’s left after sieving:


Now, return the beef slices and gravy back to the pan and cook for another hour.

Serve with potato gnocchi  and a dusting of parmesan cheese:


Francuska Salata

In Croatia, Christmas dinner is a multi-course extravaganza. First off, a beef soup is served, followed by starters, then a pot roast, followed by roast veal and potatoes, and then a dessert.

The starter is typically a platter of cured meats and cheeses, bread, and Francuska Salata. Francuska Salata literally translates as French Salad, but actually translates as Olivier Salad. It is a dangerous move to over-fill on starters as it will be a struggle to eat later courses.

To make Francuska Salata, cook carrots and potatoes and dice finely.


Mix with peas, chopped boiled egg, finely chopped gherkins, salt, pepper, lemon juice and mayonnaise.


Serve with bread, meat and cheeses, but don’t eat too much!



Similar to fritule, Bakalar is eaten in Croatia on days of fast; especially on Good Friday and Christmas Eve.

Bakalar is known in the UK as salt cod, the cod is salted and dried for preservation. In Croatia, they sell it in the supermarkets on the shelves as it is fully dried out (see here for an example). In the UK salt cod can be bought from Italian Delicatessens, but tends to be not quite as dried out.

The bakalar needs soaking for two to three days in water to rehydrate and remove the salt. Change the water frequently, and keep covered in a fridge.


Remove as much of the skin and bones as you can, then place the code in a saucepan and bring to the boil. Once it is soft enough, remove the flesh from the bones. Remove from the pan, but retain the liquid.

Peel a few potatoes, slice, and alternate layers of potato and cod in the saucepan.

When cooked, add plenty of good extra virgin olive oil, enough to make even Jamie Oliver embarrassed. The olive oil is one of the key ingredients in this dish, and needs to be good quality – look for a good green colour and fresh smell. We’re lucky to get home-pressed olive oil from Marijana’s relatives in Croatia.


Don’t even think of mashing the mixture. In Croatia, the tradition is to firmly grasp the lid of the saucepan and shake vigorously; this breaks up the potatoes and mixes in the cod.


Add a generous handful of parsley and four to five clothes of finely chopped garlic, and mix well together.


The starch released from the potato, and the oil give a creamy consistency, and this is truly delicious. Season to taste, and enjoy with a glass of good white wine.



Fritule are similar to mini doughnuts. In Croatia, it is traditional to eat fritule on days of fast, such as Good Friday or Christmas Eve.

Soak a handful of raisins in rum. Beat two eggs, add 50g of vanilla sugar and beat in 200g of yogurt. Then, mix in 300g of self-raising flour and then the rum and raisins. You can also add other flavours, such as cinnamon or lemon rind.


Heat a pan of oil, to a medium-high heat; you want the fritule to cook fully throughout before the outer exterior gets too dark.


Turn once while cooking so that each side is evenly cooked, before removing with a slotted spoon and leaving to dry on kitchen towel. Finish by dusting with icing sugar.