RunKeeper Visualisations

My last post described how I’m using the Twitter API to receive tweets off the live stream.

Since then, I’ve used the API to filter for tweets containing the #runkeeper hashtag, and used that to scrape the user’s activity from the RunKeeper site (including the GPS points from the user’s exercise). I’ve stored that information in a MongoDB, which has allowed me to do some simple visualisation:

The above video (best played at 720p) shows activities plotted against time of day (the sun overhead in the video indicates midday for that region).

I haven’t charted this to confirm, but to my eyes it looks like amount of exercise activity peaks around sunrise and sunset, with almost none at night time (which isn’t really a surprise).

For the curious, this is what the colour of the dots indicate:

let createSphere (latitude:float) (longitude:float) (activityType:string=
    let color = match activityType with
                | "RUN" -> "Red"
                | "WALK" -> "Green"
                | "BIKE" -> "Blue"
                | "MOUNTAINBIKE" -> "Orange"
                | "SKATE" -> "Yellow"
                | "ROWING" -> "Cyan"
                | "HIKE" -> "Brown"
                | "OTHER" -> "Black"
                | _ -> "Black"

On a local scale, the actual GPS traces are of interest:


Activities are present in the more-populated regions of the UK. The blue and orange traces indicating cycling and mountain biking activities are clearly visible.

NOTE: if you think the map looks weird, it doesn’t have the usual aspect ratio – I’m not using a Mercator projection do plot the points, but am simply plotting the longitude and latitude linearly.

On an even smaller scale, landmarks around London are visible:



The GPS data contains altitude information, so there are more interesting visualisations that could be done, including generating contour plots. Also, the above only contains three days worth of data – with a larger data set it would be possible to plot to determine whether activities peak on a weekend etc.



The 3D plot of the globe was drawn using POV-Ray. The 3D globe model was from grabcad, with the converted to a POV-Ray model using PoseRay.

The UK outline was obtained from the NOAA.

The code was implemented in F# (which was a pleasure to get back to after the C++ I’ve been doing recently). I did try the MongoDB.FSharp library to store my data records, but they failed to deserialise from the database. In any case, I wanted more control over the data types saved (I wanted to store the GPS data as GeoJson3DGeographicCoordinates, with the first point stored separately as a GeoJson2DGeographicCoordinates with an index on this value). I could have created .NET classes and used the BSON serializer, but it seemed more effort than writing directly to the DB (and this is about the only time I’ve seen the benefit of C# implicit conversions, but I can live without them in F#).

Why use the Twitter API, and why not scrape the RunKeeper site directly? That’s because of times – the RunKeeper website displays the activity time, but it is displayed in local time, and it’s not clear whether that’s in the user’s timezone, or the timezone of the activity. It seems cleaner to instead assume that the tweet has been posted as soon as the activity is finished and store that time as UTC (this assumption may of course not be true, but the results seem realistic).

Show me the Code!

The code’s not in a bad shape, but I would like to tidy it up a little before releasing it into the world. I’m busy with other things at the moment, but if I get much interest I can go ahead and do that…

Sipping from the twitter stream

I’ve been playing around with connecting to twitter’s streaming API, and displaying a live stream of tweets returned.
To do this, I was originally using DotNetOpenAuth, along with the HttpClient, which worked fine for the sample stream, but would return authentication errors for the filtered stream. I looked at the HTML message in fiddler2, and the oauth parameters weren’t ordered, which twitter requires. Instead, I’m using TwitterDoodle, which uses HttpClient.
The C#5/.NET4.5 async/await code is quite elegant – it’s not CPU intensive so it’s fine running on the UI thread, without blocking. My first instinct prior to C#5 would have been to use RX for this, but now if I’m doing something simple I’d stick to async/await, only using RX if doing something more complex like buffering or batching.

    private async void ProcessTweets()
      using (var t = new TwitterConnector())
        var response = await t.GetSampleFirehoseConnection();
        var res = await response.Content.ReadAsStreamAsync();
        using (var streamReader = new StreamReader(resEncoding.UTF8))
          // streamReader.EndOfStream can block, so instead check for null
          while (!cts.IsCancellationRequested)
            var r = await streamReader.ReadLineAsync();
            if (r == null) { return; }

    private void ProcessTweetText(string r)
      if (!string.IsNullOrEmpty(r))
        var tweetJToken = JsonConvert.DeserializeObject<dynamic>(r);
        var tweetObj = tweetJToken["text"];
        if (tweetObj != null)
          var tweetText = tweetObj.ToString();

The equivalent F# async code obviously looks quite similar, with the added goodness of Type Providers. Time dependent, I am planning to do some more analysis of tweets which would be a good fit for F#.

type tweetProvider = JsonProvider<"SampleTweet.json"SampleList=true>

type MainWindowViewModel() =
  inherit ViewModelBase()
  let items = new ObservableCollection<string>()
  member x.Items
    with get () = items
  member x.ProcessTweet tweet =
    let tweetParsed = tweetProvider.Parse(tweet)
    match tweetParsed.Text with
    | Some(v->  x.Items.Add v
    | None -> ()
  member x.ProcessTweets =
    let a = async {
      use t = new TwitterConnector()
      let! response = t.GetSampleFirehoseConnection() |> Async.AwaitTask
      let! result = response.Content.ReadAsStreamAsync() |> Async.AwaitTask
      use streamReader = new StreamReader(resultEncoding.UTF8)
      let rec processTweetsAsync (s:StreamReader=
        async {
          let! r = s.ReadLineAsync() |> Async.AwaitTask
          // streamReader.EndOfStream can block, so instead check for null
          if r <> null then
            x.ProcessTweet r
            return! processTweetsAsync s
      do! processTweetsAsync streamReader
    a |> Async.StartImmediate
  member x.GoCommand = 
    new RelayCommand ((fun canExecute -> true), 
      (fun action -> x.ProcessTweets))

Picking with DirectX 11 and Bullet Physics

I updated the falling cubes demo to separate out the rendering and physics into separate WinRT components – see the WinRT subfolder under . The performance of this wasn’t noticeably different when consumed by a WP8 DirectX application, but I then switched over to a “DirectX with XAML application”, so that the application was hosted via C#, and consumed the WinRT components via a DrawingSurfaceBackgroundGrid, and performance fell off a cliff.

I created the C# host as I intended to put as much game logic as I could into an F# portable library, and the only way to do this is to have a C# hosting application assembly as WP8 apps, unlike Windows Store apps, don’t allow WinRT(P) components to be created in anything other than C++.

Instead, of fighting the performance issues, I decided that it would be better to implement the game logic purely in C++ – see the WinRT subfolder under . This replaces most uses of the C++/CX extensions with standard C++ types, and implements picking, loosely based off

It gives a childish sense of satisfaction to do something as simple as destroy a wall of bricks so that the wall collapses.

Once I get time, the next step is to make the wall collapse more realistically by adding spring constraints between adjacent blocks, and to remove those springs once they are stretched beyond some elastic limit.

Bullet Physics demo DirectX C++ app on Windows Phone 8

I’ve finally gotten around to updating my original Windows Store Bullet Physics demo ( to build for the Windows 8 final release.

While I was at it, I also got it working for Windows Phone 8 – code is on github There weren’t many changes. In building Bullet for ARM, I got around lots of parameter alignment issues by defining BT_USE_DOUBLE_PRECISION. This was the quickest thing to do, and may have performance implications but the performance running on my Nokia Lumia 920 seems encouraging. It wouldn’t have been a much bigger change to instead see if ARM is defined.

ViewGene now has mapping

The newly-released version of my genealogy/family tree Windows 8 app now has Bing Maps integration, which allows the migration paths of all ancestors to be plotted.

I wanted to see visually how my ancestors moved, after researching how my ancestors moved from villages in the 17th century into towns, and then larger towns.

I’d love to see anyone else’s migration paths!

The below is just a made up sample – I’ll share my map after sanitising the data.


Try it in the Windows Store.

Option pricing in F# using the Explicit Finite Difference method

It’s been a little while since I’ve coded any F#, so I’ve done a little coding kata to polish off the rust.

I’ve converted the explicit finite difference option pricing example from (Paul Wilmott Introduces Quantitative Finance).


I followed similar steps as those I performed in my previous blog post on the binomial tree option pricing; I converted the VBA code into very similar looking imperative F# code, before refactoring out the loops into recursive calls.

It was a useful exercise as converting the algorithm made me actually focus on the mechanics of how it worked, much more than simply translating VBA code into imperative F#. It’d be interesting to hear in the comments whether people find it easy to read.

The code is available


This blog has been quiet for a while, but I’ve been busy working on a Windows Store genealogy application, ViewGene.

This is quite a niche application, but it was written to cope with a couple of use cases which are missing from other family tree websites and applications – it’s useful to have it open in conjunction with your favourite family tree software or website.

The first problem is that many of the family tree websites, when showing the whole tree (pedigree chart) of ancestors, compress the tree to make the best use of screen real estate. This is fine to save paper, but it does make it very difficult, when navigating the tree, to know what the generation is of any ancestor on the screen.


ViewGene ensures that ancestors of any generation all appear at the same vertical level – this makes it easy to see which branches of the tree need more investigation. This view wouldn’t necessarily work well when printed out, but touch-based pan and zoom do mean that it’s easy to zoom out to see the ‘shape’ of the tree, and to zoom in to see the individual details, meaning that it works for interactively investigating the data.

The second use-case is to validate and see the lifelines of the ancestors. Many of the family tree websites allow free-form entry of dates. This means that it is easy to mistype and enter a date of the wrong century, or to enter semantically incorrect dates such as a child born before the father. The Timeline Fan chart allows for some of these problems to be easily seen. The chart also allows the user to see, for example, which ancestors were alive on a given census year.


The current release includes very limited validation, to check that dates match allowable GEDCOM formats, but does not check for semantically incorrect dates. Also, there is very basic date inference for missing dates. If I devote any more time to the application, I’ll want to improve both of these points – it should be possible to validate that children were conceived whilst both parents were alive, and that siblings can’t have been born in the same gestation period etc.

Windows Store application observations

The chart functionality is quite generic, and would work well on any touch-screen platform (such as iOS), much of the development effort was spent in building an application following Windows Store idioms.

The first was to ensure that the application works when snapped (though this could do with further improvement). The second was to use semantic zoom, which definitely makes it easier to navigate through a large number of individuals in the GEDCOM file). Adding a very large number of individuals to the GridView results in Invalid Quota exceptions being thrown. Even if this weren’t the case, there would be a usability issue in navigating through so many individuals. To fix this, if there are a large number of individuals in the GEDCOM file the application adds a further level of navigation through the first letter of the surname.

It would be quite easy to add images to the application (such as pictures of ancestors, or scans or pdfs of documentation), but it is quite tricky in a Windows Store application – the GEDCOM file contains paths to images, but it is not possible from a Windows Store application to open files other than via the open file dialog. It may be possible to work around that by instead presenting an open folder dialog (which would allow access to all files and sub-folders within that folder) and then allowing the user to choose the GEDCOM file from a list, but that definitely feels clunky.

The Pedigree Chart is implemented as an ItemsControl, populated with either Boxes or Lines, with an ItemsTemplateSelector to distinguish between them. It works well, and there are no obvious performance issues, but the ItemsControl does crash if too many items are added, so the number of ancestors are limited to 1000.

The timeline fan is implemented as C++/CX WinRT component, deriving from SurfaceImageSource, with all drawing being performed in Direct2D. If I get time to invest in this, I want to switch out to a VirtualSurfaceImageSource, so that the amount of information presented is customised to the zoom level on the screen.

Non Windows Store issues

There are a few issues not related to a Windows Store application.

The application currently does not have editing – the GEDCOM format is too flaky to be a reliable interchange format; it does not round-trip without the possibility of corrupting the user’s information. Ideally, the application would allow connecting to a website to obtain the user’s family tree information, but does not provide a user-accessible API. does, and if I get time I intend to investigate that.

It is also trickier than it needs to be to implement mapping – it looks fairly easy to integrate with Bing maps, but the problem is that location data in the GEDCOM file doesn’t necessarily match up with the required data for plotting points on the map. It is possible to ask the data to enter that information, but then the application should maintain it. This isn’t too much of a problem, but it should be maintained per GEDCOM file, and to allow the user to re-import updated GEDCOM files would rely on GEDCOM merging.

The GEDCOM parser is written in C#, mostly as a GoF state machine, with some functional elements. When I originally started coding this up, I intended to start by creating a clunky procedural looking parser (which maintains the state with many flags), then show how an OO GoF pattern was nicer, then show how a F# implementation is nicer still. If I get time I still intend to create a F# version of the parser.

Speaking of F#, the layout logic for the timeline fan is implemented in a functional way, but would be nicer in F# – the problem is that it then needs to communicate with the WinRT component to do the drawing. F# can’t directly interop with WinRT, so it would be necessary to create a C# wrapper or consumer of the F# library just for this – and it seems overkill.

Overall, developing a Windows Store app is fairly easy if you have Silverlight or WPF skills. There are quite a number of features I want to add to the application, but how much time I’m going to dedicate to it depends on feedback or download numbers of the current version of the app.

Dalmatinska pašticada

Pašticada is a beef pot roast, requiring many hours of preparation, so is usually served at Christmas or Easter.

This recipe quite closely follows the recipe here.

The beef cut in Croatian recipes is usually described as top round, but butchers in the UK who were shown the pictures from Croatian recipe books say that the cut of beef to get is Silverside:



Carefully trim off the white fat that the butchers leave on. Spike the beef all around with a knife, and insert pieces of garlic and pancetta:



Marinate in a bowl of red wine vinegar. Cover in a fridge, and leave to marinate for at least overnight, and ideally 24 hours, turning occasionally.

After marinating, dry off the beef, and fry off in a mixture of lard and olive oil until brown on each side. Remove from the heat, and fry off a kilo of finely chopped onions and some pancetta in the same oil. Once softened, return the beef along with chopped carrots, celery, parsley, a handful of dried prunes, a whole bulb of garlic, a few cloves, and a few peppercorns.

After half an hour, add some beef stock, half a bottle or so of full bodied red wine, a spoon of fruit jam, and some tomato concentrate. Leave to cook on a low heat for two to three hours.

Once the meat is cooked, remove from the pan and slice:


Remove the juice, and put through a colander and then push through a fine sieve.


You don’t want to liquidise the mixture, as it will end up thicker and bittier than desired, instead of a really rich gravy. This is what’s left after sieving:


Now, return the beef slices and gravy back to the pan and cook for another hour.

Serve with potato gnocchi  and a dusting of parmesan cheese:


Francuska Salata

In Croatia, Christmas dinner is a multi-course extravaganza. First off, a beef soup is served, followed by starters, then a pot roast, followed by roast veal and potatoes, and then a dessert.

The starter is typically a platter of cured meats and cheeses, bread, and Francuska Salata. Francuska Salata literally translates as French Salad, but actually translates as Olivier Salad. It is a dangerous move to over-fill on starters as it will be a struggle to eat later courses.

To make Francuska Salata, cook carrots and potatoes and dice finely.


Mix with peas, chopped boiled egg, finely chopped gherkins, salt, pepper, lemon juice and mayonnaise.


Serve with bread, meat and cheeses, but don’t eat too much!



Similar to fritule, Bakalar is eaten in Croatia on days of fast; especially on Good Friday and Christmas Eve.

Bakalar is known in the UK as salt cod, the cod is salted and dried for preservation. In Croatia, they sell it in the supermarkets on the shelves as it is fully dried out (see here for an example). In the UK salt cod can be bought from Italian Delicatessens, but tends to be not quite as dried out.

The bakalar needs soaking for two to three days in water to rehydrate and remove the salt. Change the water frequently, and keep covered in a fridge.


Remove as much of the skin and bones as you can, then place the code in a saucepan and bring to the boil. Once it is soft enough, remove the flesh from the bones. Remove from the pan, but retain the liquid.

Peel a few potatoes, slice, and alternate layers of potato and cod in the saucepan.

When cooked, add plenty of good extra virgin olive oil, enough to make even Jamie Oliver embarrassed. The olive oil is one of the key ingredients in this dish, and needs to be good quality – look for a good green colour and fresh smell. We’re lucky to get home-pressed olive oil from Marijana’s relatives in Croatia.


Don’t even think of mashing the mixture. In Croatia, the tradition is to firmly grasp the lid of the saucepan and shake vigorously; this breaks up the potatoes and mixes in the cod.


Add a generous handful of parsley and four to five clothes of finely chopped garlic, and mix well together.


The starch released from the potato, and the oil give a creamy consistency, and this is truly delicious. Season to taste, and enjoy with a glass of good white wine.