Introducing mmbot, a C# Hubot port

I’ve been playing recently with github’s Hubot chat bot, written in Coffee script on Node. I wanted to connect it to our jabbr instance with the least amount of friction and to provide our team with an office jukebox, build automation and various meme-oriented distractions. There is a pretty good adapter for Hubot to do this but after having a few issues I decided it might be easier to do a port to C# so I could more easily write scripts and connect to jabbr (which is written in C# and has some ready made APIs).

The result is mmbot, a hubot port that follows the same basic architecture of Hubot with minimal changes. The basic goals of mmbot are thusly:

  1. Provide a chat bot written in C# with all the functionality of Hubot but with a script environment more familiar to .Net devs.
  2. Hubot scripts should be easy to convert into mmbot scripts.
    This may mean that there are some weird overloads in the API for writing scripts that look like hubot scripts but it should still be very usable, customizable and familiar to .Net devs.
  3. ScriptCS style scripts should be automatically picked up and run from a scripts folder
  4. Hubot scripts should be easy to convert into mmbot scripts.
    There are some blockers here in the NuGet package resolution and dynamic loading of scripts

Currently there are 2 adapters for mmbot – jabbr and HipChat. The jabbr adapter is the most used at the moment but the HipChat one should also be working.

Scripts can be written in code by implementing IMMBotScript or by dropping a scriptcs csx file into a scripts folder beside the executable. The pre-compiled approach gives you the power of async/await and the speed of using dynamic while scriptcs means you don’t need to create dll. The experience of porting Hubot scripts has so far been pretty painless as the API was designed to make this process incredibly easy.

Here is the hubot math script in scriptcs form

And as an IMMBot script

Notice that there is an Http fluent style helper for creating requests and processing the responses using HttpClient and Json.Net.

Starting mmbot

Starting mmbot is easy. You can choose to configure him by environment variables or by passing in config parameters in code


What scripts does it have?

mmbot currently has the following scripts

  • Searching for images, animated gifs, cats, pugs, maps, youtube videos
  • Urban Dictionary definitions
  • Ascii art generator
  • Mustache me (place a mustache on someone’s face)
  • Xkcd comics
  • Spotify player /office jukebox with playlist, query, album, track playing and queuing and volume control.
  • Jetbrains TeamCity build server – querying build status, starting builds etc.

Thanks to Damian Karzon for contributing some of the scripts. Check out the current catalog at http://github.com/petegoo/mmbot

What’s next?

Next for mmbot is to try to implement a few cooler features like loading and saving scripts from gists, starting from scriptcs (some issues here currently) and to get the script catalog expanded.

Go check out the mmbot code on github.

Making the SysTray transparent on Windows Phone

I’ve used a semi-transparent AppBar on windows phone a number of times to get the visual effect I wanted and to stop the dreaded jumping frame when transitioning pages but I didn’t think it would be possible with the SysTray. Turns out that the SysTray in Windows Phone can be made transparent.

This is pretty useful if you want to keep an image background that spans the entire height of the screen. Simply add the attributes below to your root PhoneApplicationPage element.

shell:SystemTray.IsVisible="True"
shell:SystemTray.ForegroundColor="Yellow"
shell:SystemTray.Opacity="0"

and the result should look like this:

systray

XAML surround templates for Resharper

Lately I’ve been more active than usual about extolling the virtues of Jetbrains Resharper. Despite the recent additions to the Visual Studio 2013 refactoring tools in my mind resharper is an absolute must have.

In the Build 2013 talk by Tim Heuer on “What’s New in XAML” he showed a neat trick of being able to surround one or more xaml elements in a StackPanel with a quick shortcut. I thought, this should be possible with resharper surround templates today so here goes.

  1. From the Resharper menu choose “Templates Explorer”

    image

  2. Select XML (no XAML  scope right now) and click the new template button
    image
  3. This should open a new document with the string “$SELECTION$” in it. This is simply the tag replacement for the currently selected text in the editor. Put a <StackPanel> on a line above and a closing </StackPanel> below.

  4. Then give the template a description like ooh, I dunno “StackPanel” and save it.
  5. In template explorer, drag that bad-boi onto the quicklist.
  6. Repeat for a Grid.

Now you can simply select a bunch of XAML and hit CTRL+E, CTRL+U. Type the number of the template in the quick list and you are done. Instant gratification.

image

Fibre driven development

So, I’ve been a bit quiet recently. There’s been no blogging and even my twitter activity has dwindled of late so what gives?

Well I quit my old job building software for the legal market and started a new role at Marker Metro, building apps exclusively for the windows eco-system (phone and win 8). During this transition I have been relatively quiet as I wrestle with imposter syndrome and ramp up, therefore I am not blogging or making authoritative statements about the stuff that I am doing. I should however probably be blogging my experiences of learning the new tech and tools. Hmmmm, note to self…

Why move?

Productivity

In my previous role I was finding myself in more and more meetings, training people more, fielding questions and generally getting distracted from the reason I do this job, writing code. Don’t get me wrong, I still got to write code and I did enjoy some these other activities to a degree but here’s the thing – you can’t write great code in the 30-45 minute breaks between other activities. You need prolonged periods of focus for that. I was getting increasingly frustrated with what I saw as a lack of productivity in myself.

Culture

I’m not going to say much here except that a company culture can wander off course gradually and almost imperceptibly but steering it back can be a mammoth (some say impossible) task.

Stagnation

I learned a heck of a lot from some of the smartest people I have ever met but the subject matter and the slowing rate of change were such that I was finding myself searching for alternative sources of challenges outside of the day-to-day. This is not such a bad thing but if I wasn’t doing that I was unhappy. I needed to be happy just working, anything else is a bonus.

So what to do?

Well, I figured that I’ve been in the same job for 6 years since moving from the other side of the world to New Zealand so I need to find out what opportunities are out there. Rumors had been circulating internally that there were no other good employers in NZ. Pah!

There is a huge problem in our industry, there are far too many developers working in silos being rockstars in their own garages impressing only their mums. Especially in the .Net community. I had a little exposure to the community through speaking at Tech-Ed NZ and user groups so I decided to expand my circle of influence and see what came of it. I went along to meetups, gave a few more talks, started tweeting more, posted a few more blog articles and started participating in open-source.

I met some awesome people like @kiwipom, @kiwidev and @nigelsampson and really started enjoying being a part of the dev community. From that came a DM on twitter and an informal chat before I found my new role. I suppose the lesson here is, get out more ya freaks! And for fucks sake get a github account. What does a potential employer want to know? Can this muppet code? –> github account. Can this code-monkey interact with other reasonable humans without freaking them out, killing them or indecently exposing themselves? –> meetups/the twitters

Don’t get me wrong, I’m not an advocate of this “define your personal brand” style of self-promotion. It just seems awfully contrived and disingenuous. Just get out there and start interacting, there are incredibly smart and generous people in this community with a lot to share.

Why this move?

To be honest I wasn’t 100% sold on being a Windows Phone / Windows 8 developer, it is an unproven market and in the case of Windows 8 an immature platform. I have never been a UX kind of guy, I have ashamedly in the past hidden behind that lazy developer’s mantra “I don’t really do UI stuff, you know? I’m more of a back-end kind of guy”. I mean I’d spent the last 6 years building APIs, services, an enterprise workflow system, messaging doo-daddys……so going to an informal chat with Keith and Ben at Marker Metro I wasn’t sure this would be the right move for me. I was curious, but in no way convinced.

But, I left that chat hoping desperately that I would get a follow up. Why? Well here’s the thing – I’ve been inspired by @kellabyte’s mantra “Do Hard Shit!”, this isn’t a recommendation on daily fibre intake but rather a call for people to challenge themselves and pick tasks that they otherwise would not, in the belief that valuable learning will inevitably come of it. There are mountains of things I don’t know about WinRT, Xaml, Mobile development and UX, and here are a bunch of people that know a metric fuckload about things I have no idea about. What’s more, the sheer passion in the way that Keith and Ben spoke about their work and the plans for the company is infectious. Oh, and they have Epic beer on Fridays.

So now I’m head-down soaking up the experiences and knowledge trying to provide value and genuinely enjoying it. Hopefully that means I will be able to share some of that here soon.

Pipelining a query provider onto Linq to Objects

Recently I’ve seen a few problems occur in our codebase that made me think, there must be an obvious solution to this. Basically we have a bunch of data provider classes that expose a GetQuery method that returns an IQueryable<T>, we use these generically in our UI controls such that the control will have no knowledge of the data provider it has, only that it can get the queryable via GetQuery and due to deferred execution the control is able to schedule it on a background thread instead of blocking the UI thread.

This is all very cool and works fine until a specific data provider needs to do some kind of processing that isn’t supported by OData. You see once we reach the limits of the OData query provider we have to transition to Linq to Objects if we want to do a contains or something similar (like if the category is in this predefined list). This presents us with a problem as the data provider must choose to evaluate the OData query on the UI thread so that it can post-process the result and hence block the UI thread leaving our users with an experience that is not quite fast and fluid.

Well, I sat down to try to figure this out and realised it’s as easy as calling AsEnumerable. This may be obvious to you dear reader but I just never knew this trick. You see the AsEnumerable extension method on an IQueryable will not immediately evaluate the query like ToArray or ToList. Instead it will simply move into Linq to Objects as the query provider writing an As operator into the existing expression tree thus building a pipeline from Linq to OData to Linq to Objects retaining the deferred execution behavior. The result is the code below.

 

public class MainWindowViewModel {
    private readonly ObservableCollection<Title> titles = new ObservableCollection<Title>();

    public async Task EvaluateQuery() {
        IQueryable<Title> queryable = GetDataProvider().GetQuery();
        // At this point we have not yet sent a request over the wire
        // Now evaluate our queryable, it will execute a pipeline of two steps, Linq to OData | Linq to Objects
        var result = await Task.Factory.StartNew(() => queryable.ToArray());
        // We now update our UI bound property
        titles.Clear();
        result.ToList().ForEach(titles.Add);
    }

    public ObservableCollection<Title> Titles {
        get { return titles; }
    }

    private DataProvider GetDataProvider() {
        // Normally this would be some sort of factory code to get a data provider
        return new DataProvider();
    }
}

public class DataProvider {
    public IQueryable<Title> GetQuery() {
        return new NetflixCatalog(new Uri("http://odata.netflix.com/Catalog/"))
            .Titles
            .Where(t => t.Name.Contains("dude"))
            .AsEnumerable() // Transfer into Linq to Objects for this part but keep deferred execution
            .Where(s => IsMatch(s.Name))
            .AsQueryable();
    }

    public bool IsMatch(string name) {
        // Do some complicated filtering logic that can't be represented on an OData URI
        return true;
    }
}

My experience submitting an app to the Windows Store

I thought I should document the experience I have had in getting an app submitted to the Windows Store. I guess this process is new but the pains in getting an app submitted can be quite extraordinary.

October 18th 2012

Created developer account.
Some issues with entering the account details of my credit card.
Used a different card, all good.

October 19th 2012 (1 Day)

Submitted the app I had been working on. Sweet! (It’s not my best work, I’m not normally a UI guy but I like to try to challenge myself and someone might find it useful, pay me for the advanced features, I dunno)

Uh-Oh. Stuck on first step as I need to verify my account details. This involves getting a code from a temporary “holding” charge on my credit card account.

No worries, lets just check the account, Hmmmm 6 charges on my account instead of one, at least they’ll all be refunded eh? Now, which one to choose…lets start at the latest one….
Second?….
(Some time later)
Ok one more attempt I am told and then I’ll be locked out of verification, lets choose the last one….
Ok. No more attempts and I am locked out of ever verifying without help from support.
From outside the U.S. you can only contact support by email or online chat if within U.S. working hours.
Lets try the online chat, Chat window opens, I am told that I am chatting with …for the sake of anonymity lets call him Dave.
Dave…Dave?

Some time passes, I try reconnecting to chat, 2 different browsers, I even get dressed go to work logon there and I still get Dave, and Dave is still not answering. What gives Dave? I assume Dave is doing something with hookers, hotel rooms and cocaine. I need to stop watching TV.

October 20th 2012 (2 Days)

I email support using the online form. I tell them that I have no more attempts to verify and therefore I cannot verify my account via the dashboard.

October 21st 2012 (3 Days)

Get a reply from support. They confirm that I need to verify my account via the dashboard. sigh. They did also say that I could send them my statement. Ok, slightly worried about security but what the hell…

October 23rd 2012 (5 Days)

Must have been a busy day on the 22nd. I send my credit card statement to MS support. I consider censoring the iTunes app charges to save their feelings but think better of it in case they think its porn.

October 25th 2012 (7 Days)

I don’t have a reply yet but I realise that I now have a 7th charge on my credit card that has been reimbursed immediately but the first 6 charges have not been reimbursed yet. Oh well, maybe I’ll get shares.

October 30th 2012 (12 Days)

Try the chat again, get through to, let’s call him Bob. Bob has a look and confirms that there is a problem with their end, he will chase it up for me. Sweet! Progress!

October 31st 2012 (13 Days)

Get a reply from support, Bob’s done the job and inserted the boot in the posterior, GO BOB!! Support ask me to clear my browser cache and try again. I get flashbacks of my days in Dell tech support, asking people to reboot their machines as it took roughly the same time then as smoking a cigarette and getting a curly wurly from the vending machine.

November 1st 2012 (14 Days)

Reply from support! Should now be solved. Click the link and be on your merry way young traveller!

Reply from me: Nope. Same error message. Tried 2 browsers, reformatted my machine and reinstalled windows.

Not that I thought it was necessary to reformat, it was due for a clean anyways, besides it put the browser cache thing to bed in a suitably drastic way.

November 6th 2012 (19 Days)

I send a prompting email asking for details on what is happening. My tone is starting to change in an attempt to invoke some empathy. You know that tone that you use when you think you are hard done by but you realise that being a dick about it is more likely to cost you, like, “I’m sure you can understand how frustrating this could be….”. I like to pretend at this point that I’m wearing a suit, leaning on a counter with my elbow and making circular movements with the stems of my glasses. I wish I wore glasses, I think I could really use the new injection of body language expressions effectively.

Reply from support, “I have a team looking into the matter”. Looks like the glasses did the trick!

Another reply from support (definitely the glasses). Try again, team says we’re good.

Reply from me: “Nope, still the same”. There may have been some desperate blubbering tone in this email. I think I’d just watched Ghost or something.

November 7th 2012 (20 Days)

Reply from support: “Thanks for letting us know…..we’ll keep you posted”

November 14th 2012 (27 Days)

I try again just for the shizzles and the gizzles and Bam! It now works!! I can proceed!

I send them an email to tell them that they resolved it and that they still haven’t reimbursed me for the 6 credit card charges.

November 20th 2012 (33 Days)

App failed submission. Needs a privacy policy and it crashed at some point apparently.

I add a privacy policy and fix some bugs that may have caused issues if the user performed a certain action inside the app. I resubmit my app.


November 21st 2012 (34 Days)

Quick response from certification. Failed again. I wonder why it was so fast this time, is it like getting a recheck on your car warrant? Anyhoo, now the problem is just that it crashed.

The thing is, there is only a PDF file attached to the dashboard notice with a bunch of placeholders for screenshots, all of which are blank. I find some testers notes that say:

”B:: This app stopped responding during the review and we couldn’t complete the review of it.”

Ignoring the fact this guy has the coolest name ever, I could be mean and deconstruct the grammar here but the real tragedy is the lack of info. I mean when? on what hardware? what were you doing at the time? Did you try shaking it?

November 22nd 2012 (35 Days)

(I can’t believe you’re still reading this, well done, it gets a little better but you could just cut your losses if you’re about to miss the insult-boo/compliment-cheer roulette part of tonight’s X-Factor)

I use the online chat to ask Windows Store support folks about the lack of detail. This time I get……Brock. Brock tells me that he’ll have a look. Brock then stops talking and/or responding.

I reconnect my chat and get Randy (It’s really hard to pick a mildly amusing name without appearing culturally or racially offensive). I tell Randy everything again and he tells me that Brock’s machine crashed. I jokingly suggest it was my app, but he quite seriously reassures me that it isn’t. I guess irony is tough on chat rooms.

While chatting to Randy we figure out that the PDF certification report is trying to tell me that the app crashed on launch, that is why it failed. Randy tells me to contact technical support now as there is nothing he can do. Oh, and apparently you pay for tech support but they may not charge me and that most developers are entitled to a free issue or two.

November 25th 2012 (38 Days)

I try to raise a tech support request call using the website, after picking a bunch of filters you are told that they will charge you $259.00 per issue. I start filling out the form, telling them that I want my free issue please otherwise I’m not interested. I mean $259.00 for an event log entry seems a bit steep.

On submitting the form I am told my session has expired and I have to login again. Twice. Aghhhh!!!!! The session timeout I have estimated is around 3 minutes. I use notepad as my word processor and copy and paste as my close close friends to get around this limitation. Don’t get me wrong, I can type fast, I just change my mind often. You know, you start with mother-goat comparison insults, proof-read it, think better of it then work you way down through the Monty Python lexicon of mild insults until gradually you think that you are being relatively polite and constructive.

Next stage in the tech support issue raising dance workflow is verifying your payment method. This is a basic address verification.

I fill in my street address.

It asks me for a U.S. State. Hmmmm, lets just change the country. Wait, What? I can’t change the country. Ok, lets pick Alabama, its fun to say. Nope rejected, can’t complete the form, can’t submit a tech support support request.

Game over. Back to the forums for some witty repartee.

December 6th 2012 (49 Days)

Reimbursed for 6 erroneous charges. Thank you to the support team. I can’t say enough about how helpful they have been in the recent emails.

In the mean time I have also learned that the support team have no access to the testers. I can only assume there must be a backlog of apps to fail and they don’t want the testers disturbed by trivial things like questions about why apps fail.

I’ve also come to the conclusion that a large vector based map of the world (Yes, I know) in the background of my app could affect startup perf although I don’t really see it on a Surface RT or Intel machine.I’ve removed it and resubmitted the app. Here’s hoping…

To be continued….

Creating an NHibernate OData service using ASP.Net Web API Fall Update

NHibernate Query Service

 

Overview

The Fall Update to ASP.Net Web API re-implements the OData support from the beta releases. This time it is using the ODataLib libraries from the OData team. Getting it to work with NHibernate however can be a bit of a nightmare. As we’ll see though, it is possible. The code for this article is available on github.

Setting up the database

There is a SQL 2012 backup of the database in the root folder which requires a login “queryservice”.

Implementation

There are a number of issues to solve out of the box when trying to serve NHibernate objects directly over ASP.Net Web API.

“You should be using DTOs”, they will say

A lot of people will argue that you should really be serving DTOs instead of your actual model classes over the service boundary. They are normally correct but as this is the query side of a sort of CQRS implementation, my concerns are different. I actually just want to serve queryable data to the client.

 

Providing an NHibernate session

To execute any command in Nhibernate the API call must be made in the context of “Session”. This session represents the lifetime of the database connection and the first level cache of retrieved objects and their change tracking.

We could just construct our session in our controller action but that would involve some boilerplate repetition. Another issue is that, due to the compositional nature of action filters and hence the Web API OData implementation, we need the session to survive beyond the lifetime of the action method so that it can service the late bound evaluation of LINQ operators like Where and Expand.

To solve this issue the web api sample here uses constructor based dependency injection and Ninject to pass the session to the api controller and dispose of it when the request is complete. using an override of the Dispose method on the controller.

Serialization of Dynamic Proxies

NHibernate uses dynamic proxy types which are derived versions of model types, generated at runtime. When code tries to evaluate any properties on instances of these types, commands will be issued to the database in order to lazily fetch the data to populate the instance.

This lazy loading means that any serializer will cause the endless fetching of all data related to an entity regardless of whether it was requested or not.

To solve this issue for the JSON.Net serializer we use a ContractResolver to serialize the dynamic proxy class as if it were its base type. We then use a custom JsonConverter to write null out for any proxy instance or an uninitialized collection, the collection equivalent of the dynamic proxy.

Processing the $expand OData operator

The $expand operator allows the requester to specify the depth of the data that they want retrieved. This is equivalent to the Include operator in Entity Framework and the NHibernate LINQ equivalent is Fetch. To implement this the sample uses an Action Filter which will run its Executed method after the Web API OData action filters. In this it will apply any expands directives and force the enumeration of the results.

Unfortunately the ASP.Net Web API throws an exception when an $expand querystring parameter is found. This is due to the fact that it has not yet been implemented in the fall update and is not expected till 2013. Therefore we also need to derive our own QueryableAttribute and override the validate method to allow $expand.

 

Outstanding Issues

* Need to implement odata model and enable $metadata support.
* Need to switch on syscache2 support for sql dependency based cache eviction.