Tags: , | Categories: Articles Posted by RTomlinson on 1/3/2013 5:43 AM | Comments (0)

All exceptions in our architecture at Tombola are caught and logged. We use an exception driven approach because the thought of someone having a poor experience on the site and experiencing errors is painful for us and just isn’t good enough. To then consider that we aren’t aware of every issue customers may be having is both bad for business and makes us, as a team, look shite.

We have two distinct types of exception; custom/handled exceptions and unhandled exceptions. The latter being the scary ones. The custom exceptions are there to control flow and business logic (for example, it’s a valid exception if someone attempts to deposit over their deposit limit). Throughout our service and repository layer we handle these differently but in all cases they are logged (to a SQL backend). I won’t cover the reasoning behind this approach as Terry has already covered the topic in this post and detailed the performance impact in this post.

When exceptions occur we want to know about them instantly. With a spare 60” TV in the office it made sense to make use of it with useful information.

This is the resulting product (for the sake of the demo this is me firing sample messages). Apologies for the poor quality. It’s worth playing in HD.

What the video doesn’t show is that each exception type plays a sound when it is displayed (using HTML5 audio).

How it was build

When an exception occurs, for any country (currently we operate in UK, Spain and Italy), an NServiceBus message is fired off from our database logger to a MSMQ queue on an app server. This is feature toggled to allow us to turn it on/off and looks like this (an example of a software exception from the logger):

if (FeatureToggles.EnableExceptionMessageOnBus)
{
  try
  {
    tombolaBus.SendException(message, exceptionStr, ExceptionType.Software);
  }
  catch (Exception exception)
  {
    Log("Failed to add message to bus", exception);
  }
}

Once the message reaches the queue on the app server we have a “handler” process running as a windows service using the NServiceBus host. This process is incredibly simple. It acts as an NServiceBus server endpoint to handle exception messages. When it receives a message from the bus (taken off the queue) it calls a SignalR proxy to fire the message to all connected clients:

public class ExceptionMessageHandler : IHandleMessages<ExceptionMessage>
{
  private static readonly ILog log = LogManager.GetLogger(typeof(ExceptionMessageHandler));
  public void Handle(ExceptionMessage message)
  {
    log.DebugFormat("New message received: {0}", message.Message);

    SignalRProxyConnection.SendException(message);
  }
}

The website dashboard

The dashboard (as you see from the video) was build using twitter bootstrap and SignalR 1.0 alpha. It consists of a single hub that calls a javascript function on the client to display the message. The client takes the country and exception type and styles it up to display in the correct lane. The hub is almost not worth showing….but I will:

[HubName("exceptionHub")]
public class Exceptionhub : Hub
{
  public void SendException(ExceptionMessage exception)
  {
    Clients.All.displayException(exception);
  }
}

In order to call this hub in the web project from another project, as I have done with the ExceptionMessageHandler, SignalR provides this functionality with the use of a proxy (IHubProxy). Once the proxy is created you can invoke method calls on the remote hub and pass parameters.

public static class SignalRProxyConnection
{
  private static bool connected;
  private static IHubProxy proxy;
  private static HubConnection hubConnection;
  private static string connectionUrl = ConfigurationManager.AppSettings["SignalRDashboardUrl"];

  public static void SendException(ExceptionMessage message)
  {
    if (!connected)
    {
      Connect();
    }

    if (hubConnection.State == ConnectionState.Connected)
      proxy.Invoke("SendException", message);
  }

  private static void Connect()
  {
    hubConnection = new HubConnection(connectionUrl);
    proxy = hubConnection.CreateHubProxy("ExceptionHub");
    hubConnection.Start().Wait();
    connected = true;
  }
}

You can see here I’m calling the “SendException” method on the ExceptionHub and passing the message that was received from the queue on the app server.

Further enhancements

There is a great deal more that could be done to enhance this project. There is no security implementation on the hub to stop anyone creating a proxy connection and firing random messages at it. This isn’t a huge problem for us, as it’s an internal application that’s not accessible from outside the domain.

There’s also a lack of persistence. If a message is fired and there are no connected clients then the message is lost (well this is not strictly true because it’s still in our database, but that involves somebody actively looking). There is scope to use a NOSQL persistence store to warehouse this data and retrieve it for newly connected clients.

Conclusion

The beauty in this solution is just how simple these technologies make it. The heavy lifting is done by SignalR, NServiceBus and to some extend twitter bootstrap. There is very little code involved, yet the solution is distributed and extensible. It enables us to be proactive in solving customer problems before customer support inform us and we can actively look to make our product better.

blog comments powered by Disqus