Category Archives: Software development

Gathering performance metrics using InfluxDB and Grafana #coding @grafana @InfluxDB

Collecting metrics in a log file is pretty effective when you have one process doing that logging. You have the data and can do whatever you need to with it. This approach has served me well for many years, but occasionally you want to log metrics from multiple processes to the same source at the same time. Trying to pull together data from 30+ log files and make sense of it is difficult. You can log it to SQL Server or some other datastore, but then what, more often than not you want to graph the metrics over time.

I thought I’d give some opensource tools a try, let’s walk through how to setup InfluxDB as the datastore and Grafana as the dashboard to make sense of it all.

Download

(please review licenses before doing so)
https://portal.influxdata.com/downloads#influxdb
https://grafana.com/grafana/download?platform=windows

I’m using https://dl.influxdata.com/influxdb/releases/influxdb-1.2.4_windows_amd64.zip
and https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.3.2.windows-x64.zip

Initial Setup

You will need to look further into security, licenses and your specific needs, but for me it’s as simple as extracting the two zip files.

Start the servers

Run influxd (.exe) from influxdb-1.2.4-1
Run grafana-server (.exe) from grafana-4.3.2\bin

Where does data get stored?

InfluxDB seems to use C:\Users\{current-user}\.influxdb
Grafana seems to use subdirectories.
Please let me know if you find more.

Getting a metric into InfluxDB

First you (like with any DBMS) need to create a DB instance, I did mine via the command line (though if you read the documentation you can do it via HTTP too):

influx -execute "CREATE DATABASE MetricDb1"
influx -execute "CREATE RETENTION POLICY "two_hours" ON "MetricDb1" DURATION 2h REPLICATION 1 DEFAULT"

Note: influx.exe is in influxdb-1.2.4-1, it is best to run from within that folder to avoid path problems.

As you can imagine, with those settings you are going to lose a metric some time in excess of two hours after it was written, so choose careful and read up on the documentation.

You might think the next command is CREATE TABLE… but it’s not required, it’s magic…

Inserting and selecting data from command line

For the insert the retention policy name is required, kind of like a schema name, but it’s very similar to SQL, though very reduced functionality:

influx -execute "INSERT INTO MetricDb1.two_hours cpu,host=sausages,region=eu_north value=0.69"

For the select you can leave off the retention policy when you can assume it is the default one, but do note the two full stops:

influx -execute "SELECT * FROM MetricDb1..cpu"

Or you can specify it:

influx -execute "SELECT * FROM MetricDb1.two_hours.cpu"

Inserting data from C#

using (var client = new HttpClient
{
    BaseAddress = new Uri("http://localhost:8086/"),
    Timeout = TimeSpan.FromMilliseconds(10000)
})
{
    await client.PostAsync("/write?db=MetricDb1", new StringContent("cpu,host=sausages,region=eu_north value=0.69"), CancellationToken.None);
}

Of course, opening the client every time is not how you would code it for a real application, but you get the idea.

Graphing it

  1. Open grafana web site http://localhost:3000
  2. Login with admin user, password admin (yeah, you might want to change that)
  3. From the Home Dashboad (should be the landing page when you login), click Add data source
  4. Give it a name
  5. Choose InfluxDB
  6. Url is http://localhost:8086
  7. Access is direct
  8. No Http Auth (until you configure some)
  9. Database is MetricDb1
  10. No User or Password (until you configure them)
  11. Click Add
  12. Go to @ (grafana logo) -> Dashboards -> New
  13. Click Graph
  14. Click on Panel Title then Edit. Or find some other way to edit the graph.
  15. On the Metrics tab you should be able to enter your query. I chose FROM two_hours cpu SELECT field (value) mean () GROUP BY time ($__interval) fill (null)
  16. Choose a Group by time interval, I chose >1s
  17. I wanted a bar chart, so I chose Bars option for Draw Models on Display tab.
  18. Save the dashboard when you are happy with it via disk icon or CTRL+S.

That should be all you need to get started and see data in the dashboard. There are options for auto-refresh and all manner of good things so do make sure you explore grafana. I find Last 5 minutes very useful to start with and Refresh every 5s.

Disconnected Entity Framework entity problems #dev #EntityFramework

Entity Framework is both a nice ORM and a confusing one, by default you get features that work well within a context scope, but outside (I.e. disconnected) it doesn’t work nicely. For example exceptions get thrown or navigation properties are emptied.

For work I wrote my own repository classes (as a lot of people do) that sit on top of Entity Framework and I use it a particular way. I’m more than happy with it and I can mix stored procedures with it when I need to.

I’ve put together some test cases that show some of the problems and some manual ways you can use it differently:

https://github.com/piddle/Piddle.EntityFramework.DisconnectedRepo

Granted, this might not be that useful to anyone other than me, but I don’t know until the gits voice an opinion.

Visual Studio replace Carriage Return Line Feed (CR LF) with comma and space @VisualStudio

To flatten a list of text into a CSV row you can use Visual Studio (which is where I normally am anyway).

Replace In Files (SHIFT + CTRL + H)
Find what: \r\n
Replace with: ,
(that is a comma then a space)
Look in: Current Document
Use Regular Expressions ticked

Visual Studio Find in Files RegEx @VisualStudio

I always forget how to exclude matches when I use Find in Files, so I’ll write about it here and there’s no excuse for not remembering 🙂

The problem is searching for a string and getting lots of matches that indeed contain that string, but you don’t want those ones.

For example I want to find Outbound, but not OutboundSystem, that’s something different and fills up my find results with matches.

I can “Find in Files” with “Use Regular Expressions” and this:

Outbound(?!System)

Which does a negative forward lookup for System and excludes those ones I don’t want.

How I help IntelliSense to provide more useful information #coding #JavaScript

OK, maybe it’s a hack rather than help, but there’s a lot that IntelliSense can figure out and some “basic” stuff it can’t. JavaScript is dynamic, often the type isn’t provided until runtime and this hinders IntelliSense, this is what I try to help it with. I’m using Microsoft Visual Studio Ultimate 2013 Update 2 with the corresponding Web Essentials.

Here’s starting simple, this works well:

var Person = function (who, bikes)
{
    this.Who = who;
    this.Bikes = bikes;
};

var p = new Person("Tim", ["bluebell", "rudy", "nexus"]);
console.log(p.Who.toUpperCase() + " has " + p.Bikes.join("; "));

“TIM has bluebell; rudy; nexus” is output to the JavaScript console in Microsoft Internet Explorer 11. IntelliSense can figure out what Who and Bikes is, so toUpperCase and join came from the auto-completion list it provides.

Wow, you think, IntelliSense is clever, it is, but take away the type information, for example I create a Log function and execute it somewhere else, let’s say in another script, this still outputs the same:

// Script 1
var Person = function (who, bikes)
{
    this.Who = who;
    this.Bikes = bikes;
};

Person.prototype.Log = function ()
{
    console.log(this.Who.toUpperCase() + " has " + this.Bikes.join("; "));
};

// Script 2
var p = new Person("Tim", ["bluebell", "rudy", "nexus"]);
p.Log();

But IntelliSense within the Log function does not know Who is a string and has the toUpperCase function and that Bikes is an array and has the join function.
Since the types are going to be fixed, I’ll never want anyone to do something like new Person(1, 2), I can comment the Person constructor function, but that doesn’t help when writing the Log (or other) functions.

IntelliSense has an API, so you can log messages to the Output (JavaScript Language Service) panel in Visual Studio like so:

intellisense.logMessage("Hello :)"); 

But you only want to do that if it intellisense is defined, so…

if (typeof intellisense !== "undefined")
{
    intellisense.logMessage("Hello :)");
}

That code will even work in production now, the if statement block contents will not be executed unless you (or the JavaScript runtime) defines a global called intellisense, but in Visual Studio it will be run.

So we can help IntelliSense by faking some values in the Person constructor function, so it can figure out what the types are:

// Script 1
var Person = function (who, bikes)
{
    this.Who = who;
    this.Bikes = bikes;

    // Help IntelliSense
    if (typeof intellisense !== "undefined")
    {
        this.Who = "";
        this.Bikes = [];
    }
};

Person.prototype.Log = function ()
{
    console.log(this.Who.toUpperCase() + " has " + this.Bikes.join("; "));
};

// Script 2
var p = new Person("Tim", ["bluebell", "rudy", "nexus"]);
p.Log();

Now the Log function has auto-completion for Who and Bikes, yay! You can also stick a logMessage call in there too, so you can see when it’s helping you.

This is an especially useful trick when you have dependency injection and complex object parameters (more on this another time I hope) and can be used anywhere (not just in constructor functions).

If you are struggling with IntelliSense (I always am), then I hope this helps in some way, please post a comment with your experiences.

How I share JavaScript files between web projects #coding #JavaScript

It’s the first wall to climb when you have multiple web projects with common code, how to share scripts between web projects without duplication, copy paste etc. Sure, I have nugget packages for libs, but do I need to have the same scripts in every web project (and branches thereof too)? My last count was 15 copies of everything (15 TFS workspaces), if I make a change to only one of my scripts, that is 14 that are potentially out of date and it’s easy to miss one!

TFS workspaces are certainly not as helpful as they could be, so I resorted to windows symbolic links / symlinks (CreateSymbolicLink function in kernel32.dll) as a means of sharing files and folders and then cloaking the folders in my TFS workspace. I can live with manually including additions and deletions as a result of the cloaking, the upside is that I can link extra scripts into one web project and share the rest with the other projects. It is perfect for an administration project that has extra scripts from the normal projects and we don’t want to have those admin scripts included in the normal projects.

So, I have a good solution that reduces the duplication, but it didn’t play well with IntelliSense (go to definition). I’m using Microsoft Visual Studio Ultimate 2013 Update 2 with the corresponding Web Essentials and IntelliSense doesn’t like these links, specifically, configuring the _references.js file in “Tools -> Options -> Text Editor -> JavaScript -> IntelliSense -> References” doesn’t seem to work for me, maybe I’ve just not got it correct yet, anyway, I gave up in favour of a different approach. It also falls flat when someone doesn’t have it configured correctly, it requires everyone to have it configured, and it’s fixed regardless of project.

Fortunately (and deliberately) the relative path to each Scripts folder is the same level of depth. So rather than use the above way to configure it, I opted for placing a reference to it in each script I wanted it, like so:

/// <reference path=”~/../../Shared/Scripts/_references.js” />

And in the _references.js the references are like so (of course there are a LOT more than just this one):

/// <reference path=”~/Scripts/angular.js” />

where the symlinks are like so:

XYZ/Solution1/Project1/Scripts/_references.js -> XYZ/Shared/Scripts/_references.js

XYZ/SolutionN/ProjectN/Scripts/_references.js -> XYZ/Shared/Scripts/_references.js

and

XYZ/Solution1/Project1/Scripts/angular.js -> XYZ/Shared/Scripts/angular.js

XYZ/SolutionN/ProjectN/Scripts/angular.js -> XYZ/Shared/Scripts/angular.js

This means I only have one copy of the scripts on my hard disk, the _references.js are shared between each project etc and I’m not constantly fighting to keep them all up to date.

Additional references can be defined in other files and referenced where required, for example the administration ones, but I tend to only keep core script references in the _references because if you get a circular reference then IntelliSense stops working!

If you are struggling with sharing scripts, then I hope this helps in some way, please post a comment with your experiences

Detouring a constructor with Microsoft Fakes #dev #coding #tdd

I wanted to ensure every SmtpClient instance that was created by a library was also disposed of, to do this I first needed to Shim SmtpClient using Microsoft Fakes:

  1. Add Fakes Assembly for System.
  2. Edit Fakes/System.fakes
    Add

    <ShimGeneration>
      <Add TypeName="SmtpClient"/>
    </ShimGeneration>
  3. Save the file and close
  4. Rebuild, open the System.4.0.0.0.Fakes reference and under System.Net.Mail.Fakes you should see ShimSmtpClient.

My idea was to count up every time the constructor was called and to count down every time one was disposed, to do this I created a counter:

int instancesNotDispose = 0;

Incremented it in the Constructor Shim, that is wrap the constructor and do some more things rather than completely detouring the constructor:

ShimSmtpClient.Constructor =
    x =>
    {
        // Execute outside of Shims otherwise you'll be in StackOverflowException territory
        ShimsContext.ExecuteWithoutShims(
        () =>
        {
            // Use reflection to get the default constructor and invoke it, the magic here is the instance is stashed into x!
            var constructor = typeof(SmtpClient).GetConstructor(new Type[0]);
            constructor.Invoke(x, new Object[0]);
        });

        instancesNotDispose++;
    };

Then decremented it in the Dispose Shim:

ShimSmtpClient.AllInstances.Dispose =
    x =>
    {
        // Execute outside of Shims otherwise you'll be in StackOverflowException territory
        ShimsContext.ExecuteWithoutShims(x.Dispose);

        instancesNotDispose--;
    };

Then I can call my library a bunch of times to do whatever it does and sends some e-mails, then I can assert they have all be disposed:

Assert.AreEqual<int>(0, instancesNotDispose, String.Format("Doh! {0} were not disposed", instancesNotDispose));

Assuming there aren’t other SmtpClient constructors that do not call the default constructor then this code should work fine, if there are other constructors called that do not call the default constructor then this Assert will also help find those. There are other ways, for example, stashing each instance in a List and ensuring each is in both the Constructor List and the Dispose List, but this counting approach was enough for my needs for now.