Tag Archives: C#

Gathering performance metrics using InfluxDB and Grafana #coding @grafana @InfluxDB

Collecting metrics in a log file is pretty effective when you have one process doing that logging. You have the data and can do whatever you need to with it. This approach has served me well for many years, but occasionally you want to log metrics from multiple processes to the same source at the same time. Trying to pull together data from 30+ log files and make sense of it is difficult. You can log it to SQL Server or some other datastore, but then what, more often than not you want to graph the metrics over time.

I thought I’d give some opensource tools a try, let’s walk through how to setup InfluxDB as the datastore and Grafana as the dashboard to make sense of it all.

Download

(please review licenses before doing so)
https://portal.influxdata.com/downloads#influxdb
https://grafana.com/grafana/download?platform=windows

I’m using https://dl.influxdata.com/influxdb/releases/influxdb-1.2.4_windows_amd64.zip
and https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.3.2.windows-x64.zip

Initial Setup

You will need to look further into security, licenses and your specific needs, but for me it’s as simple as extracting the two zip files.

Start the servers

Run influxd (.exe) from influxdb-1.2.4-1
Run grafana-server (.exe) from grafana-4.3.2\bin

Where does data get stored?

InfluxDB seems to use C:\Users\{current-user}\.influxdb
Grafana seems to use subdirectories.
Please let me know if you find more.

Getting a metric into InfluxDB

First you (like with any DBMS) need to create a DB instance, I did mine via the command line (though if you read the documentation you can do it via HTTP too):

influx -execute "CREATE DATABASE MetricDb1"
influx -execute "CREATE RETENTION POLICY "two_hours" ON "MetricDb1" DURATION 2h REPLICATION 1 DEFAULT"

Note: influx.exe is in influxdb-1.2.4-1, it is best to run from within that folder to avoid path problems.

As you can imagine, with those settings you are going to lose a metric some time in excess of two hours after it was written, so choose careful and read up on the documentation.

You might think the next command is CREATE TABLE… but it’s not required, it’s magic…

Inserting and selecting data from command line

For the insert the retention policy name is required, kind of like a schema name, but it’s very similar to SQL, though very reduced functionality:

influx -execute "INSERT INTO MetricDb1.two_hours cpu,host=sausages,region=eu_north value=0.69"

For the select you can leave off the retention policy when you can assume it is the default one, but do note the two full stops:

influx -execute "SELECT * FROM MetricDb1..cpu"

Or you can specify it:

influx -execute "SELECT * FROM MetricDb1.two_hours.cpu"

Inserting data from C#

using (var client = new HttpClient
{
    BaseAddress = new Uri("http://localhost:8086/"),
    Timeout = TimeSpan.FromMilliseconds(10000)
})
{
    await client.PostAsync("/write?db=MetricDb1", new StringContent("cpu,host=sausages,region=eu_north value=0.69"), CancellationToken.None);
}

Of course, opening the client every time is not how you would code it for a real application, but you get the idea.

Graphing it

  1. Open grafana web site http://localhost:3000
  2. Login with admin user, password admin (yeah, you might want to change that)
  3. From the Home Dashboad (should be the landing page when you login), click Add data source
  4. Give it a name
  5. Choose InfluxDB
  6. Url is http://localhost:8086
  7. Access is direct
  8. No Http Auth (until you configure some)
  9. Database is MetricDb1
  10. No User or Password (until you configure them)
  11. Click Add
  12. Go to @ (grafana logo) -> Dashboards -> New
  13. Click Graph
  14. Click on Panel Title then Edit. Or find some other way to edit the graph.
  15. On the Metrics tab you should be able to enter your query. I chose FROM two_hours cpu SELECT field (value) mean () GROUP BY time ($__interval) fill (null)
  16. Choose a Group by time interval, I chose >1s
  17. I wanted a bar chart, so I chose Bars option for Draw Models on Display tab.
  18. Save the dashboard when you are happy with it via disk icon or CTRL+S.

That should be all you need to get started and see data in the dashboard. There are options for auto-refresh and all manner of good things so do make sure you explore grafana. I find Last 5 minutes very useful to start with and Refresh every 5s.

Disconnected Entity Framework entity problems #dev #EntityFramework

Entity Framework is both a nice ORM and a confusing one, by default you get features that work well within a context scope, but outside (I.e. disconnected) it doesn’t work nicely. For example exceptions get thrown or navigation properties are emptied.

For work I wrote my own repository classes (as a lot of people do) that sit on top of Entity Framework and I use it a particular way. I’m more than happy with it and I can mix stored procedures with it when I need to.

I’ve put together some test cases that show some of the problems and some manual ways you can use it differently:

https://github.com/piddle/Piddle.EntityFramework.DisconnectedRepo

Granted, this might not be that useful to anyone other than me, but I don’t know until the gits voice an opinion.

Detouring a constructor with Microsoft Fakes #dev #coding #tdd

I wanted to ensure every SmtpClient instance that was created by a library was also disposed of, to do this I first needed to Shim SmtpClient using Microsoft Fakes:

  1. Add Fakes Assembly for System.
  2. Edit Fakes/System.fakes
    Add

    <ShimGeneration>
      <Add TypeName="SmtpClient"/>
    </ShimGeneration>
  3. Save the file and close
  4. Rebuild, open the System.4.0.0.0.Fakes reference and under System.Net.Mail.Fakes you should see ShimSmtpClient.

My idea was to count up every time the constructor was called and to count down every time one was disposed, to do this I created a counter:

int instancesNotDispose = 0;

Incremented it in the Constructor Shim, that is wrap the constructor and do some more things rather than completely detouring the constructor:

ShimSmtpClient.Constructor =
    x =>
    {
        // Execute outside of Shims otherwise you'll be in StackOverflowException territory
        ShimsContext.ExecuteWithoutShims(
        () =>
        {
            // Use reflection to get the default constructor and invoke it, the magic here is the instance is stashed into x!
            var constructor = typeof(SmtpClient).GetConstructor(new Type[0]);
            constructor.Invoke(x, new Object[0]);
        });

        instancesNotDispose++;
    };

Then decremented it in the Dispose Shim:

ShimSmtpClient.AllInstances.Dispose =
    x =>
    {
        // Execute outside of Shims otherwise you'll be in StackOverflowException territory
        ShimsContext.ExecuteWithoutShims(x.Dispose);

        instancesNotDispose--;
    };

Then I can call my library a bunch of times to do whatever it does and sends some e-mails, then I can assert they have all be disposed:

Assert.AreEqual<int>(0, instancesNotDispose, String.Format("Doh! {0} were not disposed", instancesNotDispose));

Assuming there aren’t other SmtpClient constructors that do not call the default constructor then this code should work fine, if there are other constructors called that do not call the default constructor then this Assert will also help find those. There are other ways, for example, stashing each instance in a List and ensuring each is in both the Constructor List and the Dispose List, but this counting approach was enough for my needs for now.

AjaxControlToolkit version 7.0123 with .NET 4.5

Here’s my steps and workarounds from the very beginning (in Visual Studio 2012), hopefully some part of this will help fix whatever error you are encountering.

File -> New Project
.NET Framework 4.5
Visual C# -> Web
ASP.NET Web Forms Application
Add AjaxControlToolkit version 7.0123 (dll is actually 4.5.7.123 which is January 2013 I believe) via NuGet

Open Default.aspx
Add a calendar to the BodyContent / MainContent:

<asp:TextBox ID="myTextBox" runat="server" />
<ajaxToolkit:CalendarExtender ID="myCalendar" runat="server" TargetControlID="myTextBox" Format="dd/MM/yyyy" />

Run it up (F5 will do)

It might error about ASP.NET Ajax 4.0 scripts:
0x800a139e – JavaScript runtime error: AjaxControlToolkit requires ASP.NET Ajax 4.0 scripts. Ensure the correct version of the scripts are referenced. If you are using an ASP.NET ScriptManager, switch to the ToolkitScriptManager in AjaxControlToolkit.dll.

I’ve also seen it error about:
‘MsAjaxBundle’ is not a valid script name. The name must end in ‘.js’.

No bother, let’s remove that reference from the Site.Master, so:

    <asp:ScriptManager runat="server">
        <Scripts>
            <%--Framework Scripts--%>
            <asp:ScriptReference Name="MsAjaxBundle" />
            <asp:ScriptReference Name="jquery" />
            <asp:ScriptReference Name="jquery.ui.combined" />
            <asp:ScriptReference Name="WebForms.js" Assembly="System.Web" Path="~/Scripts/WebForms/WebForms.js" />
            <asp:ScriptReference Name="WebUIValidation.js" Assembly="System.Web" Path="~/Scripts/WebForms/WebUIValidation.js" />
            <asp:ScriptReference Name="MenuStandards.js" Assembly="System.Web" Path="~/Scripts/WebForms/MenuStandards.js" />
            <asp:ScriptReference Name="GridView.js" Assembly="System.Web" Path="~/Scripts/WebForms/GridView.js" />
            <asp:ScriptReference Name="DetailsView.js" Assembly="System.Web" Path="~/Scripts/WebForms/DetailsView.js" />
            <asp:ScriptReference Name="TreeView.js" Assembly="System.Web" Path="~/Scripts/WebForms/TreeView.js" />
            <asp:ScriptReference Name="WebParts.js" Assembly="System.Web" Path="~/Scripts/WebForms/WebParts.js" />
            <asp:ScriptReference Name="Focus.js" Assembly="System.Web" Path="~/Scripts/WebForms/Focus.js" />
            <asp:ScriptReference Name="WebFormsBundle" />
            <%--Site Scripts--%>

        </Scripts>
    </asp:ScriptManager>

Now becomes:

    <asp:ScriptManager runat="server">
        <Scripts>
            <asp:ScriptReference Name="jquery" />
            <asp:ScriptReference Name="jquery.ui.combined" />
            <asp:ScriptReference Name="WebForms.js" Assembly="System.Web" Path="~/Scripts/WebForms/WebForms.js" />
            <asp:ScriptReference Name="WebUIValidation.js" Assembly="System.Web" Path="~/Scripts/WebForms/WebUIValidation.js" />
            <asp:ScriptReference Name="MenuStandards.js" Assembly="System.Web" Path="~/Scripts/WebForms/MenuStandards.js" />
            <asp:ScriptReference Name="GridView.js" Assembly="System.Web" Path="~/Scripts/WebForms/GridView.js" />
            <asp:ScriptReference Name="DetailsView.js" Assembly="System.Web" Path="~/Scripts/WebForms/DetailsView.js" />
            <asp:ScriptReference Name="TreeView.js" Assembly="System.Web" Path="~/Scripts/WebForms/TreeView.js" />
            <asp:ScriptReference Name="WebParts.js" Assembly="System.Web" Path="~/Scripts/WebForms/WebParts.js" />
            <asp:ScriptReference Name="Focus.js" Assembly="System.Web" Path="~/Scripts/WebForms/Focus.js" />
            <asp:ScriptReference Name="WebFormsBundle" />
        </Scripts>
    </asp:ScriptManager>

Next is ToolkitScriptManager, the snippet above now becomes:

    <ajaxToolkit:ToolkitScriptManager runat="server">
        <Scripts>
            <asp:ScriptReference Name="jquery" />
            <asp:ScriptReference Name="jquery.ui.combined" />
            <asp:ScriptReference Name="WebForms.js" Assembly="System.Web" Path="~/Scripts/WebForms/WebForms.js" />
            <asp:ScriptReference Name="WebUIValidation.js" Assembly="System.Web" Path="~/Scripts/WebForms/WebUIValidation.js" />
            <asp:ScriptReference Name="MenuStandards.js" Assembly="System.Web" Path="~/Scripts/WebForms/MenuStandards.js" />
            <asp:ScriptReference Name="GridView.js" Assembly="System.Web" Path="~/Scripts/WebForms/GridView.js" />
            <asp:ScriptReference Name="DetailsView.js" Assembly="System.Web" Path="~/Scripts/WebForms/DetailsView.js" />
            <asp:ScriptReference Name="TreeView.js" Assembly="System.Web" Path="~/Scripts/WebForms/TreeView.js" />
            <asp:ScriptReference Name="WebParts.js" Assembly="System.Web" Path="~/Scripts/WebForms/WebParts.js" />
            <asp:ScriptReference Name="Focus.js" Assembly="System.Web" Path="~/Scripts/WebForms/Focus.js" />
            <asp:ScriptReference Name="WebFormsBundle" />
        </Scripts>
    </ajaxToolkit:ToolkitScriptManager>

But of course this fails with:
Could not load file or assembly ‘System.Web’ or one of its dependencies. The system cannot find the file specified.

Remove the Assembly=”System.Web” part from the ScriptReference so we have have:

    <ajaxToolkit:ToolkitScriptManager runat="server">
        <Scripts>
            <asp:ScriptReference Name="jquery" />
            <asp:ScriptReference Name="jquery.ui.combined" />
            <asp:ScriptReference Name="WebForms.js" Path="~/Scripts/WebForms/WebForms.js" />
            <asp:ScriptReference Name="WebUIValidation.js" Path="~/Scripts/WebForms/WebUIValidation.js" />
            <asp:ScriptReference Name="MenuStandards.js" Path="~/Scripts/WebForms/MenuStandards.js" />
            <asp:ScriptReference Name="GridView.js" Path="~/Scripts/WebForms/GridView.js" />
            <asp:ScriptReference Name="DetailsView.js" Path="~/Scripts/WebForms/DetailsView.js" />
            <asp:ScriptReference Name="TreeView.js" Path="~/Scripts/WebForms/TreeView.js" />
            <asp:ScriptReference Name="WebParts.js" Path="~/Scripts/WebForms/WebParts.js" />
            <asp:ScriptReference Name="Focus.js" Path="~/Scripts/WebForms/Focus.js" />
            <asp:ScriptReference Name="WebFormsBundle" />
        </Scripts>
    </ajaxToolkit:ToolkitScriptManager>

It seems the new web forms project template adds ‘Microsoft.AspNet.ScriptManager.MSAjax 4.5.6’ package, this appears to conflict with the toolkit, so remove this via “Manage NuGet Packages”
Visual Studio might still leave the dll in your bin directory even after a clean, make sure you manually clean that out.

So there we are, works for me, best of luck.

EDIT:

Silly me, I forgot, remove the Name too (e.g. Name=”WebForms.js” ) for each that have a Path. If you don’t you will get errors like so:
The assembly ‘AjaxControlToolkit, Version=4.5.7.123, Culture=neutral, PublicKeyToken=28f01b0e84b6d53e’ does not contain a Web resource that has the name ‘WebForms.js’. Make sure that the resource name is spelled correctly.

ASP.NET Web Forms Response.Write in an UpdatePanel #dev #web

I recently had a problem at work where I wanted to use some code that writes to the Response.Output TextWriter, this works fine until you stick an UpdatePanel around the line that writes to it. Various searches on the internet didn’t reveal any clues as to how to do this, all that I found suggested alternatives and used words like “doesn’t work” and use “PostBackTrigger” (which, by the way, negates the whole point of an UpdatePanel).

For fear of being labelled a hack-it together guy, I came up with two solutions, one is rock solid, the other is less so, but works wonderfully until they change how UpdatePanels work (which is unlikely given the fact that Web Forms is a dying technology, if you disagree then we will never be friends).

The basis of the solution

A control (anything inheriting from WebControl) is given a TextWriter on RenderBeginTag, RenderEndTag, … etc. Ah (ching, the penny drops), maybe this is different from Response.Output when we are processing an Async Postback? Indeed it is!

An example where Response.Output.Write doesn’t work

This is the standard Web Forms template application with the content removed.

<%@ Page Title="Home Page" Language="C#" MasterPageFile="~/Site.Master" AutoEventWireup="true" CodeBehind="Default.aspx.cs" Inherits="ResponseWriteUpdatePanel._Default" %>
<asp:Content runat="server" ID="FeaturedContent" ContentPlaceHolderID="FeaturedContent">
</asp:Content>
<asp:Content runat="server" ID="BodyContent" ContentPlaceHolderID="MainContent">
    <asp:UpdatePanel runat="server">
        <ContentTemplate>
            <% Response.Output.Write("Hello"); %>
            <asp:TextBox Text="Text Here" ID="MyTextBox" runat="server" />
            <asp:Button Text="Button" ID="MyButton" runat="server" />
        </ContentTemplate>
    </asp:UpdatePanel>
</asp:Content>

Click the button and the Response.Output.Write destroys the response:

Unhandled exception at line 1, column 126350 in http://localhost:49213/bundles/MsAjaxJs?v=J4joXQqg80Lks57qbGfUAfRLic3bXKGafmR6wE4CFtc1

0x800a139e – JavaScript runtime error: Sys.WebForms.PageRequestManagerParserErrorException: The message received from the server could not be parsed. Common causes for this error are when the response is modified by calls to Response.Write(), response filters, HttpModules, or server trace is enabled.

Details: Error parsing near ‘1|#||4|Hello245|updatePanel’.

Here’s a server control that captures the TextWriter and sticks it into the Response.Output (this is the less so option):

using System.Web.UI;
using System.Web.UI.WebControls;

namespace ResponseWriteUpdatePanel
{
    public class MyResponseHelper : WebControl
    {
        public override void RenderBeginTag(HtmlTextWriter output)
        {
            this.Page.Response.Output = output;
        }

        public override void RenderEndTag(HtmlTextWriter output)
        {
        }

        protected override void RenderContents(HtmlTextWriter output)
        {
        }
    }
}

Then add to the page, like so:

<%@ Page Title="Home Page" Language="C#" MasterPageFile="~/Site.Master" AutoEventWireup="true" CodeBehind="Default.aspx.cs" Inherits="ResponseWriteUpdatePanel._Default" %>
<%@ Register TagPrefix="tim" Namespace="ResponseWriteUpdatePanel" Assembly="ResponseWriteUpdatePanel" %>
<asp:Content runat="server" ID="FeaturedContent" ContentPlaceHolderID="FeaturedContent">
</asp:Content>
<asp:Content runat="server" ID="BodyContent" ContentPlaceHolderID="MainContent">
    <asp:UpdatePanel runat="server">
        <ContentTemplate>
            <tim:MyResponseHelper ID="MyOutput" runat="server"></tim:MyResponseHelper>
            <% Response.Output.Write("Hello"); %>
            <asp:TextBox Text="Text Here" ID="MyTextBox" runat="server" />
            <asp:Button Text="Button" ID="MyButton" runat="server" />
        </ContentTemplate>
    </asp:UpdatePanel>
</asp:Content>

Voila, it works.

Next, the more rock solid solution which avoids tinkering with the Response.Output:

using System.Web.UI;
using System.Web.UI.WebControls;

namespace ResponseWriteUpdatePanel
{
    public class MyResponseHelper : WebControl
    {
        public HtmlTextWriter Output { get; set; }

        public override void RenderBeginTag(HtmlTextWriter output)
        {
            Output = output;
        }

        public override void RenderEndTag(HtmlTextWriter output)
        {
        }

        protected override void RenderContents(HtmlTextWriter output)
        {
        }
    }
}

And the page directly uses the TextWriter from the control instead of the Response.Output, like so:

<%@ Page Title="Home Page" Language="C#" MasterPageFile="~/Site.Master" AutoEventWireup="true" CodeBehind="Default.aspx.cs" Inherits="ResponseWriteUpdatePanel._Default" %>
<%@ Register TagPrefix="tim" Namespace="ResponseWriteUpdatePanel" Assembly="ResponseWriteUpdatePanel" %>
<asp:Content runat="server" ID="FeaturedContent" ContentPlaceHolderID="FeaturedContent">
</asp:Content>
<asp:Content runat="server" ID="BodyContent" ContentPlaceHolderID="MainContent">
    <asp:UpdatePanel runat="server">
        <ContentTemplate>
            <tim:MyResponseHelper ID="MyOutput" runat="server"></tim:MyResponseHelper>
            <% MyOutput.Output.Write("Hello"); %>
            <asp:TextBox Text="Text Here" ID="MyTextBox" runat="server" />
            <asp:Button Text="Button" ID="MyButton" runat="server" />
        </ContentTemplate>
    </asp:UpdatePanel>
</asp:Content>

Of course, should you have two UpdatePanels on the same page then you would use a MyResponseHelper within each and use that instance.

Microsoft Visual Studio 2012 and #disappointing lamda support #coding #programming #csharp

Two things I wanted in 2012, edit and continue on methods containing lambda expressions and also to be able to use them in the watch and immediate windows…

Modifying a statement which contains a lambda expression will prevent the debug session from continuing while Edit and Continue is enabled.

(new Object[] { 1, 2 })
{object[2]}
[0]: 1
[1]: 2
(new Object[] { 1, 2 }).Select(x => x.ToString())
Expression cannot contain lambda expressions

Still disappointed!
Come on Microsoft, two features I’ve wanted since you first introduced lambdas, how long will it take?

Windows Forms or WPF UI updates from a different thread #coding #programming #csharp

OK, we’ve all wanted to update the UI from a different thread at some point, it is annoying, that damn exception you get…
System.InvalidOperationException
{“Cross-thread operation not valid: Control ‘listBox1’ accessed from a thread other than the thread it was created on.”}

I tend to come back to the same approach time and time again, there may be others, but here’s an example I knocked up for a friend because this is easier to read than explain over the phone, upon clicking button1 I want to add some text to listBox1 without locking up the UI:

        private static void LogToList(SynchronizationContext ctx, ListBox listBox, String text)
        {
            ctx.Post((t) => { listBox.Items.Add(t); }, text);
        }

        private void button1_Click(object sender, EventArgs e)
        {
            Task.Factory.StartNew((ctx) =>
            {
                for (int i = 0; i < 10; i++)
                {
                    LogToList((SynchronizationContext)ctx, listBox1, DateTime.Now.ToString());
                    Thread.Sleep(1000);
                }
            }, (Object)SynchronizationContext.Current);
        }

For those who haven’t used it before, Task was introduced in .NET 4 under what is known as the Task Parallel Library (TPL), from what I can tell it is seen as the replacement for BackgroundWorker, but there’s no reason why you can’t do the same with a BackgroundWorker as we did before .NET 4.

In WPF I’ve previously also used System.Windows.Threading.Dispatcher.BeginInvoke in a similar-ish way. To get the Dispatcher instance, what you want is System.Windows.Application.Current.Dispatcher.

This also reminds me that if you’re doing a lot of logging / updating of a control then setting UndoLimit to zero is a good thing if you don’t want to leak / consume what can be vast amounts of memory for an undo buffer you don’t want or need.