Gathering performance metrics using InfluxDB and Grafana #coding @grafana @InfluxDB

Collecting metrics in a log file is pretty effective when you have one process doing that logging. You have the data and can do whatever you need to with it. This approach has served me well for many years, but occasionally you want to log metrics from multiple processes to the same source at the same time. Trying to pull together data from 30+ log files and make sense of it is difficult. You can log it to SQL Server or some other datastore, but then what, more often than not you want to graph the metrics over time.

I thought I’d give some opensource tools a try, let’s walk through how to setup InfluxDB as the datastore and Grafana as the dashboard to make sense of it all.

Download

(please review licenses before doing so)
https://portal.influxdata.com/downloads#influxdb
https://grafana.com/grafana/download?platform=windows

I’m using https://dl.influxdata.com/influxdb/releases/influxdb-1.2.4_windows_amd64.zip
and https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.3.2.windows-x64.zip

Initial Setup

You will need to look further into security, licenses and your specific needs, but for me it’s as simple as extracting the two zip files.

Start the servers

Run influxd (.exe) from influxdb-1.2.4-1
Run grafana-server (.exe) from grafana-4.3.2\bin

Where does data get stored?

InfluxDB seems to use C:\Users\{current-user}\.influxdb
Grafana seems to use subdirectories.
Please let me know if you find more.

Getting a metric into InfluxDB

First you (like with any DBMS) need to create a DB instance, I did mine via the command line (though if you read the documentation you can do it via HTTP too):

influx -execute "CREATE DATABASE MetricDb1"
influx -execute "CREATE RETENTION POLICY "two_hours" ON "MetricDb1" DURATION 2h REPLICATION 1 DEFAULT"

Note: influx.exe is in influxdb-1.2.4-1, it is best to run from within that folder to avoid path problems.

As you can imagine, with those settings you are going to lose a metric some time in excess of two hours after it was written, so choose careful and read up on the documentation.

You might think the next command is CREATE TABLE… but it’s not required, it’s magic…

Inserting and selecting data from command line

For the insert the retention policy name is required, kind of like a schema name, but it’s very similar to SQL, though very reduced functionality:

influx -execute "INSERT INTO MetricDb1.two_hours cpu,host=sausages,region=eu_north value=0.69"

For the select you can leave off the retention policy when you can assume it is the default one, but do note the two full stops:

influx -execute "SELECT * FROM MetricDb1..cpu"

Or you can specify it:

influx -execute "SELECT * FROM MetricDb1.two_hours.cpu"

Inserting data from C#

using (var client = new HttpClient
{
    BaseAddress = new Uri("http://localhost:8086/"),
    Timeout = TimeSpan.FromMilliseconds(10000)
})
{
    await client.PostAsync("/write?db=MetricDb1", new StringContent("cpu,host=sausages,region=eu_north value=0.69"), CancellationToken.None);
}

Of course, opening the client every time is not how you would code it for a real application, but you get the idea.

Graphing it

  1. Open grafana web site http://localhost:3000
  2. Login with admin user, password admin (yeah, you might want to change that)
  3. From the Home Dashboad (should be the landing page when you login), click Add data source
  4. Give it a name
  5. Choose InfluxDB
  6. Url is http://localhost:8086
  7. Access is direct
  8. No Http Auth (until you configure some)
  9. Database is MetricDb1
  10. No User or Password (until you configure them)
  11. Click Add
  12. Go to @ (grafana logo) -> Dashboards -> New
  13. Click Graph
  14. Click on Panel Title then Edit. Or find some other way to edit the graph.
  15. On the Metrics tab you should be able to enter your query. I chose FROM two_hours cpu SELECT field (value) mean () GROUP BY time ($__interval) fill (null)
  16. Choose a Group by time interval, I chose >1s
  17. I wanted a bar chart, so I chose Bars option for Draw Models on Display tab.
  18. Save the dashboard when you are happy with it via disk icon or CTRL+S.

That should be all you need to get started and see data in the dashboard. There are options for auto-refresh and all manner of good things so do make sure you explore grafana. I find Last 5 minutes very useful to start with and Refresh every 5s.

Lighter tyres #cycling

I’ve always gone for the reasonably priced 26″ Schwalbe tyres. Marathon Green Guard were my normal all round choice for almost puncture free riding, but they are heavy, not Marathon Plus weight, but not light either. (see link below for comparison). The standard Marathon Kevlar (old) tyres were lighter but weren’t robust enough for my needs.

Years pass, and a more expensive tyre catches my eye, the Marathon Supreme and coming in around 440g it’s a good weight saving:

I’ve never had a folding tyre before, they look very wonky to begin with, but look good once fitted:

Because I had an old Marathon Plus on the rear, it’s reduced the weight by around 750g! I just hope they don’t yield punctures, the reviews are very positive, but only time will tell whether they are worth double the money!

The best comparison is http://www.bicyclerollingresistance.com/tour-reviews/compare/schwalbe-marathon-2015-vs-schwalbe-marathon-plus-2015-vs-schwalbe-marathon-supreme-2016

That site shows the weight saving and fractional rolling benefits too, but also less puncture resistance.

Tips learned this winter #cycling

I’ve been a bit lazy recently with writing anything down, but that doesn’t mean I’ve not been cycling or indeed discovering new tricks and tips.

Tip 1

This winter I loved my toe covers, cheap, easy to fit and I just leave them on my shoes with minimal hassle compared with overshoes. I own two pairs now. They keep my feet warm enough and even some water out too, enough to keep my toes dry on a wet commute. £9.50 at time of writing… http://www.wiggle.co.uk/dhb-toe-cover-overshoe/

Tip 2

Base layers are expensive, but don’t necessarily need to be bought from a cycle shop, you can save money by looking around. I recently bought four from a DIY shop at £8.49 each, unfortunately they are £9.99 at time of writing… http://www.screwfix.com/p/helly-hansen-kastrup-baselayer-crewneck-black-large-42-chest/55368

Tip 3

Disposable gloves for bike maintenance. I like nitrile ones at just under 8p per glove… http://www.screwfix.com/p/skytec-utah-nitrile-powder-free-disposable-gloves-blue-large-100-pack/1705g

Tip 4

Never give up, just because your gloves don’t keep your fingers warm, you just haven’t found the right ones yet… I like the look of lobster neoprene gloves… maybe I’ll try them next year.

Tip 5

For every rubbish tool you have there’s a good one that cost just a few pounds more, not every tool is worth the extra, but I’m definitely glad I spent the extra on decent wire cutters, chain whip, floor pump and tyre levers to name but a few. Last weekend I used my wire cutters again and I remembered how much I appreciate them every time I use them.

Tip 6

Muc Off or similar top brand of degreaser and bike cleaner. Degreaser is so good, I don’t use a lot over the course of a year, but every time I clean my chain and bike I’m glad I have it and can’t imagine how I ever cleaned it without it.

Tip 7

Ignore and pity the fools with mega expensive road bikes that don’t ride all year round, they may go fast in summer, but fast is expensive and a whole load more maintenance and hassle. Buy yourself a cheap and really awful road bike, try riding it to put yourself off the idea of wasting a lot of money on one.

Tip 8

Rags and old socks, don’t throw them in the bin, they are perfect for bike maintenance and cleaning, as are old scrubbing and tooth brushes.

Tip 9

On a slightly icy Monday morning, do be careful in the office car park, they won’t have gritted it…

Tip 10

Check brakes frequently in winter, a sticking caliper can be enough to help you loose traction on that office car park…

Happy cycling, spring is on its way!

New lights and rucksack #cycling

I’ve never been one to recommend without having used it for while. Enough evaluation time has passed and I’m now ready to extol the virtues of my recent purchases.

A bag is a bag? No, mostly I prefer an office pannier bag, a rucksack trumps in terms of convenience, but also gives you a sweaty back. Though the fancy bag helps a little with fancy looking back ventilation:

It’s still more sweaty than not having it on your back.

(Osprey Talon 22)

But I’m going to stick with it because it’s more convenient (and the blue matches bluebell nicely LOL). The overall weight is slightly less because I don’t need a rack either now. It is also nicer to not have the weight on the back wheel and the rattling around the contents used to get, so the bike moves over bumps better. I’m happy with the bag and my new setup.

Now, onto my favourite subject, commuting lights. See my last post for an introduction http://wp.me/p1Tdgj-11j.

Here they are:

Don’t ever be fooled by beam patterns, heat maps or photos, the only way of knowing how good something is is to get your hands on one and try it on a wet road. The packaging (and website) show impressive patterns, which never match reality.

The Busch + Müller IXON IQ Premium LED Front Light Set – 1922QMLA is pretty good and I like plugging it in rather than taking batteries out and charging separately. The fact that it takes AA batteries gives me a lot of options with this light. It also lasts around 5 hours on full power so I don’t need to charge it every day. It switches to low power when the batteries are low. Looking at run time for low power is pointless, I don’t buy a high output light to run on low power. If this were the only light that I’d bought, then I’d probably rave about it, but, the Lumotec IQ-X light is even better…

The 20 extra lux on paper doesn’t do it justice, it’s a great light, even on my old and low powered supernova S dynamo hub, it far exceeds the output of the IXON IQ Premium, however I will keep using both because I like to be prepared at the least favourable time of year. The IQ-X is a nice looking light too. Anyone wanting to get a dynamo light should definitely consider this light.

I consider both of these Busch + Müller lights affordable and reasonably priced, even when the Pound is weak against the Euro. So there’s no excuse for not treating yourself when your existing lighting isn’t up to scratch.

Dark so early? Time to buy new lights #cycling @bike24net

In the UK we’ve just lost British Summer Time daylight saving and my gosh did I notice the loss of daylight, that hour really did change my commute home. I love riding in the dark provided I’ve got adequate lighting, but my main light is no longer good enough and my spare is just about usable, so it was time to replace them.

Most of the lights I looked at were massively expensive, powerful and put more light on the sky than on the road.

In September 2012 I bought a hub dynamo and a Busch & Müller Lumotec IQ Fly RT front light from bike24.com. I like the light, it’s courteous to other road users and efficiently puts light on the road while still making me very visible, so I wanted something similar but with a bit more power.

Fortunately Busch & Müller now have a more powerful dynamo light and a very good battery light that don’t break the bank, even with the awful pound to euro exchange rate.

So I’ve ordered them from bike24.com:

Busch + Müller Lumotec IQ-X LED Front Light – 164RTSNDI-01
(http://en.bumm.de/products/dynamo-headlights/lumotec-iq-x.html)

Busch + Müller IXON IQ Premium LED Front Light Set – 1922QMLA
(http://en.bumm.de/produkte/akku-scheinwerfer/ixon-iq.html)

There aren’t such good reviews of the IXON IQ, apparently it breaks easily, well, I’ll just have to be careful with it 🙂

Fingers crossed they both arrive safely and work as well as I hope they do, it’d be nice to have something that lasts another 4 years without any issues, the cheap lights on Amazon etc won’t that’s for sure.

The weather has also become cold too, it’s only the start of November and I’m already in my winter jacket and gloves!

Time for a change of setup #cycling

For a long time now I’ve commuted with a pannier (office) bag and a bar bag. These have worked really well, especially on 18.5 miles commutes verses a rucksack on my back which put too much pressure on my body. The bar bag was a useful place to keep keys and valuables as well as my front light battery pack. I also didn’t like a sweaty-rucksack-back on hot days.

I see a lot of commuters with rucksacks and my bags are starting to show their age, so I decided to try a rucksack for a week before investing in new ones. My commute for some time has been a lot less than when I started with this setup, but I kept the same setup. Currently my commute is only 12.5 miles and the weather was hot, so it felt like a good time to try.

After a couple of weeks at 12.5 miles I can confirm:

  • One bag is better than two; carrying two bags around is more hassle
  • A rucksack is an easy bag to carry
  • Clipping bags on and off wastes time
  • A bar bag makes it difficult to fit a front light (or a second in my case), I have to have an accessory bar hanging the light below the bag
  • Luggage on the bike does seem to weigh the rear end down
  • On hot days you get a very sweaty back with or without a rucksack, so it doesn’t make much of a difference
  • A rucksack is lighter, no rack is required either
  • A rucksack is cheaper
  • A rucksack doesn’t require fitting stuff to my bike or replacing it once it is worn out or breaks

The flip side is that for longer commutes I’d still probably want to mount the bag on the bike, but I think I’d try and wean myself off the bar bag. Besides, I’m not looking to commute any more than 12.5 miles 🙂

My chain was getting worn, so I cleaned up bluebell, removed the rack, bar bag mount and accessory bar and set her up:

She’s a little more lively but still heavy, the rucksack is working well and I can always put the rack back on, it’s only 4 hex socket cap bolts after all. (Yes, that is a ~960g rear tyre, well spotted – once worn out I will replace with something a little lighter)

One day I might even fit the drop bars that my first Thorn Raven came with, but there’s other maintenance I need to do on here at some point before then.

Happy riding.

First ride #cycling

Garden / grass riding wasn’t very successful, but first ride on the road and he’s got it mastered: 
He seems good at scooting and starting too.