In this blog post we’re going to be looking at what some people might call “big” data. No that doesn’t mean big in the conventional sense, it means big in the sense that the single file dataset is 10 Gb in size, and I wanted to make a “big data” pun.
The data in question is a record of NYC’s 311 complaints since 2010, the 6th most popular dataset on the opendata website. “311” is a complaints hotline in NYC, for those interested in following along or investigating the data themselves, it is freely available from the open data website.
Today we’re going to cover
Creating a data source and importing the data
First look at the data to determine interesting fields
In my ever onward quest to show to the world how easy it is to get up and started with Logscape, today I’m going to use a Logscape docker container in order to build visualisations based off some publicly available CSV files in no time at all. If you’ve never used the Logscape docker image, then check out my previous blog.
Today we’re going to be analysing data made available via the gov.uk website, which offers statistics for crashes in the UK for the year of 2015. The specific dataset is available for download here.
Logscape 3.2 introduced native JSON support, meaning that when working with JSON data there’s no need for datatypes, instead Logscape automatically pulls the keys from your structure.
This removes the sometimes daunting configuration step, and instead lets you get straight down to business with visualising your data. With that in mind, today we’re going to be embracing our inner geek, and get to work visualising some JSON from the game EvE Online™.
Introduction Logscape Analytics’ are incredibly powerful, however, are you using them to their full potential? In this blog post we’re going to go over some of the less used analytics, show you how to use them, and hopefully inspire you to use your Logscape instance in new and exciting ways. So, without further ado let’s get into some searches. Continue reading →
Systems tend to have workflows – where an object or ticket are passed around different systems. As someone responsible for monitoring such a system, I need to be able to keep track of the events to ensure that customers get served and orders processed. My main aim in this example is to track how long it takes to process tickets.
This kind of assumes you already have a Logscape environment running. If you don’t, download it now and get started!
The long awaited version 3 update to Logscape is finally here.
In version 3 we welcome multiple new and exciting features. We’ve improved search performance, zoning and the search UI as well as a series of bug fixes. Version 3 is sure to surprise even the most experienced user. This release is bigger, better and faster! If you want to find out all you need to know about these excellent new features, and how this update will affect you, read on.
If you just want to get started, download it for free right now from our Website.
If you’ve already tried it, why not tell us what you think? We’d love to hear your feedback. Simply message us and let us know.
Recently we’ve been working on creating new learning materials for the release of Logscape 3.0.Materials appropriate for both the Logscape expert and an individual just picking Logscape up for the first time. The first person to be addressed by this was of cof course the beginner, as such here’s a 10 minute introduction to the basics of Logscape 3.0.
Hopefully this help some of our newer users, and keep an eye out for more advanced tutorials!
Today marks the last in the series of three blogs around Microsoft HPC by guest writer Ben Newton, we hope the articles have helped to demonstrate the time and thought that goes into the development of a Logscape App, for the final section Ben covers the development of the actual app that will run inside the logscape environment. You can find the past articles below. You can find more of Ben’s work on his Github page, or his LinkedIn.
So you have written an app or log – it’s brilliant, it grabs all the data you need and runs like greased lightning. All you need to do now is ensure your output file has a nice clean format – preferably one that means Logscape does all the work for you! So here are some of my top tips.
1) Add a full time stamp to every line. You wouldn’t believe how much trouble can be caused by people using just times or dates. At the best, you have to struggle to get your data properly organised. At worst, you end up with a mess and data appears in the wrong place on the graph. Do it right, set the date and time!
2) Add a time zone to that stamp. My computer will never move time-zone, surely it’ll be fine? Don’t count on it. British Summer Time changing the system time on half your servers, servers being reset to US time, data centres moving locations… All these things can and will happen. Adding the time zone to the stamp gives you a cast iron assurance that the data will always be correct. That peace of mind is worth a few bytes.