In this blog post we’re going to be looking at what some people might call “big” data. No that doesn’t mean big in the conventional sense, it means big in the sense that the single file dataset is 10 Gb in size, and I wanted to make a “big data” pun.
The data in question is a record of NYC’s 311 complaints since 2010, the 6th most popular dataset on the opendata website. “311” is a complaints hotline in NYC, for those interested in following along or investigating the data themselves, it is freely available from the open data website.
Today we’re going to cover
Creating a data source and importing the data
First look at the data to determine interesting fields
In my ever onward quest to show to the world how easy it is to get up and started with Logscape, today I’m going to use a Logscape docker container in order to build visualisations based off some publicly available CSV files in no time at all. If you’ve never used the Logscape docker image, then check out my previous blog.
Today we’re going to be analysing data made available via the gov.uk website, which offers statistics for crashes in the UK for the year of 2015. The specific dataset is available for download here.
For a long time, we’ve had the web app available on our app repository – This covers Nginx, Apache and a whole host of other formats, and whilst functional, it hasn’t been touched in a long time, so it looks a little bit less than pretty.
Given the popularity of Apache and NGINX, and the fact they both use the same out of the box log format, we’re going to give them a dedicated app, and a brand new look.
We’re hoping this works out for everyone. If you’re currently using the existing web app you can continue to do so. But if you’re specifically running NGINX or Apache, and want a change of pace, then read on.
Here at Logscape it should go without saying that monitoring is sort of a big deal. Some would even go as far as to say it’s even our “thing”. To go with that we’ve collated a collection, of what we think might be the best 10 monitoring talks people should watch. Regardless of whether you’re looking to implement a logging tool, build your own or are just a developer, these talks are worth the time.
It’s finally that day, Logscape is now on docker hub. As such I’m going to be walking you through the process of getting Logscape running, and once you’ve got the hang of it, you’ll be able to download, run and start using Logscape all within 60 seconds. Monitoring in a heart beat. Continue reading →
Now it’s undeniable, we techies love to argue about anything we can. Emacs or Vi? Tabs or spaces? Dark theme or Light theme? Brackets on the method line, or the next? to name but a few. We can even see examples of these arguments if you follow discussions on Twitter.