Native JSON Support

json_bumper.sh-600x600Working with JSON in Logscape 3.2

Logscape 3.2 introduced native JSON support, meaning that when working with JSON data there’s no need for datatypes, instead Logscape automatically pulls the keys from your structure.

This removes the sometimes daunting configuration step, and instead lets you get straight down to business with visualising your data. With that in mind, today we’re going to be embracing our inner geek, and get to work visualising some JSON from the game EvE Onlineā„¢.


 

Lets get started

I’ve got a simple program that subscribes to a Redis endpoint, it’s a fairly simple endpoint, all it does is provide up to date JSON information about the world of EvE, the problems begin when you try to eye ball one of these lines, it’s fair to say they’re not particularly small, and they’re not particularly easy to understand at a glance.

So naturally, let’s take a look at one of the log lines.

{“package”:{“killID”:53309774,”killmail”:{“solarSystem”:{“id_str”:”30003686″,”href”:”https:\/\/public-crest.eveonline.com\/solarsystems\/30003686\/”,”id”:30003686,”name”:”O1Y-ED”},”killID”:53309774,”killTime”:”2016.04.15 14:42:02″,”attackers”:[{“shipType”:{“id_str”:”11200″,”href”:”https:\/\/public-crest.eveonline.com\/types\/11200\/”,”id”:11200,”name”:”Taranis”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/11200_128.png”}},”corporation”:{“id_str”:”1000167″,”href”:”https:\/\/public-crest.eveonline.com\/corporations\/1000167\/”,”id”:1000167,”name”:”State War Academy”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Corporation\/1000167_128.png”}},”character”:{“id_str”:”96360193″,”href”:”https:\/\/public-crest.eveonline.com\/characters\/96360193\/”,”id”:96360193,”name”:”Fasher”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Character\/96360193_128.jpg”}},”damageDone_str”:”2001″,”weaponType”:{“id_str”:”2456″,”href”:”https:\/\/public-crest.eveonline.com\/types\/2456\/”,”id”:2456,”name”:”Hobgoblin II”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/2456_128.png”}},”finalBlow”:false,”securityStatus”:-0.8,”damageDone”:2001},{“shipType”:{“id_str”:”11200″,”href”:”https:\/\/public-crest.eveonline.com\/types\/11200\/”,”id”:11200,”name”:”Taranis”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/11200_128.png”}},”corporation”:{“id_str”:”1000168″,”href”:”https:\/\/public-crest.eveonline.com\/corporations\/1000168\/”,”id”:1000168,”name”:”Federal Navy Academy”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Corporation\/1000168_128.png”}},”character”:{“id_str”:”96356199″,”href”:”https:\/\/public-crest.eveonline.com\/characters\/96356199\/”,”id”:96356199,”name”:”Horex Coogan”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Character\/96356199_128.jpg”}},”damageDone_str”:”1736″,”weaponType”:{“id_str”:”3170″,”href”:”https:\/\/public-crest.eveonline.com\/types\/3170\/”,”id”:3170,”name”:”Light Ion Blaster II”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/3170_128.png”}},”finalBlow”:true,”securityStatus”:-0.9,”damageDone”:1736}],”attackerCount”:2,”victim”:{“alliance”:{“id_str”:”99005338″,”href”:”https:\/\/public-crest.eveonline.com\/alliances\/99005338\/”,”id”:99005338,”name”:”Pandemic Horde”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Alliance\/99005338_128.png”}},”damageTaken”:3737,”items”:[{“singleton”:0,”itemType”:{“id_str”:”222″,”href”:”https:\/\/public-crest.eveonline.com\/types\/222\/”,”id”:222,”name”:”Antimatter Charge S”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/222_128.png”}},”quantityDropped_str”:”13″,”flag”:31,”singleton_str”:”0″,”quantityDropped”:13,”flag_str”:”31″},{“singleton”:0,”itemType”:{“id_str”:”15331″,”href”:”https:\/\/public-crest.eveonline.com\/types\/15331\/”,”id”:15331,”name”:”Metal Scraps”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/15331_128.png”}},”quantityDropped_str”:”2″,”flag”:5,”singleton_str”:”0″,”quantityDropped”:2,”flag_str”:”5″},{“singleton”:0,”itemType”:{“id_str”:”8517″,”href”:”https:\/\/public-crest.eveonline.com\/types\/8517\/”,”id”:8517,”name”:”Medium F-S9 Regolith Compact Shield Extender”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/8517_128.png”}},”quantityDestroyed_str”:”1″,”flag”:20,”flag_str”:”20″,”singleton_str”:”0″,”quantityDestroyed”:1},{“singleton”:0,”itemType”:{“id_str”:”563″,”href”:”https:\/\/public-crest.eveonline.com\/types\/563\/”,”id”:563,”name”:”Light Ion Blaster I”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/563_128.png”}},”quantityDestroyed_str”:”1″,”flag”:30,”flag_str”:”30″,”singleton_str”:”0″,”quantityDestroyed”:1},{“singleton”:0,”itemType”:{“id_str”:”9944″,”href”:”https:\/\/public-crest.eveonline.com\/types\/9944\/”,”id”:9944,”name”:”Magnetic Field Stabilizer I”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/9944_128.png”}},”quantityDropped_str”:”1″,”flag”:13,”singleton_str”:”0″,”quantityDropped”:1,”flag_str”:”13″},{“singleton”:0,”itemType”:{“id_str”:”11563″,”href”:”https:\/\/public-crest.eveonline.com\/types\/11563\/”,”id”:11563,”name”:”Micro Auxiliary Power Core I”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/11563_128.png”}},”quantityDropped_str”:”1″,”flag”:11,”singleton_str”:”0″,”quantityDropped”:1,”flag_str”:”11″},{“singleton”:0,”itemType”:{“id_str”:”222″,”href”:”https:\/\/public-crest.eveonline.com\/types\/222\/”,”id”:222,”name”:”Antimatter Charge S”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/222_128.png”}},”quantityDropped_str”:”13″,”flag”:30,”singleton_str”:”0″,”quantityDropped”:13,”flag_str”:”30″},{“singleton”:0,”itemType”:{“id_str”:”222″,”href”:”https:\/\/public-crest.eveonline.com\/types\/222\/”,”id”:222,”name”:”Antimatter Charge S”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/222_128.png”}},”quantityDropped_str”:”1386″,”flag”:5,”singleton_str”:”0″,”quantityDropped”:1386,”flag_str”:”5″},{“singleton”:0,”itemType”:{“id_str”:”563″,”href”:”https:\/\/public-crest.eveonline.com\/types\/563\/”,”id”:563,”name”:”Light Ion Blaster I”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/563_128.png”}},”quantityDestroyed_str”:”1″,”flag”:29,”flag_str”:”29″,”singleton_str”:”0″,”quantityDestroyed”:1},{“singleton”:0,”itemType”:{“id_str”:”222″,”href”:”https:\/\/public-crest.eveonline.com\/types\/222\/”,”id”:222,”name”:”Antimatter Charge S”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/222_128.png”}},”quantityDestroyed_str”:”13″,”flag”:33,”flag_str”:”33″,”singleton_str”:”0″,”quantityDestroyed”:13},{“singleton”:0,”itemType”:{“id_str”:”563″,”href”:”https:\/\/public-crest.eveonline.com\/types\/563\/”,”id”:563,”name”:”Light Ion Blaster I”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/563_128.png”}},”quantityDropped_str”:”1″,”flag”:32,”singleton_str”:”0″,”quantityDropped”:1,”flag_str”:”32″},{“singleton”:0,”itemType”:{“id_str”:”31526″,”href”:”https:\/\/public-crest.eveonline.com\/types\/31526\/”,”id”:31526,”name”:”Small Hybrid Burst Aerator I”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/31526_128.png”}},”quantityDestroyed_str”:”1″,”flag”:92,”flag_str”:”92″,”singleton_str”:”0″,”quantityDestroyed”:1},{“singleton”:0,”itemType”:{“id_str”:”5443″,”href”:”https:\/\/public-crest.eveonline.com\/types\/5443\/”,”id”:5443,”name”:”Faint Epsilon Scoped Warp Scrambler”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/5443_128.png”}},”quantityDropped_str”:”1″,”flag”:5,”singleton_str”:”0″,”quantityDropped”:1,”flag_str”:”5″},{“singleton”:0,”itemType”:{“id_str”:”25861″,”href”:”https:\/\/public-crest.eveonline.com\/types\/25861\/”,”id”:25861,”name”:”Salvager I”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/25861_128.png”}},”quantityDropped_str”:”1″,”flag”:34,”singleton_str”:”0″,”quantityDropped”:1,”flag_str”:”34″},{“singleton”:0,”itemType”:{“id_str”:”222″,”href”:”https:\/\/public-crest.eveonline.com\/types\/222\/”,”id”:222,”name”:”Antimatter Charge S”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/222_128.png”}},”quantityDestroyed_str”:”13″,”flag”:28,”flag_str”:”28″,”singleton_str”:”0″,”quantityDestroyed”:13},{“singleton”:0,”itemType”:{“id_str”:”563″,”href”:”https:\/\/public-crest.eveonline.com\/types\/563\/”,”id”:563,”name”:”Light Ion Blaster I”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/563_128.png”}},”quantityDestroyed_str”:”1″,”flag”:28,”flag_str”:”28″,”singleton_str”:”0″,”quantityDestroyed”:1},{“singleton”:0,”itemType”:{“id_str”:”41055″,”href”:”https:\/\/public-crest.eveonline.com\/types\/41055\/”,”id”:41055,”name”:”Heavy Karelin Scoped Stasis Grappler”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/41055_128.png”}},”quantityDropped_str”:”1″,”flag”:5,”singleton_str”:”0″,”quantityDropped”:1,”flag_str”:”5″},{“singleton”:0,”itemType”:{“id_str”:”563″,”href”:”https:\/\/public-crest.eveonline.com\/types\/563\/”,”id”:563,”name”:”Light Ion Blaster I”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/563_128.png”}},”quantityDropped_str”:”1″,”flag”:27,”singleton_str”:”0″,”quantityDropped”:1,”flag_str”:”27″},{“singleton”:0,”itemType”:{“id_str”:”222″,”href”:”https:\/\/public-crest.eveonline.com\/types\/222\/”,”id”:222,”name”:”Antimatter Charge S”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/222_128.png”}},”quantityDropped_str”:”13″,”flag”:32,”singleton_str”:”0″,”quantityDropped”:13,”flag_str”:”32″},{“singleton”:0,”itemType”:{“id_str”:”222″,”href”:”https:\/\/public-crest.eveonline.com\/types\/222\/”,”id”:222,”name”:”Antimatter Charge S”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/222_128.png”}},”quantityDestroyed_str”:”13″,”flag”:29,”flag_str”:”29″,”singleton_str”:”0″,”quantityDestroyed”:13},{“singleton”:0,”itemType”:{“id_str”:”6073″,”href”:”https:\/\/public-crest.eveonline.com\/types\/6073\/”,”id”:6073,”name”:”Medium Compact Pb-Acid Cap Battery”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/6073_128.png”}},”quantityDropped_str”:”1″,”flag”:5,”singleton_str”:”0″,”quantityDropped”:1,”flag_str”:”5″},{“singleton”:0,”itemType”:{“id_str”:”5493″,”href”:”https:\/\/public-crest.eveonline.com\/types\/5493\/”,”id”:5493,”name”:”Type-D Restrained Expanded Cargo”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/5493_128.png”}},”quantityDestroyed_str”:”1″,”flag”:5,”flag_str”:”5″,”singleton_str”:”0″,”quantityDestroyed”:1},{“singleton”:0,”itemType”:{“id_str”:”563″,”href”:”https:\/\/public-crest.eveonline.com\/types\/563\/”,”id”:563,”name”:”Light Ion Blaster I”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/563_128.png”}},”quantityDropped_str”:”1″,”flag”:31,”singleton_str”:”0″,”quantityDropped”:1,”flag_str”:”31″},{“singleton”:0,”itemType”:{“id_str”:”9944″,”href”:”https:\/\/public-crest.eveonline.com\/types\/9944\/”,”id”:9944,”name”:”Magnetic Field Stabilizer I”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/9944_128.png”}},”quantityDestroyed_str”:”1″,”flag”:12,”flag_str”:”12″,”singleton_str”:”0″,”quantityDestroyed”:1},{“singleton”:0,”itemType”:{“id_str”:”6001″,”href”:”https:\/\/public-crest.eveonline.com\/types\/6001\/”,”id”:6001,”name”:”1MN Y-S8 Compact Afterburner”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/6001_128.png”}},”quantityDropped_str”:”1″,”flag”:19,”singleton_str”:”0″,”quantityDropped”:1,”flag_str”:”19″},{“singleton”:0,”itemType”:{“id_str”:”31752″,”href”:”https:\/\/public-crest.eveonline.com\/types\/31752\/”,”id”:31752,”name”:”Small Anti-Thermal Screen Reinforcer I”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/31752_128.png”}},”quantityDestroyed_str”:”1″,”flag”:93,”flag_str”:”93″,”singleton_str”:”0″,”quantityDestroyed”:1},{“singleton”:0,”itemType”:{“id_str”:”31740″,”href”:”https:\/\/public-crest.eveonline.com\/types\/31740\/”,”id”:31740,”name”:”Small Anti-Kinetic Screen Reinforcer I”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/31740_128.png”}},”quantityDestroyed_str”:”1″,”flag”:94,”flag_str”:”94″,”singleton_str”:”0″,”quantityDestroyed”:1},{“singleton”:0,”itemType”:{“id_str”:”222″,”href”:”https:\/\/public-crest.eveonline.com\/types\/222\/”,”id”:222,”name”:”Antimatter Charge S”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/222_128.png”}},”quantityDropped_str”:”13″,”flag”:27,”singleton_str”:”0″,”quantityDropped”:13,”flag_str”:”27″},{“singleton”:0,”itemType”:{“id_str”:”563″,”href”:”https:\/\/public-crest.eveonline.com\/types\/563\/”,”id”:563,”name”:”Light Ion Blaster I”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/563_128.png”}},”quantityDestroyed_str”:”1″,”flag”:33,”flag_str”:”33″,”singleton_str”:”0″,”quantityDestroyed”:1}],”damageTaken_str”:”3737″,”character”:{“id_str”:”96337668″,”href”:”https:\/\/public-crest.eveonline.com\/characters\/96337668\/”,”id”:96337668,”name”:”Mubix”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Character\/96337668_128.jpg”}},”shipType”:{“id_str”:”16240″,”href”:”https:\/\/public-crest.eveonline.com\/types\/16240\/”,”id”:16240,”name”:”Catalyst”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Type\/16240_128.png”}},”corporation”:{“id_str”:”98388312″,”href”:”https:\/\/public-crest.eveonline.com\/corporations\/98388312\/”,”id”:98388312,”name”:”Pandemic Horde Inc.”,”icon”:{“href”:”http:\/\/imageserver.eveonline.com\/Corporation\/98388312_128.png”}},”position”:{“y”:26244834761.75,”x”:-401723349635.93,”z”:-513393252981.4}},”killID_str”:”53309774″,”attackerCount_str”:”2″,”war”:{“href”:”https:\/\/public-crest.eveonline.com\/wars\/0\/”,”id”:0,”id_str”:”0″}},”zkb”:{“locationID”:40233450,”hash”:”18f2ab1c7999394d7deaedc2e5715871d9d706b3″,”totalValue”:4472955.62,”points”:14,”href”:”https:\/\/public-crest.eveonline.com\/killmails\/53309774\/18f2ab1c7999394d7deaedc2e5715871d9d706b3\/”}}}

Seems big right? In reality, we get events significantly larger than this.

Piping this data into Logscape, we get the following.

k8YkiEz

As you can see, all of my fields are automatically pulled out, so let’s start building some visualizations.

QfTnaXu

Within just a few minutes, I’ve managed to import my data, search it – and build a visualization based upon the most active alliances within my dataset. One of the key things to note here, is that per JSON event, it is possible to have an unlimited number of attacks, in the search syntax, you’ll notice the ‘[]’ field, that means, perform this for every single value returned here, as such, we’re taking a count of tens of thousands of attackers, despite only searching 5,900 events and 454 unique alliance names.

So this shows, pulling in an entire JSON array, but what if I was just interested, in say, the usage of a certain ship?

Wf71iOC

To get this information it again has to fully expand the attackers array, and then count a single attribute within the array.

What if we want to go even deeper? I want the item, on a ship, that belongs to an attacker.

PSSXlqV

As you can see, Logscape JSON support means that I can work with huge data sets, and get value from it in no time at all.

I hope you enjoyed this brief presentation of JSON support in Logscape 3.2

If you’re interested in Logscape, you can download it for free.