Logscape 2.0.4 is out!

Logscape 2.0.4 is now available for download. It features GeoIp field extraction and D3Datamaps for visualisation. Part of the release also included updating to D3V3

The release includes work around improving Journaling reliability with some of the technology being used (MapDB and Persistit).

Into next year we will continue to drive the product to extract more performance as well as improve the user experience by adding high-level workspace filters and linkage. In other work we are also looking to provide a light-https-agent which can be used in a ‘disconnected’ environment such as cloud or through remote network links.

Web Log Analysis with GeoIp and Logscape 2.0.4

Understanding customer profiles is key for any web business. It allows you to gain geographic insights into visitor behaviour.

Important questions can be answered:

“How many people visited our site, from which city, which country…. and what did they do?”

Logscape 2.0.4 adds support for GeoIp in 2 ways.

  1. We now include MaxMinds GeoCityLite. It is exposed as a groovy-bound variable that allows you to interact with their java-api and bind it to your data. See the bottom of the page for more details.
  2. GeoIP is boring, until you show it on a map! For this purpose (and being D3.js fans) we have integrated using the awesome: http://datamaps.github.io/

Step 1: Extracting the GeoIp Country Code

First you need to extract a public IP address as a field. Then use that within the GeoIP field (Groovy script based). In the following example I’m taking a weblog ClientIP and extracting the country:

Import the  Web Access Log by adding a DataSource: - - [29/Nov/2013:05:16:57 -0800] "GET /videos/AndyCoatesInterview.webm HTTP/1.1" 206 19180929 "http://www.logscape.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36"

Mapping the ClientIP using the DataTypes page with a ‘weblog’ DataType:weblog-datatype

Extract the ClientIP into a country field (note the use of geoipLookup variable):

View the results on the DataType page

Step 2: Using the County Code on a Search

So lets create search and view the results in a table:


Now lets take the country field and use the chart(map) renderer:

Nice! Mouse wheel zoom and drag.

Step 3: Displaying a City-hit break down using a Bubble overlay

But we aren’t finished. Id like to show the Citys as bubbles ontop the the heatmapped world view. To do this we need a City field as follows, it returns Json which the browser will render onto the Geo-Projection:


The Json field results:

The search now has added the city.count()  to display a count based breakdown:

Note the chart legend shows the distribution using a quantised heat-map range over 6 colours.


Best Regards, Neil. 

THE MaXMIND GeoLite notice:

This product includes GeoLite data created by MaxMind, available from http://www.maxmind.com

Its a great tool btw! You can configure Logscape to use your own MaxMindGeopDB by setting the system property -Dgeoip.db=/opt/file/GeoIp.dat