Web Log Analysis with GeoIp and Logscape 2.0.4

Understanding customer profiles is key for any web business. It allows you to gain geographic insights into visitor behaviour.

Important questions can be answered:

“How many people visited our site, from which city, which country…. and what did they do?”

Logscape 2.0.4 adds support for GeoIp in 2 ways.

  1. We now include MaxMinds GeoCityLite. It is exposed as a groovy-bound variable that allows you to interact with their java-api and bind it to your data. See the bottom of the page for more details.
  2. GeoIP is boring, until you show it on a map! For this purpose (and being D3.js fans) we have integrated using the awesome: http://datamaps.github.io/

Step 1: Extracting the GeoIp Country Code

First you need to extract a public IP address as a field. Then use that within the GeoIP field (Groovy script based). In the following example I’m taking a weblog ClientIP and extracting the country:

Import the  Web Access Log by adding a DataSource: - - [29/Nov/2013:05:16:57 -0800] "GET /videos/AndyCoatesInterview.webm HTTP/1.1" 206 19180929 "http://www.logscape.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36"

Mapping the ClientIP using the DataTypes page with a ‘weblog’ DataType:weblog-datatype

Extract the ClientIP into a country field (note the use of geoipLookup variable):

View the results on the DataType page

Step 2: Using the County Code on a Search

So lets create search and view the results in a table:


Now lets take the country field and use the chart(map) renderer:

Nice! Mouse wheel zoom and drag.

Step 3: Displaying a City-hit break down using a Bubble overlay

But we aren’t finished. Id like to show the Citys as bubbles ontop the the heatmapped world view. To do this we need a City field as follows, it returns Json which the browser will render onto the Geo-Projection:


The Json field results:

The search now has added the city.count()  to display a count based breakdown:

Note the chart legend shows the distribution using a quantised heat-map range over 6 colours.


Best Regards, Neil. 

THE MaXMIND GeoLite notice:

This product includes GeoLite data created by MaxMind, available from http://www.maxmind.com

Its a great tool btw! You can configure Logscape to use your own MaxMindGeopDB by setting the system property -Dgeoip.db=/opt/file/GeoIp.dat