Since January we have been diving into the depths of HTML5, Scala, Websockets and other amazing technologies that have recently emerged. Logscape 2.0 is our take at leveraging these tools to provide the most fluid, intuitive tool that is designed for interactive log analysis. Sure Logscape is more than log analysis, but at the end of the day – the audit trail that tells the truth about what happened comes in the form of data located on disk or in memory: usually a log file.
So, whats new?
From our site you will see the sexy new html5 interface, it continues to amaze me how well this technology runs on mobile devices. In many cases we see an iPad 2.0 outperform a Windows i3 desktop when it comes to SVG rendering. The HTML interaction is very smooth and fluid. Testing Logscape 2.0 on the mobile platforms came with a few minor challenges but to see it on a tablet, interact and work with data is a great feeling. The power of the mobile web is truly becoming the new powerful interface.
Logscape 2.0 is Free – as in Beer
Log analysis tools are changing, their value propositions are changing, and so are we. You can now download and use the Logscape Manager and any number for Forwarders for free. This allows you to get started with minimum hassle, and then scale at fixed costs. More on this in a later post.
The 2.0 design semantic
We wanted to ‘bin’ the old flex front end and create a new look. There were many lessons learned on the road and its not often you get a fresh start. So with that in mind, we needed to make the search page more interactive, and easier to navigate. We have also adopted industry standards visualization like d3.js, while adding the ability easily plug in new visualizations. Each form or selection allows you to quickly refine results by typing a couple of letters. Everything is click-to-edit; like an interactive document. We also wanted dashboards to be different.
Dashboards to Workspaces
We have also thrown out the ‘dashboards’ concept. They are replaced with ‘Workspaces’ – the idea being that they provide a richer experience of mashed-up search visualizations. They also allow you to embed html content directly within the page. Ok, so nothing new there, but most of the time when I look at a dashboard, I’m thinking, it’s just a single page with pretty charts. That’s great, but I need to know about these other things (what else is happening). I need other views/facets – to be able drill into a search or a different view. You can get stuck pretty quickly. Our solution is to allow each Workspace to link to any other Workspace or Search page. Put this in the context of a page with integrated help and hyperlinking navigation, and you have the ability to provide users with ‘decision trees’ or analysis workflows. All of this flexibility brings you a bespoke semantic visualization network that drives your users down the correct paths when finding and fixing issues.
<a href=”Workspace=Home – System Runtime”>- System Runtime</a>
Links – System Runtime:
Dynamic Field Discovery
Unstructured data can contain multiple elements of structure. It’s increasingly common to dump JSON or XML into log files. Or print Key:Value patterns such as ‘user:joe.blogs’. This data is interesting, it tells you something about system behavior. Logscape 2.0 learns about your data, so when you hit search it will dynamically pick out these fields (i.e. user”) and make them searchable. From there you can quickly refine your focus to particular users or incoming IPAddresses, and spot unexpected behavior without having to think about what might be contained within. Logscape will provide you with a summarized breakdown of what fields are available, in a clickable popup. Check out work/audit.log and work/vsaudit.log for how we use it ourselves. The following example shows a popup displaying summary values for ‘COMMITTED’ – the values have been magically extracted from the highlighted line.
CPU:9 MemFree:183 MemUsePC:18300.00 DiskFree:109157 DiskUsePC:0.00 SwapFree:6182
Plotting the CPU field and changing to a line chart gives the following:
Making data searchable means it needs to be imported by adding directories, filemasks etc. We frequently find that many deployments have variations on a theme – for example some apps might be installed on different drives, or slightly different paths, or a myriad of nested directories in a particular location. Logscape 2.0 introduces wildcards which follow standard conventions. For example: ‘*’ represents a directory name. /*Server/ represents and directory ending with ‘Server’. For multiple directory recursion ‘**’ can be used. For example:
DataSource Path: /JBossServer/Cluster-*/JVM-*
As we grow, so do our customers. We find more and more that it makes sense to structure deployments according to our customers’ data center design. Zoning is the ability to shape your Logscape deployment across regions, timezones, datasets, subnets – much like a network map – thereby “zoning” or clustering related sets of Agents. It is the key ingredient in scaling. We supported this in previous version, but in Logscape 2.0 it is more intuitive. Dot-notation is now part of the agent-role and includes a hierarchical fall-up. In practice this means that you install a Manager as lab.Manager, any IndexStores with lab.uk.IndexStore and UK related Forwarders into the UK zone. lab.uk.Forwarder. The image below shows how Zoning allows Logscape to scale to support multiple-geographic regions.
With multi-tenancy it is important to limit the views of Users. In Logscape 2.0 this has evolved from assigning a user a set of DataSource Tags (i.e. include myserver-logs) to the intriduction of DataGroups. DataGroups (found on the user tab) as the name implies is a set of DataSource tags which are included or excluded. The also inherit behaviour from other sets of DataGroups. In all, this capability allows for complete control needed in the modern enterprise. In case of lockdown scenarios there is the ability to ‘disable’ a DataGroup which will prevent the data from being visible.
Logscape Apps on GitHib
All Logscape Apps are moving to github. Not everything has is there yet – we’re still working on it – but we’ll finish soon. Visit apps.logscape.com