Logscape 2.0 – Whats New?

Since January we have been diving into the depths of HTML5, Scala, Websockets and other amazing technologies that have recently emerged. Logscape 2.0 is our take at leveraging these tools to provide the most fluid, intuitive tool that is designed for interactive log analysis. Sure Logscape is more than log analysis, but at the end of the day – the audit trail that tells the truth about what happened comes in the form of data located on disk or in memory: usually a log file.

So, whats new?

From our site you will see the sexy new html5 interface, it continues to amaze me how well this technology runs on mobile devices. In many cases we see an iPad 2.0 outperform a Windows i3 desktop when it comes to SVG rendering. The HTML interaction is very smooth and fluid. Testing Logscape 2.0 on the mobile platforms came with a few minor challenges but to see it on a tablet, interact and work with data is a great feeling. The power of the mobile web is  truly becoming the new powerful interface.

Logscape 2.0 is Free – as in Beer

Log analysis tools are changing, their value propositions are changing, and so are we. You can now download and use the Logscape Manager and any number for Forwarders for free. This allows you to get started with minimum hassle, and then scale at fixed costs. More on this in a later post.

The 2.0 design semantic

We wanted to ‘bin’ the old flex front end and create a new look. There were many lessons learned on the road and its not often you get a fresh start. So with that in mind, we needed to make the search page more interactive, and easier to navigate. We have also adopted industry standards visualization like d3.js, while adding the ability easily plug in new visualizations. Each form or selection allows you to quickly refine results by typing a couple of letters. Everything is click-to-edit; like an interactive document. We also wanted dashboards to be different.

d3-wheel

Dashboards to Workspaces

We have also thrown out the ‘dashboards’ concept. They are replaced with ‘Workspaces’ – the idea being that they provide a richer experience of mashed-up search visualizations. They also allow you to embed html content directly within the page. Ok, so nothing new there, but most of the time when I look at a dashboard, I’m thinking, it’s just a single page with pretty charts. That’s great, but I need to know about these other things (what else is happening). I need other views/facets – to be able drill into a search or a different view. You can get stuck pretty quickly. Our solution is to allow each Workspace to link to any other Workspace or Search page. Put this in the context of a page with integrated help and hyperlinking navigation, and you have the ability to provide users with ‘decision trees’ or analysis workflows. All of this flexibility brings you a bespoke semantic visualization network that drives your users down the correct paths when finding and fixing issues.

 blog-Home
Workspace – Home (link on RHS highlighted)

<a href=”Workspace=Home – System Runtime”>- System Runtime</a>
Links – System Runtime:  

blog-Home-RT
Workspace – System Runtime

Dynamic Field Discovery

Unstructured data can contain multiple elements of structure. It’s increasingly common to dump JSON or XML into log files. Or print Key:Value patterns such as ‘user:joe.blogs’. This data is interesting, it tells you something about system behavior. Logscape 2.0 learns about your data, so when you hit search it will dynamically pick out these fields (i.e. user”) and make them searchable. From there you can quickly refine your focus to particular users or incoming IPAddresses, and spot unexpected behavior without having to think about what might be contained within. Logscape will provide you with a summarized breakdown of what fields are available, in a clickable popup. Check out work/audit.log and work/vsaudit.log for how we use it ourselves. The following example shows a popup displaying summary values for ‘COMMITTED’ – the values have been magically extracted from the highlighted line.

CPU:9 MemFree:183 MemUsePC:18300.00 DiskFree:109157 DiskUsePC:0.00 SwapFree:6182

blog-keyValue

Plotting the CPU field and changing to a line chart gives the following:

blog-KV-CPU

DataSource Wildcards

Making data searchable means it needs to be imported by adding directories, filemasks etc. We frequently find that many deployments have variations on a theme – for example some apps might be installed on different drives, or slightly different paths, or a myriad of nested directories in a particular location. Logscape 2.0 introduces wildcards which follow standard conventions. For example: ‘*’ represents a directory name. /*Server/ represents and directory ending with ‘Server’. For multiple directory recursion ‘**’ can be used. For example:

DataSource Path: /JBossServer/Cluster-*/JVM-*

Zoning

As we grow, so do our customers. We find more and more that it makes sense to structure deployments according to our customers’ data center design. Zoning is the ability to shape your Logscape deployment across regions, timezones, datasets, subnets – much like a network map – thereby “zoning” or  clustering related sets of Agents. It is the key ingredient in scaling.  We supported this in previous version, but in Logscape 2.0 it is more intuitive. Dot-notation is now part of the agent-role and includes a hierarchical fall-up. In practice this means that you install a Manager as lab.Manager, any IndexStores with lab.uk.IndexStore and UK related Forwarders into the UK zone. lab.uk.Forwarder. The image below shows how Zoning allows Logscape to scale to support multiple-geographic regions.

DataGroups

With multi-tenancy it is important to limit the views of Users. In Logscape 2.0 this has evolved from assigning a user a set of DataSource Tags (i.e. include myserver-logs) to the intriduction of DataGroups. DataGroups (found on the user tab) as the name implies is a set of DataSource tags which are included or excluded. The also inherit behaviour from other sets of DataGroups. In all, this capability allows for complete control needed in the modern enterprise. In case of lockdown scenarios there is the ability to ‘disable’ a DataGroup which will prevent the data from being visible.

blog-DataGroup

Logscape Apps on GitHib

All Logscape Apps are moving to github. Not everything has is there yet – we’re still working on it – but we’ll finish soon. Visit apps.logscape.com

GitHub-Mark-120px-plus

Cheers Neil.

All Data is Not Created Equal

The cost of analyzing log files and operational data in many companies is starting to add up. Lifting and shifting data around is expensive anyway, and per-gigabyte vendor fees are making it even more expensive.

In the past 6 months, we’ve heard from a lot of companies who’ve placed hard limits on the amount of operational data they’re willing to collect and index in their centralized log management service. Many are actively enforcing an artificial “data ceiling” to make sure that only data that’s pre-defined as highly valuable gets indexed, and everything else gets ignored.

This makes a lot of sense as a cost-control mechanism. Not all data is created equal, and some data is always going to be more valuable than other data. But it goes without saying that – as long as you have a good way to analyze it – the more data you have, the deeper and more complete the picture you can get about what’s going on. Cutting apparently lower value data out of the picture may cut your costs, but it also cuts away at your valuable insights.

With the launch of Logscape 2.0, we’re inviting companies everywhere to expand into a more holistic approach to log file analysis and operational analytics. We’re helping our customers break the data ceiling with cost-effective and massively scalable analytics for high-value AND lower value data using localized and centralized log management. Oh, and we’re also helping them index unlimited data volumes free, so they can get up and running quickly and scale over time to analyze ALL of their operational data, not just what they can afford to collect.

So from the team at Logscape, we hope you enjoy the new release – give it a spin and let us know what you think.

We’ll be sharing more stories with you about how our customers are getting deeper insights from implementing more holistic, cost-effective and massively scalable operational analytics very soon.