Logscape 3.22 Arrives

It’s Alive! Download Logscape 3.22researcher_translation

Today Logscape 3.22 becomes available to the public, we’re really excited and hope everyone’s going to love the improvements that come with it. We’ve packed in numerous performance tweaks; but we’ve also started to focus heavily on UI/UX to make the Logscape experience better for you, our users. In case you’ve missed it, you can grab the newest release from the website. Without further ado, let’s get onto some of the highlights of the 3.22 release.  Continue reading

Failover: Keeping your Environment alive!

Why Everything I Have Is Broken

Computers break… it’s a fact of life! Sometimes it’s a nice quick fix, such as the one cunningly suggested by Randall Munroe. Sometimes it can take hours of trawling through logs. Regardless of the reason, in an ideal world,you want to fix the problem as soon as possible; but the next best thing is to have a Failover – another server that works just as well!

In a Logscape environment, your Management agent is the central point of your environment. It controls alerts, provides users access and runs the entire system: without it, you have nothing. So how do you make sure that your environment is resilient against a Management Agent failure? Simple… you add another one!

The recent 3.2 Logscape release has added new and improved Failover capabilities, making it easier to provide a seamless environment for your users. Having recently implemented this feature in my environment, I thought I’d share with you both the benefits – and the possible pitfalls – of installing this useful bit of kit.

You will need:

  • An existing Logscape Environment with a subscription (Failover is not supported without a license) running at least version 3.2.
  • A server you wish to make your new Failover Agent.
  • A little bit of understanding about ports
  • A pinch of bravery.

Continue reading

Native JSON Support

json_bumper.sh-600x600Working with JSON in Logscape 3.2

Logscape 3.2 introduced native JSON support, meaning that when working with JSON data there’s no need for datatypes, instead Logscape automatically pulls the keys from your structure.

This removes the sometimes daunting configuration step, and instead lets you get straight down to business with visualising your data. With that in mind, today we’re going to be embracing our inner geek, and get to work visualising some JSON from the game EvE Online™.


 

Continue reading

Logscape 3.2 Touches Down

ssksLVBLogscape version 3.2 is now available for public download, you can get it now from the Logscape Website.

A brief rundown of Logscape 3.2 brings with it, and what we’re going to cover today…

  • File Explorer
  • JSON Support (Including JSON Arrays)
  • Failover Overhaul
  • Performance and Stability Changes

 


 

Continue reading

Converting Splunk Searches into Logscape

universal_converter_boxConverting Splunk searches into Logscape

Logscape and Splunk share a lot of overlap, and there is one question we get asked quite often by people looking to migrate from Splunk to Logscape.

 How do we convert Splunk searches and Workspaces into Logscape?

Unfortunately, there’s no magic cure or just click here style solution. Fortunately it is significantly easier than you think.

We’re going to cover converting Splunk searches into their Logscape equivalent.

Continue reading

boot.properties – 5 hints for Logscape Environments

A Logscape Agents is incredibly powerful: it might be a Forwarder shipping data or an IndexStore receiving it. It might even be a Management Agent providing the web front end. Regardless of what it may become, they all start from the boot.properties file. This small, innocuous looking file sitting in the Logscape folder is what makes the difference between a powerful, resource consuming Manager and a small, lightweight forwarder. Here are 5 useful tips for dealing with this file. Continue reading

Advanced data analytics and use-cases in Logscape

Introduction
self_descriptionLogscape Analytics’ are incredibly powerful, however, are you using them to their full potential? In this blog post we’re going to go over some of the less used analytics, show you how to use them, and hopefully inspire you to use your Logscape instance in new and exciting ways. So, without further ado let’s get into some searches. Continue reading

3.03 is here (and now)

Performance :
For this release we carried out more work around execution performance.
Single threaded benchmarking takes 2 profiles. Search page and Workspace oriented execution. When a Search is executed from the Search page it builds the facets stats to support adhoc analysis; it also streams a large set of events to the JettyWebServer. All the extra work yields about a 40% overhead, and we were seeing about 80k events per second for a single thread (30-40 discovered fields). The Workspace execution plan yields 120k per second, per thread.
The execution plan follows these steps:
1. Identify log files in the selected time period and meet the system field criteria (i.e. _agent, _type, _tag etc)
2. Select the time-series buckets associated with each resource
3. Scan the time-series buckets and build data-type patterns, synthetics and discovered fields for each event. (Using indexed fields is much faster that synthetics – 3.03 enhancement)
4. Aggregate and pump data using map-reduce execution of the functions(avg, count etc) (3.03)
5. Jetty Aggregate the incoming streams and drive the interface using websockets
6. Websocket events then send status messages, notification of replay-events (3.03), facets and updated histogram data.
Note I: 3.03 – marks where performance improvements were made.
Note II: A single thread processing 100,000 events is sustainable, 16 threads should process an equivalent of 1,600,000 events per second (in theory). Scalability depends upon I/O subsystem performance relating to disk-io, os-buffers and network.
Note III: Logfile processing is carried out with 1-thread per request.
Important: Before upgrading remember to: 1) backup your config, 2) backup the downloads and space folders (in case of reversion). 3) make sure all agents are online!
Release notes:
1. Fix summation problem where only the first event was evaluated
2. Further performance improvements on search performance and UI interaction
3. Ability to index any  field; discovered or synth (yields faster performance and requires reindexing)
4. Improved data types page for debugging and benchmarking
5. Datasources now use natural keys instead of UUIDs. This should combat DS duplication when importing.exporting. Note: ids only generated on new DS’s being saved
6. You can set the java.tmp.io.dir in boot.properties (boot.properties sets it to work/tmp by default. The directory is cleared on rebooting logscape. When upgrading you will need to perform this operation manually.
7.  Networking now uses faster lz4 compression. This will make offline agents break if not updated!
8. Geo-maps now use a chloropleth palette
9. Workspace linking now forces correct filtering when driven via URL clicks
10.  Fixed random hs_err crashing caused by ChronicleQ fixed
11. Search page chain.button now saves state and auto-runs search when auto-run is enabled
12. Rickshaw charts now format numbers with ‘,’ on mouse-over
13. Syslog no longer prints to stdout

Correlated Alerts in Logscape

computer_problemsIn my experience Correlated Alerts are something the average user doesn’t touch on, either thinking that they don’t need them, or believing (falsely) that setting up a correlated alert is much more difficult than it is. While correlated events can be used on almost any form of data, my personal opinion is that they’re at their best when dealing with data such as audit or webserver logs, but truly they will excel in any environment that makes use of error messages or codes. Today, I’m going to walk you through setting up a correlated alert, just to show how easy it is.

Continue reading