3.10 – Log Analysis performance leaps and bounds (and a new face)

Early access is now availnew-homeable to 3.10 while it undergoes client sign-off. Log-analysis just got even better! We introduce new analytic functions (percentile, eval) and a set of performance improvements.

3.10 also has a new Home Workspace and improved time-series navigation.

In version 3.03 we introduced the ability to ‘index’ a field. This improved performance and allowed the user to control the balance between search-time execution and ingest/index-time execution. Storing fields in the Index massively helps search performance, particularly where the field is expensive. In choosing, ‘Index’ on the field-editor you are doing the work once, and the search performance is improved at the expense of re-indexing whenever the groovy-script or code changes. The particular use-case this feature was developed for, was geo-ip mapping (i.e. to city or country) but it can be applied to everything. Woohoo!

In version 3.10 we introduce:

  1. POST-Aggregation inline evaluation: ‘eval()’  (eval(field/1000.0) for post-agg conversion i.e. convert bytes to Gigabytes or derive a unit-cost or something else! Use this in the post-agg way; note the use of ‘+’

    Note: EACH is used to apply the expression to every value

    Example 1: CPU: | CPU.avg(,POST) +POST.eval(CPU * 3.5)
      the same except with grouping: CPU: | CPU.sum(_host,POST) eval(EACH * 13.5)
    Example 2: (multi-expr)
    CPU: | CPU.avg(,POST) +POST.eval(CPU 3) +POST.eval(CPU 9,GB2)
    Example 3:(with grouping):
    CPU: | CPU.avg(_host,POST) +POST.eval(CPU * 100.22)
    Example 4: Boolean flagging (shows 1 where value == true) (cool!)
    CPU: | CPU.avg(_host,POST) +POST.eval(CPU < 30)
    Example 5: Constant value:
    * | _type.equals(agent-stats) eval(50,Mid) eval(50 * 2,Max))
    Example 6: Combine field values (percentile * avg will determine outlier surprise)
    *  | CPU.percentile(,90) CPU.avg(,AVG) buckets(10) eval( p90th * AVG, surprise)
    NOTE: eval() is using Parsii – more examples here: https://github.com/scireum/parsii/blob/master/src/test/java/parsii/ParserTest.java

  2. FUNCTION: percentile() – allows you the plot a range of percentiles, either for a single-aggregate value (i.e. CPU across all machines or by region – where you see the percentiles values:  1,5, 25, 50, 75, 95, 99
    Example: CPU.percentile()
    Example: CPU.percentile( ,90) // determine 90th percentile
  3. PERFORMANCE: LogFile Facet cache. That is to store the facets of all fields on a data-type for a particular file where the field summary is true, or the field is a discovered key-value pair. Why? Because on the search page, the left hand navigation panel is populated with the values structures and they are expensive to calculate. 3.04 improves performance by 40% when performing adhoc searches.
    Note: Changing the DataType will require reindexing to rebuild the Facet cache (stored in work/DB/facets/N.idx)
  4. PERFORMANCE: summary.index(): Every search analytic that builds a search visualization processes the data according to its algorithm in order to create it (count, avg etc). The problem is that if you have searches that run over the same data again and again for long periods of time, i.e. 3 months, or 6 months, then these searches will be very slow due to the amount of processing/volume of data. The introduction of a ‘summary.index()’ allows for pre-build search results at a particular granularity (10 minute default). When searching against the summary.index the results are retrieved, re-bucketed to fit the search and passed back to the visualisation.
    How do you use it?
    1) execute desired search with summary.index(write) and then
    2) with the identical search use summary.index(read).
    3) summary.index(delete) will delete the whole index regardless of time-period
    How do I maintain the index?  (i.e. schedule a periodic (hourly?) search that uses the ‘summary.index(write)’ command to keep extending the index. Each entry contains a bloom filter to prevent over-writing, so when in doubt you can execute a long ‘write’ to fill in any gaps that may have occurred.
    What chart can I use? You can use any visualization line, pie etc – as long as the underlying analytic applies.  i.e. probably not a X.by(Y)
    Where is the data stored?: in work/DB/.sidx/I-xxxx. The data is stored in 10m buckets, and a search execution will re-quantize them to the correct end-bucket. ie. roll them out to 1 day or hourly.
    How do I delete it?  (‘summary.index(delete)’) will remove the data stored on disk. work/DB/sidx/I-xxxxxx folder.
    Will it work at different zooms? i.e. can I start at 6 months and keep zooming in? Yes, the source data is being aggregated out to bigger ‘chunks’, you an zoom all the way into a 10minute bucket.
  5. A new Home Workspace. (install it by clicking on logscape-home.config on the Deployment page). This workspace features C3 with zoom and pan functionality that can be used to visually control the entire workspace.
  6. Workspace HTML Editing with syntax highlighting and folding. Now when editing a html widget on a workspace you get a nice html editing environment where you can more easily interact with your code.
  7. Optional (client-server side) control of workspace-filtering in Workspace-Charts. Until now each chart would choose whether or not an event was filterable in the legend (client-side). This choice was based upon the legend, charts like pie and maps would always re-run the search. However people was to allow the search to re-run, expecially when it cannot be filtered at the legend. i.e. CPU.percentile() will show cpu across all hosts, however the legend shows percentile values; now you can re-run the search by enabling the flag on the edit panel.

Other fixes/enhancements:

  • Workspace editing is now much more fluid when resizing or moving panels around
  • Search panel resizing is no longer thrashing on resize events making it smoother
  • Search page: click facet to remove from the search bar now works regardless of parameters i.e. click to remove _filename.count(_host)
  • Further internationalisation enhancements
  • Drilldown from Search->sources will now split multi-tags to force an ‘AND’ search