Docker 1.5 came out a few weeks ago and with it the new stats api arrived. Before 1.5 there was no standard way to collect the metrics of running docker containers without writing custom scripts to parse files stored in the proc memory file system.
Developing a responsive highly available web application is a complex task with many subcomponents. In the old days , a web application used to consist of a web server, usually Apache with a few cgi scripts to provide content dynamically. As the web grew in size so did the stack, the number of subcomponents,hosts involved increased in number and complexity. A simple web application can now easily consist of a web server, a load balancer, a few databases and a web framework such as Django or Rails. In this blog post we are going to look at the different Logscape Apps needed to monitor a web farm.
MongoDB is a popular open source nosql database. It’s gain in popularity has been due to ease of use, speed and options for scalability.The MongoDB App collects health and operational metrics for MongoDB cluster or group mongo instances.
Monitoring and analysing network traffic is difficult due to the sheer volume of data produced by network devices. Even if you do have the data, an experienced network engineer is also needed to decipher and analyse the log data adding to the total cost of ownership of network operations. Any downtime can be expensive and has a direct impact on business operation.
Using a Log Analysis tool improves a network teams ability to find the root cause of a problem and sometimes detect a problem before it occurs. Once embeded in the team workflow it makes analysing network traffic routine and efficient. In this blog post we are going to continue our look at analysing Cisco ASA data using the CiscoApp
In this blog post we are going to take a look at some of the sensors that come with Collectd and how to chart the information. Here are some useful links on searching in Logscape.
Before we get straight into the searches lets take some time and understand how Collectd outputs data.
Collectd Metric Data
The collectd daemon stores data from other linux host on disk by the server name and the plugin. The load plugin is set by default on most Linux distributions and is stored like this.
In this set up many collectd daemons are forwarding their metric data to one central host.
Using the Collectd DataType
To chart the data in Logscape you need a passing familiarity with the how to search using a data type.
To execute a search I would need to know which Collectd plugin I am interested in and what metrics it outputs. The table of all collectd plugins can be found here. Here’s an example which charts the load of a host svr0001
| _type.equals(collectd) plugin.equals(load) value.max(id) _host.equals(svr0001)
One thing to note from the search is the use of the id field. This field is a unique key which identifies each unique metric value by host, plugin and instance.
Lets take a look at a few other search examples
Today businesses face an array of external and internal threats to their corporate network. Protecting business operations from security threats requires vigilance, experience and excellent tools.
Recently one of our customers found that alert emails from a trade capture system where being blocked by external mail servers. Further investigation showed that all traffic from the company mail server had been blacklisted and were being marked as spam.
Fortunately the sysadmin had Logscape installed actively monitoring all Cisco router traffic.
Outbound Connection Analysis
The Composite Blocking List (CBL) keeps a list of blacklisted mail servers with suspicious traffic. Many mail servers use this list to reject suspicious mail traffic. The first step for the sysadmin was to analyse all outbound traffic coming from within the company.
The first search he built showed all outbound mail connections recorded by the Cisco router grouped by the outbound mail-servers ip address. In a search like this you would expect to see connections to a short list of company approved mail server. This wasn’t the case.
| _type.equals(cisco-asa) dstAddress.count() dstPort.equals(25)
On the 10th of April there is a distinctive spike in mail traffic. The blue parts of the search results represent expected mail activity but the multi coloured spike indicates suspicious activity. Continue reading
Logging and monitoring system health is a hot topic where operational engineers manage large server estates. There are many solutions out there that solve a piece of the puzzle of how the metrics are generated, where the metric data is stored and how it is then visualized.
In this blog post we are going to take a look at Collectd and how to integrate this with Logscape. Collectd is an excellent monitoring backend for collecting operating system metrics. Collectd has around 90+ plugins including hardware sensors such as temperature and power usage. Metric data by itself is of little use unless you can visualize it in some way or fire alerts based on trends in the systems under supervision.
This table shows some of the available sensors being collected. There are abount 32 different sensors from 8 different hosts being imported in this environemnt. Here is dashboard of system health KPIs.
CPU Temperatures and Wattage metrics depend on the server hardware. Continue reading