Today marks the last in the series of three blogs around Microsoft HPC by guest writer Ben Newton, we hope the articles have helped to demonstrate the time and thought that goes into the development of a Logscape App, for the final section Ben covers the development of the actual app that will run inside the logscape environment. You can find the past articles below. You can find more of Ben’s work on his Github page, or his LinkedIn.
Today Ben Newton returns for the second in a series of three Blog articles covering his progression through building a monitoring solution for Microsoft HPC through Logscape, todays article covers Data collection, both in where the data was sourced, and how he chose to format the data. You can find more of Ben’s work on his Github page, or his LinkedIn.
Data! Data! Data!” he cried impatiently. “I can’t make bricks without clay.
-Sherlock Holmes, The Adventure of the Copper Beeches
Today we have a blog article by Guest writer Ben Newton, Ben manages a HPC Grid where Logscape is used in anger for all their management needs. This is the first of a three part series by Ben on how they use Logscape, and built their monitoring Solution. You can find more of Ben’s work on his github page, or his LinkedIn.
Microsoft HPC Server 2012 – More Compute, More Monitoring
640K ought to be enough for anybody.
– Unknown… but not Bill Gates! Continue reading
So you have written an app or log – it’s brilliant, it grabs all the data you need and runs like greased lightning. All you need to do now is ensure your output file has a nice clean format – preferably one that means Logscape does all the work for you! So here are some of my top tips.
1) Add a full time stamp to every line. You wouldn’t believe how much trouble can be caused by people using just times or dates. At the best, you have to struggle to get your data properly organised. At worst, you end up with a mess and data appears in the wrong place on the graph. Do it right, set the date and time!
2) Add a time zone to that stamp. My computer will never move time-zone, surely it’ll be fine? Don’t count on it. British Summer Time changing the system time on half your servers, servers being reset to US time, data centres moving locations… All these things can and will happen. Adding the time zone to the stamp gives you a cast iron assurance that the data will always be correct. That peace of mind is worth a few bytes.
Docker 1.5 came out a few weeks ago and with it the new stats api arrived. Before 1.5 there was no standard way to collect the metrics of running docker containers without writing custom scripts to parse files stored in the proc memory file system.
Sitting at my desk, I was approached and asked this question.
Do you guys support Platform Symphony?
An interesting question, really we “support” most anything, Logscape has the base functionality for most tasks, and product “apps” just build upon that functionality, the app provides the workspaces, data types and sources, as well as the tools that Logscape will use to find the data you want. But at the end of the day it’s still Logscape, just it’s running a groovy script to fetch your metrics.
So what is the process that goes into making a Logscape App?
Release 2.5.1 includes enhancements to JMX functionality, improvements to Logscape roll detection aswell as significant search performance optimisation.
2.5.1 Introduces new linking functionality. This allows you to flawlessly link your workspaces and transfer context between them providing a troubleshooting workflow for non-expert users of the Logscape system.
The release notes are here.
Deploying Logscape in the cloud has its challenges. Logscape is a cpu and i/o intensive application and finding an optimal configuration between disk, i/o and cost requires extensive benchmarking and tests.
Recently we started looking at Amazon. We have now have three stages to our benchmarking.
In preparation for release Logscape as a SAAS solution, the Collectd App has been updated. The CollectdApp is one of the first apps to support Logcape Cloud.
- write_graphite – This release now uses the graphite plugin to import collectd metrics.
- Workspaces Update – the workspaces have all been updated
- Zip – The app is now available as a zip. Whereas before it was distributed as a config file.