As of v1.35.0 the Netdata Agent can now run Metric Correlations (MC) itself. This means that, for nodes with MC enabled, the Metric Correlations feature just got a whole lot faster!
The Netdata Metric Correlations feature uses a Two Sample Kolmogorov-Smirnov test to look for which metrics have a significant distributional change around a highlighted window of interest. This can be useful when you are interested in short term “change detection” and want to try answer the question “what else changed around this time?”.
Our original implementation was via a Python FastAPI based cloud service that wrapped up the scipy.stats.ks_2samp() function of the popular open source SciPy package.
While this has worked well, it requires the underlying raw input data (all metrics!) to be sent over the network to the cloud service for the statistical test computation to happen. Results are then returned to the Netdata Cloud frontend to filter for only those metrics that have changed the most (according to the statistical test). Obviously, this means some latency while all this data gets sent around, so users could be waiting anywhere from 5 to up to 25 seconds for results in some cases.
So, to give users a faster option, we decided to re-implement the whole algorithm on the Agent itself in C (here is the PR if you’d like to geek out a bit). This means that Netdata Agents now have a new /api/v1/metric_correlations endpoint that can run the MC algorithm without having to send any data anywhere.
Below you can see the typical latencies on the default cloud based Metric Correlations service range from 5 to 25 seconds.
In comparison we can see that the latencies for the agent based Metric Correlations tend to be between 100 milliseconds to maybe 5 seconds typical upper range.
Getting started
Following some amazing optimizations by our CEO (yes our CEO!) it is actually more efficient to run MC on the agent than it even was previously to prepare the input data to send to the MC cloud service, as such it has been enabled by default since v1.35.0-22-nightly.
This means you can get blazing fast metric correlations, out of the box, just by updating your nodes to the latest nightly release.
What came next
Running the Metric Correlations computation on the Netdata Agent paved the way for Metric Correlations across all your nodes in Netdata Cloud, and for more algorithms: Metric Correlations today finds the metrics that changed most between a highlighted period and a baseline, with KS2 and volume algorithms, on raw values or anomaly rates.
Learn more
If you would like to learn more, check out the Metric Correlations documentation or feel free to leave some feedback in our launch community post or come chat in our discord.