NetApp metrics collector for Graphite: OPM vs Harvest

So you have Graphite + Grafana up and running and want to get metrics in there from your NetApp storage systems and discovered there are two options:

Your next question was probably which one is right for me?

The answer is, as with many things, “it depends”.

Some history here.  A long time ago (maybe 2yrs now) I created Harvest and socialized it internally.  The product team that owns OPM saw it, recognized the value of integrating with open source metrics platforms, and decided to create the “External Data Provider” feature which they shipped in the OPM 1.1 release. Meanwhile, Harvest continued being used internally and at some customers with continual feedback/enhancement loops occurring. Harvest then went “public” in September on the ToolChest. So that’s how we got here and might help you understand why there are two options and give context to some of the differences.

To compare the two I created this table:

OPM vs Harvest as data source

Let’s step through each one:

  1. Polling interval.  OPM has a fixed polling interval of every 5 minutes from the clusters.  It uses this interval for its own DB and UI and the metrics sent to Graphite share this interval. Harvest collects every 1 minute by default.  The time is however configured per poller, and any time (lower or higher) can be set; the lowest you can go  is largely dependent on the API response time from the cluster.  I don’t think there are a lot of use cases to go lower than 1 min (data overload to view and to store!) but it’s possible.  For large clusters it’s also quite possible that Harvest won’t be able to keep up at 1 min intervals and you will have to either reduce the objects collected or poll less frequently.

  2. Counter inventory.  OPM chose simplicity here and has three presets to chose from with a varying level of details. It does not allow further customization or selection of your own counters. Today it has about 115 counters in the most detailed preset. Harvest went for flexibility and uses a config template to control the counters collected.  The default one is quite comprehensive with more than 500 counters and is the result of my field experience at NetApp and lots of feedback from others. You can also create your own templates to add or remove counters based on your need. Over time I expect to add more, for example NFSv4 is on my to-do list.

  3. Dashboards provided.  OPM doesn’t have any.  Hopefully someone will make some and post them to the NetApp Communities! Harvest does include dashboards, a bunch of them.  Maybe so many it can be a bit overwhelming!  I plan to create some YouTube videos of stepping through them so you can better understand what you’re looking at; another item on my to-do list.

  4. Data ONTAP 7-mode.  OPM is built from the ground up for cDOT and has no support for 7-Mode.  Harvest was built for both cDOT and 7-Mode and supports both just fine.

  5. Storage capacity information. OPM doesn’t do storage capacity, and since there is no “External Data Provider” in Unified Manager (OCUM) you can’t get capacity info from it either.   Harvest has a poller type for OCUM 6.x and can fetch capacity info at many levels (aggr, vol, lun, qtree, aggr, etc) allowing cDOT dashboards that have a blend of performance and capacity information. I didn’t add a poller type for UM 5.2 (and don’t plan to) so you don’t have capacity info for 7-mode systems; sorry.

  6. Scaling. OPM does a lot of analytic processing and has some documented maximums for clusters it can monitor. With cDOT 8.3 it also uses a collection method from the cluster that is more efficient (not individual API calls but fetching an archive file that the cluster has prepared for it) making it lighter weight for the monitored cluster.   Harvest doesn’t do any analytic processing but rather calculates a the metric value, normalizes it, and sends it off to Graphite.  As such there are no explicit maximums of a cluster it can monitor. It uses API calls to gather all data (even on cDOT 8.3) and for large clusters that might mean collection can’t complete in 1 min and you have to adjust frequency or collected objects as mentioned earlier. But, you do have a tool that can collect metrics from any cluster size or configuration.

  7. Ease of use. OPM has a menu driven interface and all monitored clusters will automatically send metrics to Graphite.  And since it’s part of OPM there is nothing extra to install or maintain.  Harvest is driven by a config file which needs to be edited manually; installation of a new cluster means editing that config file.  It also has to be installed separately and maintained over time.

  8. Downloadable from.  OPM is an official product and is available on the NetApp Support Site. Harvest is a ToolChest tool and is available on the NetApp Support ToolChest site.

  9. Supported by.  OPM, including the “External Data Provider” feature, is supported by NetApp Support and you can open a support case if something doesn’t work or you need help. Harvest is a ToolChest tool and is not supported by NetApp Support.  If something doesn’t work you can either fix it yourself (Harvest is written in Perl and delivered as code) or you can post your question on the NetApp Communities site.

So the choice of OPM vs Harvest is really a matter of requirements and preference. Both have pros/cons but hopefully with the info above you will be able to pick the right solution for your environment.

Comments are always welcome!

Share Comments
comments powered by Disqus