https://www.beginswithdata.com/ Recent content on It all begins with data Hugo -- gohugo.io en-us Tue, 14 Oct 2025 08:31:03 +0100 https://www.beginswithdata.com/2025/10/14/gcs-buckets-relocate/ Tue, 14 Oct 2025 08:31:03 +0100 https://www.beginswithdata.com/2025/10/14/gcs-buckets-relocate/ If you’ve ever had to move a Google Cloud Storage bucket from one storage location to another you know it’s not easy or cheap. Maybe you needed to move it to a different region to be closer to your compute instances for better performance, or perhaps to reduce cost. In the past, this meant a manual, multi-step process of creating a new bucket, transferring the data, and then updating all your applications to point to the new bucket. https://www.beginswithdata.com/2025/05/14/aws-s3-tools-with-gcs/ Wed, 14 May 2025 00:00:00 +0000 https://www.beginswithdata.com/2025/05/14/aws-s3-tools-with-gcs/ Introduction Google Cloud Storage (GCS) is a service for storing objects in Google Cloud. While GCS provides native XML, JSON, and gRPC APIs, Google recommends using a higher level interface such as GCS Fuse or the Cloud Storage client libraries when possible. Higher level interfaces simplify integration, provide optimal GCS performance, and are actively developed with new features and bug fixes. Many customers have existing software built for Amazon S3 using the AWS SDK. https://www.beginswithdata.com/2025/04/28/n2-vs-c4-showdown/ Mon, 28 Apr 2025 00:00:00 +0000 https://www.beginswithdata.com/2025/04/28/n2-vs-c4-showdown/ If you’ve been deploying N2 VMs on Google Cloud for the last few years it’s time to make a change. The latest generation 3 and 4 machine series deliver more performance at lower cost. How they do it isn’t one thing but a mix including more powerful CPUs, Titanium infrastructure enhancements including Hyperdisk and offload adapters, updated hypervisor software and NUMA aligned scheduling logic, and more. The tech is amazing, but most folks just care about what it means for their workloads. https://www.beginswithdata.com/2024/12/02/gcs-buckets-ip-filter/ Mon, 02 Dec 2024 00:00:00 +0000 https://www.beginswithdata.com/2024/12/02/gcs-buckets-ip-filter/ A Google Cloud Storage (GCS) bucket is accessible to all internet and Virtual Private Cloud (VPC) network client requests that include valid credentials. To further restrict access based on source IP addresses configure the Bucket IP filtering feature. When enabled, only requests from permitted IPv4 and/or IPv6 address ranges for internet and VPC traffic are allowed, and all other requests are denied. In this blog I will demonstrate how to use bucket IP filtering feature and recover from a situation where you have been locked out. https://www.beginswithdata.com/2024/10/08/hyperdisk-shared-gfs2/ Tue, 08 Oct 2024 08:00:00 +0100 https://www.beginswithdata.com/2024/10/08/hyperdisk-shared-gfs2/ Building highly available and performant applications often requires shared storage accessible by multiple VMs. Traditionally shared storage has been made available to VMs in a single zone and was not fault tolerant to zone failure. Google Cloud’s Hyperdisk Balanced HA multi-writer technology solves this problem by combining cross-zone replication with multi-writer access from VMs across two zones. This solution opens up new possibilities for applications requiring both resilience and performance including: https://www.beginswithdata.com/2024/02/01/google-cloud-storage-max-throughput/ Thu, 01 Feb 2024 09:00:00 +0100 https://www.beginswithdata.com/2024/02/01/google-cloud-storage-max-throughput/ TLDR; Accelerate large single file GCS transfers from 200 MiB/s for a single stream, to over 8 GiB/s with parallel streams This guide will teach you how to parallelize your Google Cloud Storage (GCS) file transfers for dramatically increased throughput vs. single stream transfers. We’ll cover GCS API choices, optimized upload/download strategies, library and tool support, and best practices. Let’s get to it! GCS API Support Google Cloud Storage (GCS) offers a JSON API (Google’s own specification) and an XML API (compatible with AWS S3). https://www.beginswithdata.com/2023/11/20/google-cloud-read-only-disk/ Mon, 20 Nov 2023 08:31:03 +0100 https://www.beginswithdata.com/2023/11/20/google-cloud-read-only-disk/ TLDR; High performance low cost sharing of static data with read-only persistent disk attachments Both Google Cloud Compute Engine and Kubernetes Engine allow you to attach a Persistent Disk (PD) in either read-write or read-only mode. For the Compute Engine VM boot disk, you’ll usually create it from an existing image or snapshot and attach as read-write. Data disks on the other hand can be created empty, or seeded with data from an existing image or snapshot, and are usually also attached in read-write mode. https://www.beginswithdata.com/about/ Sat, 21 Oct 2023 14:17:32 +0100 https://www.beginswithdata.com/about/ About this site This is a blog about technology. Usually it will include topics related to cloud computing, storage, open source monitoring, and the container ecosystem. I maintain this blog using Hugo, an awesomely fast and powerful static site generator and have chosen the Icarus theme. The site is then published and deployed using Cloudflare Pages which builds, deploys and maintains SSL certs automatically and distributes it all worldwide using their CDN. https://www.beginswithdata.com/2023/09/10/google-cloud-netapp-volumes-smb/ Sun, 10 Sep 2023 08:31:03 +0100 https://www.beginswithdata.com/2023/09/10/google-cloud-netapp-volumes-smb/ I just kicked the tires of Google Cloud NetApp Volumes service and it is pretty slick! This service is delivered by Google Cloud using NetApp ONTAP and has full integration with IAM, Cloud Monitoring, Cloud Logging, Cloud Console, and the Google Cloud SDK. If you need Windows file shares using SMB on Google Cloud then this is the managed service for you! There is a quickstart guide in the official documentation, but plenty of steps are “left to the reader”. https://www.beginswithdata.com/2019/11/09/2019-11-09-using-google-bigquery-sandbox-tutorial/ Sat, 09 Nov 2019 17:00:27 +0100 https://www.beginswithdata.com/2019/11/09/2019-11-09-using-google-bigquery-sandbox-tutorial/ I recently presented a session about Google BigQuery at the Google Cloud Summit in Amsterdam. While preparing for the session I was reminded how easy it is to get started, at no cost and without a credit card, to explore your own data. In my presentation I showed a short demo working with a dataset I uploaded, and then combining it with a public dataset. This blog allows you to execute the demo yourself, for free! https://www.beginswithdata.com/2017/07/13/2017-07-13-openshift-with-netapp/ Thu, 13 Jul 2017 19:31:03 +0100 https://www.beginswithdata.com/2017/07/13/2017-07-13-openshift-with-netapp/ I love containers, and if I have to manage them at scale give me Kubernetes. Recently I started playing around with OpenShift, which is built on top of Kubernetes, and found even more to like. For me, the ability to create ‘routes’ (enabling external access to cluster services) and a GUI to see and influence what is happening, was already enough to show its value above the base Kubernetes. It all begins with data™ and bringing NetApp storage to the party was a top priority. https://www.beginswithdata.com/2017/06/23/2017-06-23-fsck-k8s/ Fri, 23 Jun 2017 16:05:03 +0100 https://www.beginswithdata.com/2017/06/23/2017-06-23-fsck-k8s/ I like to break things. Kubernetes is really good at maintaining desired state so I decided to play whack-a-mole with my pod: # kubectl get pods NAME READY STATUS RESTARTS AGE etcd-2372957639-s3107 2/2 Running 1 1d # kubectl delete pods etcd-2372957639-s3107 pod "etcd-2372957639-s3107" deleted # kubectl get pods NAME READY STATUS RESTARTS AGE etcd-2372957639-s3107 2/2 Terminating 1 1d etcd-2372957639-tl5xh 0/2 ContainerCreating 0 4s After dozens of ‘whacks’ I noticed it got stuck with a status of ContainerCreating. https://www.beginswithdata.com/2017/06/14/2017-06-14-techsummit-k8s-video/ Wed, 14 Jun 2017 08:31:03 +0100 https://www.beginswithdata.com/2017/06/14/2017-06-14-techsummit-k8s-video/ In this 35 minute talk recorded at TechSummit.io Amsterdam you will learn about deploying stateful apps in Docker containers on Kubernetes. First I will describe how app owners build their stateful apps using StatefulSets and PersistentVolumeClaims, and how these are underpinned by infrastructure resources like PersistentVolumes and StorageClasses. If your stateful app is a single instance database these basics might be enough, but with a scale-out database a bit of application specific glue is needed to manage this scaling gracefully. https://www.beginswithdata.com/2017/02/06/ndvp-usage-solidfire-san/ Mon, 06 Feb 2017 17:00:00 +0100 https://www.beginswithdata.com/2017/02/06/ndvp-usage-solidfire-san/ In this blog I will explain how to configure and use the NetApp Docker Volume Plugin (nDVP) with SolidFire storage. To get up and running follow these three main steps: Install the nDVP binary on your container host Create a config json file Create and manage volumes from Docker and if it doesn’t work, check Troubleshooting :-) [1] Install the nDVP binary on your container host Install the nDVP software (a single binary) on your container host. https://www.beginswithdata.com/2017/02/05/ndvp-install-coreos/ Sun, 05 Feb 2017 17:00:00 +0100 https://www.beginswithdata.com/2017/02/05/ndvp-install-coreos/ In this blog I will provide the steps to install the NetApp Docker Volume Plugin (nDVP) on CoreOS manually and automatically with cloud-config. First, some introductions: NetApp Docker Volume Plugin (nDVP): Provides direct integration with the Docker ecosystem for NetApp storage including ONTAP, E-Series, and SolidFire. The nDVP allows for Docker integrated storage management including provisioning/deprovisioning, attaching/detaching, and cloning of NetApp storage. The nDVP software (a single binary) must be installed on the container host and reads its configuration from a JSON formatted config file when it is started. https://www.beginswithdata.com/2017/01/02/docker-swarm-on-container-lab/ Mon, 02 Jan 2017 01:03:55 +0100 https://www.beginswithdata.com/2017/01/02/docker-swarm-on-container-lab/ If you used Using CentOS 7, KVM, and CoreOS to build a container lab and have a KVM host or two ready to go here is your first project: Docker Swarm. In this 3 minute long screencast we will: Build three CoreOS container hosts Create a 3 node swarm (1 manager, 2 workers) on them Create a web app with 3 replicas Verify we can access the web app and it is load balanced by Swarm Scale up to 15 replicas Simulate failures by destroying the 2 worker nodes and verify the web app stays up Remove the last node as final clean up The ascii screencast has no audio but allows copy/paste. https://www.beginswithdata.com/2016/12/30/centos7-kvm-coreos/ Fri, 30 Dec 2016 16:41:57 +0100 https://www.beginswithdata.com/2016/12/30/centos7-kvm-coreos/ Containers are the next big thing. Bigger than Virtual Machines. No question about it. To learn about containers, Docker is the logical place to start. Docker Machine provides a Docker engine based environment that is point and click simple for your PC. It’s awesome. This is your first stop to get a taste. Soon you’ll want more. Lots more. Your list of ‘things to check out’ will start to grow: https://www.beginswithdata.com/2016/11/14/harvest-1-3-released/ Mon, 14 Nov 2016 08:31:03 +0100 https://www.beginswithdata.com/2016/11/14/harvest-1-3-released/ I am pleased to announce that NetApp Harvest v1.3 is now available on the NetApp Support Toolchest! This feature release includes lots of new counters, dashboard panels, and of course, bug fixes. Major new features in version 1.3: Support for ONTAP 9.0 and 9.1 Support for OnCommand Unified Manager 6.4, 7.0, 7.1 Updated dashboards that leverage Grafana 3.x features New counters and dashboard panels: MetroCluster (FCVI) Backend disk/tape adapters NFS v4 and NFS v4. https://www.beginswithdata.com/2016/11/09/pibakery-to-the-rescue/ Wed, 09 Nov 2016 21:41:08 +0100 https://www.beginswithdata.com/2016/11/09/pibakery-to-the-rescue/ Together with my colleague Clemens Siebler we were preparing for an upcoming Hacker’s night night we’ll run at NetApp Insight Berlin. We have a challenge for participants that includes a few different bits of tech, including some Raspberry Pis. I volunteered to handle the Pis; hardware and software is my task. Prototyping on one Pi I got everything working as wanted. But for the event, one Pi isn’t enough. Okay, 30 Raspberry PIs. https://www.beginswithdata.com/2016/09/09/insight-2016/ Fri, 09 Sep 2016 19:47:52 +0000 https://www.beginswithdata.com/2016/09/09/insight-2016/ This year at NetApp Insight I’ll be presenting three sessions and want to pack the house so please join me! 62051-2 - Open Source Monitoring Using Harvest, Graphite and Grafana This one is brand new, and I’ll cover the solution, custom counter collection templates and more. So if interested in Harvest/Graphite/Grafana don’t miss it! Schedule it: [Las Vegas, Berlin] 59851-2 - Monitoring Performance Using NetApp OnCommand Performance Manager and API Extensions I’m lending a hand to Dhiman (primary presenter) because I’ve delved into the APIs a bit with Harvest and can share my experiences. https://www.beginswithdata.com/2015/12/22/optimizing-azure-expressroute-netapp-private-storage/ Tue, 22 Dec 2015 21:49:05 +0000 https://www.beginswithdata.com/2015/12/22/optimizing-azure-expressroute-netapp-private-storage/ Not too long ago NetApp, Microsoft, and Interxion came together to sponsor a hybrid cloud lab with NetApp Private storage for Azure nearby my office in Amsterdam. When someone was needed to set this up I jumped at the chance! For those unaware of the solution, the basic premise is to use the Hyperscaler providers for elastic compute while maintaining ownership of the data and gaining the rich data management features of NetApp. https://www.beginswithdata.com/2015/11/20/awesome-performance-scaling-netapp-clustered-data-ontap/ Fri, 20 Nov 2015 12:54:11 +0000 https://www.beginswithdata.com/2015/11/20/awesome-performance-scaling-netapp-clustered-data-ontap/ I just got back from NetApp Insight 2015 in Berlin and what a time it was! One highlight for me was seeing the power of NetApp Clustered Data ONTAP scaling a database workload from 22k IOPs on a Hybrid array all the way up to 1.2 Million IOPs on an all flash one with no disruption, while at the same time maintaining hundreds of snapshots and clones. Wow. What other storage system on the planet can do that? https://www.beginswithdata.com/2015/11/11/grafana-2-5-support-with-netapp-harvest/ Wed, 11 Nov 2015 19:02:47 +0000 https://www.beginswithdata.com/2015/11/11/grafana-2-5-support-with-netapp-harvest/ UPDATE (11-Nov-2016): Grafana 3.1.1 support is available with NetApp Harvest v1.3. Download NetApp Harvest v1.3 from the NetApp Support Toolchest. If you must use Harvest 1.2.2 and desire dashboard updates keep reading this blog entry. Recently Grafana released v2.5 with a host of new features: Included in this major release was some additional error checking on singlestat dashboards to ensure only one metric series was passed; previously multiple were allowed and it would just display the first one (see Issue 2745 if interested). https://www.beginswithdata.com/2015/10/09/netapp-metrics-collector-for-graphite-opm-vs-harvest/ Fri, 09 Oct 2015 10:37:58 +0000 https://www.beginswithdata.com/2015/10/09/netapp-metrics-collector-for-graphite-opm-vs-harvest/ So you have Graphite + Grafana up and running and want to get metrics in there from your NetApp storage systems and discovered there are two options: OnCommand Performance Manager ‘External Data Provider’ feature NetApp Harvest collector on the NetApp ToolChest Your next question was probably which one is right for me? The answer is, as with many things, “it depends”. Some history here. A long time ago (maybe 2yrs now) I created Harvest and socialized it internally. https://www.beginswithdata.com/2015/09/25/cpu-utilization-monitoring-of-many-nodes/ Fri, 25 Sep 2015 09:00:39 +0000 https://www.beginswithdata.com/2015/09/25/cpu-utilization-monitoring-of-many-nodes/ I got an email from a colleague asking if I had any ideas for a customer need of presenting CPU busy across a set of systems. He wanted both average and percentile values over a specified timeframe. In his case it was NetApp cDOT storage array nodes, but the concept could be applied to whatever group of systems you happen to be interested in. I hadn’t yet used the repeat function added in Grafana 2. https://www.beginswithdata.com/2015/09/16/netapp-advanced-performance-monitoring-with-harvest-graphite-and-grafana/ Wed, 16 Sep 2015 19:26:50 +0000 https://www.beginswithdata.com/2015/09/16/netapp-advanced-performance-monitoring-with-harvest-graphite-and-grafana/ I’m pleased to announce that NetApp Harvest is now posted to the NetApp ToolChest! NetApp Harvest is a data collector that fetches a wide set of performance metrics from NetApp cDOT and 7-mode storage systems, and capacity info from Unified Manager 6+, and posts it to an open-source Graphite metrics server. It also imports 19 dashboards containing over 500 graphs into Grafana to make viewing it all a snap. I am just thrilled with the results and I hope you are too. https://www.beginswithdata.com/2015/09/14/installing-graphite-and-grafana-on-rhel-6-7-or-ubuntu-in-under-30-minutes/ Mon, 14 Sep 2015 14:31:19 +0000 https://www.beginswithdata.com/2015/09/14/installing-graphite-and-grafana-on-rhel-6-7-or-ubuntu-in-under-30-minutes/ The open source metrics software Graphite is brilliant. To get it up and running however is not so. The ‘official’ documentation is quite generic, and the packaged versions (on EPEL and via PIP) have their own issues on some Linux flavors. So I usually end up searching Google to hopefully find a working recipe for the Linux flavor of the day on someone’s blog. And when I’ve done that invariably it still doesn’t work. https://www.beginswithdata.com/2015/09/13/volume-move-as-impressive-as-ever/ Sun, 13 Sep 2015 20:46:57 +0000 https://www.beginswithdata.com/2015/09/13/volume-move-as-impressive-as-ever/ Back in 2012 when NetApp first added SAN protocol support to clustered Data ONTAP I created a short video to show how easy and non-disruptive it is to move a volume (containing one or more LUNs) around in a cluster. Although a bit dated, the feature is as powerful as ever! From an operational perspective being able to move data without impacting users or reconfiguring clients is huge. If you want to re-balance for capacity or performance (including from spinning disk to hybrid to all-flash!