Private GCS via Internal Load Balancers: No Client Auth Required

For web serving use cases it’s common to put a load balancer in front of Cloud Storage providing a custom domain name with TLS, path flexibility with a URL map, and a single place to apply policy.

External Application Load Balancers (ALB) can use backend buckets that are shared publicly, or are kept private by granting a Google managed LB service account access. Many organizations use the private option to prevent direct API access to the bucket by unauthenticated users, as well as to avoid org policy exceptions for public access prevention and security findings that come with allUsers access.

Visually, the private option looks like this, where the External ALB adds authentication:

Great, that works for External ALBs, but what about Internal ALBs?

Unfortunately as of today there is no private option per the Limitations: Private bucket access isn't supported, so the backend bucket must be publicly accessible over the internet.

It sounds crazy that the bucket must be publicly available on the Internet in order to share it publicly on our internal network, but this appears to be the situation today. We can use Bucket IP filtering to block Internet access to the bucket, but from an IAM perspective it will still be open. Not good enough. Time to get cracking and see if we can build a solution that keeps the bucket private ourselves!

This is what we get out of the box with no auth headers added anywhere and a 401 response from the private bucket:

To fix this up we’ll need to add auth headers ourselves. Using a traffic extension on the forwarding rule, we’ll call a small Go ext_proc service running on Cloud Run for every request. The service holds a Cloud Storage scoped OAuth token in memory and returns a header mutation that sets authorization: Bearer <token>. The load balancer applies the header, GCS authenticates the request, and the client gets the data being none the wiser:

No signed URLs, no service account keys, no proxy VMs. Clients just fetch objects, and the bucket stays private.

Turning theory into practice

1. Private bucket and identity: Bucket with public access prevention enforced and the objectViewer role granted to a service account.

2. Callout service: Under 400 lines of Go in a 5.4 MB distroless image, deployed to potentially dozens of tiny 1 vCPU / 128 MiB Cloud Run instances.

3. Load balancer: A serverless NEG and backend service for the callout, a backend bucket for GCS, and a regional internal forwarding rule in front of both.

4. Traffic extension: Attach the callout extension to the forwarding rule and configure it to process request headers only.

5. Optional HTTPS: Add a private CA, a regional managed certificate, and a private DNS zone if you want TLS and a friendly hostname.

For the how and why details, check the repo. The rest of this post is more of a “lessons learned”.

Lessons learned

1 - Don’t call the metadata server on the request path. The Cloud Run metadata server caches tokens well enough to call it per request. Or so I thought. But checking response time metrics after a 20k request test run I found response times for this cached token were 2.5 ms avg, but up to nearly 189 ms at worst. Caching the token in service RAM reduces this to under a microsecond. Considering the other steps in the callout flow take about 6 ms avg, adding latency for the token fetch itself, especially at the tail, would be impactful. Better to add logic to refresh the token in the background and serve from memory.

2 - The metadata server won’t mint a fresh token until the old one is nearly expired. The first request for a token is valid for 30 minutes. Request another five minutes later and you get that same token with 25 minutes remaining. So at what point does the metadata server mint a new one? Through my testing, at 4 minutes remaining. I added refresh logic to save it to service RAM at 3 minutes remaining to avoid slowing any client request.

3 - Go sees 2 CPUs, but it turned out not to matter. runtime.NumCPU() returns 2 on 1 vCPU and smaller instances. And there are actually two vCPUs available to your container as well. But on a 1 vCPU instance if your code uses both CPUs concurrently it can only do so half the time and will be throttled the other half adding jitter, reducing processing efficiency, and potentially causing Adaptive concurrency tuning (ACT) to reduce incoming request concurrency.

I suspected this was costing me throughput, so I added code to read cgroup quotas and set GOMAXPROCS to match. While testing, I found that even cgroup quotas cpu.cfs_quota_us didn’t work properly, at least when using the CPU boost feature with less than 8 vCPU. It seems the boosted level is reported continuously even after the boost has ended. Looking at other options I found the cgroup cpu.shares shows 1024 shares for each vCPU configured in the instance. This was starting to get quite complicated so I decided to measure the impact of setting GOMAXPROCS to 1 or 2 for my workload and to be honest I couldn’t see any difference; same scaled instance count, comparable CPU utilization and response time. So I deleted the code. Tuning this likely pays off when you’re saturating CPU, but for this workload where 1 vCPU runs at around 30% utilization and the instance scaling_driver is something else it doesn’t really matter.

4 - The solution scales up smoothly, and the additional callout latency is minor. With a 32 KiB file and 100 concurrent clients fetching it from a GCE VM in the same region I observed around 3,800 req/sec @ 25 ms avg response time with 6 Cloud Run instances active. Quadrupling the clients to 400 increased serving to 12,750 req/sec @ 28 ms avg response time with 14 instances serving. At both load points the callout latency contributed under 6 ms to the overall latency and I didn’t see a single HTTP error. These results are once things were warmed up, but even from a cold start (e.g. --min-instances=0) the very first request had total latency of 342 ms, and the immediate requests thereafter at around 70 ms:

5 - Stop guessing why Cloud Run scaled. Coming from an infra background I was interested in testing different Cloud Run shapes to find the best price/perf. Initially, my approach was to test with --min-instances=1 to see how much throughput, and at what latency, I could get from a single instance. Next, I’d extrapolate the needed instance count for my workload. Don’t do this. Cloud Run’s autoscaler is more advanced and uses multiple signals to scale and distribute work. Instead of the extrapolation, start your benchmark and then check the Cloud Monitoring recommended instances metric label scaling_driver. It will show you how many instances would be needed for each autoscaling driver. During tests with 100 and 400 clients system was always the greatest value and two sizing elements I’d been investigating were off the table: neither --concurrency nor act was anywhere near producing a scaling decision:

6 - Upsizing resources bought nothing. I bumped the service to 2 vCPU to see what changed in scaling:

The CPU recommendation dropped from 3 to 2, but the count needed for system stayed the same at 6. Throughput and latency were also comparable. I see no reason to double the cost; the 1 vCPU / 128 MiB instance is already more than this workload needs. Let Cloud Run do its thing and scale them wide and fast.

7 - As request rates get higher, the bucket will be your temporary constraint. Cloud Storage ramps a bucket up gradually, starting at around 5,000 object reads per second. In my test I naturally scaled past 5,000 and hit some errors, 429s and 5xxs mostly, while the bucket was ramping. If you see higher latency or errors be sure to evaluate the full chain: ALB, Callout, and Storage bucket. And especially when testing request bursts, or higher request levels, make sure your bucket is warm.

What about security?

The Envoy service extension does not provide any authentication when streaming requests to the callout. As a consequence, the Cloud Run service has to be deployed with --allow-unauthenticated. To reduce access make it internal using --ingress=internal and remove endpoints using --no-default-url. Even so, any client inside the VPC that can reach the callout can get a token back. To limit the blast radius, make the token itself less useful: * Least privilege IAM: Grant the service account read access to the one bucket and nothing else. * Downscoped token: Scope the token to devstorage.read_only, so it can’t be used for writes, or for any other Google API, even if permissions on the SA are expanded.

Conclusion

This solution leverages specific features of storage, networking, and serverless compute to meet a business requirement. While the code that adds an auth token will hopefully be replaced by a built-in feature soon (like it is for External ALBs), the callout design pattern itself is adaptable for other use cases where headers need to be manipulated, or request details logged.

Use a disposable project, run through it, and see how it fits your environment! As always, comments welcome below.