Two storage systems launched within weeks of each other in 2006, and made opposite bets about the same question: how smart should the wire between storage and compute be. Hadoop’s HDFS split off as its own project that February. Amazon’s S3 launched that March. One embedded genuine intelligence directly into its protocol. The other refused to. A decade later, the dumb one had out-evolved the smart one so completely that the smart one is now, per current sources, functionally obsolete outside regulated on-premise holdouts.

That’s not a story about better engineering. HDFS was excellent engineering, correctly suited to the constraints of its moment. It’s a story about what a commodity interface can do that a smart one structurally cannot: let the ecosystem on either side of it evolve independently, without asking permission.

The Smart Bet

HDFS colocated storage and compute on the same physical nodes, on purpose. Its defining principle was sending the computation to the data rather than moving the data to the computation — genuinely sophisticated scheduling, with the wire protocol itself encoding cluster topology, node roles, and data-locality awareness. A HDFS client wasn’t just fetching bytes; it was participating in a coordination scheme that assumed intimate knowledge of the specific cluster it was talking to.

This was the correct engineering call in 2006. Moving a large dataset across the network to reach elastic compute was genuinely more expensive than moving a small job to wherever the data already sat. HDFS ran the majority of the world’s big-data workloads for roughly a decade, from around 2008 through the mid-2010s, precisely because that constraint was real and the smart wire addressed it well.

The cost of smartness showed up later. A protocol that embeds coordination assumptions can only be spoken correctly by clients that share those assumptions — the same cluster, the same vendor’s implementation, the same protocol version. Nobody outside the Hadoop ecosystem could reimplement HDFS’s wire protocol and call the result compatible, because the intelligence was the lock-in. There was nothing dumb enough to copy.

The Dumb Bet

S3 shipped a REST API — PUT, GET, DELETE, LIST, a bucket name, a signature — that knows nothing about what’s inside the bytes it stores and coordinates nothing about where compute happens. It rode on HTTP, which by 2006 was already the most universal transport in existence, and inherited byte-range requests — the Range header, the 206 Partial Content response — from RFC 2616, a specification seven years older than S3 itself. Amazon didn’t invent partial reads. It got them for free by choosing to speak an already-commodity protocol instead of building a new one.

This is the same discipline the meterkast runs on: a wire that stays deliberately dumb, measuring bytes the way a utility meter measures kilowatt-hours, indifferent to what’s flowing through it. The meter doesn’t care whether the electrons power a kettle or a server. S3’s API doesn’t care whether the bytes are a Parquet file, a JPEG, or a database backup. All the intelligence — what the data means, how to query it, where to run the computation — stays entirely on either side of the wire, never inside it.

A dumb wire is what a stranger can copy. There’s no coordination scheme to reverse-engineer, no cluster assumptions to inherit, nothing but a shape — and anyone can build to a shape.

Why Dumb Out-Evolves Smart

The consequence of that difference compounds in a direction HDFS’s design could never access. Because S3’s wire carries no embedded assumptions, both sides of it became free to multiply independently.

One storage, many computes. DuckDB, Spark, and Trino can all point at the same bucket simultaneously, with zero coordination between them, because none of them are actually talking to each other — each is independently resolving against the same commodity interface. DuckDB’s own range-request mechanism, pulling exactly the columns a query needs out of a Parquet file, never needed an AWS SDK. It needed a GET with a Range header, which is a twenty-seven-year-old feature of HTTP itself.

One API, many storages. That same compute stack doesn’t care whether the bucket sits at AWS, at Cloudflare R2, or at StackIT — the German sovereign cloud operated by the Schwarz Group, the company behind Lidl and Kaufland. R2, launched in 2021, is S3-compatible specifically to eliminate egress fees; its own migration pitch is “switch your endpoints… keep using your current tools.” StackIT, selected by the European Commission in April 2026 as one of four providers in a €180 million sovereign-cloud tender, offers the identical proposition for an entirely different reason — EU data residency, outside the reach of the US CLOUD Act — and its own documentation states migration “typically only requires an endpoint change.” Two companies, two unrelated motivations, both able to compete without asking a single customer to rewrite a line of code, because the interface between storage and compute was never smart enough to lock either side in place.

HDFS could offer neither freedom. Colocated storage and compute can’t multiply independently of each other, because they were never separated in the first place — and a protocol that embeds cluster-specific coordination can’t be spoken by a stranger’s storage or a stranger’s compute engine, because “stranger” was never a case the protocol was designed to handle.

The Physics That Took a Decade

None of this made S3 the better bet in 2006. It made S3 the bet that was waiting to become better, once a technology-bound constraint compounded away. The same mechanism the classic latency table exposes: network bandwidth kept improving year over year in a way HDFS’s colocation advantage never could, because colocation’s whole value proposition was avoiding network cost in the first place. Once 10 Gb network cards became standard through the early-to-mid 2010s, the expense that justified smart coordination stopped being decisive, and the flexibility of a commodity wire — evolve either side freely, swap vendors without rewriting anything — started outweighing whatever efficiency HDFS’s smartness had originally bought.

S3 didn’t get smarter between 2006 and 2016. The physics underneath the original question simply moved far enough that the dumb bet’s real advantage — an ecosystem that could evolve on both sides of the wire independently — finally mattered more than the smart bet’s original efficiency. DuckLake, DuckDB Labs’ own table format, is the clearest sign of how settled this has become: a serious database team, building new infrastructure from scratch, chose S3-compatible object storage as bedrock without a second thought, the way a builder trusts a pipe thread that’s been fixed for a century.

Commodity, Not Smart

The lesson runs against the instinct that produced InfoWorld’s advice to build one comprehensive control plane across every cloud — a new, smart abstraction, custom-built to understand the differences between vendors. That instinct is HDFS’s instinct: embed the intelligence in the wire, and trust that intelligence to keep paying for itself. It rarely does, because a smart wire can only be spoken by clients that share its assumptions, which is precisely what makes it a control point rather than a commodity — and control points get exploited by whoever ends up holding them, the moment holding them turns out to be profitable.

The better answer was already sitting in production, proven over twenty years: keep the wire dumb, put the intelligence on either side of it, and let both sides multiply independently. One storage, many computes. One API, many storages. Neither freedom required anyone to be clever about the interface. Both required someone, in 2006, to resist the temptation to be.

This post is part of the Gutenberg/Semantic series. Related: The Meterkast Pattern on the dumb wire that stays commodity on purpose, The Latency Table Already Knows the Gutenberg Layer Compounds and the Semantic Layer Doesn’t on the physics that took a decade to compound, and IKEA-Oriented Development on sticking to the screw everyone already owns.

Sources: Cloudflare R2 — Distributed Object Storage for the Edge, Cloudflare; STACKIT Object Storage, STACKIT (Schwarz Digits); “Three concepts cloud architects overlook”, David Linthicum, InfoWorld, August 7, 2026; RFC 2616 — Hypertext Transfer Protocol (HTTP/1.1), IETF, June 1999.