As you can imagine, over the years I've had the privilege (and sometimes the pain) of working across a wide range of database technologies.
My journey started in the early 90s with Sybase System 4—long before I realized that its kernel would later become the foundation of Microsoft SQL Server.
From there, I moved into the era of desktop databases:
- dBase
- Clipper
- FoxBase
These systems brought data closer to the developer. You could actually touch the data, shape it, build interfaces quickly. Clipper, in particular, stood out—we used it to connect AutoCAD to databases, effectively creating early GIS-like systems before that term was mainstream.
The rise of enterprise systems
In the late 90s and early 2000s, Microsoft SQL Server became one of my go-to platforms.
We built what I would call one of the earliest forms of SaaS—customer intelligence platforms—before "SaaS" was even a term people used.
From there, like many engineers, I inevitably crossed paths with Oracle Database.
The era of scale
As data volumes exploded in the mid-2000s:
- Teradata entered the picture
- Apache Hadoop clusters became the new frontier
- Systems like Netezza emerged for specialized analytics
This was the beginning of distributed data—but also the beginning of complexity creeping in everywhere.
Cloud changes the game
Around 2012, Amazon Redshift arrived.
It felt revolutionary at the time:
- Run large datasets on relatively small hardware
- Spin up clusters quickly
- Pay for what you use
We used to joke about "pizza box servers" (1U racks), but Redshift made that abstraction even cleaner.
That said—performance required deep understanding of data distribution, sort keys, and modeling. It worked well, but only if you respected its constraints.
The zero-admin promise
Then came Snowflake (founded 2012, breakout ~2018).
It was, honestly, an act of beauty:
- Separation of compute and storage
- Minimal operational overhead
- Elegant user experience
The lakehouse movement
Next up: Databricks, built on Apache Spark.
Anyone who has worked with Spark knows: there's a lot more going on under the hood than meets the eye.
Early Spark was powerful—but also:
- Buggy
- Hard to tune
- Operationally heavy
Databricks did a strong job productizing it:
- Better UX
- Managed infrastructure
- Competitive cost structure vs Snowflake
Today, those two are going head-to-head.
And then... a duck showed up
A few months ago, I evaluated a newer entrant: MotherDuck built on DuckDB.
I'll be honest—I wasn't sold on the name at first.
But once I got hands-on, it became clear: this thing is deceptively powerful.
- Lightweight
- Can run directly in the browser
- Syncs seamlessly with the cloud
- Extremely fast for analytical workloads
We ran a direct comparison:
- Same dataset (our full warehouse)
- Same query patterns
- Comparable compute assumptions
The real kicker (and what actually matters)
Because we designed our stack with modularity in mind—using dbt—
Switching from Databricks to MotherDuck was:
- Re-rendering dbt models
- Minor tweaks (catalogs, functions)
- Done in under 3 weeks
So what's the takeaway?
After all these systems, generations, and paradigms:
Every generation promised:
- Better performance
- More scale
- Easier operations
But the real advantage came from:
- How quickly you can adapt
- How portable your logic is
- How loosely coupled your system becomes
Final thought
If you haven't looked at MotherDuck / DuckDB yet, it's worth exploring.
Not because it will replace everything overnight—but because it challenges assumptions about what a data platform needs to be.
And those moments are usually where the next shift begins.