I tried three tools to configure Databricks and here's what actually worked

For weeks, we've been debating about PowerShell scripts, Terraform configs, and figured Bicep templates didn't entirely work for managing our Databricks workspace.

Each tool promised to be "the one". Then Bicep extensions arrived, and everything changed.

Here's what I learned about choosing the right tool for configuring (Azure) Databricks, and why the answer isn't what you'd expect.

The problem I didn't know I had

It started innocently enough. We had been focusing mainly on Azure infrastructure. We declared the infrastructure using Azure Bicep. After locking down the environment, we reached a point where we had a Databricks workspace and needed some configuration.

That's when we hit the wall.

Bicep could create the Databricks workspace instance – that became easy. But configuring what's inside the workspace? Catalogs, schemas, clusters, credentials? That required something else entirely. The Databricks API was separate from Azure Resource Manager, and Bicep didn't speak that language.

So we did what everyone does: we cobbled together a solution.

"Just run this PowerShell script with a module. Oh, wait, but first you need to set these environment variables. And make sure you're using the right API version. Actually, let me just do it myself."

It worked, but it felt wrong. We had beautiful, declarative Bicep templates for our Azure infrastructure. And then we had... scripts. Scripts that needed documentation. Scripts that broke when APIs changed. Scripts that only two people really understood.

I thought the solution was obvious: we just needed "proper" infrastructure-as-code for the Databricks configuration. So I did what any sensible person would do: I started researching.

And that's when I discovered something worse than no automation.

I discovered choice paralysis.

Everyone had an opinion. PowerShell folks swore by the flexibility and direct API access. Terraform people pointed to the mature Databricks provider and massive community support. And Bicep? Well, we already knew Bicep couldn't do it. Or so we thought.

The problem wasn't just choosing a tool. It was that we'd already invested in Bicep for Azure, and now we needed to add another tool just to configure what's inside our Databricks workspaces. Our infrastructure-as-code story was about to become infrastructure-as-multiple-tools.

But I had to make a decision. So I did what I had to do: I tried all three.

The Terraform debate

Before diving into anything, we had to address the elephant in the room: Terraform.

The team had mixed feelings. Some had used it before and loved it. Others had heard the horror stories about state files and drift. I sat down and made a list.

The case FOR Terraform:

• Mature Databricks provider with hundreds of resources
• Declarative approach — just describe what you want
• Massive community and Stack Overflow support
• Works across cloud providers if we ever need that
• Proven at scale in many organizations

The case AGAINST Terraform:

• We'd need to manage state files (where? how? who has access?)
• Yet another tool to learn and maintain
• Another language (HCL) that not everyone on the team knew
• Drift management when people inevitably click around in the UI
• We're already using Bicep for Azure — now we'd have two IaC tools

The arguments went back and forth. "But everyone uses Terraform!" versus "Do we really need all that complexity?" Some pointed to the mature provider. Others worried about the learning curve and operational overhead.

In the end, we decided to do a quick exploration, but with skepticism. If we were going to add Terraform to our stack, it needed to be significantly better than the alternatives. Not just "industry standard" — actually better for our specific situation.

Meanwhile, I suggested: "Why don't we just start with PowerShell? It's simple, we already know it, and we can always migrate later."

That seemed reasonable. So PowerShell it was. At least to start.

PowerShell: Where we started

PowerShell felt like home. I'd been writing PowerShell for years. The Databricks API was well-documented. How hard could it be?

Turns out, not hard at all. That was the problem.

I started with a simple script to create clusters. Then I needed to manage secrets. Then Unity Catalog. Before I knew it, I had a folder full of scripts, each one slightly different from the last. Some used environment variables, others took parameters. Some had error handling, others... didn't.

The worst part? It worked. Really well, actually.

PowerShell gave me complete control. When I needed to do something weird, I could easily leverage Invoke-DatabricksApiRequest. The API client was straightforward, and with easy commands available to me, it worked great.

But here's what nobody tells you about PowerShell for infrastructure management: you're not building automation, you're building a framework.

Every script needed error handling. Every script required logging. Every script needed some way to handle state — "did I already create this thing?" Parameters need validation. Some operations needed retries. And don't even get me started on trying to make these scripts idempotent.

Three months in, I realized I'd spent more time building infrastructure around my PowerShell scripts than I had actually managing infrastructure.

When PowerShell actually makes sense

Don't get me wrong. PowerShell isn't the real problem. It's perfect for certain things:

• One-off operations: Need to bulk-update cluster configurations? PowerShell is your friend.
• Debugging: When something's broken and you need to poke around the API manually, PowerShell's interactivity is unbeatable.
• Complex workflows: When you need actual programming logic (not just resource declarations), PowerShell shines.

But for managing infrastructure state over time, especially around permissions? I was fighting an uphill battle, and I knew it.

Terraform: The industry standard that almost broke us

After my PowerShell adventure, Terraform seemed like the obvious next step. Everyone uses Terraform, right? There's a reason it's the industry standard.

The Databricks provider was mature. Hundreds of resources supported. Thousands of questions answered on Stack Overflow. This was going to be easy.

It was not easy.

Look, Terraform is brilliant. The declarative approach makes sense. The state management is solid. The provider ecosystem is unmatched. But somewhere between terraform init and my fifth state file conflict, I started to wonder if I'd made a mistake.

Here's what happened: I'd define a cluster in HCL. Run terraform apply. Great, cluster created. Then someone would modify that cluster in the Databricks UI (it happens). Next time I ran terraform plan, it wanted to destroy and recreate everything.

"Just use terraform import," they said. So I did. Except now my state file had drifted from my code. And my code was getting complicated — workspaces, modules, remote state. I was spending more time managing Terraform than managing Databricks.

The breaking point came on a late afternoon. I'd been debugging a state conflict for four hours. Four hours. The cluster existed. Terraform thought it didn't. Or maybe it did? The state said one thing, but terraform refresh said another.

I fixed it eventually. But as I sat there, staring at my state file in VS Code, I realized I was optimizing for Terraform, not for Databricks.

Where Terraform is actually perfect

Here's the thing. Terraform certainly didn't fail me. I just wasn't the right user for it.

Terraform is phenomenal when:

• You're managing multi-cloud infrastructure: AWS + Azure + GCP? Terraform has you covered.
• You have a dedicated platform team: People who eat, sleep, and breathe Terraform.
• Complex dependencies matter: When Resource B depends on Resource A's output, and Resource C needs both.
• You need that mature ecosystem: Want to manage Databricks, Snowflake, and GitHub in one place? Terraform's probably your only option.

But I wasn't doing any of that. I was managing Databricks on Azure. That's it. Just Azure.

And that's when the experience around Bicep extensions changed everything.

Bicep: The tool I dismissed

I'll be honest: I'd written off Bicep completely.

"Bicep only does Azure resources," I thought. "Databricks isn't really an Azure resource. Sure, I can create the workspace, but I can't configure anything inside it."

This was true until it wasn't.

Bicep extensions changed everything. Suddenly, Bicep wasn't just for Azure Resource Manager. It could be extended to manage anything with an API. And someone had built a Databricks extension.

I was hesitant. I'd already invested time in PowerShell and Terraform. Did I really want to learn another tool and probably another programming language?

But here's what got me: the simplicity.

targetScope = 'local'

param workspaceUrl string

extension databricks with {
  workspaceUrl: workspaceUrl
}

resource cluster 'Cluster' = {
  clusterName: 'analytics-cluster'
  sparkVersion: '13.3.x-scala2.12'
  nodeTypeId: 'Standard_DS3_v2'
  numWorkers: 2
  autoterminationMinutes: 120
}

That's it. No state files to manage. Just declare what you want, and Bicep handles the rest.

And because it's Bicep, I got all the good stuff for free:

• Type safety: Typo a property name? Bicep catches it before deployment.
• IntelliSense: All the available properties, right there in VS Code.
• Native Azure integration: Managing the Databricks workspace resource and the internal configuration in the same language.

My first real deployment

I decided to test it on something real: migrating our Unity Catalog configuration to Bicep.

I expected pain. I'd spent weeks getting this right in Terraform, and before that, even longer with PowerShell scripts.

It took me an afternoon to hook Bicep into Databricks' API.

One Bicep file. Catalogs, schemas, external locations, credentials — everything. And because Bicep is declarative, I didn't need to worry about order of operations. It just figured it out.

The best part? When I handed this off to my team, they got it immediately. No Terraform state to explain. No PowerShell framework to document. Just Bicep templates that looked like the Azure ARM templates they already knew.

The real question: What are you actually trying to do?

After months of using all three tools, I realized something important: I'd been asking the wrong question.

The question wasn't "which tool is best?" The question was "what problem am I solving?"

For me, the problem was simple: manage Databricks configuration on Azure in a way that my team could understand and maintain.

For you? The problem might be completely different.

Here's the decision matrix I use now:

Reach for PowerShell when:

• You need one-off operations or scripts
• You're debugging or exploring the API
• You need actual programming logic and control flow
• You already have a library of PowerShell scripts

Reach for Terraform when:

• You're managing multiple cloud providers
• You have complex cross-resource dependencies
• You need to manage Databricks alongside other non-Azure services
• You have a team that already knows Terraform

Reach for Bicep (with extensions) when:

• You're primarily on Azure
• You want type safety and IntelliSense
• Your team already knows Azure and Bicep templates
• You can invest time in extensions

The uncomfortable truth? There's no universally correct answer. The "best" tool is the one that fits your actual situation, not the one with the most GitHub stars.

What I wish I'd known earlier on

Looking back, here's what would have saved me weeks of frustration:

Start simple. I wasted time building complex PowerShell frameworks when I could have switched to DSC. Don't add complexity until you actually need it.

Ship first, optimize later. My Terraform setup was over-engineered from day one. Modules, workspaces, remote state — all before I'd deployed a single cluster to production.

Master one tool deeply before adding another. I was bouncing between three tools, never really learning any of them properly. Pick one, get good at it, then evaluate if you actually need something different.

Your team's knowledge matters more than the tool's capabilities. A tool your team can confidently use is better than a "better" tool they don't understand.

The biggest lesson? Stop debating and start deploying. I spent probably 40 hours researching and testing tools. In hindsight, I could have picked any of them and been productive.

My recommendation (with caveats)

So what do I actually use now?

PowerShell for 90-95% of my Databricks work. Yes, even after trying all three tools. It's not perfect, but it's pragmatic. I've streamlined my scripts, built some reusable functions, and accepted that sometimes "good enough" beats "theoretically optimal." The team knows it, it works, and we can ship.

Bicep extensions for the remaining 5-10%. Here's the thing: Bicep extensions for Databricks aren't GA (generally available) yet. That matters. It's preview technology, which means potentially breaking changes. Plus, you need to keep your Databricks extension files separate from your regular Azure Bicep files

Terraform? We decided against it. Not because it's bad — it's excellent. But the operational overhead didn't justify the benefits for our specific situation. If we were multi-cloud or had a dedicated platform team, different story.

Here's the uncomfortable truth: there's no "best" tool. There's only the best tool for your situation right now.

We're still mostly using PowerShell because:

• The team knows it inside and out now
• It's stable
• No preview or GA concerns to worry about
• It works with our existing workflows

But I'm watching Bicep extensions closely. When they hit GA, when the tooling matures, when the separation concern gets addressed? We'll revisit. That's the real lesson: infrastructure decisions aren't permanent. They evolve as the tools evolve.

The tool doesn't matter as much as you think

Six months ago, I thought choosing the right tool was everything. I read blog posts, watched YouTube talks, and debated in Teams channels.

Now I know better.

The best infrastructure code isn't the code written in the "best" tool. It's the code that your team actually maintains. It's the code that solves real problems. It's the code that ships.

These days, I spend less time debating tools and more time deploying infrastructure. I write Bicep templates for Databricks. When they don't work, I debug them. When they need to change, I change them. And you know what? It's boring. Which is exactly what infrastructure management should be.

Next up? I'm working on extending the Databricks Bicep extension with better examples and documentation. Because the real work isn't choosing tools — it's helping people actually use them.

Want to try the Bicep Databricks extension? Check out the documentation and examples on GitHub.