Azure DevOps Databricks

How to privately install Python packages on Databricks cluster with Azure Artifacts

Learn how you can privately install Python packages on your Databricks clusters

Gijs Reijn
Gijs Reijn
• 4 min read
Privately install Python packages on Databricks
Privately install Python packages on Databricks

Are your Python dependencies secure? Do your compliance requirements prevent you from using public repositories for installing dependencies?

While NPM is a different ecosystem, recent events highlighted once again the importance of carefully downloading and installing packages you rely on. One wrong package can compromise your enterprise.

As workspace admins, you want to ensure that users using your Databricks instance default to installing privately. If they want to install from PyPI.org explicitly, they should still be allowed to do so (there are other methods to close this off).

In this tutorial, you will learn how to connect to Azure Artifacts to install Python packages privately.

Prerequisites

Assuming you want to follow along, you're going to need:

  • An Azure DevOps Services account.
  • A running Databricks instance.
  • The Databricks CLI v0.264.0 or above.

Step 1 - Set up Azure Artifacts

To be able to download and install Python packages from Azure Artifacts, you first need one.

  1. Go to https://dev.azure.com/.

  2. Select the project you want to work in.

  3. In the left navigation, select Artifacts.

    select-artifact.png

  4. If this is your first feed, select Create Feed. Otherwise, select + New Feed.

    create-artifact.png

  5. Enter a Name for the feed (for example, PrivatePython).

  6. (Optional) Provide a Description to help other users understand the purpose of this feed.

  7. Under Visibility, choose one of the following:

    • Organization – allow all members of the organization to use the feed.
    • Project – restrict access to the current project.
    • Specific people or groups – grant access only to selected users.

  8. Leave the Upstream sources to allow packages to flow from external registries (in this case, PyPI.org).

  9. Select Create.

You have now successfully created an Azure Artifacts feed. We purposely left the "Upstream sources" enabled, even though it still goes to an open-source repository. This is for demo purposes to illustrate it downloads packages through Azure Artifacts.

Step 2 - Upserting the credential scope and secret

By default, users who want Python packages installed should always go to the private repository, unless they explicitly don't specify the --index-url. You, as workspace admin, can therefore set two important values in the instance:

  • A Databricks secret scope named databricks-package-management.
  • A secret key linked to the scope with the name: pip-index-url.

When both values are set, each cluster that is spawned will use the configured settings.

  1. Create a Personal Access Token (PAT) in Azure DevOps:

    1. Go to https://dev.azure.com/<organizationName>.
    2. In the upper-right corner, select your profile picture and then select Security.
    3. Under Personal Access Tokens, select + New Token.
    4. Enter a Name (for example, artifacts-python).
    5. Set the Organization to your Azure DevOps organization.
    6. Set the Expiration to a suitable period (for example, 30 or 90 days, depending on your compliance policy).
    7. Under Scopes, select Packaging (Read).
    8. Select Create and copy the PAT value. You will use this in later steps.

  2. Authenticate with Databricks CLI (if not already logged in):

    databricks auth login

  3. Create a secret scope in Databricks for package management:

    databricks secrets create-scope databricks-package-management

  4. Configure access control lists (ACLs) for the scope:

    databricks secrets put-acl databricks-package-management admins MANAGE
databricks secrets put-acl databricks-package-management users READ

  5. Store the private index URL as a secret. Replace with the full Azure Artifacts Python feed URL, including your PAT token.

    databricks secrets put-secret --json '{"scope": "databricks-package-management", "key": "pip-index-url", "string_value":"<index-url-value>"}'

    Example format for the value:

    https://<username>:<pat-token>@pkgs.dev.azure.com/<organizationName>/<projectName>/_packaging/<feedName>/pypi/simple/

At this point, you have securely stored the package index URL and credentials in Databricks. Your users won’t need to specify manually --index-url or --extra-index-url when installing packages.

Step 3 - Test out library installation

To verify that the default index URL is correctly being used from the secret scope and secret you created earlier, you need a cluster to test with. By installing a standard package such as pandas, you can confirm that the package is resolved through your private Azure Artifacts feed instead of PyPI.org.

  1. Create a demo cluster in Databricks:

    1. In the Databricks workspace, go to Compute.
    2. Select + Create Cluster.
    3. Enter a Cluster Name (for example, demo-cluster).
    4. Choose a small cluster configuration (for example, Standard_DS3_v2) suitable for testing.
    5. Select Create Cluster.

  2. Start the cluster if it does not start automatically.

  3. Install a Python library from the private feed:

    1. In the cluster details page, select the Libraries tab.

    2. Select Install New.

    3. Choose PyPI as the library source.

    4. In the Package field, enter:

      pandas==2.3.2

    5. Select Install.

    install-pandas.png

    This test confirms that users no longer need to specify --index-url or --extra-index-url. Instead, the cluster uses the default value stored in the Databricks secret scope.

  4. Verify the installation succeeds. The pandas package should install successfully without additional configuration.

  5. Check Azure Artifacts:

    1. Go back to your Azure DevOps portal at https://dev.azure.com/<organizationName>.
    2. Navigate to the project and select Artifacts.
    3. Open your feed (for example, PrivatePython).
    4. Under Packages, confirm that pandas now appears as a cached package in your feed.

You have now validated that Databricks can resolve and install Python packages directly from your Azure Artifacts feed, ensuring secure and private access across your workspace.

Summary

You have successfully installed a Python package through the Azure Artifacts. Users who want to install Python packages through notebooks, Databricks Asset Bundles (DABs), or any other mechanism will consistently be enforced by the secret scope. Please note that when they explicitly define the --index-url, it will take precedence over the default.

Even though you have used the "Upstream sources" to fetch packages from PyPI.org, it illustrates that if you push the packages yourself to the Azure Artifact feed, you can install them from here.