We’re excited to announce that our partnership with Databricks has expanded with full support for Unity Catalog through Databricks Partner Connect.
While Unity Catalog helps simplify data discovery in Databricks, that requires ongoing certification from a Data Quality Monitoring tool. Without that, Databricks users won’t know if they can trust the data they pull from Unity Catalog.
That alone limits their access to real self-service data in Databricks, because they’ll constantly depend on data engineers for help certifying the data, causing time-consuming bottlenecks. Even worse, they might blindly use bad data, causing hidden bugs in reports, inaccurate machine learning models, and low-quality products and services.
That’s why we’ve strengthened the Lightup integration with Databricks to fully support Unity Catalog.
Unity Catalog Support
As a Databricks technology partner, Lightup offers a prebuilt connector enabling users to connect Databricks to Lightup for modern Data Quality Monitoring with AI-powered Anomaly Detection and Integrated Incident Alerts.
After Databricks launched Unity Catalog, “a unified governance layer for data and AI within the Databricks Intelligence Platform,” we expanded the Lightup integration with Databricks even further to include full support for Unity Catalog, streamlining the process of connecting Databricks Workspace Catalogs to Lightup.
Before, when connecting to Lightup through Databricks Partner Connect (DPC), users could choose their Databricks SQL warehouse and schemas, but the Catalog option was locked to the default catalog hive_metastore.
Now, Lightup fully supports Databricks Unity Catalog, enabling Databricks users to automatically share their Databricks Workspace Catalogs with Lightup by clicking Partner Connect — without manually configuring each one separately.
How to Get Started
From your Databricks Workspace, click Partner Connect and select Lightup.
Then, use the drop-down menus to select the Catalogs and schemas to use in Lightup.
Lightup automatically grants the same privileges based on your Databricks Workspace.
The necessary resources and tokens are automatically generated to connect the selected Catalogs to Lightup.
After the resources have been established to connect to Lightup, sign in to your Lightup account by clicking Connect to Lightup.
Lightup automatically sets up a free 30-day trial, so you can test out the features and functionality.
Finally, you’ll be taken to your Lightup Workspace, where you’ll see your Databricks Catalogs in the Explorer tab.
Key Benefits
- Get quick, easy access to Lightup directly from Databricks Partner Connect.
- Automatically share Databricks Catalogs and Schemas with Lightup, eliminating manual connector configuration for each Databricks Catalog connection.
- Unlock access to other Catalogs, beyond the default Hive metastore.
- Reduce Data Quality bottlenecks by empowering data stewards, data analysts, and non-technical business users with self-service access to Lightup’s no-code Data Quality Checks, without requiring Databricks SQL expertise.
Enabling Self-Service Checks for Unity Catalog Data
Lightup enhances the usability of Unity Catalog data, especially for data stewards, who need an easy way to check data consistency and quality requirements without relying on data engineers.
By offering self-service access to no-code/low-code Data Quality Checks, Lightup enables non-technical business users who know the context of the data — but may not know Databricks SQL — to step in and write their own checks to ensure the data is trustworthy and correct for their purposes. (Learn more about how Lightup empowers enterprise organizations to democratize Data Quality from engineers to executives for accessible, trusted data.)
Giving non-technical business users the ability to self-serve their own Data Quality Checks for their datasets in question allows enterprise organizations to:
- Accelerate Data Quality rule authoring
- Mitigate risks of downstream Data Quality issues
- Increase data reliability and trust
- Reduce Data Quality bottlenecks
- Support a data-driven culture with data democratization
And this is just the beginning. Stay tuned for more enhancements as Lightup continues to innovate with Databricks and all our partners to deliver the best experience for our joint customers.
