Ensuring Reliable Data for Self-Service Analytics: How Lightup and Dremio Work Better Together

Bryan Feuling

July 16, 2024

Data lakes, data warehouses, databases, data lakehouses — data is vast, varied, and scattered everywhere in modern enterprise data stacks. Traditional methods of running analytics in these complex environments is an ongoing challenge for enterprises due to:

Costly and time-consuming data management processes
Scarce and expensive data engineering resources
Poor-quality data

That’s where Lightup and Dremio come in, providing easy access to no-code Data Quality Checks for self-service analytics. While there are a variety of vendors that provide Data Quality solutions based on specific data sources, there are very few that connect directly to Dremio for Data Quality Monitoring.

Dremio for Self-Service Analytics

Built on Apache Arrow as the only Open Unified Lakehouse Platform, Dremio’s advancements in data virtualization, cataloging, and query optimization gives analysts and data engineers easy access to query any dataset across on-premises and cloud sources, unifying data quickly for self-service analytics.

Regardless of where data is stored — legacy data warehouses, transactional databases, or data lakes — Dremio knows where it is, how to query it, and how to retrieve it for analysis — super-powerful, super-streamlined, super-modern self-service analytics that keeps up with the speed of changing business requirements.

While Dremio doesn’t load or store data itself, it operates like a modern data platform, giving analysts a unified system to query any and all connected data sources.

The best part? Dremio virtually eliminates data silos and resource-intensive data management tasks, such as ETL workflows required to extract, transform, and load data into a single location before running analytics the typical way.

Lightup and Dremio Integration

Enhancing Dremio’s self-service analytics capabilities, the Lightup integration with Dremio allows analysts and data engineers to write no-code, low-code, and custom SQL queries to check the quality of the data that Dremio returns.

Lightup pushes Data Quality Checks down to Dremio, where they get pushed down to the data source(s) Dremio is connected to — without moving or copying data. This allows Lightup to query any data source connected to Dremio, without requiring a direct connection to the data source itself.

Simply put, wherever you run a Dremio SQL query, you can also run a Lightup Data Quality Check — without moving or copying data. That means our joint customers can deploy referential integrity checks on any Dremio supported data source, including data as code, metastores, local and cloud-based object stores, and even CSV files.

Referential Integrity Checks
Dremio supports referential integrity checks that can help answer questions such as, “Is the city in my Redshift table also present in the CSV file in my S3 Bucket in column A?” Before loading a CSV from a Redshift table into S3, simply create a Lightup Data Quality Check to see if there are data values in the CSV that aren’t in S3. If Lightup detects discrepancies, Lightup will send notifications, so the CSV can be fixed before it’s loaded into S3.

Key Benefits

Improve Analytics Efficiency
Simplify analytics workflows by querying, analyzing, and running Data Quality Checks across multiple sources with Dremio and Lightup, without moving data between systems and loading it into one place for analysis and data scans.

Ensure Data Reliability
Run referential integrity checks at scale with automatic slicing to ensure data consistency and accuracy across disparate data sources like Amazon Redshift, Amazon S3, and MongoDB for reliable data insights and accurate analytical outputs.

Reduce Operational Costs
Bypass ETL processes, accelerating time-to-insight while reducing operational costs associated with data movement, replication, and transformation by deploying pushdown queries through Lightup and Dremio.

Expand Data Quality Coverage
Expand Lightup’s Data Quality coverage to include Dremio’s wide range of supported data sources, such as NoSQL databases and CSV files.

Fortifying Self-Service Analytics with Self-Service Data Quality

Lightup’s integration with Dremio marks a significant advancement in simplifying self-service analytics workflows for modern enterprises. By eliminating the complexities of data management and offering an easy way for analysts and data engineers to access, query, and check data anywhere it resides, this integration fortifies self-service analytics with self-service Data Quality at enterprise scale.

Whether your goal is to unify data access for self-service analytics, enable easy-to-use Data Quality Checks for Dremio users, expand Data Quality coverage across more data sources, or reduce operational costs associated with conventional analytics and Data Quality workflows, Lightup and Dremio provide a powerful modern solution designed to meet complex enterprise requirements.