Comparing query design efficiencies in Power BI, with text files, lakehouse & database tables

I wanted to share the results of a few experiments I recent conducted with one of my favorite sets of sample data. This will take at least two blog posts to cover, but I will summarize them here:

Compare data load & transformations with CSV files vs a Fabric lakehouse using the SQL Server connector:
Loading 20 million fact rows from CSV files vs a Fabric lakehouse, using Power Query.
Same comparison with deployed Power BI model.

Comparing Fabric data transformation options & performance:
Loading & transforming the same data with Power BI Desktop, Fabric Gen2 dataflows, Fabric pipelines and Spark notebooks.

Comparing semantic model performance in Fabric and Power BI:
Report & semantic model performance comparing the same data in Import mode, DirectQuery and Direct Lake.

Continuous Delivery & Version Control for Power BI

For the business intelligence professional, DevOps can be a confusing topic because it intersects with many disciplines and philosophies. I’m hopeful that a bit of reflection on my own experience over the years provides some valuable insight. I have seen DevOps and version control implemented on dozens and perhaps hundreds of projects, with different degrees of sophistication and control.

If you work in a software development environment where DevOps is practiced as part of your team’s development culture, you know that DevOps truly is a holistic methodology, including practices can be very extensive and regimented. Software development tends to be a linear process, whereas data and BI projects are more iterative in nature. Frankly, DevOps purists can be down-right militant about enforcing all the rules, which many BI specialists tend to skirt, in an effort to move quickly.

If you are a business intelligence analyst or developer and suddenly find yourself working on a team where DevOps is practiced, you will likely find the process to be more strict and less flexible than typical ad hoc BI development. The key is to strike the right balance for your organization’s needs.

There is certainly a flavor of DevOps that seem to be over-engineered and counter-productive. DevOps practices exist because they address critical needs in a software development environment, but you should find the right balance for your organization’s project needs. Be mindful that the very purpose for business intelligence is to deliver insights and reporting insights directly to business users, which entails quick iterations through the entire process – from requirement refinement to report delivery, with several steps in-between.

Direct Lake memory: hotness, popularity & column eviction

I just read that the Miss Universe contestant from Panama was evicted from the Miss Universe pageant. I don’t know…