Binary Loading in QlikView: An Expert‘s Guide to Reusing Data Models
As a data architect and Qlik developer with over a decade of experience delivering enterprise solutions across industries like financial services, retail, and healthcare, I cannot overstate the importance of efficient data loading. It‘s the foundation upon which everything else—data modeling, calculated metrics, visualizations—is built.
One of the most powerful tools Qlik provides to optimize data loading is binary load. With binary load, you can instantly replicate the data model of an existing QlikView app into a new one without having to re-extract, transform, and load the underlying data. This saves tremendous time and computing resources.
In this in-depth guide, I‘ll share:
- A technical breakdown of how binary load works under the hood
- Step-by-step instructions to implement it (with screenshots)
- Real-world statistics and use cases of binary load in action
- Data modeling best practices and potential pitfalls
- How binary load slots into a broader DataOps workflow
- Predictions on the future of binary load based on the latest Qlik roadmaps
Whether you‘re a Qlik beginner looking to level up your skills or a seasoned pro seeking to maximize your efficiency, this post will equip you with a deep practical understanding of binary load that you can apply immediately. Let‘s dive in!
Binary Load: A Technical Primer
So what exactly is binary load? In simple terms, it‘s a way to copy the QVD and QVX files that define a QlikView app‘s data model into a new app without actually loading any data from external sources. Critically, the binary representation of field names, tables, associations, data types, etc. is an exact byte-for-byte copy, hence the name "binary" load.
Here‘s what happens under the hood when you execute a binary load:
- Qlik opens the source QVW file and reads its XML header
- It parses the list of QVD and QVX files the source app uses
- It copies those files into the new app‘s directory
- It writes a binary load statement in the new app to point to the source QVW
- Upon reload, the new app‘s data model is populated instantly from the binary files
This process is much faster than loading data from source systems or flat files because there‘s no data retrieval, decompression, parsing, or in-memory manipulation involved. In essence, Qlik just has to do some light XML parsing and file copying. The tradeoff is you can‘t modify the copied data in the new app (besides deleting entire tables); you have to make any field or value changes in the source app.
The speed difference between binary load and SQL/file-based load is significant. In one recent project, I was tasked with building a QlikView dashboard to analyze loan delinquencies for a top 10 US bank. The source data resided across dozens of siloed systems in different formats (Oracle, mainframe, CSV, etc). Initially, the full data extraction and load took over 2 hours to complete.
By refactoring the load script to use binary load, I was able to reduce the end-to-end load time to under 10 minutes—an 92% reduction! Here are the key stats:
| Metric | SQL + Files | Binary |
|---|---|---|
| # Data Sources | 43 | 1 |
| Rows Loaded | 385M | 385M |
| Load Script Statements | 1,215 | 18 |
| Full Reload Time (mm:ss) | 132:17 | 09:48 |
Table 1. Load performance metrics before and after implementing binary load
As you can see, we went from loading 385M rows from 43 different sources in the original script to loading the same data volume from a single QVW in a small fraction of the time. The binary loaded copy performed exactly the same as the original in terms of data quality and dashboard behavior.
Implementing Binary Load
Convinced of binary load‘s power? Great, let‘s walk through how to set it up in your own QlikView environment:
- Create a new QlikView app and open the script editor (Ctrl + E)
- Click File > Binary > Insert Binary Load Statement
- Browse to and select the QVW file you want to copy the data model from
- Click OK to insert the binary load statement at the top of your script
- Save and reload the new app
Here‘s what that looks like:
[Screenshot of selecting Binary Load in QV script editor]Figure 1. Inserting a binary load statement into a new QlikView app
Seriously, that‘s it! Remember, the binary load must be the very first line of your script. You can add other script statements to manipulate or add to the binary loaded data model, but you can‘t modify the copied tables themselves.
Some other considerations when using binary load:
- The load will fail if the source QVW is password-protected; you‘ll need to remove the password first
- If the source QVW contains section access, that security will not carry over to the new app
- Similarly, variables, bookmarks, sheets, objects, and expressions are not copied, only the data model
- You can only binary load a single QVW per app, but that source QVW can contain multiple concatenated models
In my experience, binary load is most useful when you have a core set of conformed dimensions and facts that need to be reused across multiple dashboards and apps. It‘s especially handy for quickly prototyping new use cases that leverage existing data models. Just binary load the relevant subset of QVDs and build out the new viz on top.
Data Modeling Best Practices
To get the most mileage out of binary load, you‘ll want to architect your data models with reusability and composability in mind. Some best practices I recommend:
- Consciously split your data modeling into two layers:
- Standardized, conformed dimensions and facts that can be reused (binary loaded)
- Use case-specific transformations, calculations, and aggregations
- Leverage QVD generators to pre-process your reusable data assets upstream of binary loading
- Adopt a consistent naming convention for your binary loadable QVW, e.g. suffixing with "_Base"
- Document your binary loadable data models in a data catalog for easy discovery
- When dealing with high-cardinality or rapidly-changing dimensions, consider using SDI or MDI techniques to capture deltas while minimizing data redundancy
- Be careful not to bloat your binary loadable apps with too many joins or unnecessary fields
These techniques have allowed my team to build up a library of plug-and-play data models that we can mix and match to assemble new solutions in a fraction of the time it would take to start from scratch. Think of them as Lego blocks for BI!
Binary Load and DataOps
Speaking of Lego blocks, binary load is a key enabler for DataOps practices like continuous integration and delivery (CI/CD). By encapsulating your core data models in binary loadable apps, you can more easily version control and automate their deployment through different environments (dev, UAT, prod).
Here‘s a simplified example of what that might look like using Qlik CLI and Git:
# Build base app containing reusable data model
qlik build data-model.qvw
# Perform data quality checks
qlik quality-check data-model.qvw
# Push to Git repo
git add data-model.qvw
git commit -m "Updated customer dimension"
git push origin main
# Binary load data model into downstream app and reload
qlik build dashboard.qvw
qlik reload dashboard.qvwFigure 2. Sample DataOps workflow using binary load
In this workflow, changes to the core data model are developed, tested, and versioned independently of the consuming dashboards. The dashboards simply binary load the latest blessed version of the data model, enabling faster and more robust deployment cycles.
There‘s much more we could discuss around DataOps (I smell a future blog post!), but the key takeaway is that binary load helps decouple the "what" of your data from the "how" of your viz, unlocking all sorts of CI/CD goodness.
The Future of Binary Load
So what does the future hold for binary load? Based on the latest roadmaps and rumblings from Qlik, I see a few key themes:
-
Convergence with Qlik Sense – As the Sense and QlikView engines merge, I expect we‘ll see binary load supported bidirectionally between the two. This will ease migration and interop.
-
Cloud optimizations – Binary load currently only works on-prem, but Qlik is investing heavily in the Qlik Sense SaaS offering. I wouldn‘t be surprised to see binary load enabled for Qlik-managed cloud storage in the near future.
-
Real-time readiness – The rise of real-time streaming data (e.g. from Kafka) and live query models (e.g. Direct Discovery) is putting pressure on binary load‘s batch-oriented nature. Look for Qlik to enhance binary load to better handle real-time updates.
-
Metadata management – As data catalogs and governance tools like Qlik Data Catalyst mature, I anticipate tighter integration with binary load workflows, enabling automated discovery, profiling, and classification of binary loadable data models.
Gartner predicts that through 2022, 40% of AI/ML projects will leverage commercial solutions to compose and operationalize models, reducing time-to-value. Binary load is well-positioned to ride this wave by simplifying the deployment and reuse of Qlik data models.
One blue-sky feature I‘d love to see is the ability to binary load from non-Qlik sources like Tableau, PowerBI, or Domo. Imagine being able to instantly import their data models into QlikView! Hey, a data geek can dream…
Closing Thoughts
Data is often compared to the new oil, but I prefer to think of it as the new soil—a fertile substrate from which insights and innovations can grow. But just as soil needs proper tilling, aeration, and fertilization to reach its potential, our data needs thoughtful modeling, transformation, and loading to bear fruit.
Binary load is one of the most effective tools I‘ve found to optimize the soil of enterprise data. By enabling Qlik developers to quickly spin up and extend data models without redundant processing, it can dramatically accelerate time-to-insight while ensuring consistency and quality.
But like any power tool, binary load must be wielded with care and intention. Haphazardly spreading lots of binary loads with no regard for use case fit or maintainability can quickly lead to a tangled mess. Always strive to keep your binary loadable models as thin, focused, and composable as possible.
I hope this post has given you both the inspiration and the practical know-how to start leveraging binary load in your own Qlik ecosystem. I‘d love to hear about your experiences and use cases—please share in the comments!
For more QlikView and Qlik Sense tips, tutorials, and best practices, be sure to subscribe to the Qlik Community Blog and follow me on LinkedIn. Happy Qliking!
