How to Get Real-Time Data in Microsoft Fabric
Imagine you’re at a busy airport, and every minute counts for travelers trying to catch their flights. With real-time data, airport management can keep track of flight statuses and passenger flow instantly. Picture a data platform using real-time analytics to monitor security wait times and gate changes. This lets staff send quick alerts to passengers about delays or faster routes to their gates. It even helps optimize staffing at security checkpoints based on how many people are there. That’s the power of real-time data—it keeps everything running smoothly and helps travelers navigate their journeys with ease!
In this blog, we’ll explore three ways to get (near) real-time data using Microsoft Fabric tools. We’ll talk about the pros and cons of each option.
First up is Fabric Real-time Intelligence, which we’ll compare to Fabric Mirroring. We’ll also touch on why you might want to consider batch-driven data ingestion instead.
Real-Time Intelligence
Fabric Real-time Intelligence is your go-to for handling event-based data, streaming data, and logs. It’s a no-code solution for ingestion, transformation, storage, modeling, visualization, alerts, and more. This means it’s super easy to get started!
However, if you’re used to working with non-event-based or batch data, it might take some time to adjust to interpreting event-based data.
Ingestion and Transformation
When you need to ingest event-based data or reflect changes in real-time, Real-time Intelligence is the way to go. Both ingestion and transformations happen as soon as the events come in. The data you ingest is stored in a KQL database in an Eventhouse, where you can organize it into bronze, silver, and gold tables, often part of the medallion architecture.
You can apply transformations right in the event stream that handles ingestion or in the KQL database with update policies as data comes in. Plus, it’s easy to create a real-time dashboard to visualize your data and choose how often you want it refreshed—anywhere from continuously to once a day. A cool feature is that you can set alerts when certain thresholds are reached, sending you an email, Teams message, or even triggering another Fabric item.
Points to Consider
Of course, not everything is perfect. Fabric Real-time Intelligence has some downsides. One major concern is its unpredictable Capacity Units (CU) consumption and storage costs. While you don’t pay directly for CU consumption, it draws from your total available CU, so it’s important to have an idea of its expected use. Since event-based data can be hard to predict, estimating CU consumption and storage can be tricky without knowing the size of the events/messages you’re processing.
Also, the transformations you can do on incoming data are somewhat limited. You can use no-code visual building blocks or write SQL, but even the SQL is a subset of T-SQL, which means fewer functions and less flexibility.
Finally, the sources you can connect to are still limited. For instance, getting data from an on-premises Microsoft SQL server isn’t natively possible.
Fabric Mirroring
Given the limitations of Real-time Intelligence, you might want to consider other options. But keep in mind, Fabric Mirroring is only a good choice if you don’t need actual real-time data and can settle for near real-time.
Fabric Mirroring gives you a near real-time replica of your source database, which is stored in a lakehouse in Fabric, making it accessible from all other Fabric tools. The setup is pretty straightforward, and Microsoft has good documentation for it.
One of the biggest perks of Fabric Mirroring is that any compute used to replicate your data doesn’t count against your Fabric Capacity. Plus, you get 1 TB of free storage for this replica per Fabric Capacity tier (e.g., F2 means 2TB and F8 means 8TB), which can save you money on data ingestion and storage.
Downsides
But, like any tool, it has its downsides. You need a high level of security access on the source systems to set it up, or you’ll need to work closely with someone who does. Additionally, it lacks support for some popular databases, like on-premises PostgreSQL. Microsoft offers open mirroring, which lets you use your own application to write data to the Fabric Mirroring system, but that adds complexity and resource consumption.
Mirroring – Microsoft Fabric | Microsoft Learn
Another thing to note is that Fabric Mirroring currently has no SLA (yet), meaning there’s no guarantee that your replicated data will always be up to date. This also means there’s no promise about service uptime, so you could be without up-to-date data for a while. Plus, Microsoft could decide to start charging for the service at any time, and you’d have to accept whatever price they set.
Batch-Driven ELT
If neither of the above options works for you, or if the uncertainty is too much, you can always go back to good old batch-driven data loading. While this won’t give you real-time data, scheduling your process regularly means you have full control over how often your data gets updated.
Fabric offers a variety of options for batch-driven ingestion, like Notebooks, Pipelines, or dataflows. The upside is that you have a lot of flexibility and control over transformations, plus there’s plenty of documentation and troubleshooting help available. On the downside, it might require more work to implement and won’t be as real-time as Real-time Intelligence or as cheap as Fabric Mirroring.
Concluding Thoughts
In the end, there’s no one-size-fits-all solution. Each tool has its own benefits and downsides. Hopefully, this blog has given you some insights to help you decide which option best suits your (near) real-time data ingestion needs.
Below some reference material:
1Fabric Capacity Estimator | Microsoft Fabric
2 Mirroring – Microsoft Fabric | Microsoft Learn
Mirroring for SQL Server in Microsoft Fabric (Preview) | Microsoft Fabric Blog | Microsoft Fabric
