Skip to main content
FlareDB is a streaming database for running Apache Beam data pipelines and making the results queryable. It unifies pipeline execution, storage, and SQL in a single Rust-based system. Write pipelines using any Apache Beam SDK: Java, Python or Go. FlareDB executes them on a Rust engine, materializing the results as queryable tables in the same system.

​
The problem

A data pipeline describes how data should move and transform, but running one is only half the job. Results typically land as output files, or get pushed to a separate database, before anyone can actually query them, for a dashboard, another pipeline needing that data, or a person running SQL. Each hop is a system to operate and a boundary the data has to cross. FlareDB replaces this into a single system.

​
How it works

​
One system for streams and tables

FlareDB is built around a unified streams-and-tables architecture:
  • Streams represent data in motion, the events flowing through a pipeline.
  • Tables represent data at rest, materialized, persistent, and queryable state.
Streams and tables are two views of the same underlying data. Streams are produced by the transforms in a pipeline and a table is the materialized state of that same data. FlareDB brings this model directly into pipeline execution — as a pipeline runs, its PCollections are persisted as a materialized state that is ready to query.

​
Write a pipeline

Using any Apache Beam SDK: Java, Python or Go, describe how your data should be transformed. Pipelines are standard Beam: no proprietary syntax, no FlareDB-specific pipeline API.

​
FlareDB runs it

Submit the pipeline to a running FlareDB instance. FlareDB breaks the pipeline into stages, schedules them to run, and streams data through each transformation step on its Rust execution engine until the pipeline completes.

​
Write results to tables

Add a FlareDbIO.write() step to persist your results as a named table. From there the data is queryable with standard SQL through flare sql, FlareDB’s interactive SQL shell, or any client speaking FlareDB’s SQL interface — and reusable as input to another Beam pipeline via FlareDbIO.read().

​
Capabilities

  • Pipeline output is a table - results are written to a table. Query them with SELECT, WHERE, COUNT(*) via flare sql.
  • Pipelines compose through tables - one pipeline’s output table can be read as another pipeline’s input, without an intermediate storage layer.
  • Arrow end to end - data is represented as Apache Arrow through pipeline execution, storage, and query execution.
  • DataFusion query engine - SQL against FlareDB tables executes on Apache DataFusion. flare sql and FlareDbIO.read().fromQuery() use the same query path.
  • No vendor lock-in - FlareDB runs unmodified Apache Beam pipelines through the standard Beam portability layer, no FlareDB-specific API.

​
Get started

Quickstart

Install the CLI, start an instance, and run your first pipeline.

Run a Beam pipeline

Add the FlareDB runner to an existing Apache Beam project.

FlareDB I/O

Read from and write to FlareDB tables with the Beam I/O connector.

SQL shell

Query the local warehouse with the interactive SQL shell.
Last modified on October 6, 2026