Skip to main content
FlareDB is a streaming database for running Apache Beam pipelines and working with the data they produce. Write pipelines using Apache Beam SDKs such as Python, Java, or Go. FlareDB executes those pipelines on a Rust-based engine and provides a unified way to process, store, and query the data.

​
One system for streams and tables

FlareDB is built around a unified streams-and-tables architecture:
  • Streams represent data in motion — the events flowing through a pipeline.
  • Tables represent data at rest — materialized, persistent, and queryable state.
Streams and tables are two views of the same underlying data. Streams are produced by tramsfroms/operators in the pipeline, while a table represents the materialized state of the same data. FlareDB brings this model directly into pipeline execution. As a pipeline runs, its PCollections are persisted as materialized data, allowing processing and storage to live in the same system. The resulting data can then be queried as tables using SQL or serve them to application, dashbaords and agents.

​
How it works

  1. Write a pipeline - Using any Apache Beam SDK (Python, Java, Go), describe how your data should be transformed.
  2. FlareDB runs it - Submit the pipeline to a running FlareDB instance. FlareDB breaks your pipeline into stages, schedules them to run in order, and streams data through each transformation step until the pipeline completes.
  3. Write results to tables - Add a FlareDbIO.write() step to store your results.

​
Get started

Quickstart

Install the CLI, start an instance, and run your first pipeline.

Run a Beam pipeline

Add the FlareDB runner to an existing Apache Beam project.

FlareDB I/O

Read from and write to FlareDB tables with the Beam I/O connector.

SQL shell

Query the local warehouse with the interactive SQL shell.