Run a Beam pipeline

FlareDB runs Apache Beam pipelines as jobs. Build your pipeline with the Beam SDK as usual, then set FlareRunner as the runner to submit it to a running FlareDB instance.

Prerequisites

A running FlareDB instance. See the Quickstart.
JDK 17.
An Apache Beam project. The examples in this guide use Gradle and Beam 2.76.0; Maven is supported as well.

1. Add the runner dependency

Add the flaredb-runner artifact to your Beam project.

Gradle
Maven

build.gradle

dependencies {
    implementation 'com.flare-db:flaredb-runner:0.2.0'
}

pom.xml

<dependency>
  <groupId>com.flare-db</groupId>
  <artifactId>flaredb-runner</artifactId>
  <version>0.3.2</version>
</dependency>

2. Configure the pipeline

Set FlareRunner as the runner and configure the FlareDB instance and application JAR:

WordCount.java

WordCountPipelineOptions options =
    PipelineOptionsFactory.fromArgs(args).as(WordCountPipelineOptions.class);

options.setRunner(FlareRunner.class);
options.setJobEndpoint("127.0.0.1:8099");
options.setUberJar("build/libs/wordcount-0.2.0-all.jar");

Pipeline pipeline = Pipeline.create(options);

The JAR is staged to FlareDB so your pipeline code is available when the job runs. Point setUberJar to the fat JAR produced by your build. See the full pipeline in the WordCount example.

3. Build the pipeline

Build a self-contained (uber) JAR containing your pipeline and its dependencies.

./gradlew shadowJar

Use the build output’s JAR path in setUberJar.

4. Run the pipeline

With FlareDB instance running, submit the pipeline:

./gradlew run

Pipeline options

Define your options by extending FlarePipelineOptions, For Example:

WordCountPipelineOptions.java

public interface WordCountPipelineOptions extends FlarePipelineOptions {}

Option	Usage	Description
Runner	`setRunner(FlareRunner.class)`	Pipeline runner.
Job endpoint	`setJobEndpoint("host:port")`	URL of the FlareDB job service. Defaults to `127.0.0.1:8099`.
Uber JAR	`setUberJar("/path/to/app-all.jar")`	Path to the fat JAR staged to workers.
Job name	`setJobName("my-job")`	Name of the submitted job.

These options can also be provided as command-line arguments:

./gradlew run --args="--jobEndpoint=127.0.0.1:8099 --uberJar=/path/to/app-all.jar"

Pipeline logs

A JOB-ID is generated automatically for each submitted job. Use it to view logs for a specific job, or stream the most recent logs when you do not provide an ID. Usage:

  flare logs                  # Stream logs for the most recent job
  flare logs <JOB_ID>         # Stream logs for a specific job

The JOB-ID is logged during job submission via the runner SDK.

Next steps

FlareDB I/O — read from and write to FlareDB tables from a pipeline.
SQL shell — query the data your pipeline produces.

Intoduction

Guides

Run a Beam pipeline

Prerequisites

1. Add the runner dependency

2. Configure the pipeline

3. Build the pipeline

4. Run the pipeline

Pipeline options

Pipeline logs

Next steps

​Prerequisites

​1. Add the runner dependency

​2. Configure the pipeline

​3. Build the pipeline

​4. Run the pipeline

​Pipeline options

​Pipeline logs

​Next steps

Prerequisites

1. Add the runner dependency

2. Configure the pipeline

3. Build the pipeline

4. Run the pipeline

Pipeline options

Pipeline logs

Next steps