Pipelines

Pipelines are transformation jobs that execute a specified transformation (referred as model) continuously and maintains an incremental view of the output in our destination. Rubicon support two kinds of pipelines:

1. Production pipeline

Used for models in the main Github branch. Rubicon assumes any pipeline for a model in main branch to be in-production and supports

• Continuous ETL of the transformation.

• Automatic deployment of new versions of pipeline when a new version of the transformation is checked-in

• Sending metrics to your observability platform

1. Development pipeline

Used for testing your models while developing them. Rubicon assumes any pipeline for a model in a non-main branch is development and creates a materialized view of the transformation for data but does not continuously update after it has processed all the data up until the time the pipeline was created.

Creating a new pipeline

Step 1: Go to https://app.rubicon.io/pipelines. Select the Github branch for the model

Step 2: Pick any of the available models in the branch that you want to build a pipeline for.

Step 3: Pick a destination for the output of your transformation.

Step 4: Deploy the pipeline.