Documentation

Rubicon is a modern streaming platform that enables developers to build complex data pipelines using SQL. Rubicon run continuous transformations on the changes in the data from your data sources and maintains a materialized view in the destination of your choice completely eliminating the complexity of running scheduled jobs and providing real-time updates for use cases like search, customer facing dashboards and AI driven features like recommendations.

Step 1: Connect Source

On the Sources page, select your data source from our supported connectors list. Fill out the setup form using the instructions embedded in the dashboard, then click Save & test to create your connector. Check here for the list of supported sources.

Step 2: Connect Destination

On the Destination page, select your destination from our supported destinations list. Fill out the setup form using the instructions embedded in the dashboard, then click Save & test to connect your destination. Check here for the list of supported destinations.

Step 3: Install Github App on Rubicon

All your transformations are written in the form of SQL statements in your Github repository. Follow these instructions to connect Rubicon App to your Github repo - This allows us to start new version of pipelines automatically when you modify your transformations.

Step 4: Wait for the initial sync to complete

If your source is an OLTP database, Rubicon has to do an initial sync before the data is available in Rubicon to build pipelines. The initial sync can take a few hours depending on the size of your data source. You will be notified by email when your sources are synced and ready to be used in your first transformation. You can find instructions here on how to write your first transformation here using the demo source and destination already available to you while you wait for the data to sync.

Step 5: Write your transformation

On the Models tab, write your transformation as a SQL statement and submit it. See the documentation here for details on how to write transformation.

Step 6: Build a pipeline

Go to https://app.rubicon.io/ and select the Git branch you want to deploy the pipeline in on the top right-hand side. The main branch is considered production and supports continuous transformation. Other environments are considered development and support running transformation on a bounded amount of data.

Step 7: Invite users (optional)

Click on your icon at the right and invite users to the platform. They will be able to join your team