Core Concepts
Last updated
Last updated
All data in Rubicon is treated as tables and schemas. Rubicon does not require you to define the schema and instead infers the schema from the source itself. When you connect a source to Rubicon, it starts reading the data in a streaming fashion and makes it available to process on Rubicon platform. You can then write transformations in SQL and build a pipeline that maintains a materialization of the transformation into a destination of your choice.
Sources are databases or event streams that you want to use for building real-time streaming pipelines. Rubicon supports a variety of sources including all major OLTP databases, event platforms. The data from sources is represented as tables in Rubicon and available in a single namespace. So, you can join data from Postgres with data in Kafka. You can see the full list .
Destination is the system you want the materialized view of your pipeline output to be saved. This can be a data warehouse, a SQL database like MySQL or specialized database like ElasticSearch to power your search. You can see the full list of supported destinations .
Transformations are defined in SQL in files referred to as models. The SQL statement can refer tables from a connected source or other models enabling you to build complex pipelines without defining an explicit DAG. You can learn more about transformation .
A pipeline runs a transformation continuosly and streams the results into a materialized view of a destination. You can learn more about how to setup pipelines .