Moving from dbt Core to dbt Cloud streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed software service.
Explore our 3-part-guide series on moving from dbt Core to dbt Cloud. This series is ideal for users aiming for streamlined workflows and enhanced analytics:
If your team is using dbt Core today, you could be reading this guide because:
You’ve realized the burden of maintaining that deployment.
The person who set it up has since left.
You’re interested in what dbt Cloud could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices.
Moving from dbt Core to dbt Cloud simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With dbt Cloud, you gain access to features like cross-team collaboration (dbt Mesh), version management, streamlined CI/CD, dbt Explorer for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently.
It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity.
What is dbt Cloud and dbt Core?
dbt Cloud is the fastest and most reliable way to deploy dbt. It enables you to develop, test, deploy, and explore data products using a single, fully managed service. It also supports:
dbt Core is an open-source tool that enables data teams to define and execute data transformations in a cloud data warehouse following analytics engineering best practices. While this can work well for ‘single players’ and small technical teams, all development happens on a command-line interface, and production deployments must be self-hosted and maintained. This requires significant, costly work that adds up over time to maintain and scale.
Your existing dbt project source code should live in a Git repository. In this section, you will connect your existing dbt project source code from Git to dbt Cloud.
(Recommended) Connect with one of the native integrations in dbt Cloud (such as GitHub, GitLab, and Azure DevOps).
This method is preferred for its simplicity, security features (including secure OAuth logins and automated workflows like CI builds on pull requests), and overall ease of use.
Explore these additional configurations to optimize your Git setup further:
Log into dbt Cloud using OAuth connections to integrate with your source code platform. It automatically links to the repository using one of the native integrations set at the account level. enterprise
Set up groups for dbt project access with those configured for repository access to streamline permissions.
The most common data environments are production, staging, and development. The way dbt Core manages environments is through target, which are different sets of connection details.
Integrating with features such as job scheduling or version control, making it easier to manage the full lifecycle of your dbt projects within a single platform.
Streamlining the process of switching between development, staging, and production contexts.
Making it easy to configure environments through the dbt Cloud UI instead of manually editing the profiles.yml file. You can also set up or customize target names in dbt Cloud.
Adding profiles.yml attributes to dbt Cloud environment settings with Extended Attributes.
Using Git repo caching to protect you from third-party outages, Git auth failures, and more. enterprise
Set up development environment — Set up your development environment and development credentials. You’ll need this to access your dbt project and start developing.
dbt Core version — In your dbt Cloud environment and credentials, use the same dbt Core version you use locally. You can run dbt --version in the command line to find out which version of dbt Core you’re using.
When using dbt Core, you need to think about which version you’re using and manage your own upgrades. When using dbt Cloud, leverage "Versionless" so you don’t have to.
Each environment is roughly equivalent to an entry in your profiles.yml file. This means you don't need a profiles.yml file in your project.
Development tools — Set up your development workspace with the dbt Cloud CLI (command line interface or code editor) or dbt Cloud IDE (browser-based) to build, test, run, and version control your dbt code in your tool of choice.
If you've previously installed dbt Core, the dbt Cloud CLI installation doc has more information on how to install the dbt Cloud CLI, create aliases, or uninstall dbt Core for a smooth transition.
Explore these additional configurations to optimize your developer setup further:
Custom target names — Using custom target.names in your dbt projects helps identify different environments (like development, staging, and production). While you can specify the custom target.name values in your developer credentials or orchestration setup, we recommend using environment variables as the preferred method. They offer a clearer way to handle different environments and are better supported by dbt's partial parsing feature, unlike using {{ target }} logic which is meant for defining the data warehouse connection.
Review the dbt commands supported for dbt Cloud development. For example, dbt init isn’t needed in dbt Cloud as you can create a new project directly in dbt Cloud.
dbt Cloud environment variables must be prefixed with DBT_ (including DBT_ENV_CUSTOM_ENV_ or DBT_ENV_SECRET).
If your dbt Core environment variables don’t follow this naming convention, perform a “find and replace” in your project to make sure all references to these environment variables contain the proper naming conventions.
dbt Cloud secures environment variables that enable more flexible configuration of data warehouse connections or git provider integrations, offering additional measures for sensitive values, such as prefixing keys with DBT_ENV_SECRETto obscure them in logs and the UI.
Setting project level and environment level values
dbt Cloud environment variables order of precedence
Environment variables in dbt Cloud are managed with a clear order of precedence, allowing users to define values at four levels (highest to lowest order of precedence):
The job level (job override) or in the IDE for an individual developer (personal override). Highest precedence
The environment level, which can be overridden by the job level or personal override.
A project-wide default value, which can be overridden by the environment level, job level, or personal override.
The optional default argument supplied to the env_var Jinja function in the code. Lowest precedence
This section outlines the considerations and methods to set up your dbt Cloud environments and jobs for orchestration. The following categories are covered in this section:
To use the dbt Cloud's job scheduler, set up one environment as the production environment. This is the deployment environment. You can set up multiple environments for different stages of your deployment pipeline, such as development, staging/QA, and production.
dbt Core version — In your environment settings, configure dbt Cloud with the same dbt Core version.
Once your full migration is complete, we recommend upgrading your environments to "Versionless" to always get the latest features and more. You only need to do this once.
Configure your jobs — Create jobs for scheduled or event-driven dbt jobs. You can use cron execution, manual, pull requests, or trigger on the completion of another job.
Note that alongside jobs in dbt Cloud, discover other ways to schedule and run your dbt jobs with the help of other tools. Refer to Integrate with other tools for more information.
Explore these additional configurations to optimize your dbt Cloud orchestration setup further:
Custom target names — Use environment variables to set a custom target.name for every corresponding dbt Cloud job at the environment level.
dbt commands — Add any relevant dbt commands to execute your dbt Cloud jobs runs.
Notifications — Set up notifications by configuring email and Slack alerts to monitor your jobs.
Monitoring tools — Use monitoring tools like run history, job retries, job chaining, dashboard status tiles, and more for a seamless orchestration experience.
dbt Explorer — If you use dbt Explorer and run production jobs with an external orchestrator, ensure your production jobs run dbt run or dbt build to update and view models and their metadata in dbt Explorer. Running dbt compile alone will not update model metadata. In addition, features like column-level lineage also requires catalog metadata produced through running dbt docs generate. teamenterprise
Building a custom solution to efficiently check code upon pull requests is complicated. With dbt Cloud, you can enable continuous integration / continuous deployment (CI/CD) and configure dbt Cloud to run your dbt projects in a temporary schema when new commits are pushed to open pull requests.
Workflow of continuous integration in dbt Cloud
This build-on-PR functionality is a great way to catch bugs before deploying to production, and an essential tool for data practitioners.
Set up an integration with a native Git application (such as Azure DevOps, GitHub, GitLab) and a CI environment in dbt Cloud.
Create a CI/CD job to automate quality checks before code is deployed to production.
Run your jobs in a production environment to fully implement CI/CD. Future pull requests will also leverage the last production runs to compare against.
In this section, you’ll be able to validate whether your models run or compile correctly in your development tool of choice: The dbt Cloud IDE or dbt Cloud CLI.
In your development tool of choice, you can review your dbt project, ensure it's set up correctly, and run some dbt commands:
Run dbt compile to make sure your project compiles correctly.
Run a few models in the dbt Cloud IDE or dbt Cloud CLI to ensure you’re experiencing accurate results in development.
Once your first job has successfully run in your production environment, use dbt Explorer to view your project's resources (such as models, tests, and metrics) and their data lineageData lineage provides a holistic view of how data moves through an organization, where it’s transformed and consumed. to gain a better understanding of its latest production state. teamenterprise
If your team is using dbt Core today, you could be reading this guide because:
You’ve realized the burden of maintaining that deployment.
The person who set it up has since left.
You’re interested in what dbt Cloud could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices.
Moving from dbt Core to dbt Cloud simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With dbt Cloud, you gain access to features like cross-team collaboration (dbt Mesh), version management, streamlined CI/CD, dbt Explorer for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently.
It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity.