Conquering DevOps for Data

Challenges Involved in DevOps for Large-Scale Data Applications

Developers who are building data applications and want to incorporate DevOps workflows face five common difficulties. These challenges are related to the data integration and database change management necessary as part of developing large-scale data applications.

Creating isolated environments

Standing up isolated, ACID-compliant, SQL-based compute environments typically requires procuring, creating, and managing separate cloud resources for each stage of the pipeline. Developers must be able to create isolated environments quickly.

Schema changes

New features often require schema changes, which can be expensive operationally since they require developers to coordinate their code changes with updates to the database schema. Developers need a way to reduce schema changes to make the process more economical.

Seeding preproduction environments with production data

Using test data to validate feature changes is far from ideal. Even if the schema is the same between production and preproduction, production data is still necessary to properly validate code changes. With traditional databases, seeding preproduction environments with production data is so costly and time-consuming that it can result in schedule delays or compromised product quality.

Scaling preproduction environments

Historically, preproduction environments have been smaller in scale due to the costs associated with procuring, standing up, and managing production-scale environments. This translated into slow development cycles. However, it’s much more affordable to scale with today’s cloud data solutions.

Handling errors with the automated release of a database and related application code

With traditional systems, developers have to make backups before releasing new changes and then restore data in the event of a failure. But both backup and restore operations are very time-consuming and costly. The larger the amount of data, the more expensive and time-consuming these operations are. Speedy error handling and rollbacks are essential for efficiency.

How the Snowflake Data Cloud Simplifies DevOps for Large-Scale Data Applications

In fact, each of these challenges can be solved with modern cloud solutions, particularly with the Snowflake Data Cloud. Let’s examine exactly how.

Instantly create any number of isolated environments

With Snowflake, developers can stand up as many isolated, ACID-compliant, SQL-based compute environments as needed, without the traditional process of procuring, creating, and managing separate cloud resources for each stage of the pipeline. Using standard SQL queries, it’s possible to create new environments in seconds as well as auto-suspend idle environments and delete environments when testing is complete.

Reduce schema change frequency with VARIANT data type

Snowflake supports JSON and other semi-structured data natively using a proprietary VARIANT data type. There’s no need to define the schema ahead of time. Once loaded, semi-structured data can be queried using SQL or even joined with other structured data. Because it’s unnecessary to modify and retest code every time new parameters are added, data pipelines are simplified. Additionally, rather than add new columns and change schema frequently, many developers choose to store their data in JSON and apply a schema at query time (Schema-on-Read). Using VARIANT columns in this way significantly reduces the DevOps operational burden.

Rapidly seed preproduction environments with production data

Snowflake offers two ways to seed a preproduction environment with production data. Secure Data Sharing is used when the environments are on separate Snowflake accounts, and zero-copy cloning is used when the environments are on the same account. Secure Data Sharing enables access to live data from a provider account to one or many consumer accounts and is typically used to share data with partners or with other departments. Zero-copy cloning creates a copy of live data instantly in metadata, without the need to duplicate or move data, saving storage costs and time.

Instantly scale environments to run jobs quickly and cost-effectively

Scaling issues are easily overcome with Snowflake’s per-second pricing structure. Customers pay only for the time needed to run the job, no matter the cluster size, so DevOps teams can scale the production environment to run a big process in a fraction of the time and scale it down when the process ends.

Easily roll back with Snowflake Time Travel

To address the challenges that come with automatic updates, Snowflake offers Time Travel. This capability simplifies the effort required to handle errors and rollbacks in a CI/CD process. With Time Travel enabled, objects such as tables, schemas, and databases that have been changed or deleted can be easily restored or accessed programmatically at a point in time within the previous 90 days without having to manage and maintain costly backups.

Beyond these features, the Snowflake Data Cloud makes DevOps simpler by enabling the following:

Use your preferred programming language: Snowflake supports ANSI SQL, Python, Node.js, Go, .NET, Java, and other popular languages, so developers can code in their preferred language and leverage their existing tools.
Near-zero maintenance: Because the Snowflake Data Cloud is delivered as a service, it automates many of the tasks that traditionally burden DevOps teams. There’s no need to manage infrastructure, install patches, or perform backups. Snowflake rolls out software updates automatically so all environments are always running on the latest version. And high availability and security are built in.
Real-time integration with external services: Using Snowflake’s External Functions feature, developers can call third-party or custom services that are stored and executed outside of Snowflake. The External Functions feature helps simplify DevOps processes by reducing the number of tools and steps.

Snowflake for DevOps

Snowflake enables developers to build data-intensive applications with no limitations on performance, concurrency, or scale. Thanks to its multi-cluster, shared data architecture, it scales horizontally and vertically on demand, delivering fast response times regardless of load. And because it is delivered as a service, Snowflake improves developer productivity without the need to maintain infrastructure.

To see how Snowflake can support your DevOps workflows, sign up for a free trial.