Simplified End-to-End Development for Production-Ready Data Pipelines, Applications, and ML Models

In today’s world, innovation doesn’t happen in a vacuum; collaboration can help technological breakthroughs happen faster. The rise of AI, for example, will depend on the collaboration between data and development. We’re increasingly seeing software engineering workloads that are deeply intertwined with a strong data foundation. 

Whether you’re part of a global data team or a solo developer, Snowflake’s AI Data Cloud is a single platform that helps you run development tasks (building apps, pipelines, ML models) right alongside your data. This means smoother production and better results, all on one platform. We’re making it even easier and better by letting you use your favorite tools (like pandas and Notebooks) and eliminating technical barriers that slow you down. 

Snowflake offers a secure, streamlined approach to developing across data workloads, reducing costs and reliance on external tools. This means faster development and happier data teams. Let’s dive deeper into what we announced. 

Snowflake now offers data teams a suite of intuitive tools designed to simplify development and accelerate workflows. This suite extends seamlessly across Snowflake's offerings, including Snowpark, Native Apps, Streamlit and more, for building anything with your data. Whether you're a data engineer, data scientist or app developer, Snowflake prioritizes a user-friendly experience while maintaining robust functionality. 

See how we’ve streamlined development across our platform with the following features: 

• Snowflake Notebooks, in public preview (PuPr), lets data teams seamlessly combine Python, SQL and Markdown in an interactive workspace. Explore and experiment with data, visualize results, share insights — all in one place. Streamline workflows and accelerate data journeys from exploration to production.
• The Snowflake Command Line Interface (CLI) has long been a developer favorite, boosting productivity and enabling CI/CD automation for seamless DevOps workflows. Our newly improved Snowflake CLI (general availability soon) gets a major upgrade. Revamped commands hide complex SQL for a smoother experience. Build, deploy and run workloads across Native Apps, Snowpark, Snowpark Container Services and Streamlit with ease. 
• Snowflake's new Python API (GA soon) simplifies data pipelines and is readily available through pip install snowflake. Interact with Snowflake objects directly in Python. Automate or code, the choice is yours. This suite of APIs supports Tasks/DAG, Snowpark Container Services, Tables, Warehouse, Schema and Databases. 

Snowflake is also excited to announce Snowflake Tasks, Dynamic Tables and Database Change Management (DCM), powerful new features designed to streamline your development workflow with declarative best practices and automated orchestration. 

We’ve improved Snowflake Tasks for better pipeline orchestration and job scheduling. Use Serverless Tasks for Python (private preview) to simplify orchestration with Snowflake managed compute for Snowpark Python. Serverless Tasks Flex (PrPr) offers flexible cost optimization by up to 42%.  For faster workflows, choose event-driven Triggered Tasks (PuPr) with 10-second latency or Low Latency Tasks (PrPr) for reduced task-scheduling intervals, down to 15 seconds. Finally, Tasks Backfill (PrPr) automates historical data processing within Task Graphs.

Additionally, Dynamic Tables are a new table type that you can use at every stage of your processing pipeline. Whether you’re processing batch data that needs to be refreshed daily or near real-time data that needs to be processed in minutes, Dynamic Tables allow you to create data pipelines that are easy to build, operate and evolve. 

More than 2,900 customers are using Dynamic Tables today with approximately 200,000 Dynamic Tables actively running (as of April 2024). We are also expanding the use of Dynamic Tables by supporting lower latency (<10 seconds) for streaming data processing, soon in private preview (PrPr), and adding support for Snowflake managed Iceberg tables (PuPr).

Follow this quickstart to test-drive Dynamic Tables yourself. 

To simplify your delivery lifecycle, Database Change Management (DCM) to CREATE or ALTER (PuPr) makes it easy to declaratively manage changes across Snowflake objects at scale, directly from your Git repo (or stage). Store your DDL scripts in source control, and Snowflake will automatically apply the necessary changes (create, alter, execute) to maintain consistency across objects. This eliminates the need for complex third-party tools and cumbersome state files, freeing you to focus on building high-quality, production-grade data applications.

You don’t work in isolation, and building great data products means collaborating across different teams and functions, all while ensuring your workloads have a scalable infrastructure. To make this easy, we’re bringing you Git integration (PuPr) for seamless collaboration, access to Snowpark pandas for scalable data processing and a simplified CI/CD workflow. 

Snowflake integrates with GitHub, GitLab, Azure DevOps and Bitbucket. Sync your code and manage SQL scripts, Snowpark functions and more within Snowflake. In this initial public preview, you can access, read and write files within your Git repository. For writing and collaborating on changes, you can continue using tools that integrate with Git, such as VS Code.

Many data teams running pandas locally are met with slow, single-threaded execution and painful out-of-memory errors on production use cases. Snowflake has continuously expanded the capabilities of Snowpark to enable Python users to build scalable data pipelines, ML models, apps and more, using Snowflake’s elastic, multilingual processing engine. Snowpark now adds distributed pandas support with the launch of Snowpark pandas API into public preview. With just the change of an import statement, developers can use the same pandas syntax they know and love while benefiting from Snowflake’s performance, scale and governance. 

Simplify CI/CD setup for your projects by connecting your Snowflake Notebooks with your Git repository. This will allow you to trigger the Snowflake CLI GitHub Action, automatically executing CLI commands to orchestrate your CI/CD pipeline. View run history results for faster debugging. See it in action on our public repo.

Additionally, Notebooks’ scheduler helps automate repetitive tasks, test-runs, data processing, model training and report generation, freeing your team to focus on higher-level analysis.

We are excited to introduce Snowflake Trail, a rich set of curated observability capabilities that provide enhanced visibility into data quality, pipelines and applications, empowering developers to monitor, troubleshoot and optimize their workflows with ease. 

Snowflake Trail expands the built-in observability features for Tasks and Dynamic Tables to Snowpark and Snowpark Container Services, giving developers visibility into their Snowpark code and resource usage, so they can quickly diagnose and debug apps and pipeline development. With log data right inside of Snowflake, requiring no additional data transfer or complex setup, developers can easily explore and manage their pipelines and apps using metrics, logs and distributed tracing, and even send targeted error messages to alerts and notification tools. Best of all, Snowflake Trail is built with OpenTelemetry standards, schema and open ecosystem integrations in mind. Snowflake telemetry and notification capabilities are integrated with some of the most-favored developer tools, including Datadog, Grafana, Metaplane, PagerDuty and Slack. Alternatively, users can simply use Snowsight (for supported use cases), where developers can monitor and trace directly within Snowflake.

Behind Snowflake’s commitment to innovation and accessibility, we are excited to bring these tools to help empower you and your organization to a more streamlined path to production. We invite you to learn more about each of these features through their corresponding documentation.