OM1 Simplifies Its Environment and Saves Time and Money by Switching to Snowpark

OM1’s mission is of monumental magnitude: Transform healthcare by elevating how industry, providers, payers and patients obtain and leverage real-world insights by incorporating cutting-edge technology with real-world data to improve the advancement of research and patient outcomes.

Partnering with major research universities and hospital systems, pharmaceutical companies and insurers, OM1 collects and analyzes massive stores of patient data — to the tune of billions of rows of data that cover more than 350 million people. In addition to offering these rich data sets to clinical researchers looking to cure the world’s most pernicious diseases, the company also builds AI products that help better understand, compare and predict patient outcomes for chronic conditions that affect millions of Americans every day. 

There are no higher stakes than the health of millions of people. The sheer size and complexity of OM1’s data sets require a reliable, scalable platform to meet the industry’s rigorous demands for privacy, accuracy and credibility. But OM1 faced a variety of challenges with its previous data processing environment. 

“We were on a mission to simplify our entire ecosystem,” OM1 CTO Eric Schrock says. “While we had moved our data to Snowflake, we had a complex ecosystem of compute spread across AWS, Databricks and custom software. While this system worked, it came with fairly high cost and overhead. From a business perspective, it’s all about efficiency. We want our data engineers to spend their time innovating and solving hard problems, not maintaining platforms.” 

To simplify its ecosystem, OM1 moved its data processing from its previous platform to Snowflake — unleashing greater efficiency, cost savings and performance in the process.

Sitting at the forefront of medical breakthroughs and personalized healthcare, OM1 is built on complex data. “Finding ways to understand and encode what’s happening in the real-world patient’s journey is not straightforward,” Schrock says.

Take, for example, a patient suffering from psoriasis. Billing and coding data can convey the diagnosis, but it doesn’t account for factors like location, severity or condition over time. Yet those details can be illuminating for doctors or researchers. Instead, that data is often buried in clinicians’ notes. OM1 unlocks such unstructured data, taking singular health information and making it valuable to a greater medical community.

But as OM1’s capabilities grew, so did the complexities associated with its data processing systems. After migrating its data storage to Snowflake back in 2017, OM1 moved to a managed Spark-based model for processing. But the team found itself constantly navigating reliability issues. As OM1 Senior Software Engineer for Machine Learning Ryan Boch explains, “When we were processing data with our previous platform, we had challenges with stability, job scaling and costly maintenance.” 

Deploying models was a complex and disjointed process that involved setting up pipelines that manage Spark clusters, run external notebooks and execute dbt models. This was not an easy task, even for experienced engineers. Executing and testing the pipelines involved a manual process that required Spark cluster management, large data transfers and executing code in different environments.

Snowpark and Snowflake’s robust ecosystem of partners now provide a simpler, faster and more reliable experience for OM1’s engineers. A main driver of this improved experience is Modelbit, an ML engineering platform that simplifies model deployment into production environments. With its seamless integration with Snowpark — which also supports Hex, the tool OM1 uses to author Python code — Modelbit cut out many of the steps that previously bogged down productivity. “We needed a whole framework to deploy models previously,” Vice President of Product Engineering Neil Davies says. “Whereas with Modelbit and Snowpark, it’s just so much easier.”

The savings have been undeniable. “SQL is the lingua franca for our data analysts, clinical informaticists, data scientists and statisticians. Keeping it simple is essential,” Shrock says. By eliminating this complex, costly environment, moving to Snowflake has unlocked OM1’s capacity for innovation.

The effects of moving more compute to Snowflake hasn’t just streamlined OM1’s more complex ML operations; it’s made a sizable impact throughout the organization — down to its most basic and fundamental operations. 

Given the sensitive nature of patient data, successful and efficient de-identification is core to OM1’s business. The company uses a “scrubber” to redact personal health information (PHI) from textual notes, which allows it to grant broader access rights to that data without fear of violating HIPAA and other privacy regulations. But previously, it required jumping to an external Spark ecosystem, one that required exfiltrating data out of Snowflake and being proficient in a different programming language. 

However, by using Snowpark Container Services, which allows containerized software to be deployed directly in its Snowflake environment, OM1 found that it could seamlessly apply an industry-leading, anonymizing large language model into its dbt pipelines. What used to be a complex Rube Goldberg machine of notebooks, clusters and custom code became a single SQL call directly within the dbt models.

The reception within OM1 “has been unanimously positive,” Associate Director of Engineering Aaron Williams says. “Behind the scenes, Snowpark Container Services handles a lot of the scaling and complexity and everybody who has switched to the solution has been happy.” Not only has it simplified the experience, it has enhanced overall performance and reliability, including seamless adoption of GPU instances for even greater performance. OM1 can now scrub more than 100 million records in less than half an hour — while saving 75% in costs.

Envisioning the wide-ranging future applications of innovations like these isn’t difficult. The future of medicine is personalized insights based on each patient’s unique journey, and AI is the means to get there. OM1 PhenOM®, for example, is an AI-powered digital phenotyping platform. This innovation takes complex signals and interactions shared by patients with similar conditions, characteristics or outcomes, and synthesizes them into unique “fingerprints” that help highlight patients that might be at risk or may benefit from a particular treatment.

The ability to easily package AI/ML models through Modelbit and Snowpark is opening new opportunities for PhenOM. Now, data scientists can iterate quickly and generate models that can be easily invoked from SQL or APIs, which opens the possibility of delivering models as a service, either as an API or a Snowflake Native App. 

These developments all result in better products for OM1, and the trickle-down effects are, quite literally, life-changing. By providing stronger, more reliable and accurate data to their customers — the healthcare providers, the medical researchers — OM1 can help them treat patients more effectively, speed closer to scientific breakthroughs and ultimately, save lives.