MarketWise Reduces Costs and Ops Burden While Accelerating Insights with Snowflake Hybrid Tables

With easy access to markets and a growing desire to control their own financial future, people are turning to self-directed, or do-it-yourself, investing more these days. But making sense of financial markets can sometimes be overwhelming to the average person — whether it’s a retiree looking to maximize their savings or a Gen Zer just getting into stocks. More and more, these self-directed investors are looking for expert insights to help.

MarketWise — a holding company that includes a network of affiliate brands, including Stansberry Research and TradeSmith — aims to give people the tools to make smart investment decisions. Offering newsletters, research reports and software, the company is built on democratizing access to financial market analysis and insights, putting more data and information into the hands of millions of self-directed investors. 

With each brand operating independently, however, MarketWise had several specific needs when it came to building its ideal data foundation — most notably, speed of delivery and data security and governance control. The company had been using MySQL instances on Amazon Redshift for business intelligence data but found it overly complex and limiting in its ability to scale. When Chief Architect Ron Stiffler joined MarketWise in 2018, he assessed the system. Given his own experiences working with both Redshift and Snowflake, he recalls that the decision was clear: “We really needed to switch this over to Snowflake.” 

MarketWise chose to migrate from Amazon Redshift to Snowflake to create a unified data platform that can separate compute from storage and provide some nuance to data access. Given MarketWise’s structure — Ron Stiffler’s MarketWise Solutions team provides the basic technology infrastructure and support to each of the subsidiary brands — managing access to data would be vital. “Snowflake’s role-based access controls are critical for how we do things because we have such a separation between our end customers,” Stiffler says. 

With granular oversight of the entire company’s information, MarketWise has confidence that affiliates are only able to access their own particular data. And each brand, through its own Snowflake environment, can work on subsets, separate from MarketWise or the other sister brands. These individual data teams can be working on brand-specific BI or developing better products, like pipelines that deliver timelier market insights for their subscribers. “We’re all about data democratization,” Stiffler says. “So if data teams can do it themselves, we’re happy to show them how.”

That kind of self-service ethos is made possible because of Snowflake’s ease of use. Dynamic Tables, for one, help simplify the team’s reporting process, thanks to their adaptability to new data and changing requirements. “It has been a game-changer for us,” Stiffler says, noting how easy it is to simply create a semantic layer between reporting data and other systems, like Salesforce. “Instead of creating and maintaining a bunch of SQL scripts that need to be run by some external system, or a bunch of tasks that somebody else has to monitor, Dynamic Tables makes it a lot easier to get ahead of things.”

For instance, MarketWise works with various payment processors that have to handle different transactions and inquiries. To get that data more quickly, MarketWise can use Snowpipe Streaming integrations with AWS to ingest it directly in Snowflake. The feature essentially removes half the time and infrastructure MarketWise had been using and ultimately helps business teams access insights faster without having to interact directly with underlying data structures. Once the data lands in Snowflake, MarketWise uses Dynamic Tables to do incremental streaming transformation declaratively.

With full cutover to the Snowflake AI Data Cloud expected before 2025, East Health Trust PHO is looking aStiffler’s team supports multiple divisions across MarketWise, so it needs to easily track the workflow state of numerous data engineering pipelines in real time, especially when it comes to executing extract, transform and load (ETL) jobs. Finding that standard tables did not work optimally on certain small tables that required near-constant writing, MarketWise tried using MySQL — to mixed results — and then tried DynamoDB to keep jobs from stepping over one another. But that solution was essentially a workaround, requiring more overhead and infrastructure to maintain a separate database. It also lacked seamless functionality because of the constant data movement it demanded, and it limited the team’s ability to scale with the business. 

Snowflake Hybrid Tables, however, centralized the data and didn’t require moving it unnecessarily outside of Snowflake. This helped reduce MarketWise’s architectural footprint, while the ability to handle large volumes of writes — and look up or update individual rows quickly — also substantially improved the team’s reporting capabilities. If an API is down and leads to stale data for the business, Hybrid Tables now gives Stiffler’s team the granular context to triage immediately — which means it’s easier and faster for cross-functional stakeholders to get what they need, too. to enjoying the benefits and efficiencies gained from reducing the time to complete some data processing jobs that took most of a day to an hour. 

Using Hybrid Tables to track the current state of import and export ETL jobs has led to many practical business benefits. With a simple API call, the company can now get an accurate audit log of its data movement — what data is leaving the system and when, where it’s going — a time-consuming process that had previously been handled manually. For example, data ingestion from billing systems occurs every 10 minutes, and MarketWise now uses Hybrid Tables to track that in real time. So if a customer signs up for a subscription, MarketWise can be confident that it’s using fresh, accurate ingested data, which helps ensure downstream that the customer receives a registration email and will receive future subscription charges or payment reminders.

MarketWise also uses Hybrid Tables to track — and ensure accuracy — for all of the data it’s exporting to Salesforce. If one of MarketWise’s brands, which rely on Salesforce Marketing Cloud, sends 1,000 emails over an hour for a specific marketing campaign and suddenly that number drops to 10, the data team can immediately see that there’s an issue and have granular context to identify and address it.

Removing these complexities and superfluous builds also frees up engineering resources, which Stiffler is eager to apply toward emerging technologies like generative AI. The team, for instance, is exploring ways to analyze how articles or newsletter topics are resonating with readers through text-based LLMs — figuring out how marketing copy on, say, cryptocurrency has performed historically. And Stiffler himself has been experimenting with Cortex AI, using it to summarize large bodies of raw text down to its relevant pieces. 

The predictive analytics team is also looking to build various search apps that could help efficiently go through massive data sets, like customer emails, call transcripts or SEC filings, and present information in a user-friendly format using a Streamlit app. “I hate shipping data around,” Stiffler says. “So, if I can add sentiment scores to my data or documents and not have to export it over here, run it there, then push it back over, it makes things faster and cleaner. I love bringing the compute down to the data.” 

With Snowflake’s security and governance, matched with the growing number of AI and ML tools that can be applied in-platform, MarketWise now has the freedom to explore these new innovations with confidence. 

“We're always trying something new,” Stiffler says, “And Snowflake helps us innovate with speed and confidence.”