5 Reasons Snowflake Isn’t Just Another Data Warehouse
Many IT Architects have approached the deployment and provisioning of data warehouse solutions the same way for decades. Analytic and insight users wanted data, and data was what they got. ETL processes were constructed and executed. Databases were populated with complex joins and structures that allow users to access data in various ways. Servers were dedicated to hosting the data warehouse environment.
This approach was “successful” if data availability was the only measure of success. But for many organizations, the exceptions to this approach (non-structured data, spikes in usage, etc.) created additional time, effort, and budget. Data warehouse operations and infrastructure became an IT organization within the IT organization, adding cost and bureaucracy.
To address these concerns, a new player entered the game; Snowflake.
The label “data-warehouse-as-a-service” was hung on the company as it approached its historic IPO in 2020. While the moniker does align with the outcome of using the platform, it fails to spotlight the many strengths and benefits of the platform.
Snowflake is a cloud-based system that removes the need for organizations to provision and maintain infrastructure (hardware and software) for their insight and analytical needs. This distinction also allows Snowflake to bring to the market several unique strengths.
Separate Storage and Compute
Snowflake distinguishes between storing data and the processes that utilize the data. This separation enables the platform to divide costs across those two concepts. An organization pays for storage in the data cloud and a separate fee for processing (compute).
Scalability
The ability to increase and decrease the capacity and usage of the Snowflake system provides excellent processing scalability. No longer are IT and data warehouse architects responsible for building and maintaining environments based on maximum usage scenarios; the Snowflake system allows for independent usage configuration. One user may have complex search algorithm needs, and another may have simple query needs. Both are serviced by the Snowflake environment but at different sizes and fees. It’s a unique consumption-based model that breaks from the one-size for all data warehouse environments of the past.
Performance
The resources to power complex process queries or data load processes are dynamic. An expensive powerhouse server sitting idle for most of the time is no longer a reality. Compute resources are a function of the cloud deployment, and the ability to add horsepower when needed alleviates many of the complaints of long-running processes associated with traditional large data warehouse initiatives. If more processing capacity is necessary, Snowflake can automatically adjust.
Data Format Flexibility
Snowflake’s ability to work with structured, semi-structured, and unstructured data (a recently added capability) is very beneficial for innovation. Data consumers have a single source for all data, which helps them optimize their work and eliminate silos and handle multiple data sources. Big Compass used this capability in a recent project with one of our logistics industry customers. You can read about the story here.
Beyond Traditional ETL Processes
Snowflake provides a variety of ways to build data warehouses and data lakes, and populating these is now straightforward and clear. Snowflake uses traditional bulk and augmented data load capability from existing data sources (Internal Stages) like other data warehouse platforms. The powerful innovation the platform delivers is its ability to consume cloud system data (referred to as External Stages in Snowflake) in multiple ways: bulk loading, continuous auto ingestion, and direct query. Big Compass leveraged the auto ingestion processing feature for a large data warehouse project. This streamlines the data load capabilities and goes well beyond the ETL processes used in the past.
Conclusion
Big Compass relies on Snowflake to deliver on the promise of “source to value .” This demands consistent, reliable, affordable, and proven data movement and aggregation so data can be used for insights and operations. Snowflake also aligns with our focus on data agility (the ability to move and combine data for insights and automation) and provides the opportunity to move beyond heavily SQL and technical expertise to build and leverage an organizations’ data.