In two earlier posts we discussed the Pros and Cons The Benefits Of 3rd Party Restaurant Data Warehousing and in another To SaaS or Not? That's the Question, we outlined where possible cost differences between Do it Yourself (DIY) and 3rd Party Software as a Service (SaaS) solutions. While I didn’t start with the thought in mind to create a trilogy on DIY I guess I just felt compelled to address the (cost difference) elephant in the room. Therefore, in what could now be considered post # 3 on this topic we’ll add the dollars and cents to see just how much a difference exists between DIY and SaaS solutions.
Getting What You Pay For
The biggest differences when making this comparison are going to be found in the up-front and ongoing costs. DIY solutions will have higher up-front costs and one could argue they then have relatively low ongoing maintenance fees however, this doesn’t take into account all of the costs associated with staff needed to manage and maintain the system nor does it consider any changes needed as the business evolves. Furthermore, it ignores the real possibility that a self constructed data warehouse has a much higher probability of failing to meet the needs of the business (something that isn’t discovered until you’ve invested all of the time, effort and money to build out your new solution), just when those lower maintenance fees were supposed to kick in. I guess this is where you could insert the phrase – you get what you pay for… but more on this later.
A scant 5 years ago if you were considering a DIY data warehouse solution for your restaurant business you’d be faced with challenges associated with purchasing hardware, software, network infrastructure and then building out the space to house your new system or finding a suitable co-location facility. Today, you can turn to Amazon Redshift or Google cloud based storage to remedy these hardware and networking needs. Ironically, in going to Amazon/Google you are buying into the benefits of a SaaS subscription versus buying the assets.
However, while these services reduce your overhead by providing you with the resources to house your data they don’t address the resources you’ll need (consultants or specialized IT team members) to create the data model (schema), or to get your data loaded and organized so you / your team can begin using it.
Road Map For Success
The design and analytical efforts begin with crafting your data model which will serve as your road map for integrating the different aspects of your operational environment. Most importantly, a correctly crafted data model is needed if you’re going to use your data to answer questions about your business. Failure to correctly craft the data model will, unfortunately, only be seen when answers to questions are unable to be rendered (often times this doesn’t show up until well after the DW has been in operation). The only fix for this is to go back and start over – more on this in a minute but first here’s a snap shot of some required skills needed to build you new data warehouse - IT Knowledge Required.
Creating a DIY data warehouse requires you to have IT team members with deep skills in the following areas:
- Creation of a restaurant centric data model (Schema)
- Data warehouse design
- Business intelligence (BI) requirements
- Database architecture, design, development, and administration
- Extract, transfer and load (ETL) process
OK, so you can rent some storage space in the cloud and the company you rent from will be providing the hardware, software, networking services, infrastructure to house everything as well as support the systems and environment they’re maintained in, so what’s the big deal – the big deal is, if you don’t know a Snowflake from a Star (types of Schema used to create a dimensional data warehouse) you’re going to need specialized help (consultant, new hire or SaaS provider – the choice is yours). Let’s first look at a few of the items you’ll need from the IT Knowledge list above:
How do you get data into a warehouse?
To build a data warehouse, you first need to copy the raw data from each of your data sources, cleanse, and optimize it. The process of getting data into a data warehouse is called ETL: Extract, Transform, Load. Often times these processes are dismissed by IT as being a simple data extraction solution. They are far from it!
1. Extract the data from the source system.
Ultimately have to acquire or create a middleware solution to extract and transport the data you want to load into your data warehouse. In this process, one of the challenges that you’ll need to avoid is copying all the data every time you sync. Some of your data sources could have hundreds of thousands of records, and copying every record during every sync isn’t a good idea. This optimization is when things really get complicated. Once your extraction and transport solution is complete you’ll then have to create the programs to transform and load the data as well.
2. Transform the data.
Transformation refers to the programs that need to be written in order to properly move data from the legacy operational environment into the data warehouse. There is a large amount of effort required to move data from one environment to another and includes an understanding around the following functionality: filtering data, summarizing data, reformatting data, and selecting data from multiple disparate sources.
During transformation, data is cleansed, denormalized, and pre-calculated so that it’s ready for analysis. Cleansing the data means that you resolve any inconsistencies (i.e., links that have been tagged in different ways, etc.). Denormalizing optimizes the data warehouse’s ability to read data, and pre-calculations include the calculations that you know you’ll need frequently (e.g., net sales calculation, comp rules, etc.).
2a. Data Model (Schema)
To my knowledge, a restaurant centric data model does not exist off the shelf, meaning you’ll need to add someone to your team or hire a consultant with skills in this area (note, just because someone knows what a Snowflake or Star schema is or has the skill to create a data model does not mean they’re familiar with the restaurant business or more importantly your restaurant business, your needs or objectives). This lack of understanding around things like Comp Rules, 53rd week, etc., raises the stakes that something will be omitted or overlooked in the creation of the data model. As mentioned, making a mistake in the creation of the data model will have a major impact on the performance of your systems ability to address the needs of all the stakeholders in your business. That’s a costly mistake. Also, each time you want to add new data e.g., (guest experience, speed of service, loyalty, reservation) etc., into your data warehouse you’ll have to perform this process again and hope that it’s also architected correctly.
On the other hand, SaaS service providers, like Mirus, that focus on the restaurant sector have created a proven data model for restaurants. This streamlines the process which, in turn, shortens the time to completion from months to weeks and ensures a successful implementation because you can see it and validate its capabilities before you buy.
3. Load the transformed data into the warehouse.
Once your data is loaded into your warehouse, it’s officially ready for analysis.
On-going costs
Building a data warehouse is not a one and done proposition, it’s an iterative process and will change over time as your needs change. However, many companies considering DIY don’t often stop and think about their data warehouses’ sustainability / costs, such as, how upgrades, new integrations (Data Model, ETL) and ongoing maintenance will be handled.
Some of the future on-going costs that companies forget about are:
- Data formats changing over time
- The time / cost of adding new data integrations
- The time / cost of fixing broken data integrations
- Requests for new features, new dimensions, etc.
I can’t emphasize on-going costs enough. For example, if you’re connecting to an Application Programming Interface (API) - based service (which you most likely will at some point), you’ll encounter frequent updates being made to the API. If you want uninterrupted access to your data, you’ll need to be prepared to quickly respond to these updates. In short, as long as you have a data warehouse, you will need to have the IT infrastructure to maintain and support your current as well as future needs (and that’s costly). With a SaaS partner all of this is taken care of.
Quantifying the Costs
OK, as mentioned, this is not a tutorial on how to build a data warehouse but rather a snapshot of some of the challenges a DIY data warehouse presents and associated costs to create and maintain. That said, a data warehouse’s costs can be classified in several ways – recurring and one-time expenditures and by capital and operational expenditures. The classifications include the rental or purchase of hardware, software, consulting and ongoing development and maintenance. The creation of a data warehouse is generally done in stages, where initial expenditures are made in year one and successively lower one-time costs are made in successive years the costs for the DIY data warehouse can be broken into two areas; one-time and recurring.
Assumptions
1. Fees
Consulting Fees for software development capped at $150K - but could be higher ($250K).
Years 2, 3, etc., include new integrations (Food Cost, Guest Sat, Loyalty, etc.) being added to data warehouse.
2. Employees
One pecialized database skilled IT employee will be required to keep the solution operational.
A second IT Person will be necessary as deployment ramps up. After all, this solution has to run 24 hours a day, 365 days a year.
3. Tools
Software costs for tools to mine data – typically a per user fee for DIY (Reflects 1 to 2 users).
4. Storage
Cost to house minimal data with cloud service providers (AWS, Google, etc.), includes network infrastructure, hardware, security, on-going maintenance). Cost for service will escalate as storage needs increase and as data is accessed (I/O).
5. Time
Deployment of DIY may take 6 - 9 months before operational v/s SaaS which can be deployed and operational in 6 - 8 weeks.
6. Satisfaction
If you're unhappy with the SaaS service you can cancel and move on - if you're unhappy with your DIY your options are much more limited and far more expensive.
Summary
Having access to your data is incredibly important. An effective data warehouse empowers your team to more quickly create reports that produce fact-based answers so decision-makers can make decisions that result in the best outcomes for the business. If you’re a believer in data-driven decision making as we are, then there really aren’t many projects of greater strategic significance.
While there are more options today making it easier to create your own data warehouse there’s compelling evidence indicating that there remain a number of hurdles / costs to successfully completing the project.
- Gartner once estimated as many as 50 percent of data warehouse projects would have only limited acceptance or fail entirely.
Solid cloud based providers like Mirus deliver the results you need and are accountable for the entire solution not just your data storage. Additionally, unlike DIY, SaaS providers continuously invest in software development, infrastructure and human capital. For example, Mirus operates as an agile development environment and as such, releases new software updates every 4 – 6 weeks.
These investments ensure that the latest features are available, infrastructure is able to support a client’s ongoing needs and there’s ample highly qualified talent to address any questions / issues that might arise.
The choice is yours, DIY or SaaS, but given the costs, time to develop and possibility that a DIY solution might ultimately fail to deliver, its safe to say that SaaS solutions are far faster to deploy, safer and less a much less costly approach than a DIY system.
Thoughts?
Do you think a DIY data warehouse project has a higher potential to fail? Why?
Do you think a SaaS solution will cost less / more than DIY data warehousing over the long run? Why?
About Mirus:
Mirus is a multi-unit restaurant reporting software used by operations, finance, IT, and marketing.
For more information, please visit: www.mirus.com
Watch Mirus reporting demonstrations and client insight on our YouTube Channel
If you enjoyed this blog, please share this post by using the social buttons at the top of the page and make sure to leave your thoughts in the comment section below!
