The One Graph You Need If You're Thinking of Building a Data Warehouse
If you are a CTO or CIO, you’re likely to fall into one of these categories:
“We want to build and maintain our own data warehouse.”
“We already create dashboards on our data.”
“We want to build our own internal system in order to be more competitive.”
We hear you.
But we also believe there is a danger to this.
It is tempting to want to focus all the resources of an IT department on building a data warehouse and data pipeline. After all, building is fun! You want something unique that your competition doesn’t have. You want to solve your own problems and use your own people to do it. You are passionate about your work– you love new technology and new ideas.
But more resources you dedicate to building your own data platform, the greater the impact you should expect from that system. As the payoff required from a build approach increases, so too does the risk of failure.
So, while you may have the desire to build a data warehouse, you need to consider if you can do it as effectively as an expert third party would.
No one expects a construction company to have the resources needed to build a unique data ingestion, normalization, and intelligence platform. And nor should you. You don’t build your own project management platform, nor do you have your own email client. You do, however, have the resources to ensure the decisions gleaned from a PM application or a data platform are valid and scaled in your organization.
The specialized knowledge needed today around data normalization, data pipeline, and data intelligence may make any internal build programs too risky and too resource intensive, but implementing the insights gleaned from these tools will require a unique internal approach that only a CTO or CIO will be able to offer.
So rather than present the binary option of a build vs buy approach, we have sourced research from Forrester to recommend a model called the Buy, Build, Extend approach.
THE OLD PARADIGM
There was a time when an enterprise had two options: buy an expensive and resource-intensive ERP, or build an expensive and resource-intensive ERP. Both approaches carried equally risky outcomes. Today, with the availability of cloud applications and open-source software, it has become less risky in general to build or buy an enterprise solution. So why not do both?
This is how it works:
Buy: Source a data platform from a specialized provider
Build: Ensure your internal systems have the proper infrastructure to implement the learnings and recommendations from a provider
Extend: Take the components of what is working and implement at scale
DON’T REINVENT THE WHEEL
If you are dead-set on building a data platform internally, this is a minimum of what you’re looking at: hiring developers or outsourcing development. Appointing HR and Legal resources to onboard new developers. Recruiting or assigning an internal project manager for development. Implementing systems to ensure results can scale across projects, across departments. Ongoing maintenance.
This doesn’t include other factors: selecting a technology stack that your team will use to develop the platform. Determine the best tech environment for your platform to be developed (Security, Immutability, Hosting, Database Structure, etc). Develop machine learning code to render predictions and recommendations based on the data warehouse. Hire a team of python or json developers that have both a knowledge of construction objects/terminology as well as a specialized knowledge in machine learning and AI-based predictive analytics.
TIME TO GET HONEST
We love seeing companies that are as passionate about data platforms as us. If you have a strategy internally that you believe will build a more effective data solution at a cheaper cost, then from a business perspective, you should. We would certainly lose credibility if we told you otherwise.
But remember, the more specialized the platform, the riskier it becomes. If a team does not have the specialized knowledge needed to pursue the build route, the riskier the project becomes. In the end, our clients are in the construction business, not the machine learning and development business. Run your systems as usual, let Briq weave it together.