In today’s world, good use of business data can result in growing revenue. Yet, for many businesses, this opportunity remains completely untapped. For most, cloud environments such as data lakes and data warehouses are not new concepts - on average, 60% of data lives in a data warehouse with the other 40% living as dark data.
Simon Hayward is VP of Domo EMEA.
Dark data is data that isn’t managed within an IT-approved, rigorous rules-based technical workflows and business processes data architecture. Dark data is often collected but not used, such as operational logging data which is archived in a data lake but not accessed.
However, this standard definition of dark data often ignores a much more valuable type of dark data: data that is already in use (such as offline data in spreadsheets or third party systems) but which isn’t integrated into the enterprise’s data architecture.
This segment of dark data has greater associated risk and missed opportunity because, unlike data which hasn’t yet been exploited, it is essential to many business functions but is being managed and processed in ways that the IT management department wouldn’t approve.
Problems caused by dark data
If dark data is not handled well, it can result in many organizational problems. Firstly, because this data is essential to the business user, they will continue to use it and will also opt for going rogue and using separate BI tools of their choice - because they often don’t need IT’s approval and can connect to data much faster.
Analysts spend up to 80% of their time seeking out and managing data required by business users but which isn’t in primary systems. This makes it a time-consuming and expensive operation.
Dark data can lead to shadow analytics - more tools, separate to the main systems, being used - and a data governance, certification nightmare without the right tools or level of visibility.
Finally, there is the issue that when data is taken out of the data warehouse it loses its security and accuracy, which could have a detrimental effect on the business. Some businesses fail because they lose access to the information that they need to reach potential clients, or because existing clients start to question the company’s trustworthiness.
The key to overcoming these barriers is to move and shape the vast volumes of data into one centralized location, creating one source of truth. This allows organizations to extract insight, helping inform business decisions at speed.
How can your business get there?
The first part of this process is to determine where the ‘dark data’ resides. Dark data takes shape in many forms, from offline Excel spreadsheets to Salesforce pipelines to monthly profit and loss statements. Gaining insights from this segregated or siloed data can take hours, if not weeks, of data analysis and reporting. Senior leads will often call upon departments to produce these reports individually before dissecting it and finding common themes to inform their next steps. This process is slow and naturally affects consistency across the business, as many departments find themselves singing from different hymn sheets.
The solution is to integrate and automate how people see data across the business, providing one consistent line of truth for all departments. This integration happens by plugging all ‘dark’ siloed data platforms into one data visualization platform. By doing so this process helps businesses answer pressing questions like “how much money did we make yesterday” or “which products are in high demand this week”. This shift in data management allows organizations to move away from time-intensive reporting toward larger data science initiatives that have an impact on the bottom line, such as building credit risk or eligibility models in developing markets.
The need for democratized data
Democratizing dark data and hosting it under one roof allows all departments to foster a culture where decision-making happens without going through IT support. Business leaders should focus on driving this behavior change, helping department leads innovate in how their teams are deployed and operate, using live data as a tool for creative and impactful decision making.
While democratizing data is crucial, effective data governance is needed to maximize an efficient and accurate use of data across the enterprise. However, data integration in the cloud doesn’t mean losing control, as each user of the central platform can be assigned a role or attribute - meaning they have certain levels of access from the moment they log in.
Integrated cloud solutions allow IT leads to certify data sets so that users know what data has been approved by the company or individual department, or IT leads can set certain levels of unlimited personalized data permissions. Administrators can create entitlement policies that govern access to specific data in a dataset for individuals and groups.
Whether it be shielding the IT team from endless data requests or activating chatbots via AI to speak with your customers, the possibilities of uncovering and using dark data are endless. Once a data-driven culture is created across your business, you are only bound by the creative use of its application.
- We've featured the best cloud storage services.
