For decades, IT has relied on conventional business intelligence and data warehousing, with well-defined requirements and pre-defined reports.
We are now in a new age of big data analytics, where discovery is part of the process, so objectives shift as new insights emerge. This requires an infrastructure and process that can quickly and seamlessly go from data exploration to business insight to actionable information.
To swiftly transform data into business value, a big data architecture should be seen as a supply chain that can manage and process the volume, variety, and velocity of data. To get started, every company needs a big data process. That process is divided into three steps:
1. Identify business goals
No one should deploy big data without an overall vision for what will be gained. The foundation for developing these goals is your data science and analytics team working closely with subject matter experts. Data scientists, analysts, and developers must collaborate to prioritize business goals, generate insights, and validate hypotheses and analytic models.
2. Make big data insights operational
It's imperative that the data science team works in conjunction with the devops team. Both groups should ensure that insights and goals are operational, with repeatable processes and methods, and they communicate actionable information to stakeholders, customers, and partners.
3. Build a big data pipeline
The data management and analytics systems architecture must facilitate collaboration and eliminate manual steps. The big data supply chain consists of four key operations necessary for turning raw data into actionable information. These include:
- Acquire and store: Access all types of data from any platform at any latency through adapters to operational and legacy systems, social media, and machine data, with the ability to collect and store data in batch, real-time and near-real-time modes.
- Refine and enrich: Integrate, cleanse, and prepare data for analysis, while collecting both technical and operational metadata to tag and enrich data sets, making them easier to find and reuse.
- Explore and curate: Browse data and visualise and discover patterns, trends, and insights with potential business impact; curate and govern those data sets that hold the most business value.
- Distribute and manage: Transform and distribute actionable information to end-users through mobile devices, enterprise applications, and other means. Manage and support service-level agreements with a flexible deployment architecture.
Once the process is established, the big data reference architecture can support these four common big data use case patterns, which enable actionable business intelligence: data warehouse optimisation, 360-degree customer analytics, real-time operational intelligence, and managed data lakes.
Data warehouse optimisation
As data volumes grow, companies spend more and more on the data warehouse environment. The problem arises when capacity in the environment is consumed too quickly, which ultimately forces organisations into costly upgrades in storage and processing power.
One way to cope with high-volume data growth is to deploy Hadoop, which presents an inexpensive solution for storing and processing data at scale. Instead of staging raw data that comes from the source systems into the warehouse, simply store original source data in Hadoop.
From there, you can prepare and pre-process the data before moving the results (a much smaller set of data) back into the data warehouse for business intelligence and analytical reporting. Hadoop does not replace the traditional data warehouse, but it provides an excellent, complementary solution.
360-degree customer analytics
Most companies want to understand their customers better to increase loyalty and retention -- and upsell products or services. To do so, you need to develop a 360-degree view of the customer.
CRM software has long claimed to do this. Today, however, new types of data about individuals abound via social, mobile, and e-commerce channels -- as well as customer service records, telematics, sensor data, and clickstream data based on Web interactions.
A true 360-degree view now means you must be able to access new data types along with traditional data ones, combine them, transform them, and analyse everything to discover new insights about customers and prospects.
This greater level of understanding, combined with big data algorithms for predictive analysis, enables organisations to predict customer behaviour more accurately and provide meaningful recommendations. Knowing your customers better, including what they are saying and doing, enables you to deliver more value to them.
Real-time operational intelligence
Real-time operational intelligence is the ability to monitor and (optimally) respond to events in real time. An example of this in sales or marketing is known as "marketing to the moment."
For example, via mobile device, a sales associate could be provided with information about a customer as soon as he or she walks into the store, including that customer's recent experiences on the store's e-commerce site.
