Almost 20 years ago, the British mathematician Clive Humby proclaimed, “data is the new oil.”
The co-founder of Tesco Clubcard was reflecting on how oil has driven industrial and commercial progress, and how data is the next frontier. But crucially, Clive was also highlighting the shared challenges between oil and data in refining, creating by-products, and processing it to make it suitable for business innovation.
Data is the lifeblood of successful enterprises. Growing organisations must tap into available data sources, consolidate them and develop ways to use this data as a valuable asset.
Through the process of exploring, organising and analysing data from every transaction and touchpoint, organisations can gain vital insights about their customers and potential market opportunities. Ideally, these insights should drive the priorities for the product roadmaps and map out where and how digital activities could maximise the business impact. Driving a business strategy with data to continuously improve and shape the customer experience is one of the greatest challenges facing organisations today.
As AI becomes more and more ubiquitous, coupled with the massive growth in connected devices and systems, the amount of data will continue to grow.
Successful organisations can leverage a wide variety of data streams to drive and optimise their customers' digital experiences. Data pipelines for AI and sensor-driven analytics must be designed with scalability, security, and availability in mind to meet the needs of modern enterprises and the expectations of end users.
So with this in mind, we caught up with Chris Gojlo, a senior Data Architect at Scrumconnect. We were keen to understand the challenges when dealing with data, but also to learn our approach to building bulk data processing tools for one of our largest clients.
Given the exponential pace of change and the scale of innovation in almost every industry, what are the main challenges that large organisations face when dealing with data?
Cloud has seen some of the important problems of the recent past regarding the mechanisms for storing and capturing large, raw datasets, mostly eliminated. Today they're rarely a point of discussion.
Data-driven insight is a key competitive advantage in any industry today, but obtaining information from the raw data can still take days or weeks. Addressing bottlenecks that slow down time-to-insight across the data discovery, transformation, processing, and production is now a common challenge. The focus has shifted from the data processing infrastructure, which has been largely commodified, to new models for effective data management.
Inefficient data governance leads to the acceptance of low-quality data circulating in the enterprise system with incomplete or invalid data points.
Poor quality datasets shared by downstream components force each consumer to create quality protection mechanisms that increase the data pipeline's complexity and increase latency in the data channel. Lack of consistent rules regarding data discovery, data reuse policies or regulations governing the creation of associations across the data are key factors contributing to the increase in time-to-insight.
As data streams can flow through different channels, using a variety of formats and semantics, it is essential to ensure that the data dictionary supports real-time data profiling and auditing. This can improve operational efficiency, ensure compliance and increase customer satisfaction as data problems are flagged as soon as they occur.
All of this contributed to the continuing trend of creating self-service data roadmaps to support data discovery, data quality scoring, lineage, and governance.
The self-service approach, coupled with the automation across data governance processes, can be used to build the capabilities required to democratise data in an organisation and reduce time-to-insight, which is a substantial bottleneck in modern data architectures.
What is the role of data architecture and data governance in building a solid bridge between the consumer of data and the data provider?
A successful enterprise data strategy should have a clear plan on how to unleash the value of the data assets to serve business purposes. It requires a systematic approach to data governance by managing data assets in terms of ownership, integrity, compliance, access methods, and relationships between different datasets.
With increased regulatory demands on business to implement compliance policies to protect sensitive data, organisations are required to build data-driven solutions aligned with the current regulatory framework and evaluate existing systems for new rules and restrictions. This emphasises creating a coherent strategy for all of the processes that define how to store, process, access sensitive data and guard against breaches. This tactic is an important factor in ensuring uninterrupted and secure access to valuable data for both internal and external customers. The data architecture and data governance model are part of this strategy.
The bridge between the data service and data recipients is always based on trust. This can be developed by providing high-quality data, with metrics, metadata describing the structure, and semantics that reflect business requirements.
Improving reliability of the service, not only operational, but also in terms of delivering data that is relevant to the customer, is a crucial part of this process too.
How were the challenges you mentioned addressed in the design of bulk data processing tools?
Understanding the problem domain and the business context is a critical starting point. By leveraging Domain-Driven Design (DDD) as a method for developing an understanding of a problem space, we were able to create and refine our set of conceptual models for the solution within a target domain.
This approach helped us with structuring the data architecture in line with the domain model, and at the same time, controlling the complexity of the solution architecture itself.
It has proved to be very useful in exposing potential misalignments between different services operating in the same domain.
As our data processing solution operates within a bounded context, with strong information architecture requirements, we had to address specific issues related to data governance, especially around ownership, data quality, integrity, security and compliance.
Two areas were particularly important for our client: the ability to audit data changes across the processing pipeline (data lineage) and data quality assurance. The self-service aspect of our solution allowed users, for example, to automatically check the quality of data coming from many different sources against a spectrum of data quality factors, without the need for manual and time-consuming effort.
Managing data traceability from a single service or exposing changes as data audit events has led to a number of improvements and benefits from compliance and security standpoint, but also for the end users. Being able to find out what happened to your data at any stage of the process is critical to building user trust.
You spoke about data quality. What is the impact of poor data quality on organisations and their operations? With massive datasets available for exploration, does this have any relevance to data science and AI?
In general, without high-quality data, companies are unable to recognise and react to changes in the market landscape, correctly assess their position relative to competitors or provide accurate and reliable analytics.
Worst of all, this can lead to ill-conceived business strategies.
There are two aspects to consider. The first one: using automation and AI to improve data quality and data management processes. Data quality is a multidimensional problem as it covers consistency, integrity, accuracy, and completeness. Traditional data profiling, combined with machine learning and semantic analysis techniques, can help to tackle more complex problems such as disambiguation and relationship extraction.
We have implemented smart data validation and correction in our bulk data processing tools to ensure that high-quality data is available to consumers regardless of their relative position in the business process or distance from the original data source.
The second aspect concerns the use of data in the data science and AI pipelines. Access to verified, consistent and reliable data is extremely important at all stages of data science projects, for example, in exploratory analysis and hypothesis testing. In machine learning, training data is the dataset of labeled images, video, audio, and other data sources used to train the ML algorithm.
To be clear: anything that might mislead the learning algorithm is not high-quality data.
A dataset, regardless of its volume, can contain biased data that does not provide enough information for the model to learn the problem, or is unrepresentative of reality in some way.
Models trained on biased data not only produce inaccurate results, they pose ethical, legal and safety problems too. We should not assume that operating with high volume of data will reduce the role of data quality checkpoints.
ML/AI can deal effectively with noise while working on data, but this is only possible if the training dataset has all quality issues resolved.
Real-time data helps organisations respond instantly to shifting conditions, improve service delivery, reduce operational risk, and make smarter, faster decisions. It enables greater agility, better customer experiences, and stronger performance across sectors where speed and accuracy matter. And in the freeze of 2022, it helped keep the lights on and our homes warm.
Data is often called the new oil, but unlocking its true value comes with challenges. So we sat down with a senior Scrumconnect Data Architect to learn how organisations can refine, govern, and leverage data to drive business innovation.
At a House of Lords event sponsored by Scrumconnect, experts discussed AI adoption challenges. Businesses struggle, policymakers lack understanding, and workers fear automation. This write-up covers key insights.
Scrumconnect is pleased to announce its successful award of G-Cloud 14 Lot 4 Framework Contract.
We are committed to continuously providing the best in tech with our people, practices and technology through our certifications.
Scrumconnect is committed to disability rights.
This enhanced government-endorsed standard demonstrates that we have implemented the essential plus security requirements for protecting your data, and that of your client’s, against unskilled internet-based cyber attackers.
We are a supplier on Crown Commercial Service’s (CCS) Cloud Compute 2 framework, a testament to our commitment to providing high-quality services to the UK public sector.
An integrated management system, designed for ISO 9001:2015 and ISO 27001:2015, operates across our business. These are regularly reviewed in order to ensure the continuous improvement to meet the needs of our customers and stakeholders.