Swimming Beneath the Surface of Data Lakes

Data comes in various shapes and sizes and plays a decisive role in several data-driven business decisions.

Let’s take an example of a retail store. A retail store uses data extensively for several key business decisions such as defining sales strategy, planning promotions, managing inventory, engaging customers, among others. To power this decision making, every minute, a variety of different data is being collected and stored. We have inventory details like product description, available stock, and price. We have customer profile containing purchase history, their feedback, reviews, etc. We also have sales and promotional plans, social media activity, competitive landscape, demand forecasts, and many more. This data varies from structured to semi-structured, to unstructured in its nature. Similarly, every domain has its own variety of data that gets generated, stored, analyzed, and used for different types of data-driven decisions.

Given the scale, complexity, and heterogeneity of such data, data storage plays a pivotal role in the overall data management process. There are different types of data, and there are different types of data stores. In this blog, we are going to look at Data Lakes and how they fare against the traditional Data Stores.

So, what are data lakes, and why did we need them? To understand this, we need to take a step back and understand the evolution of data and how we have worked with it.

Evolution of Data Storage

With increasing focus on data-driven decision making, there has been an increasing interest in collecting data across different layers of business, applications, and infrastructure. With more and more data coming in, a question kept popping up from time to time — “how do we store this data?”.

We created the classic databases and relational database management system (RDBMS), which later on became to be known as the traditional data store. We also created a logical flow, and we called it ETL (Extract → Transform → Load).

Simply put, we extracted data from a source, we transformed it to fit logical criteria of each database, and we loaded the transformed data into the database. And when a different user came in asking for the data, but in a different form, we created a different relation for them, transformed the data and loaded it into their relation for their use, and life went on.

The ETL tasks were needed to consolidate data from multiple sources and prepare the data such that it allows simplified access for analysis and additional processing. However, this soon became a time consuming and redundant activity.

While data storing and processing was evolving rapidly, there were lots of other things in the IT landscape that were evolving rapidly at the same time. With the advent of internet, the thirst of collecting data increased. Today, these are just some of the sources of constant data creation:

• IoT devices, sensors, smart wearables, etc.
• Social media activity
• Offline and online transactions
• Websites and web applications that collect data about user behavior
• Mobile applications data
• Healthcare devices, scientific devices
• Crowdsourced opinions on different matters

The thirst for data collection wasn’t just limited to the idea of collecting infinite data. The endless possibilities when working with data was the true reason why everyone started valuing data. Suddenly, we became aware of the real possibility that data is a currency. With data, anything could be done.

• Predicting the purchase patterns of different customers.
• Predicting what products people buy together.
• Augmenting intelligence by taking user feedbacks, and giving better responses in return.
• Automating from the simplest manual tasks to complex enterprise level procedures.
• Creating a real-world JARVIS.

Data storage, data transformation, and identifying the right data in the right format for the right use case became important concepts. Some fundamental assumptions regarding data storage got challenged with questions such as the following:

• How do I decide how the data should be transformed before storing it in database? What if I must transform the data a different way tomorrow, for a different need?
• Why should we transform the data before loading it into a database?
• Why can’t there be a database that can store all kinds of data together? Why the need for different databases for different kinds of data?

And ultimately, there came a time when we asked ourselves the question once again. “How do we store all this data in the best possible way so that it can be used by everybody as and how they want it?”.

And then, different types of data bodies came into the picture.

Different Strata of Data Bodies

Data lakes were built with this fundamental philosophy to store the data in its purest form. Not just that, they contain almost all data, nothing is discarded. Contrasting this with a data warehouse, lots of data manipulation happens up until the point it gets stored in the database. While structure and schema are important, data lakes give the flexibility of deciding your own schema depending on the requirement. Data ponds and Data puddles along with Data lakes form the different strata of data bodies.

Let us understand these data bodies better.

Data Lakes:

• Data lakes are adaptable, centralized, comprehensive, extensible data storage systems that allow organizations to store large amounts of structured, semi-structured, and unstructured data.
• Data lakes follow an ELT (Extract → Load → Transform) approach. Data is extracted from the source either in real time or in batches. It is then loaded into the system as-is, without any data loss. Data gets transformed next, wherein, the data gets cleaned and converted to appropriate format. This transformed data is then stored for everyone in the enterprise to use.
• Data lakes often contain data from a variety of different data sources, captures different business contexts, and cuts across multiple solutions.
• The data architecture team owns this data body and is responsible for providing/restricting access to appropriate users.

Data Ponds:

• Data ponds can be thought of as small chunks cut up from the data lakes, for the purpose of analytics, reporting, etc.
• There can be multiple data ponds in the enterprise. In fact, there can be multiple data ponds for a single business context. The data ponds hence can also be architected to be related to each other.
• The data architecture teams own the data ponds, but so do the data analytics team. The analytics team will put up a request for what data they require, how much of it do they require and in which format do they require the data from the lake, while the architecture team would make it possible and also maintain the pond.

Data Puddles:

• Data puddles are even smaller chunks from the data ponds that can be created by the users for their analytical needs.
• The data puddles can be created, maintained, and owned by both data architecture team and the data analytics teams.

So, how are these data bodies used within an enterprise? We can understand this with the help of a simple example.

Imagine a retail store!

• A retail store maintains a variety of data capturing sales, customer profile, inventory, social media activity, customer reviews and feedback, customer demographics, performance of the competitors, etc. A data lake stores all this data in its raw form.
• Now imagine a data pond that captures a slice of this data lake containing the customer behavior of all departments of the retail store. This data pond can now be used for deriving insights on customer preferences across departments, identifying popular products, and other kinds of inter-departmental insights.
• Each department of this retail store would maintain an inventory record, to manage and replenish their inventory. This would be achieved with the help of a data puddle. This data puddle is a single-function data body that would be used specifically by the respective team.

Now, let’s see how the data lakes fare against traditional data storage systems.

Data Lakes vs Traditional Data Warehouses

Data warehouses consist of a collection of different relational databases. These data warehouses are able to link different databases and allow the data to be viewed as a whole. There are a variety of reasons why data warehouses work.

• They are consistent and have a defined structure in which they store data, making it easy for users to analyze.
• They are centralized and have the entire enterprise’ data consolidated into one location.
• They are easily accessible.
• The data in data warehouse can be easily managed, taking care of things such as redundancy, insufficiency, duplication, etc.

However, with the changing landscape of data and analytics, data lakes provided some differentiating benefits over these data warehouses.

Benefits of Data Lake over Data Warehouse:

• Before data goes into a data warehouse, decisions on how the data will be used in the future are decided upfront, leaving little room for expansion. Data lakes on the other hand are flexible and allow users to ingest any kind of structured or unstructured data, without worrying about the type or schema of data. Data lakes contain data for today’s and tomorrow’s potential use.
• Along similar lines, cleanup of data becomes necessary in a data warehouse, while the concept itself is redundant in a data lake. Any analysis can be done on any historic data as and when needed.
• Traditional data warehouses consist of data that fall under the umbrella of metrics and attributes that describe them. On the other hand, data lakes allow us to store all data, in all its different shapes and sizes. As and when any data needs to be analyzed, it can be transformed to convenience.
• While data warehouses facilitate most users, it restraints the data analysts and deep data researchers. The structured, sanitized data might be convenient for most users, but becomes shallow data for the data analysts who would much rather prefer to go back to the source of the data to get the raw data.
• So, when new requirements arrive at your doorstep, often the existing data warehouses are unable to adapt to the changed needs. Data lakes are inherently unstructured. Data lakes allow the users to explore different mechanisms of reading and interpreting the data, allowing for exploration and experimentation. If something doesn’t work, you can always go back to the original data and re-interpret it.

Is everything then rosy when it comes to data lakes?

No, there are somethings that need to be taken cautiously:

Caution while using Data Lakes

• Standards on data collection, storage, distribution, and maintenance need to be clearly defined and adhered to.
• Data security can also become a concern in data lakes. An oversight of the kind of data that goes into the data lake is necessary.
• Data lakes can store all data, doesn’t mean it should store all data. Data lakes can easily become data swamps or data dumps without proper governance.
• While structure has its disadvantages such as limitations, restrictions, lack of adaptability, it also has its advantages. Unstructured data can be chaotic, can create plain unusable data or may require advance tools for basic retrieval.

Creating a Data Lake

Let us consider a simple example of managing our photo gallery. We end up creating a library of thousands of photos captured over the years. These photos contain a lot of memories of people, places, things, and events, and we often enjoy browsing through them from time to time. With advancements in machine learning to automatically detect objects and annotate images, it has become much easier to ask various questions on this photo library such as, “show an image of the last time I came to this cafe”, “make a collage of all my family pictures”, “show me the pictures of my birthday over the years”, “show me pictures of my most recent medical reports”, and so on! Data lakes can provide an effective tool to enable this functionality. Let us take a look at how this data lake can be created:

1. Data Source Identification: We are dealing with unstructured data in the form of photographs. We also get structured data in the form of tags associated with the photographs such as date, time, location, device from where the photo was clicked, the photograph dimensions, etc. Some photographs might have captions provided by the users which is another form of unstructured data.

2. Ingestion and storage: We need a way to ingest the photographs and associated metadata. This ingestion can be in a streaming mode (say, as and when we click) or a batch mode (say, end of the day we sync all photos clicked in the day). Next, we need to decide where to store the data. Data lakes are usually created using a storage system that provides high durability at low cost, that is scalable, and that provides flexible access to the data. Data lakes are usually stored in distributed object stores. Hadoop distribute file system is often the go-to open-source choice. AWS S3 or similar services provided by Azure or GCP are some of the popular choices.

3. Catalog Creation: The next key step is to build a data catalog. A data catalog will take all the metadata associated with the photographs and create a logical catalog. The metadata includes information captured by a camera, as well as the user written captions. With the advancement of image analytics, a lot more metadata can be captured by detecting different types of objects in the images (people, place, things, etc.). Data cataloging is a very important step to prevent data lakes from converting into data swaps. It also plays a crucial role in enabling powerful search and querying constructs.

4. Data Transformation and Processing: This step is responsible for transforming data into various structures and formats. This layer also performs tasks such as aggregation of data by different time and space boundaries, storing the same data in different formats, etc.

5. Data Querying: Now that the data is stored, cataloged, and transformed, we need a way to query these photos. This is where different analytics tools come to play to query these photographs in many creative ways to search photos by similar objects, look for trends, look for anomalies, create groups of similar photographs, create memory albums, and so on.

And once again, we can see here how a data lake stands apart from the traditional data stores. In a data lake, we store the data first, in this case, the photos captured over the years. And after that, we work on transforming this data into different formats as per the need of the hour. However, if we had been working with the traditional data stores, due to its rigid schema, we would have had to first decide how we want to use the data, then transform the data accordingly and finally store it in the database. By doing so, we would lose out on the flexibility of using the data in a different fashion, in the future.

Industry Examples

Data Lakes have a multitude of advantages over traditional data warehouses. Hence, in recent times, many industries have considered and adopted data lakes for their data storage requirements.

The life sciences is one such industry. The nature of life sciences is such that new and newer types of data are being generated daily. With technological advancements, the type of data that was being generated a decade ago is vastly different from the type of data being generated today. A rigid data warehouse with strict schema would not be able to handle such growth in data sources. However, data lakes, with just minor tweaks in schema can handle any type of data with ease. Data lakes are being used to store large amounts of data ranging in diverse data types including medical test records, genetic records, clinical research data, etc. This data is used in different ways for different use cases. For instance, data of genomics, electronic health records, other digital sources are used to accelerate research, optimize clinical trials, and develop new drugs. Data lakes can store such different data in a raw, unadulterated form, providing flexibility for data analysts to retrieve the data as and how they want it, without the need for extensive data transformation and structuring that data warehouses need.

The banking and financial sector is another curious case. The requirements of BFSI domain entail the need for easy access to short and long-term history for analysis, predictions, and reporting. The traditional data warehouses have worked well in this domain for data with fixed structure and applications. Data lakes prove to be an interesting addition into the mix. The BFSI domain gives great importance to personalized services and customized experiences. With the help of data lakes, different data such as transaction history, browsing patterns, social media activity, etc. can be stored together and later analyzed to come up with targeted marketing strategies.

ignio’s take

ignio deals with a variety of unstructured data such as error screenshots, semi-structured data such as incidents and change requests, and structured data such as time-series of performance and resource metrics. Data lakes provide an effective vehicle to store such data and use it to enable various use cases.

For example, ignio uses data lakes to store raw ticket dump. The tickets data contains logs of system-generated events, user-generated incidents, change requests, service requests, anomalies detected from logs, among others. This data is semi-structured in nature. It has some well-defined attributes such as timestamp and severity, and some unstructured attributes such as problem description, and resolution notes. This data is first stored as-is in its raw form in the data lake. Various text-mining algorithms are then used to catalog these tickets data by extracting information of interest. The ticket descriptions can be used to extract information such as the entity type, entity name, and the issue referred by the ticket. Similarly, the resolution notes can be used to extract the symptoms, the root-cause, and the corrective actions. The cataloged data is then used by various statistical and machine learning algorithms to mine temporal patterns, derive correlation signatures, predict future events, among others.

Conclusion

Data warehouse technologies have existed for a long time. Data lakes are a much newer concept and aren’t even explored to their full potential yet. Does that make data lakes better than data warehouses? Is the shinier new toy better than the worn old toy that gave you comfort for all these years?

Data warehouses have withstood time. Does that make them better and more reliable than the chaotic data lakes? Don’t fix it, if it ain’t broke, right? But has this attitude ever given rise to any creativity and innovation?

So, what is it? Traditional data manipulation tricks or the new age technology?

As I see it, it is not that data warehouses have gone out of fashion. It is just that the data lakes have entered the discussion. In a world where mobile phones were merely used to call friends and family, and send messages, somebody decided that it could also be used as a torch, a calculator, to watch a movie, etc. Does that mean you no longer use your mobile phone to call or send messages?

The right answer to the question lies in the right question being asked. The question isn’t, “What should I pick?”. Rather, it is, “What benefits do I get by picking either, or which one suits me the most?”.

If you have a short-lived project with a well-defined problem statement(s), limited set of users who are all well served with a common type of data, then data warehouse might work best for you.

If you are a big enterprise that is constantly growing and looking towards bringing innovative solutions to varied problems, then migrating to a data lake might serve you best.

Ultimately, the correct answer need not be one or the other. It could both be used accordingly to derive the best output. While you can continue to maintain data warehouses and use them to provide your solutions, creating and populating your lake with data for future use might work the best.