In today’s data-driven world, applications are producing and collecting vast amounts of data from various sources. Organizations need robust database solutions, to effectively and efficiently store and query this humongous data that can be in heterogenous format.
Traditionally relational database has been a popular option for this, which was slowly changed by NoSQL databases. But with increased digitization and increased focus on observability a wide variety of data is getting collecting including user transactions, activity logs, metrics, error logs, among others. The data is also getting highly connected and have complex relationships. Consequently the modern applications do not find the relational or NoSQL as the right fit. So how to do modern applications work with such diverse data needs? Read on!
How to deal with new-age diverse data needs?
Two broad approaches stood out to handle the volume and variety of data: polyglot persistence and multi-model databases.
Polyglot persistence uses different databases for different data needs. For instance, it may use MySQL for storing transactions, Elasticsearch for product inventory, Neo4j for catalog, among others. But this approach adds a significant overhead.
An alternative to polygot persistence is the multi-model database. This database allows you to natively store and access different types such as time-series, graph, logs, documents, etc. Multi-model databases offer a simpler alternative by reducing the variety of databases required to server the new-age applications.
The rise of multi-model database
According to Wikipedia, the first time the word “multi-model” was associated to the databases was on May 30, 2012, in Cologne, Germany, during Luca Garulli’s keynote “NoSQL Adoption — What’s the Next Step?”. Luca Garulli envisioned the evolution of the 1st generation NoSQL products into new products with more features able to be used by multiple use cases.
The concept of multi-model databases began to gain traction primarily in the 2010s. However, the foundational ideas behind them can be traced back to earlier times. The objective has always been to address diverse data needs without having to rely on multiple, disparate systems.
A multi-model database is a database management system that supports multiple data models against a single, integrated backend.
Multi-model databases are intended to offer the data modeling advantages of polyglot persistence without its disadvantages.
Multi-model database management systems unify multiple database systems into one. Instead of working with numerous models and finding ways to incorporate them together, multi-model databases provide a singular engine for various database types.
Why the need for Multi-Model Database?
In a modern application, there may be need for multiple data models. Following are examples of a few but not limited to,
As you can see from the table above, different type of data depending on their usage and access patterns, warrants for a different type of data model. Force fitting the data to be stored into an in-appropriate data model would lead to in-effective usage.
Storing such type of data into a multi-model database also enables us to query this across the different data models in a more effective way. We do not have to implement polyglot storage for the required data to be queried.
Implementing the database of such an enterprise application using multi-model database gives us enormous benefits, simplifies application development, eases deployment and maintenance of the database, opens possibilities of using data in a way that is more powerful as data access becomes simplified.
Let me take an example and try to explain more about where we need the power of the cross queries. In an enterprise, the various entities, and relationships between them are best represented in a graph, such as a three-tier application with a model such as:
A typical application is deployed on multiple servers running application servers, some of them dynamically getting scaled in and out. This arrangement of Applications to CPUs of Servers is best represented in a Graph. It would allow to get the list of involved CPUs very efficiently.
To characterize the behavior of application running on a server with one or more CPUs, there will be situations when you want to check the CPU utilization of the servers involved in running the application. CPU Utilization is a time series data, and hence should not be put into a graph. It could be stored as a set of time-series.
So, if you now want to find out all those Applications that had underlying CPU utilization going more than 80% in last one week, there will be a need to run a query on graph and join it with the timeseries to get this answer.
A multi-model database would allow to write these kinds of queries and can run this very efficiently to get the desired answer.
I have taken a simple example, but as the complexity of the graph increases and the interesting data points from the timeseries increase, the possibility of doing something like this using a traditional database becomes harder.
This becomes harder when we have needs of ACID compliance and not all distributed NoSQL databases are ACID compliant. Hence, having this additional type of requirement from the same database poses additional challenges.
The Benefits of Multi-Model Databases
Multi-model database comes with various advantages, some of them are:
1. Flexibility: By supporting multiple data models, such as relational, document, graph, key-value, and columnar, they allow for diverse data representation within a single database.
2. Unified Query Language: A unified query language to access and manipulate different data models, simplifying interactions and application development.
3. Data Consistency: Maintaining consistency across multiple single-model databases can be challenging. Multi-model databases can ensure ACID (Atomicity, Consistency, Isolation, Durability) properties across various data models.
4. Reduced Data Redundancy: By centralizing various data models, multi-model databases can reduce the redundancy that might arise from storing similar data in multiple databases.
5. Simplified Architecture: Reduces the need to manage and integrate multiple standalone databases, leading to a more simplified architecture.
6. Reduced Complexity: Multi-model databases can simplify data integration between different parts of an application or between different applications, as the data is stored in a common database.
7. Cost Effective: Operating and maintaining multiple database systems can be expensive. By consolidating these into one multi-model database, organizations can realize significant operational and infrastructure savings.
Challenges with Multi-Model Databases
While multi-model databases come with a lot of advantages, there are also few challenges. Some of them are:
1. Learning Curve: Multi-model databases can be more complex due to their ability to handle multiple data models. This can lead to steeper learning curve for developers unfamiliar with the various underlying data structures.
2. Maturity: Some multi-model databases may not be mature. Compared to their single model counterparts, they may not be as battle-tested.
3. Performance Concerns: Optimizing a multi-model database can be tricky. While these databases aim to offer the best of multiple worlds, they may not always achieve the performance levels of specialized single-model databases in specific use cases.
4. Migration Problems: Transitioning from single-model databases to a multi-model database or migrating data from one multi-model database to another can be complex and may require significant planning and effort.
5. Vendor Lock-in: Since multi-model databases tend to be unique in their implementations, and unlike SQL, no global standards have been established, migrating from one multi-model database to another or back to a single-model system can be challenging.
6. Limited Ecosystem: The ecosystem around multi-model databases (e.g., tools, libraries, and community support) may not be as extensive or mature as that of well-established single-model databases.
7. Features: Multi-model databases may not offer full range of features available in specialized single-model databases. For example, a multi-model database that supports graph data may not provide all the advanced graph processing features of a dedicated graph database.
Popular Multi-Model Databases
1. ArangoDB: An open-source multi-model database that supports document, key-value, and graph data models. It offers features such as ACID transactions, horizontal scaling, and a user-friendly query language.
2. OrientDB: Another open-source option that provides support for graph and document data models. It boasts of features such as ACID compliance, SQL support, and multi-master replication.
3. MarkLogic: A proprietary solution that supports document, triple-store, and relational data models. It’s known for its enterprise-level features such as ACID transactions, built-in search, and scalability.
4. Cosmos DB: Offered by Microsoft Azure, it supports document, key-value, graph, and column-family data models. It is globally distributed and provides multi-master support, tunable data consistency, and robust SLAs.
Final Thoughts
Multi-model databases are a game-changer for organizations dealing with diverse data types and complex relationships. They offer the flexibility, performance, and scalability needed to thrive in today’s data-driven world.
In conclusion, while multi-model databases offer a range of advantages, they aren’t a universal solution. We must be careful if we want to hop on the journey of multi-model databases.
The decision to adopt a multi-model database depends largely on the specific needs of the business or application in question. For projects that require diverse data handling with the ease of a unified platform, multi-model databases can be a game-changer.
As with any technological choice, it’s essential to weigh the pros and cons, test thoroughly, and ensure that the database you choose aligns with the longer-term vision of your project or business.
