In the ever-advancing realm of artificial intelligence, innovation occurs at breakneck speed. If you’ve recently explored AI applications, you’ve likely encountered applications of LLMs, such as Q&A Bots, Virtual Assistant Bots, Image Generators etc. Amid this technological whirlwind, one important piece that binds the whole application together often gets overlooked– the ‘vector database.’ But what exactly is this vector database, and why is it gaining such prominence amidst AI’s rapid evolution? In this article, we’ll delve into the world of vector databases from an AI perspective.

While the concept of vector databases has been around for roughly six decades, it wasn’t until the early 2000s that significant strides were made in their development. This progress can be attributed to academic research in techniques such as Locality-Sensitive Hashing (LSH) and metric indexing, which began to gain significant traction. Today, the critical role that vector databases play in AI-based applications is impossible to overlook, propelling them into the limelight alongside the rapid advancements in the field.

What is Vector Database?

A vector database is a data repository designed to house information in the form of vectors, which serve as numerical representations of data objects, often referred to as vector embeddings. It harnesses the potential of these vector embeddings to effectively index and explore extensive datasets containing unstructured and semi-structured data, including elements such as images, text, and sensor data. Vector databases are purpose-built to handle vector embeddings, thus providing a comprehensive solution for the management of unstructured and semi-structured data.

In a vector database, data is structured using high-dimensional vectors, each of which encompasses numerous dimensions. Each dimension corresponds to a specific attribute or property of the data object it represents. Now, let’s elucidate this concept with a simple example.

Imagine you’re a dedicated movie enthusiast with an extensive collection of films spanning multiple genres. You have a deep love for superhero movies and have organized your collection into distinct categories, such as “Lighthearted Heroic Adventures,” “Dark and Complex Superhero Epics,” “Marvel Cinematic Universe (MCU),” and “DC Extended Universe (DCEU).”

When you’re in the mood for the lighthearted, action-packed adventures of the MCU, you head to that section, enjoying films featuring iconic characters such as Iron Man and Spider-Man. Occasionally, you find yourself yearning for a cinematic journey that seamlessly combines the exhilarating Marvel action with the depth and complexity more commonly found in DC storytelling.

At this point, you turn to a cinephile friend who possesses an unparalleled understanding of superhero cinema. They know your cinematic preferences like the back of their hand and can recommend a film that masterfully merges the excitement of Marvel with the dark and intricate themes reminiscent of DC. It is during this moment that your friend surprises you with the perfect recommendation: “Doctor Strange in the Multiverse of Madness.” It’s a Marvel movie with a unique DC flavor that perfectly aligns with your cinematic cravings.

In this scenario, your friend effectively serves as a ‘human vector database’ for movies, skillfully guiding you to discover the superhero film that perfectly aligns with your cinematic cravings. Similarly, a ‘vector database’ operates with versatile applications, assisting in tasks related to text, image, audio, and more, much like your knowledgeable friend recommending the ideal cinematic choice.

Working of Vector Databases

The entire process is quite simple. Initially, we take text, images, or objects and convert them into numerical representations known as embeddings. When a user asks a query, we also transform that query into an embedding. These embeddings are then compared with other embeddings in a database, and we employ mathematical methods such as cosine similarity, Euclidean distance, or dot products etc. to discover the most suitable match. Now, let’s delve into each aspect of this vector database architecture.

 

 

Vector Embeddings

Now, you might get confused with what is an embedding or vector embeddings. So, an embedding refers to a representation of a data object, which could be a word, image, or user, within a vector space. In this vector space, each dimension of the vector corresponds to a specific feature or characteristic of the object in question. For instance, in the field of Natural Language Processing (NLP), it is common to use word embeddings to portray words as compact vectors with a fixed length. Each dimension of these vectors signifies a semantic or syntactic attribute of the word.

Various algorithms can be employed to create embeddings, such as Word2Vec, GloVe, and FastText, BGGE for NLP, as well as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) for Computer Vision (CV). These algorithms acquire embeddings through training on extensive datasets, with the primary objective being to enhance the likelihood of accurately predicting the words surrounding a given word (in the case of Word2Vec) or class labels (in the case of CNNs and RNNs) based on the provided input data. For more information on embeddings, you can refer to this blog on embeddings.

Now, let’s go back to the earlier example of movie recommendations. Imagine we have two characteristics for movies, which we’ll call ‘Marvel Essence’ and ‘DC Grit’. These characteristics are rated on a scale from 0 to 1, where 1 represents a strong presence of that characteristic. The below table shows various movies and their embeddings:

 

 

Now, if you look at the table, you can see different movies with their vector embeddings. Imagine, a user wants a movie that has the essence of Marvel but also a bit of DC. Let’s say the embedding for this user’s request looks like Marvel Essence = 0.6 and DC Grit = 0.4.

To search and find the right movie from this set of embeddings, we need a way to compare two vectors. This is often referred to as a “similarity measure” or “similarity metric.” These metrics help us determine how similar two vectors are by looking at the distance or angle between them. Now, let’s explore these similarity measures.

Similarity Measures:

When a vector database gets a question, it looks at the indexed vectors to find the closest neighbors to the query vector. To figure out which vectors are the closest, the vector database uses math techniques called “similarity measures.” There are a few different types:

· Cosine Similarity: This method rates similarity from -1 to 1. It does this by checking the angle between two vectors. If they’re completely opposite, it gets a -1, if they’re at right angles (like the corners of a square), it gets a 0, and if they’re identical, it gets a 1.

· Euclidean Distance: It rates similarity from 0 to infinity by measuring the direct distance between vectors. If the vectors are the same, it’s 0. As the values get bigger, the vectors are more different.

· Dot Product Similarity: This measure rates similarity from negative infinity to infinity. It does this by looking at the size of the vectors and the angle between them. Negative values mean the vectors point in opposite directions, 0 means they’re at right angles, and positive values show they point in the same direction.

For more details on various similarity techniques, you can refer to this blog on similarity techniques. Let’s revisit our previous example where a user wanted a movie with specific embedding values, like Marvel Essence = 0.6 and DC Grit = 0.4. We will use the Euclidean distance method to figure out which movie embeddings are most similar to what the user asked for. Since our example deals with two characteristics, the Euclidean distance formula for two dimensions looks as below:

 

 

Where D is the Euclidean distance between the points or vectors .

Below, you can see the Euclidean distance between the user’s query embedding (Marvel Essence = 0.6 and DC Grit = 0.4) and the embeddings of movies.

 

 

The smaller the distance, the closer or more similar the result. If you observe, the movie with the smallest distance is ‘Doctor Strange in the Multiverse of Madness.’ And hence, the system recommends this movie to the user. This is how similarity measures work in a vector database. However, please note that this is just a simplified example. In reality, various complex and advanced methods are used to determine similarities between vector embeddings.

Vector Indexing

Now, you might be curious about what comes next. We have an important and intriguing topic to delve into within vector databases, and that’s vector indexing.

A vector index is a specialized data structure designed to facilitate rapid and precise retrieval of vector embeddings from a vast dataset of objects. In traditional databases, data is stored in rows, each representing a particular piece of information, with columns providing additional details or links to other tables with supplementary information. This data is scalar, meaning it comprises single values in each cell, in contrast to vector data, which consists of multiple values.

In traditional database searches, we typically seek exact matches. However, with vector embeddings, we can perform something quite remarkable — search for approximate matches. We input a vector and instruct the vector index to return other vectors that are similar to the query vector. This capability enables swift searches through extensive collections of vectors. Now, let’s explore some common techniques and methods used to construct these data structures.

· Flat Indexing: Flat indexing is a method where we keep each vector just as it is, without making any changes to it. This approach is simple and easy to put into practice, and it always gives us the correct results. However, the downside is that it’s not very fast. In flat indexing, we have to calculate how similar the query vector is to every other vector in the index. Afterward, we choose the K vectors that are most similar to the query. Flat indexing is the way to go when you need perfect accuracy, and speed isn’t a big concern. If the dataset we’re searching through is small, using flat indexing can still provide reasonable search speed.

· Inverted Indexing: Inverted indexing is a method often used in text search engines. It works by making a connection between each word in a collection of documents and the documents that have that word. When it comes to vector search, we can use a similar idea by linking each part of a vector to the vectors comprising that part.

· k-NN search: k-NN (k-Nearest Neighbors) search is a method to discover the k closest neighbors to a given query vector in a space with many dimensions. You can do k-NN search in different ways, like directly comparing each vector (brute force), using tree structures (like k-d trees), or employing special methods like LSH (Locality-Sensitive Hashing).

· Approximate Nearest Neighbor (ANN) search: ANN search is a method to find nearby vectors to a given query vector in a high-dimensional space. These algorithms balance search accuracy and speed, which makes them great for huge datasets. Well-liked ANN search methods include Product Quantization and Hierarchical Navigable Small World graphs (HNSW).

In a vector database, data and associated metadata are stored, prompting the existence of two distinct indexes: one for numerical data representations (the vector index) and another for the metadata itself. Leveraging data represented as vectors, vector databases efficiently compute similarities and distances between data points. GPUs enhance these calculations through parallel processing on vectors, notably accelerating indexing and search tasks within the database. This integration of vectors and GPU processing significantly bolsters the speed and efficiency of operations within vector databases, optimizing computational performance for tasks involving extensive data analysis.

Popular Vector Databases/ Libs

Before we delve into the advantages and disadvantages of vector databases, it’s worth noting that several highly regarded vector databases and libraries have gained prominence in recent years. These tools play a pivotal role in harnessing the power of high-dimensional data and enabling efficient search and retrieval of vector embeddings. Some of the popular options in this domain include:

· Pinecone: A managed vector database service that provides scalable, real-time similarity search for applications such as recommendation systems, machine learning applications or LLM based applications.

· HNSW (Hierarchical Navigable Small World): A method and library for building and querying high-dimensional indexes, known for its excellent search performance.

· FAISS: Developed by Facebook AI Research, FAISS is an efficient and widely used library for similarity search and clustering of high-dimensional vectors.

· Milvus: Milvus is an open-source vector database that empowers users to store, search, and analyze extensive embeddings effectively.

· Annoy: Annoy is an open-source library designed for conducting approximate nearest neighbor searches within high-dimensional spaces.

· Elasticsearch with Vector Plugins: Elasticsearch, combined with specialized vector search plugins like Open Distro for Elasticsearch, can be used to build vector database capabilities.

· Some others include ChromaDB and Qdrant.

Advantages and Disadvantages

Vector databases have become essential tools in data management and analysis. They come with advantages but also have some downsides. These databases are great at dealing with complex data, which is why they’re used in many applications. To understand their pros and cons better, it helps to look at the bigger picture of how they work.

Advantages:

  • Fast Similarity Searches: Vector databases can quickly find similar data points, which is valuable in applications like image and product similarity search.
  • Scalability: Many vector databases are designed to scale horizontally and vertically, allowing them to handle large datasets and increased computational loads.

Disadvantages:

  • Complexity: Setting up and maintaining vector databases can be complex, especially for those without extensive knowledge of machine learning and database systems.
  • Data Sparsity: Sparse data can pose challenges in vector databases, as they are designed to work with dense vectors. Special handling may be needed for sparse data.

Overall, the choice of using a vector database depends on the specific requirements of the application, the available resources, and the expertise of the development team.

Applications

Vector databases find applications in various domains due to their ability to efficiently handle high-dimensional data. Some of the current applications of vector databases include:

  • Natural Language Processing (NLP): In NLP, tasks such as document retrieval and sentiment analysis, vector databases help store and search text embeddings efficiently.
  • Image and Video Search: Vector databases enable efficient image and video search, allowing users to find similar visual content quickly. This is used in image retrieval, facial recognition, and content moderation.
  • Genomic Data Analysis: Researchers use vector databases to analyze genomic data and identify patterns and associations in DNA sequences.
  • Geospatial Applications: Vector databases can store and query geospatial data for applications such as location-based services, GIS (Geographic Information Systems), and route optimization.
  • Audio Analysis: In audio processing, vector databases assist in music recommendation, speech recognition, and acoustic fingerprinting.
  • IoT (Internet of Things): IoT devices generate vast amounts of data, and vector databases help manage and analyze this data efficiently for real-time monitoring and decision-making.

Conclusion

Looking ahead, the future of vector databases is closely intertwined with the progress in AI and machine learning. Researchers are constantly improving the art of creating more effective embeddings for structured and unstructured data using deep learning techniques.

As these embeddings become more refined, vector databases will continue to evolve, requiring innovative techniques and algorithms to manage them effectively. The quest for better methods is ongoing, with experts striving to enhance how vector databases work.