Jonathan Bennion dispelling GraphRAG hype (may not be worth it when compared to a vector db) скачать в хорошем качестве

Jonathan Bennion dispelling GraphRAG hype (may not be worth it when compared to a vector db)

TLDR: I evaluated Knowledge Graph-based RAG against traditional vector-based methods using Neo4j and FAISS. This comparison was prompted by a Microsoft paper on GraphRAG that lacked clear metrics and quantifiable improvements, potentially due to vested interests in promoting their technology. Context: After reviewing Microsoft's GraphRAG paper, which didn't justify the additional engineering and compute costs for GraphRAG, especially for dynamic documents, I decided to test knowledge graph performance myself. This post (Part 2) compares graph retrieval (Neo4j) vs. vector database retrieval (FAISS), building on the database comparison from Part 1 (which was comparing the same retrievers of the vector database componenents of Neo4j vs FAISS. The goal is to assess if GraphRAG's performance (when retrieving its graph components) justifies its increased complexity and resource requirements. Methodology: My approach below, for evaluation of each GraphRAG vs a Vector DB: Set up the environment. Processed the input data (used the same document as Part 1 - the June 2024 US Presidential Debate transcript). Used the above chunks to create a knowledge graph in Neo4j with GPT-3.5-Turbo (to see if any lift from small model). Setup FAISS retriever with comparable chunking Generated ground truth from the document to evaluate performance of each. Same as Part 1: Evaluated on RAGAS metrics and plotted results of the vector database side for each. Created a function containing a cypher query to transverse the knowledge graph. Now, unique to Part 2: Evaluated on RAGAS metrics for the graph transversal retrieval of the graph database vs the FAISS vector database baseline. Microsoft GraphRAG paper I'm referring to: https://arxiv.org/pdf/ Link to code: https://github.com/j-space-b/analyses/blob...