Transcript: Connecting the Dots with Context Graphs — Stephen Chin, Neo4j

Source video ID: eW_vxrjvERk

Transcript

• 0:14 — Hello and welcome everybody to connecting the dots with context graphs. My name is Stephen Chin. I run the developer relations team at Neo4j and you are in store for the power hour of context and graphs and all of this technology. So I’m the first speaker. We have some other amazing talks after me. So I hope you enjoy all the great content which you’re going to see over the next um hour or so. So what I’m going to talk about is a bit about how we’ve all been
• 0:45 — feeling with the AI revolution where we are trapped as engineers. We are using AI coding tools or or maybe they’re using us. Where our work is being reviewed. Who Who here has their work reviewed by an agent when they check in their PRs? Yes. All of you. So we are we’re stuck in this limbo where we have amazing tools, we have amazing capabilities, but rather than us controlling them, they are controlling us. And we would like to get to a state
• 1:18 — where we’re in control of this. So we have to decide is it going to be the the blue pill where we’re stuck inside of this mire of disparate knowledge stuck in in Slack discussions and little customer threads and different enterprise systems which are all segregated and siloed. And when it when we ask the agents to make critical business decisions or our applications to make critical business decisions with with all this spread,
• 1:48 — it can’t possibly give good answers cuz it doesn’t have the context. Or do we want to dive in and and embrace the red pill, escape from the matrix, and have a system of reasoning where we actually have all these systems connected, all of our different enterprise data sources, previous decision traces, the reasoning tool calls of the tools to give us a more consolidated view of our enterprise stack,
• 2:19 — and escape from the matrix. So who who’s who’s going to who’s in the escape club? Who Who wants to break out? Okay, hopefully if you’re in the room, you’re you’re with me. Um and guess who else is with us? Gartner has now officially made context graphs as part of the AI hype cycle. So we have been officially recognized by the um the analysts of the world. They they also realize that we’re all stuck in this in this mire.
• 2:49 — Um Foundation Capital actually started this thread with their $3 trillion startup opportunity post about how context graphs are going to move forward the industry and dramatically change how we build applications. And um what I’ll do is I’ll I’ll show some demos and I’ll talk about how we can move from being stuck in this matrix, stuck in this world, and then become the superheroes of our organization and actually build the capability of the systems using technologies like knowledge graphs. So knowledge graphs are a very powerful
• 3:20 — tool for us to aggregate all this information, create the connections, create the relationships. And at a fundamental level, they they hold nodes which are are people or or things or companies or relationships. Um You have relationships between nodes where um in this case um you know, Dan, those are properties. Um lives with Anne. They He drives her car apparently. So we know who who wears the pants in this relationship. And we have some embeddings on top of the car. So we’re
• 3:52 — embedding vector information in it so we can also do similarity searches and kind of combine the best of both worlds with building information, but then also combining it with LLMs. So when we take what LLMs are really good at, this language, this reasoning, this creativity, when we combine that with what knowledge graphs are really effective at, so knowledge, context, and enrichments, then we can start doing things with our data like um storing all these relationships together, visualizing
• 4:22 — them, so we can get to the data which matters, finding hidden patterns, and then analyzing this and getting more insights which will help power the context graph demonstrations which I’m going to show you all. So here’s a a simple example of how graphs power retrieval because I think it’s it’s good to understand what the difference is between a baseline LLM. So this is a um healthcare case. What was the care plan associated with Andre Jenkins’ emphysema?
• 4:53 — And when you ask the LLM, it has broad knowledge. It It understands a lot of information. It knows what emphysema is. It knows what standard practices so it gives a a very generic answer, preventing damage to the lungs, yada yada yada. Now when we give it a rag system, so we go to vector database, now it has more context. It knows a bit about the the patient and their information and it tells you maybe recommend some activities like respiratory therapy, deep breathing, coughing exercises. So this is pretty generic medical advice. Now where we
• 5:24 — want to get to is grounded complete information where we’re pulling in who’s the patient, what was the previous diagnosis, what operations have they have. And you can see here that it it’s specifically recommending medication management, smoking cessation counseling, pulmonary rehabilitation exercise. So that Clearly the parent here has a the patient here has a history of of smoking, has had an operation. So like there’s there is certain things which are background information that was lost in the similarity search.
• 5:54 — And if our agents have this information, then like the matrix, now we’ve loaded all up up this information and we can you know, we’re like Neo. We can do kung fu, we can shoot bullets, we can do all this amazing stuff with the right memory structure in place. So this is kind of the second layer. So now we have the the grounding with with graph rag and retrieval and we can pull things out of knowledge graphs. But we need to now store the the memories, the short-term memory, the long-term memory,
• 6:25 — and the reasoning memory so that we have our complete history of what’s happened that we can build on top of this. Um so I think you all know that short-term memory is things which are happening in the current pipeline with agents, the conversation, the the current state of um activities which your your agent architecture is working on. So this can all be persistent in the knowledge graph and it gives important information in the execution pipeline. Long-term memory is really important and
• 6:55 — needs to be organized well cuz there’s so much of it. So you have to figure out how to aggregate and pull this information in, have a good domain model for representing the the different business processes, the entities, the um folks who are part of your application or part of your domain. And then you can actually store the information which your agents are working on over longer um tasks and procedures and across multiple different um user or customer interactions
• 7:26 — to give that history and context for what your application has done. And then finally, reasoning traces give us the ability to understand why decisions were made and how they’re done. So typically, what we get from LLMs is we we get the result, right? They’ll they’ll tell us, “Well, this is what I recommend. This is, you know, advise this.” But to get to that result, there is there is thinking, there’s reasoning which happens behind the scenes. And we’d like to make it repeatable where that information it’s learning
• 7:56 — from the experience of the previous traces, it has that decision providence for if there were previous decisions, you’re going to draw on that knowledge and we’ll use that to come up with better future decisions. And this also gives us a great hook in for compliance and debugging. One of the great things about knowledge graphs is they’re great for tooling, they’re great for LLMs. LLMs can build Cypher, which is the query language for knowledge graphs. They can create knowledge graphs and create structure out of um unstructured documents. But
• 8:26 — it’s also how we would represent things as humans. It’s how we would draw things on a white board. It’s how we would show things. And with these, now we are loading up the memory into a structure where we can actually start to do interesting things with our memory. So graphs are a a great use case for memory because relationships are first class within knowledge graphs. They’re part of the structure. It’s not like you have to join a bunch of tables together. It’s a natural progression.
• 8:58 — They’re highly performant from all the hop traversal. Um a lot of the graph rag research papers talk about this specifically as a major advantage to graph rag and graph AI architectures where they can navigate more complex structures at a very performant speed. Um using graph embeddings like fast RP, we can also do vector lookups, which is a great way to get a starting point or hook into the graph where we navigate using algorithms like the Louvain algorithm for community grouping.
• 9:28 — And then we get explainable decisions. We have more cross knowledge and we’re building asset compliant um solutions with things like the Neo4j agent memory package. So this is an open source package which we built on top of Neo4j. We have an open GitHub repo. We encourage other folks to contribute for it. And it brings these three concepts together, short-term memory, long-term memory, reasoning into a context graph structure. And the first demonstration I’m going to
• 9:59 — show is an example of how you can build a knowledge graph which ties your short-term your long-term memory and your reasoning memory together to answer questions from Lenny’s podcast. Who’s a Who’s a fan of Lenny’s podcast in the room? Oh, okay, a bunch of folks. Um it’s a great resource and um but it’s hard like podcasts are hard. They’re very dense. There’s a lot of connected information and topics and we’d like to be able to extract that and then
• 10:30 — understand more of the context and things which are happening with the help of AI. So, um we built a little demo, Lenny memory podcast, again an open-source project. Um what it lets you do is it has all the podcasts loaded up. Um for those of you don’t know, Lenny talks a lot about different AI topics, about product management. And um one of the things we can do is we provide the AI with different tools for accessing the memory. This is all written on top of the Neo4j agent memory APIs. And
• 11:03 — then, for example, we’re pulling back locations in the episode and it’s using that to design and build a graph and or in this case a a map and show us all the different locations of things mentioned in the film by aggregating all that context. And because we have it in a graph format, it’s not just pulling out some similar locations and getting like part of the data, we get a holistic view of the entire data set which can be navigated and queried dynamically. So, now we’ve shown what we can do with
• 11:34 — graphs, what we can do with memory. But, what we’re all here for is is context graphs, right? How can we take this and actually apply this to solve those cross-domain business problems where it’s very hard to get the information, it’s very hard to quantify why decisions are made. And um context graphs are really powerful for this because unlike a traditional audit log, they’re capturing the the why, the decision traces that happens while you’re evaluating your models.
• 12:05 — It organizes these by entities and relationships. And then it’s pulling all the knowledge from from different sources. So, rather than having conversations hidden in Slack or emails or other informal um conversations, now your app becomes a central point where they can look up previous decisions, they can get that advice, and then they can add that recommendation back to the reasoning traces for um future lookups. Broadly, the architecture is you’re you’re searching, you’re using your context graph retrieval tools from your
• 12:36 — agentic architecture. It’s using a combination of knowledge graphs, vector search, and um data science algorithms. Then they when you go through the agent loop, it’s then pushing that back into the context memory which gets added back into the graph. And subsequent queries are then pulling this back as part of your reasoning traces in your output to solve specific domain problems. So, what I’m going to show here as an example is a financial services application. Um for this, we’re going to have entities of different people and
• 13:07 — organizations, um different events for decisions, transactions, and approvals which happened during the workflow of the application. And then the context of why, what policies were applied, what risk factors are there, what was the employee reasoning behind giving a certain recommendation. And the architecture again, it’s an open-source project, you can try this out on we have a hosted version of this and you can try it out with the GitHub project and run locally. But, it’s pulling in from a variety of different data sources. So, we’ve hooked it up to
• 13:37 — a support ticket system, a CRM, and an internal business data system with 10 different MCP tools that it has access to. And then we’ve used our cloud agents to create open AI embeddings and then populated Neo4j context graph with a lot of this information so that it has a a domain domain graph and a reasoning graph which it can look into. And then finally, it’s exposed with a user interface which is a simple Next.js application that gives us a front end
• 14:08 — like what we’d want a an end user or consumer to to use for this particular use case. And for this application, what it does is it presents to you a prompt where you can ask it a bunch of questions. We’re going to ask it about Jessica Norris and see whether she should get an approval. And it’s going back to the graph and it’s querying both information about her history. So, it it knows what her bank account is, it knows that she has some related margin trades. You can see some
• 14:38 — of the Cypher queries there that were queried through the model. And you can also see the knowledge graph that we’re traversing and populating. So, this is why knowledge graphs make things explainable and auditable because now we see exactly the information which is being populated and used. You can see there was a previous rejection. And these are the sort of things which get which get lost in in disparate systems and tooling where we don’t bring all this information together in a in a queryable, in a understandable form
• 15:10 — where we can build and pull out that knowledge. Now, unfortunately for Jessica, the the AI model recommends not giving her the loan, but it gives us the reasons, the risk factors, it gives us previous decisions which should influence this. And um fraud detection patterns about why this may be a big risk for our organization. And as a an agent who’s or a a user, a human who’s using the system to make decisions, this is the sort of
• 15:40 — information you need to actually make a decision you can stand behind and you can justify to your organization. And then us as developers, now we can justify why our agentic applications are actually solving real business problems, providing grounded information that our users can rely on and are taking advantage of the latest techniques with context graphs which um as we know, Gartner approves of.
• 16:10 — All right. So, let me leave you with some resources that you can use to learn more. Um I run the devrel team. One of our big pushes is free education. So, we just want to help people to understand how to how to use graphs, how to use context graphs, how to use um AI. We have a new context graph course that we just released on GraphAcademy. Um also, it makes it really easy to get started because we in the background, we spin up a free Aura instance. So, you just have a graph database to play with for free. You can try a bunch of these techniques out before you even try it
• 16:42 — in, you know, your own production instance, your own enterprise instance. So, I hope you guys enjoyed the talk and learned a little bit about what the possibility with context graphs is. The next set of presenters, my my colleagues, um Zaid and ABK, are going to dig a little bit more into agentic use cases of context graphs. And, you know, please come chat with us either after the talk or at the Neo4j booth. We’re We’re happy to have conversations, kind of dig more into demos, dig more into your use cases, and
• 17:14 — help all of us to escape the matrix. So, thank you very much.