Transcript: CrabRAG: Why Automated Assistants Need Graph Memory, Not More Tokens — Stephen Chin, Neo4j

Watch video

AI Engineer20m 42sTranscript ✅Added Jul 30, 9:47 pm GMT+8

Source video ID: Q0VkgCyNVUg

Transcript

0:01 — [music] » My name’s Stephen Chin. I run the developer relations team here at Neo4j. And I’m excited to talk to you about something we’ve all come to love, our our crustacean friends. So, we have um um open claw mascot. We
0:33 — have a bunch of other crustaceans, and we’re going to we’re going to focus on one member of the crustacean family. I I I love crabs, so our little boy Crab D. And I think in the in the journey to to figure out how to apply agents, how to do things which are more autonomous, we’re all looking for ways where we can get better results, more accurate answers, and to actually capture all of this. But, the tools kind of work against us. So,
1:03 — um here is our our friend Crab D. He’s a he’s a personal assistant. Very happy, very eager. He wants to to help us out with our lives, maybe to help us to code, to help us to, you know, manage our email, to do different things. But, he’s got a problem. And our poor boy Crab D has a very bad memory. He wakes up every day, and his memory file flips, and now it’s a new day, and he forgets everything from yesterday. Does this happen to you
1:33 — where you you wake up, and you’re using open claw, and suddenly it’s on a new set of memory files, and remembers nothing that you actually did the previous day? He’s got a lot of tools at his disposal. I mean, we love giving our agents tools, but sometimes he doesn’t pick the right tool for the job. I don’t think either of these are going to help him drink his his bowl of soup. So, that’s not the tool which he was looking to to reach for.
2:03 — And a little bit forgetful at times. So, you know, I think I don’t remember everybody I meet, but I’m pretty good at faces. Like if I’ve if I’ve met you before, I recognize faces. It’s like pleased to meet you. Um Krabdi is not as good at that. So, very forgetful. It’s like you’re re-teaching it every day to do the same sort of tasks and we want agents which are more helpful, which are able to do more for us. So, let’s dig into how Krabdi actually works. So, it’s basically a a memory loop, right?
2:33 — So, we’re we’re prompting, we’re thinking about the response, maybe calling tools, observing what happens. But, the hard part is the memory. The hard part is what you put in context, what you’re recalling from. And the way which you have memory structured in most tools is an example of um how Open Claw structures things. Is you have a solved at MD for your agent’s memory. You have maybe um memory files. You have different tool files. You have daily memory files.
3:04 — Now, if you look at this, there’s one thing which is in common with all of these, they’re just markdown files. So, mark markdown files are great. That’s easy for us to read. Like we can we can look through it. We can quickly figure out what’s not needed and compact them. Um they’re intentionally small for agents because you have a limited context window and also you need to keep the right things at the top of the context. But, if your whole memory is a bunch of markdown files, you’re wasting a lot of tokens.
3:35 — So, um my my average agents are loading up at least 100k in tokens for each round. Um they’re doing a lot of skills. They’re adding a lot of things into the context constantly. It’s very repetitive because they they basically load up everything in the hopes that something will be useful in the context. At small scale that works where you get the results you want with a high quality model. It doesn’t work at large scale and I’m going to show a demo of large scale where
4:06 — we take open claw and we let it run loose on my home lab. So, um high demo risk, but a lot of fun and um a classic digital twin scenario. So, I think we’ll have we’ll have a lot of fun here. Um anybody use Hermes agent at all? Okay. I’m a big fan of Hermes agent. Um I think it’s got a much better memory system. It kind of at the end of each task it goes and it reflects and it adds back in new skills or new things which it needs. So, um it’s a really powerful
4:38 — system. And um you know, again, we’re relying a lot on markdown files. Skills are just basically markdown files. But we can teach the agent to do a lot of things with skills and it can it can get the right skill if it gets loaded up and then good things happen. But sometimes we don’t get the right skill loaded up. So, our our poor boy crab D here is not going to get that clam. He just doesn’t have the open clam shell skill. Lots of shrimp, no clams.
5:10 — Maybe you pick the wrong skill for the job and suddenly you’re you’re jet skiing on the on the beach. Right? This this is not this is not going to get him very far. And sometimes you you might get that clam open, but then you don’t have the skill to eat them. So, skills you need to have the right skills, the right chain of skills. Um actually we have an awesome project by one of the Neo4j folks which is just Bennet as an Arvix paper which is a
5:41 — graph for skills. So, that’s an exciting way of like like figuring out what the right skills are. But, maybe we can do better. So, um Goose is a project that’s part of the um Agentic AI Foundation. It’s a new foundation which MCP is part of. Um Anthropic is backing this. We’re We’re also a member of this. So, it’s a it’s a great automation tool for a lot of enterprise workflows. You can also use it kind of like a personal assistant. It relies heavily on MCP as the layer.
6:13 — Over 70 MCP extensions. And what it does is it treats memory just like another MCP server. So, this is great, right? It’s It’s pluggable. You can call different commands on it to retrieve memories, remember memories, um forget memories. Memories are just plain files on disk. So, now you can manipulate them. So, same great idea, same fundamental problem. We’re We’re storing the memory. We’re storing the memory of agents as
6:43 — markdown files on disks. And again, you end up with what if What if you pick the wrong tool for the job, the wrong paddle? Now, in this case, if you pick the wrong paddle, you’re genius cuz you’ve invented the most the fastest rising sport in the US, which is pickleball. Um actually, the origin of pickleball was was um a family wanted to create a new game and they just took what they had around the house, a badminton court,
7:13 — and um made up the rules along the way. So, creation can be good when you have the wrong tools. Maybe you remember everything, but it’s too much. It’s too much weight because you can’t actually solve the problem. So, our poor friend Goose here is encumbered by too many notes, too many memories. Or, most dangerously, now you have MCP tools, you’re one step away from calling the forget command and just wiping out your own memory.
7:44 — Okay, so we want to be able to do better than this. So, vector databases, right? So, we can store everything, we can create embeddings for it. Now, we actually have a a database, we can store it in a vector database. So, this is great. I mean, you have to pick the right vector database. Um and then now you can do similarity searches, so you can pull back information which is which is relevant. So, we’re doing much better. We have a large repository of knowledge, we can pull back related information.
8:15 — Um Open Claw comes with PGVector out of the box. Given an embedding, you can just start using this. Um LanceDB is a great option. I’m going to use both of those in my demo. But, the challenge here is similar what what vectors give you, which is similarity in vector space, is not the same as actual relationships. And so, you get hallucinations, you get a lot of problems when you’re relying solely on vector lookup as the answer. And it compounds with more complex
8:46 — scenarios when you’re doing things like like I’m going to show you an example of a digital twin. When you’re doing things which are very complex, they they just don’t scale and you make silly mistakes like this is obviously is not what poor crab D wanted to munch into and it’s a very expensive lunch for him. Also, it’s sometimes impossible to get to the answer even though you have all the facts because those large multi-hop reasoning chains
9:16 — don’t work on similarity searches. They’re also very expensive on traditional relational databases. And often things look similar but they’re not exactly the same. And this is one of the problems with the responses you get from a vector database is you suffer from getting facts which are related in some way and they’re not your shell. And you don’t you don’t want to take the wrong shell out of the locker room. That’s that’s very unfortunate. So,
9:47 — enter graphs. Graphs are a great way of finding the relationships, finding those identities, mapping out the paths, getting that full chain. And they’re built for this sort of connected data. So, now that you have first class nodes, which are the the circles, edges, those are the relationships between different objects, and then you can put properties on top of graphs to store information. You can also store embeddings in your graph and that gives you a way to both use vectors and graphs
10:18 — together. Um architecturally, the demo I’m going to show you is um both a vector search and a graph search. So, it uses the vector search to get the seed nodes where it starts the traversal. And then it uses a graph search pulling the the nearest neighbors and then ranking those by how related they are. And this gives you this complex multi-hop queries to solve more difficult, more domain-specific problems and to figure out where that where that reef
10:48 — is that we want to get to with all the the tasty um the tasty junk food across the ocean. And graphs are they’re accurate. So, they give you very precise information. Explainable because you can look at the graph which got returned. And auditable because now you can actually say these are the this is the context. This is the part of the graph which resulted in that answer. So, it’s very powerful and it gives you more tools as a developer
11:18 — where if you’re not getting the right answer, you know where it’s coming from. You can actually see and introspect the graph and you can change how you’re doing extraction, you can reduce duplicate nodes in the graph, and then you can get to and converge very quickly on a great answer. If you’re not a graph expert, guess what? Claude is. Claude can write Cypher better than I can. Claude can extract build entity extractors and it can do pretty much everything you need to do to get started with graphs today
11:48 — as long as you know the the basic kind of model for what you want to accomplish. And that’s what I’m going to cover in the demo. So, we’re going to do have Claude write each action into the graph as he works. We’re going to follow up by traversing, not re-reading it. And then in a fresh session, we will get the results we want to get out. Now, what I did for this um high-stakes demo is I over the past week or two, I took my home lab as the demo environment, did a full digital twin as a graph and I have two separate
12:19 — environments built off the same original markdown files. One is a vector database store, that’s our our A test. And the second is a graph store, that’s our B test. And the graph store is built on top of um Cogney. So, I’m using Cogney, which is a startup. Um they do amazing stuff in the memory space. They’ve a Neo4j backend. This is my the structure. So, we have a bunch of Proxmox servers in my my home lab. It’s really a couple computers
12:50 — around my desk. And I built a separate VLAN for the demo, so it’s segmented off my real network. So, it was trained on real network for my network, but now it’s it’s cut off. It can only answer from memory. It can’t actually look up the hosts and I can’t get dynamic information. So, let’s see how it does in a live demo. Okay. So,
13:23 — All right. Here we have our our Grab Rack cockpit. Um and I have five different questions queued up with schematics. You can see this is the same home lab schematic that you saw earlier in the slides. And um let’s let’s start with this one. So, WRN exposed end-of-life soft when exposed end-of-life software. So, we’re going to basically we’re going to try to find out if there’s anything on my network which is exposed to the network, the the internet, the WAN that’s running
13:53 — out-of-date software which put my home lab at risk, right? So, if if somebody can attack the home lab. And um you can see here that [clears throat] there there is some servers um Tinkster Lands, which is my daughter’s Minecraft server. It’s running Oh my god, Debian 8 Jessie. And let’s see how the the two agents did in looking this up. Okay. So, we got the vector response back. Couldn’t find specific details. Excluded by policy for more precise information. Yadada yada. Source is
14:24 — separately. Okay, that’s not that’s not very helpful. Now, on the graph side, it’s done a bunch of Cypher queries. Here are the Cypher queries it’s fired off. Um this is the graph traversal. And the the color coding on the graph traversal is these blue guys. These are the seed nodes. So, this came from a a vector look up and a ranking, but it didn’t stop there. It does the one hop traversals. Those are all the gray nodes. Some of the nodes get highlighted in green and those are the ones which which
14:55 — won and got into context. And you can see the answer here. So, guest name Tinksterlin, exactly as expected. Um OS version out of date and it’s flagging So, so it gives us very precise actionable information. And so that’s the difference between same same exact data. One is a vector store, one is a graph store, and you can see the difference where the the vector store is having a lot of trouble pulling the information out the relevant information out.
15:25 — Okay, let’s try another one just for fun. Um Let’s see, expose 0.0.0.0 management ports. That’s that’s bad. So, um basically you don’t want your management ports on the network exposed to the, you know, the world. And there’s a bunch of these. So, I have a a new Matrix server I set up and also HAProxy which are exposed to the internet. That’s bad. The rest of these, like my Cogny demo, my Open Claw instance, those are inside the LAN. You
15:56 — need to get into the LAN to access them. That’s That’s what you want. Okay, and let’s see how the two agents did in identifying this. So, the memory search returned some information and it’s telling me check services configuration expects pfSense rule. So, it told me to go do the job for it. Um okay, and then on the graph memory side found an open port exposed to WAN, HAProxy and OpenVPN, which are the are the two we expected. Now, this you can see the
16:27 — shape of this graph is entirely different from the previous one. And what it did is it it actually found the node for for my router, the pfSense router, and it was able to follow that directly to all of the results which related to it and then give us a like a very precise answer. All right. So, um
17:00 — So, now So, now we’ve seen our little boy Krab D with his certified Neo4j developer t-shirt is able to do a lot more, right? Now, he’s able to follow that full chain, crack, eat, do the next thing. So, he’s getting his he’s getting his clams. He’s helping me fix all the security holes in my network. Um oh, by the way, I I patched all those security holes after the demo. Um so, this was good for me, too. It found a bunch of security holes in my home lab, and then I I went and patched them later.
17:30 — And um now we have an agent which actually can do interesting things. Now, if you can imagine like I have a three or four node home lab at home, if if you have a big enterprise which has a huge data center, if you’re doing things in financial services where you have like a huge set of companies and customer records you’re trying to do, if you’re doing anything at at large scale where it doesn’t fit into the 1 million context window of the modern models, you really need a better memory system than just throwing
18:00 — things in markdown files. Our little boy crab recognizes all crew, all the crustacean friends. And he’s read the book. Um so, we just finished my my co-authors and I, Michael Hunger and Jesús Barrasa, finished Graph RAG the Definitive Guide. The full book is out on on early release. It’ll be published um in a couple months once they finish the editorial process, but super excited about this. It’s got information not
18:31 — only on Graph RAG, but also on building memory, on different industry vertical use cases, on agents. So, it’s kind of the the whole umbrella if you’re building on top of graph solutions, how you need to build applications, the technologies you need end to end. And then finally, a great free resource which everybody in this room can take advantage of is Neo4j um’s Graph Academy. So, it’s free online training, um
19:02 — dev.neo4j.com/ga-rag or the QR code below there. And um we have courses on doing agent memory, doing context graphs, and everything you need to get know to get started and to do some of the amazing stuff which I showed you on stage today. So, thank you so much for coming to the the kickoff talk for the graph track. » [applause] » You’re in the right place for all of the
19:32 — content from graph experts. Andreas Kollegger, my colleague and I crafted a great set of speakers from industry to experts. People really know about graph technology. So, hang out here, find out more, and then you can see me in the Neo4j booth. Thank you.
20:03 — » [music]