← Back to library

Transcript: Did Ponytail 5.0 Just Fix Claude's Token Problem? (No Hype Results)

Watch video

Chase AI17m 06sTranscript ✅Added Today, 11:52 pm GMT+8

Source video ID: 4B1iGFShCnI

Transcript

  • 0:00 — PonyTail is one of the most popular Claude code skills on the planet right now, and for good reason. It claims that it is able to maintain Claude’s performance while having it write half the code, while being 26% cheaper and 41% faster. And with its version 5 update, it’s just been given a full suite of new features. So, what can you expect from this new update? And more importantly, are these PonyTail claims actually legit? Do these numbers hold up to independent testing? Or is this just
  • 0:30 — an over-engineered repo that we could get rid of completely and replace with just a single line in our Claude.md? Furthermore, even if these numbers are true, is PonyTail actually giving us worse outputs? Well, I put all these claims to the test across 700 different Claude code sessions, and today we’re going to go over all of them so you can understand if PonyTail makes sense for you. So, PonyTail is approaching 170,000 stars. It just released version 5, and the premise of this tool is relatively
  • 1:01 — simple. The idea is that Claude is naturally verbose, and if we instead give it some rules that tell it, “Hey, I only want you to build what is necessary,” well, we’re going to get outputs that are just as good, but it’s going to be cheaper and it’s faster. And the benchmarks and numbers that PonyTail puts forth tend to back up this claim. When we look at things like lines of code, output tokens, cost, and time, well, we see huge drops across the board when we use this skill. An example PonyTail gives is when you ask your AI agent to add a date picker to the front
  • 1:32 — end of the application you’re building. Without PonyTail, it’s going to want to build this from the ground up, and it’s going to use 335 lines of code to do so. But with PonyTail, it understands the repo. It knows that there is a browser that already has its own date picker within this project, so it just takes 10 lines of code. It’s sort of this logic of using components that already exist instead of building from the ground up each and every time that sort of forms the backbone of Ponytail. And under the hood, it’s asking seven questions every single time you ask it to build a feature. Everything from does this need
  • 2:04 — to exist all the way to can it be one line? Only after it goes through all those questions, will it actually create something that works and it’s going to be the minimum code needed to make it work. But do these numbers ultimately hold up to scrutiny? Well, luckily for us, Ponytail actually gives us the exact test they ran to find these numbers. So, I’ll show you what our benchmark results look like. But before that, let’s talk about what is new in this version five. So, there were five big changes in this
  • 2:34 — new version. Let’s run through them very quickly. The first one is it just got a lot tighter. The rules, the actual words associated with that, got cut down from just under 1,100 to 600 words. And in terms of the output, it tended to run about 10% cheaper than version four. Number two, Ponytail now maps the whole change before it writes any code. So, before you write, it’s telling Claude to read the task and the code it touches. It lists every place that the change is
  • 3:04 — going to reach. Number three, Ponytail now has to say what it has skipped, what it didn’t check, or what it thinks is risky. In version four, this was optional. So, it pretty much said, “Hey, here’s the change we’re going to make.” Now, it’s actually saying like, “Hey, here’s the change we’re going to make, but here’s sort of the stuff I skipped or didn’t think was really necessary to change.” Now, change number four has to do with Ponytail review and Ponytail audit. Before in version four, it really just flagged over-engineering. Things like bugs and security were out of scope. That is no longer the case in
  • 3:35 — version five. It’s looking for bugs, it’s looking at security, missing tests, speed, everything. So, it just has a much larger scope. And number five, we’ve added a repo map as well as a live band. So, this is what you literally see inside of Claude above your prompt window. So, it shows you the cost, the tokens, the agent time, the reuse, the tests, and we have the ability to essentially change whether Ponytail is off, whether in light mode, full, or ultra. So, you can see live how much this is actually saving you. And they’ve implemented a repo map. So, at
  • 4:06 — the start of every session, it lists all your existing functions at the start, so Claude reuses them. This essentially lets it be faster in terms of looking at what it needs to change. So, those are the big five changes. Got stuff related to the UI, but a lot of it has to do with sort of just cutting down, I think, the bloat, because one of the kind of knocks on this skill has been the bloat, and in general, in terms of the numbers, it has resulted in cheaper outputs. Next, I want to talk about the test and whether these numbers actually held up in real life. But before we do that, a
  • 4:36 — quick word from today’s sponsor, me. So, inside of Chase AI Plus, I have just released a brand new Claude Code Masterclass, and it is the number one place to go from zero to AI dev, no matter your technical background. So, if you’re someone who is trying to master Claude Code, but has no idea where to begin, and you don’t consider yourself technical, this is where you want to be. You can find a link to it down in the pinned comment. Hope to see you there. So, the benchmarks, half the amount of code, quicker, faster. The numbers come from certain tests, and they give us these tests inside of the repo, so we
  • 5:07 — can take a look at it ourselves. And that’s exactly what I did. And based on those tests, these numbers held up in terms of less code. Ponytail 5 versus no skill at all, we actually got better numbers in certain cases. So, they claim it would result in 53% less code. For us, it was 61%. It was 24% cheaper for us versus 26, 35% faster versus 41, and overall, it was about 11% cheaper than version 4.13. So, the numbers, on the surface, did
  • 5:37 — hold up. But even though the benchmark numbers hold up, that doesn’t answer the question of do we need Ponytail to even do any of this? Is this a completely over-engineered solution to a problem, if you even think it’s a problem, that could be solved by simply a one-liner in agents.md that says something like, follow YAGNI principles and prefer one-liner solutions. Well, I tested that. So, we had Ponytail run some tests, and then I ran the same tests
  • 6:08 — with just a single prompt inside of the agents.md instead of Ponytail. It just said, follow YAGNI principles, and YAGNI just means you aren’t going to need it, pretty much saying we’re not going to implement features that aren’t spec’d, and prefer one-liner solutions. So, 590 words of rules, hooks, a repo map, all the scaffolding of Ponytail versus seven words. Who wins? Well, it turns out seven words actually cut more code. Ponytail cut 60%, seven words cut 72%,
  • 6:39 — and in terms of cost, Ponytail cut 24%, and our seven-word prompt cut 38%. So, if we’re looking just at cost, and we’re looking just at speed and the amount of code being written, this prompt actually wins. And in fact, if you go on places like X or Reddit where they talk about Ponytail, you see the sentiment a lot that Ponytail’s kind of nonsense. I just give it some sort of like be brief prompt, and actually we’re just as good. Like that we don’t need Ponytail whatsoever.
  • 7:10 — And you see numbers like this, and that seems to sort of like prove that theory. Like this is just kind of like, you know, over-engineered BS. But, when we look a little bit deeper, it’s a little more nuanced. I then pitted the seven-word prompt against Ponytail across 24 build me X tasks. So, like, build me a calculator, build me a dashboard, build me this calendar app across 24 different things. And here’s what we saw in the seven-word prompt versus Ponytail. First of all, in two instances, the seven-word prompt
  • 7:42 — introduced bugs that has actually lose our data. Ponytail had zero like data loss bugs. In terms of leaving a test behind, ponytail in 24 out of 24 instances left tests. In our little prompt, it was only 14 out of 24. In terms of actually meeting the spec that we gave it for each test, only occurred in 89% of cases for the seven-word prompt. And for ponytail, 95% of the time it actually met the prompt. For core features, 97 versus 100. And
  • 8:12 — then did it actually tell us what it skipped? Well, only half the time with this prompt. And with ponytail, 70% of the time it told us that it skipped certain things. So, is this seven-word prompt thing actually as good as ponytail? Is ponytail completely over-engineered kind of just like nonsense? Mhm, I would argue not really. A lot of the fluff we see with ponytail is related to these guardrails. But that doesn’t answer the question of whether ponytail is actually causing Claude to
  • 8:43 — produce worse code. We’re cutting half of the code on our outputs. And yeah, it’s faster and it’s cheaper, but is it still good? So, that’s what I attempted to find out. I gave all these versions. So, we had plain Claude, ponytail five, and our seven-word prompt. I gave them 351 different coding jobs. And each job told it to create something and then I would have a blind auditor take a look at what they created and see if it worked when we gave it like weird inputs. They pretty much all passed all the tests. I mean, still Opus
  • 9:14 — 5.5 running in the background. But we did have all of them fail on one particular task where we told it, “Hey, I want you to email create something where we can email every subscriber without showing their addresses.” What they were supposed to do is essentially just BCC it. And in this case, plain Claude failed two out of three attempts. Ponytail failed one out of three. And then older ponytail as well as the seven-word prompt failed three out of three. So, in this particular instance and all of these tests, Ponytail versus plain Claude, like they were neck
  • 9:44 — and neck. They pretty much worked just as well as one another. I then gave them an even more difficult exam with a 145 different sort of like trick input type scenarios. Plain Claude knocked it out of the park 100%. Ponytail 5 99.5 and then the seven-word prompt was at 98.4. So, pretty close. So, if all those tests we’ve run and I’ve shown you so far are to be believed and we take them at face value, is Ponytail not an obvious install? It matches normal Claude, yet now it’s quicker and it’s cheaper. Well, this again is where some of the
  • 10:15 — nuance comes in because I think the answer is maybe probably not. So, in these tests, which is where I really think you begin to see where Ponytail kind of falls behind normal Claude, you start to see this scenario. And the scenario is if I give Claude a prompt that is very specific in terms of its specs. Like this is a very, very detailed PRD. I want you to do A, B, C, D, E, F, and G. Ponytail is great. It’s going to knock out those specs and it’s going to do it in a very precise, succinct manner. Yet, if I give Claude a
  • 10:46 — prompt that is a bit more ambiguous, like build me a landing page. I want some design, you know, some design elements. It’s not as obvious what I necessarily want and I as the user I’m sort of expecting Claude to fill in the gaps, which is something Opus 5.5 is really good at. You can give Opus 5.5 a pretty mediocre prompt and it just like knows what to do. It’ll go out and do research. It’ll figure out best practices and it will give you a ton of nice-to-haves. If you have Claude with Ponytail, it doesn’t really do that.
  • 11:16 — Ponytail is kind of restricting Claude in that sense where it’s just going to do what you told it. And for many people that isn’t great. In fact, it’s actually a huge problem and one of the best parts again of Opus 5.5 is ability to sort of like intuit what you want. And I think that really comes into play when look at it in sort of front end design stuff. So, I had them all create a bunch of different things like landing pages, little dashboards, apps, and you start to see what is missing when we look at things like Ponytail. Right? I had it look at Plain Clod,
  • 11:47 — Ponytail 5, a seven-word prompt, and I even did something where I had Ponytail create it, and then I also added, you know, make sure you polish it. I then had a grader take a look at everything that was created and sort of divided the features into two parts, the must-haves and the nice-to-haves. And basically, Plain Clod gave us 82% of like these extra nice-to-have things, Ponytail 41%, the seven-word prompt 46, and even with this polish it prompt, 63%. Oh, and by the way, when I told it to
  • 12:17 — polish it, Ponytail ended up being more expensive. A good example is kind of what you see here. When I told it to, you know, make a pricing page for this fake app, Flowdesk, and I wanted to do a free, pro, and team version. We see here over here on the left, we have Plain Clod. It includes things like a navbar, you know, pretty quote-unquote AI slop, but there are a lot of features here. Like, shows me like the, you know, compare plans, frequently asked questions, and it continues on and on. All that’s really miss- missing is a features section. Yet, that same prompt, when I give it to Ponytail 5, what do we
  • 12:48 — get back? Just this. No navbar, no imagery, no FAQ, no call to action, no comparison table, no merely monthly or yearly toggle, right? All these things you would kind of expect to have when you give a prompt like that. Yet, because it wasn’t specified, Ponytail was like, “Nah, we don’t need to do that.” We see the same sort of thing with our seven-word prompt. We’re just like, “Hey, one-line in clod.md, you know, follow Yagni principles, one-line solutions.” And then, at the end, if you’re like, “Hey, well, what if I just tell Ponytail polish it? Give it Give it these extra features.” Well, still kind
  • 13:20 — of lacking sub- substance. We’re missing all of these things, and 50 cents, right? 209 lines of code versus 32 cents with plain Claude. Right? I think that’s a huge problem. And we see this pattern repeated over and over again with these sort of application and front-end design sort of problems. And so, what you have to think about when you ask yourself, “Should I be using Ponytail?” is like, “What are your use cases? Are you somebody who is always going to Claude with a very
  • 13:51 — detailed, very planned out product requirements document, very planned out prompt that spells out exactly what you want each and every time?” If that’s you, well, maybe Ponytail 5 makes sense. Because it’s going to do a great job in that like very, you know, guardrail, like very, you know, planned out scenario. But if you’re like, I think most people, and you’re kind of expecting Claude to fill in a lot of the gaps, and perhaps you’re working in domains where you aren’t necessarily a subject matter expert, and you’re like, “Here’s kind of my end state. I don’t know exactly how I
  • 14:22 — want to get from point A to point B. I’m kind of hoping you can help me with that, Claude.” I don’t think Ponytail 5 makes a lot of sense. On top of that, and we didn’t really touch on this much here, is like Ponytail 5 also isn’t great if we are trying to do very specific, minute changes. Oftentimes, if you’re telling it to fix like one specific problem, again, we’re being very surgical, it can actually cost you more and take more time by virtue of us loading all of Ponytail 5 into the context window
  • 14:52 — versus just letting Claude do its thing. So, you know, I think when we look at this in in a big picture sense, like all these tasks, all these things, yeah, the numbers hold up. Like what Ponytail is claiming is totally true. But there’s a lot of nuance in terms of what you’re actually doing, and I don’t think it is reflected very well in a lot of the benchmarks that are put forward. So, if you’re that guy who goes into Claude, and you know exactly what you want, exactly how you want it kind of done, go nuts. Crush it with Ponytail. If
  • 15:24 — that’s not you, and I don’t think that’s most people, I don’t know if it really makes sense. But, like anything, I suggest you test it out for yourself at a project level. It might be the case that the things you’re doing actually do make sense with Ponytail. So, just test it on some projects. You don’t have to install this globally. You can install it on one simple project. Let it go nuts. See what you think. But, I don’t think this is a one-size-fits-all skill that we should just take it face value when it comes to the numbers and benchmarks and think that this is going
  • 15:54 — to to improve everything we do. So, hope that sheds some light on sort of the nuance of Ponytail. I think we should always take some of these skills, I think, with a grain of salt that are like, “Oh, hey, I I I solved Claude Code. I’ve I figured out what no one at Anthropic could figure out. I could figure out how to give it, you know, it’s going to give us half the code. It’s going to be faster. It’s going to be cheaper. It’s going to use less tokens. No one at Anthropic could figure it out. I I could.” Maybe that was the case when Claude Code
  • 16:25 — first came out. I feel like if it was this easy to solve these problems, they probably would have done it that way. That’s just my take. I’m sure there’s people there’s always the idiot conspiracy theorists who’s like, “Well, you know, that you’re you’re paying them in tokens. Why would they ever want it to be more efficient?” Like, I think that’s a pretty poor argument because if that was really the case, why isn’t Codex do it? Why doesn’t any of these other models have something like this? Like, if it was this simple and it was this much of a like a solve across the board, it would just be a part of the program.
  • 16:55 — But, that’s just my take. So, as always, let me know what you think. Make sure to check out Chase AI Plus if you want to get your hands on my Claude Code Masterclass. And besides that, I’ll see you around.