Imagine kicking off a refactor before bed and waking up to code that compiles, passes tests, and is already committed. No babysitting. No prompting “try again”. Just progress.

That sounds like the promise of autonomous AI coding we’ve been hearing for years. The difference is that this version may actually work.

The trick isn’t a new model or a breakthrough in reasoning. It’s something much simpler. People are calling it the Ralph Wiggum approach, created by Geoffrey Huntley.

Why We Need This - The Weakness of LLMs

If you’ve spent time with AI coding agents, you’ve probably hit the same wall I have. The model starts strong, then slowly degrades. It forgets constraints. Reintroduces bugs it already fixed. Works great for new projects but struggles as the codebase grows.

Better agents and models like Claude code and Opus help, but the problem remains.

These issues are related to fundamental limitations of LLMs. In my post The 4 Ways LLMs fail, I describe how LLMs struggle when given too much context. Benchmarks like NoLiMa show that accuracy degradation. It doesn’t matter if the input document is too large, you’re calling too many tools, your agent has been looping and adding too much context, or your codebase is growing in size and complexity. It’s all the same issue. And it’s been popularized as context rot.

This is why the concept of context engineering has burst onto the scene. It’s critical to give an LLM the perfect amount of context to do its job and nothing more (avoid that dumb zone!).

So to code in the most effective way, you need to engineer the context you provide your coding agents to be super precise. You need to keep your tasks tight, crispy defined, and with very clear success criteria (you know…tests).

And if you do that, you can keep running those tasks in a loop until they are complete. And if you have a number of tasks defined? You can have those tasks running while you sleep. That’s the Ralph Wiggum approach.

The Ralph Wiggum Approach

The name comes from the Simpsons character Ralph Wiggum. He keeps trying, no matter what. That’s kinda the whole philosophy.

“In its purest form, Ralph is a Bash loop.”

Geoffrey Huntley, creator of Ralph

Wait, are you really just running your coding prompt over and over and over until it’s done? Isn’t that just the “try again” approach in vibe coding?

Well not exactly. It isn’t just a prompt in a loop. Because if you did that, you’ll still hit those context rot issues. And your prompt might loop forever fixing one issue and introducing another…

What It Actually Is

The Ralph Wiggum approach requires a healthy amount of upfront work. You’ll want to break your work into smaller work items.

Think about the world of your team working on a 2-week sprint. You spend the first day documenting all the tasks that need to get done (and the smaller the task the better). Then each developer grabs a task, works on it, and when done checks it in, and once all tests pass and it’s committed, they move onto the next task.

That’s the right way to Ralph Wiggum.

In practice, it looks something like this:

• Give the agent a clear, bounded task
• Define a hard completion signal (tests pass, specific output, etc)
• Let it loop until it succeeds or hits a cap

Overall progress also doesn’t live in the ongoing agent conversation. It lives outside the model. You write progress to a file in the filesystem and use git commits. Then, each iteration can reference that info and start over in a clean context window.

A clean and precise context window will guarantee the most success out of your LLM coding agent. That’s why this approach works.

Why This Matters - The Future of Software Engineering?

What’s interesting about Ralph isn’t the loop itself. It’s what it represents.

Software engineering isn’t just about the code. There’s a whole lifecycle that starts with understanding your users, figuring out requirements, deciding the UX design, thinking about architecture, finally getting it out to your users, etc.

As AI starts to change how we do coding, it puts pressure on the left side and right side of the “coding” step in the product development lifecycle. And this is especially true as coding becomes more autonomous using techniques like Ralph.

Teams that are deliberate on that left side in defining the right inputs will get the better outputs. Think better PRDs, better specs, and better exit criteria.

And teams that are deliberate on the right side in understanding how to validate the outputs will succeed in delivering more solutions faster. Think robust testing cultures, automated and comprehensive QA, and strong CI/CD practices.

Autonomy may not come from bigger prompts or longer context windows (or looping). It will likely come from process, culture, and being deliberate.

The craft is changing. We’re moving more towards the world of farming. We’re going to need to obsess about the inputs, the architecture, the users, and then understand how to validate if it works. And if it doesn’t, have a learning culture, adjust our inputs, and try again!

Is the future engineer more similar with a farmer?

Conclusion

The Ralph Wiggum approach works because it respects the limits of today’s models instead of fighting them.

Short contexts. Clear goals. Relentless iteration.

Write a good spec. Start the loop. Go to sleep.

If you did your job upfront and were deliberate in inputs and tests, you might wake up to working code.

And a very different way of building software.

Everyone’s chasing smarter AI. But the real breakthrough won’t be intelligence. It’ll be reliability.

Over the weekend, Andrej Karpathy’s conversation on the Dwarkesh pod made waves when he said AGI is still at least a decade away.

His reasoning came from experience. Years ago, the first self-driving car demos looked amazing. Cars gliding through cities, reacting to obstacles in real time. I also remember seeing those early GoogleX SUVs near the LinkedIn campus in 2013.

Yet a decade later, the work continues. In fact, I’m writing this inside a Waymo that just told me to get out in the middle of an intersection.

The progress has been less about demos and more about grinding out reliability and predictability.

We’re now in the same moment with AI agents.

The productivity unlocked in just the last couple of years is undeniable. I have so many go to prompts and agents helping me daily. I’m loving to code because of how fun and easy it has become. But the road from “impressive demo” and “babysitting your agent” to “production-grade reliability” requires hard, unglamorous engineering work.

It’s easy to make something work once. The hard part is making it work every time.

Reliability Is the Real Frontier

While the stakes with AI agents might not match those of autonomous vehicles, the underlying challenge is the same: reliability. Automating parts of your work or life still demands rigor. Companies can’t afford hallucinations in customer support, errors in financial workflows, or security breaches with MCP or AI browsers that get tricked into exposing your data.

Most of today’s limitations trace back to the structural weaknesses of LLMs. Many have written about context rot, security vulnerabilities, or tool misuse. I’ve also explored this in The 4 Ways LLMs Fail where I break down how these weaknesses create real-world reliability issues.

The 4 Ways LLMs Fail

AI still remains incredibly useful today for lower-stakes tasks and keeping a human in the loop, but the next wave of innovation will come from teams focused on high-stakes reliability. Building agents that don’t just “work sometimes,” but work nearly all the time.

As Karpathy said, it’s going to be a march of nines: going from 90% to 99.99% requires a lot more engineering effort, testing, and iteration.

Borrowing from Decades of Software Engineering

The good news is that we already have a playbook. The path to reliable AI will look a lot like the evolution of modern big data, web-scale, and distributed software systems. Many of the same ideas apply:

• Evals-driven development - validation loops and metrics for every model and retrieval pipelines, similar to test-driven development
• Agentic orchestration patterns like planning, parallelization, and merging - reminiscent of service oriented call graphs or MapReduce.
• Sub-agent modular design - agents focused on narrow, well-defined tasks, like microservices.
• Context engineering - where we can pull out the most relevant context from memory or underlying search engines
• Query understanding and routing - sending tasks to the right models or tools.
• Reliability engineering - retries, tracing, observability, auto-healing infra.

These aren’t new ideas. But they’re being rediscovered and reinvented in the context of AI systems. For example, here’s Anthropic defining a very familiar looking multi-agent architecture or famed ML researcher Andrew Ng introducing a MapReduce-style parallel agent approach.

A resilient service architecture is looking very similar to a resilient multi-agent architecture

The teams that bring that disciplined software engineering to AI will win the next decade.

The Paradox of Agent Workflows

AI agents excel when the task is deeply personal or open-ended. Essentially, the long tail of use cases.

But if you want to automate something your users do over and over again, building it the “old-fashioned way” might still be the right choice (today at least). Traditional software can guarantee the nines of reliability that agents don’t have yet.

Karpathy summed it up perfectly: “A lot of times, the value that I brought to the company was telling them not to use AI.”

The Next Breakthrough in AI? Software Engineering

We’re in a phase where agents are already unlocking big gains for coding, creative, or custom one-off tasks, especially in lower-stakes situations with clear evals and humans in the loop. But making them trustworthy and enterprise-grade will be a defining challenge of the coming years.

Sooner or later, we’ll be back to calling it “software engineering.” And there’s nothing wrong with that.

The good news is that AI has a lot of momentum and can unlock more and more use cases as it matures. And with increasing amount of autonomy (unlike say self-driving cars). Low-stakes today, medium-stakes tomorrow, highest-stakes in the future.

And the work to address this has already begun.

Anthropic describing the different techniques of "engineering"

“Prompt engineering” started as the first approach in making AI behave more reliably. Then we morphed to “context engineering,” realizing passing the most essential information to LLMs increases accuracy and predictability. Now, we are off to “agent engineering” that thinks about task break-downs, sub-agents, and tight feedback loops and evals to control reliability.

Sooner or later, we’ll be back to calling it “software engineering.” And there’s nothing wrong with that.

Large language models (LLMs) and AI agents that use them often get lauded as magic. But anyone using them in production or serious applications quickly learns they have Achilles’ heels.

What we commonly call hallucinations, workslop, or security vulnerabilities are not random bugs. They tend to cluster into a handful of recurring failure modes. In my experience, you can trace most of these breakdowns to four core weaknesses in how LLMs and Agents are built or deployed.

There is a lot of exciting research underway trying to tame these failure modes. But even as new ideas filter into deployed systems, many predictions that “hallucinations will disappear soon” are overly optimistic. These weaknesses are deeply structural and continue to require a lot of systems work like context engineering to solve.

Let’s go through those failure models and what to do about them.

The 4 Failure Modes

1. They Get Lost

With every new model release, AI labs love to tout their ever-growing context window length. Gemini for example, was the first model to ever support 1 million tokens. This created a ton of excitement. You could load the entire Harry Potter books into the prompt itself and ask any question! Because of this, a lot of people thought that retrieving and selecting individual chunks of information (also known as RAG) would be a thing of the past. Everybody was starting to talk about CAG (Cache Augmented Generation), where you could just throw everything into the context window and you were done.

But with all that hype came serious reality. A lot of research showed how accuracy quickly would degrade as the context window fills up.

Take the NoLiMa benchmark: it tests models on a “needle in a haystack” scenario where the relevant fact (the needle) has minimal lexical overlap with the question. As the distractor “haystack” grows, performance degrades sharply. Even modern reasoning models, and the most powerful ones, still degrade. And the newer NoLiMa-hard benchmark is showing that those are even failing.

Now, most users don’t start chats with massive context, but everyone has noticed the drift that creeps in during long, multi-turn sessions. Accuracy degrades, the model loses track, and the common fix is to “start a new chat.” That’s because LLMs struggle to reason over ever-growing context. It’s the same problem. Over time, as more is added, the original goal gets buried. This term has been popularized as context rot.

Growing context windows can poison their own knowledge leading to context rot

Context rot is a large reason why very complex agents struggle. They continue to work on their own doing these multi-turn exercises, filling up the context window and drifting from their original goal.

What to do about it:

• Be ruthless in context curation. Don’t dump the entire conversation history or every document you retrieved. Select only what’s relevant.
• Use retrieval-augmented generation (RAG) or agentic retrieval so the model only fetches the snippets it actually needs at that moment.
• For chat / agentic systems: employ context summarization and compression techniques. e.g. generate concise abstracts of prior exchanges, store those instead of full logs. Keep in mind a lot of research is actively working in this area as it’s still very finicky. We’re more or less trying to replicate how the human brain memorizes.
• Adopt modular / hierarchical agent structure rather than a “one super agent loop.” Let a planner LLM decide subproblems, then dispatch to sub-agents with narrow tasks and tight contexts. That keeps each agent’s memory and context footprint manageable. Think MapReduce for agents.

2. They Are Gullible

LLMs are at their core, pattern learners, not truth verifiers. When you feed them context, even wrong or misleading or malicious context, they often treat it as an authority. That means if the retrieval returns incorrect or outdated documents, the LLM will happily repeat them. That also means they are vulnerable to prompt-injection. In short: garbage in, garbage out - a common problem in software development.

One of the more notorious examples was with Google’s earliest AI overviews in search. It once recommended adding glue to pizza because a joke Reddit thread made it into the context. Because the LLM treated that context as fact, it repeated the absurdity.

From Google's now infamous AI Overviews launch

Models have improved tremendously, especially reasoning models that can reflect more on its output. But this still remains to be a challenge overall. LLMs need to understand if a source is reliable, trustworthy, and accurate. It might be able to understand and throw out something that’s wildly off, but subtle mistakes won’t be caught. For example, even GPT-5 will repeat an incorrect fact if provided an incorrect fact.

Even GPT-5 repeats plausible but inaccurate information passed to it

Another dangerous part of this failure mode is prompt injection leading to data exfiltration. Because LLMs and Agents will follow what they are asked to do, they can be tricked to go off course, extract some sensitive content, and post it publicly. A recent prompt injection attack against Notion where hidden white on white text in a PDF told the agent to gather confidential data from other pages.

The root problem: LLMs lack an intrinsic fact-checker. They default to completion, not verification.

What to do about it:

• Invest heavily in retrieval / search quality. Rank by trust, recency, authority. Don’t just fetch “something relevant”. Search isn’t a side quest in AI apps. It’s the main quest and you only earn the right to expand once you do it world-class. Read more about advanced techniques to improve retrieval.
• Always cite sources or return the provenance of snippets. Use prompt engineering to make the model flag “these statements come from document X.” That gives you post hoc auditability. Check out my post about building an AI answer engine like Perplexity to see how to prompt for citations.
• Use cross-checking or consensus models: for important outputs, run the query through two retrieval chains or two LLMs and compare. Divergence is a flag.
• Apply fact verification or consistency modules downstream (e.g. specialized verifiers) to validate the answer before presenting it to users.
• Treat all external inputs as untrusted and defend with a layered, zero-trust pipeline. Sanitize and filter what goes into the context, restrict what tools the model can touch, and audit and monitor outputs for attempted data exfiltration.

3. They Are Overconfident

One of the most dangerous failure modes of LLMs is their inability to say “I don’t know.” Instead, they default to completing the prompt — even if that means confidently fabricating an answer. That’s the essence of hallucination.

OpenAI’s recent article on “Why Language Models Hallucinate” suggests most evaluations measure model performance in a way that encourages guessing rather than honesty about uncertainty. Research backs this up: studies like Characterizing LLM Abstention Behavior in Science QA and ConfuseBench show that even state-of-the-art models struggle to abstain when evidence is missing or ambiguous. Meta’s Abstention Benchmark confirms the issue persists at scale, with little improvement from larger models.

What to do about it:

• Prompt explicitly for uncertainty. Encourage the model to respond with “I’m not certain” or “I don’t know” when evidence is insufficient (though results may vary).
• Clarify capabilities by defining what the model can and cannot do. When asked something out of scope, guide it to suggest alternatives.
• Train for abstention and fine-tune with abstention-aware objectives that treat refusal or uncertainty as valid outcomes.
• Use structured uncertainty checks. ConfuseBench recommends multi-step diagnosis: clarify or re-retrieve if confidence is low. While SteerConf helps align generated confidence with actual uncertainty.
• Improve retrieval quality (like before). Better, more trusted context reduces guessing and increases overall reliability.

4. They Get Overloaded With Tools

As you build more ambitious systems, you may want your LLM (and more specifically an “Agent”) to call on many APIs, tools, or functions (search, translation, summarization, databases, code execution, etc.). But more is not always better: giving an LLM access to a large toolbox can overwhelm it. It might choose the wrong tool, misuse them, or simply ignore them, effectively “paralysis by analysis.”

Recent work on tool use calibration highlights exactly this: many LLMs exhibit tool-abuse behavior. They overconfidently misuse tools or issue useless calls. A paper called SMARTCAL addresses this by letting an LLM self-evaluate tool calls and calibrate use. On benchmark tasks, it reduced tool misuse and boosted reliability somewhat.

There’s been a lot of hype around models that could combine “hundreds of APIs or MCPs” (e.g. the MCP / super-agent approach), but that hype has cooled as we see the complexity cost: too many choices, too many errors, too fragile routing logic.

MCP is an exciting standard for LLMs, though its robustness is still in progress

It’s also critical to check out the tool calling leaderboard, to see which models do better than others. Those might be the ones worth using for agentic applications.

Berkeley's tool calling leaderboard

What to do about it:

• Limit the number of tools the agent can choose from. Keep the palette narrow and highly curated.
• Adopt a planning / tool selection step: have an LLM (or light planner) decide which tool to call before handing off to the “executor” LLM.
• Use sub-agents with specialized tools instead of one all-purpose agent. Each sub-agent has a narrow objective and limited tool set. Think MapReduce for agents.
• Favor general tools (e.g. code executor or a general Python sandbox) over many micro-APIs. That way, the model learns to orient its logic around one flexible tool rather than dozens of brittle ones. We follow this approach at Dropbox.
• Monitor tool selection accuracy as a metric (check the leaderboards on tool-calling correctness). As models scale up, this remains a surprisingly brittle axis.
• Fine-tune the model to improve the calling of your tools or the growing number of tools.

Context Engineering is the Answer

In all of the above scenarios, a common theme emerges: context engineering. This is the art of retrieving and providing the right pieces of information and tools at the right time. This goes beyond prompt engineering, where you’re simply iterating on a prompt itself for an LLM, and includes:

• Any context you’re providing through retrieval (whether it’s RAG or more of an agentic retrieval solution)
• The past context that you’ve had in a multi-turn session
• Any longer-term memory from previous sessions
• Tool selection

Context engineering is the high-status job in AI right now. Deciding what belongs in the context window is the real secret sauce to accurate, reliable, and safe LLMs / Agents.

Given how important context engineering is, how do you know if you are context engineering correctly? Measurements and Evals!

Evals are a bit beyond the scope of this doc, but you’ll want to think about these areas:

How would you know if you are selecting the right context from search / retrieval? Check…

1. Recall: Did we retrieve all the chunks / facts we needed?
2. Precision: Are relevant chunks ranked higher than non-relevant chunks?
3. Relevancy: What % of retrieved chunks / facts matter?

And how would you know if the LLM is using that provided context correctly? Check…

1. Faithfulness: Is the output contradicting the retrieved chunks / facts?
2. Relevancy: Is the response answering the question correctly?
3. Citations / Formatting: Is it showing its work or following the structure you expected?

Conclusion

LLMs are powerful, but they are far from perfect. It’s tempting to treat them like oracles, but that invites failure. These four failure modes: getting lost, gullibility, overconfidence, and tool overload are predictable and addressable. Your reliability depends not just on how “good” the model is, but how thoughtfully you design the surrounding system.

A useful heuristic: good context + weak model > bad context + strong model. Even a simpler model with carefully managed context and rigorous retrieval can outperform a “state-of-the-art” model with sloppy inputs.

Most active research today is aimed at these exact weaknesses:

• making long contexts and multi-session memory more stable,
• reducing hallucinations and teaching models when not to answer,
• improving factual verification and reasoning,
• and building agentic systems that choose and use tools more effectively.

But while research pushes the frontier, real-world success still comes down to context engineering. Deciding what belongs in the window, when to prune, which tools to expose, and how to guide the model to reason (or abstain) are the levers that separate brittle prototypes from robust systems.

I wrote this post originally as an article published on LinkedIn. View the original there.

There’s been a lot of recent buzz around whether Retrieval Augmented Generation (RAG) has reached its limits.

Is RAG truly dead or just replaced by new approaches like search agents, MCP, or massive context windows?

All the newer solutions being proposed like agentic search, code-base crawlers, hybrid semantic + lexical rankers, graph databases, or even MCP tool calling are still retrieval-augmented generation under a new name.

What RAG Really Is

Large Language Models lack a lot of relevant content. First, they are trained with data up to a certain point in time (their knowledge cutoff date). AI search engines like Perplexity (or my own yaddle.ai) use real-time search results to augment the LLM’s knowledge. I even had to do this to generate up-to-date fantasy football player outlooks.

Second, a lot of applications require proprietary data. This is most common with enterprises that need their company’s data as part of an LLM’s context. That’s what we’re building at Dropbox with Dash.

At its core RAG means helping a language model with context it did not train on. When you retrieve content from an external source and feed it into the prompt, that’s RAG. It doesn’t prescribe how you fetch the data. It could be through vector search, keyword search, a graph query, retrieving from an MCP server, a searcher agent, or a combination of all of that.

MrBeanGPT retrieving some "expert" knowledge

Why are some declaring RAG “dead”?

The first RAG paper glued together the three simple steps for retrieval augmented generation:

1. Chunk documents from your corpus
2. Embed each chunk into a semantic vector
3. Retrieve top-k vectors with nearest-neighbor search and drop them into the model’s input

Basic RAG with Vector Search

Those ingredients were chosen because 2020-era LLMs were small, hallucinated often, and lacked long context. LLMs are quite effective at representing semantic meaning. By transforming chunks of text into vector embeddings, you could query them semantically: “Find me the text snippet whose meaning is most similar to this question.”

But in today’s world, chunking documents and doing semantic lookup from a vector database can be suboptimal for many document types, query types, or for modern corpora like large code bases.

So those that associate “RAG” with just “chunking + semantic search” feel it’s dated.

But those who associate “RAG” with retrieving the best context for an LLM know that RAG is alive and well, just evolved. Any retrieval loop is still the safest way to ground an LLM.

Today, you need to design your retrieval system with the understanding of many factors.

One Size of Retrieval Does Not Fit All

An essential consideration in RAG is “what do you retrieve, why, and from where?” The shape, size, and nature of your knowledge corpus heavily influence your retrieval strategy. For instance, if your corpus is small and specialized, then chunking and embedding might be sufficient. However, if your corpus is large (millions of documents) or extremely varied, you might need more refined retrieval strategies.

Structured vs. Unstructured: Is your data a set of well-structured tables and knowledge graphs, or is it a corpus of free text (e.g., PDFs, Word docs, HTML)? You’ll likely need different indexing and retrieval approaches for each.

Real-time vs. Static: If your corpus changes regularly (new product entries, daily news articles, codebase changes), your retrieval pipeline must handle rapid updates.

Complex vs. Straightforward Queries: For simple question-answering, a single query might suffice. But if you need multi-step reasoning for deep analysis, your retrieval approach might need an iterative or agent-based method

Latency / cost budget: Complex agents with reasoning cost a lot of tokens and time.

Let’s go through some other options in the retrieval toolbox.

Lexical (BM25) Retrieval

Using a lexical keyword index excels for exact terms, codes, and part numbers. A procurement bot that can’t resolve SKUs is useless. Using a ranking function like BM25 thrives on tight phrases, dates, and proper nouns, with millisecond latency on millions of docs. A classic example is part-catalog search in manufacturing or legal statute lookup where citations must match verbatim. Semantic search may only return similar part numbers etc, frustrating your users.

The range of which solution to pick based on what your users need

Hybrid Retrieval and Fused Ranking

Many real workloads need both precision and fuzziness. The common approach is a hybrid retrieval system that fetches the top-N from both a lexical index and a vector index. Then those combined results are blended in a cross-encoder type re-ranker (like Cohere’s re-ranker).

Hybrid retrieval gets you the best of both worlds

E-commerce search is the poster child for these hybrid solutions. You want users to be able to do exact search “size 7” (lexical) matching while understanding “trail running” (semantic).

This technique can be further improved by introducing contextual embeddings, as described by Anthropic, to your chunking and embedding strategy.

Graph RAG

Sometimes, your corpus isn’t a set of documents but rather a knowledge graph or a database with entities and relationships. For example, if you maintain a complex knowledge base about pharmaceuticals, you might have a graph where nodes represent active ingredients, brand names, side effects, drug interactions, and more. Then you can answer queries like “What other conditions might be impacted if the patient is taking Drug A and has a history of heart disease?”

Traversing that graph for relevant information can be more effective than chunk-based retrieval of unstructured text.

In the enterprise world (like we’re doing at Dropbox), we have org charts where people are connected to other people, but also connected to groups, documents, projects, and more.

Graph retrieval might be enough on their own, or part of a hybrid retrieval solution

Agentic RAG

For complex user queries (like deep research) or complex documents (like a code base), the newest generation of retrieval is to use agents that emulate how a human researcher might approach the problem. Instead of a single query that retrieves the top-n results in one shot, an LLM-based agent might:

1. Start with a broad keyword or semantic search.
2. Skim retrieved content.
3. Evaluate whether the retrieved results fully answer the query.
4. If not satisfied, refine the query, add new keywords, or investigate tangential topics.
5. Repeat until it has gathered enough context to produce a thorough answer.

This iterative retrieval process can significantly improve correctness and completeness. However, it comes with a trade-off: multiple queries to your database or knowledge store increase latency and cost (especially if you’re using state of the art models).

This approach is still RAG, though the LLM agent itself might invoke a retriever tool to interact with your knowledge base or invoke an MCP service to retrieve data from an external app.

Agent retrievers that fetch content, refine, and evaluate

Multi-agentic RAG

A specialized subset of retrieval scenarios emerges when the knowledge corpus in question is very complex and often has multiple distinct parts to reason upon.

For example, coding assistants / copilots may need a very sophisticated understanding of the structure of a company’s code-base, its dependency graphs, its function names, and any external documentation. If a user asks, “How does function XYZ handle edge cases for user login?” the agent can quickly surface the relevant code snippet, the documentation, or references in other parts of the code.

These retrieval systems are complex and may involve many sub-agents that specialize in retrieving more narrow understanding, which then gets merged back by a main agent (similar to MapReduce).

Agent retrievers that fetch content, refine, and evaluate

Cache Augmented Generation (CAG)

Wait a second, LLMs now have massive context windows. Gemini supports over 1M tokens. Why do we even need these complex retrieval systems? Can’t I just load my entire corpus into the context window and have the LLM figure it out? Is Cache-Augmented Generation all you need?

That’s a very tempting proposition, since the level of effort to add this to your product is very small. But in practice, you will hit four roadblocks.

Retrieval quality

After about 30K tokens, retrieval quality suffers. The NoLiMa scoring method shows active degradation.

LLMs are also very impressive at Needle in the Haystack retrieval, but still can’t quite match the keyword lookup prowess of BM25.

Freshness

If your documents are changing often, it requires significant re-loading of your data, creating the new for a complex invalidation scheme.

Latency

LLMs need to search stochastically in their vector (latent) space, which requires inference and time. The methods described above (especially the one-shot approaches) have significantly less latency and are more deterministic.

Costs

LLMs are very costly for inference, especially these state of the art models that have massive context windows. So a lot of retrievals will result in a lot of inference calls and therefore costs.

Evals, Evals, Evals

With generative AI, there’s a tendency to want to obsess over the model selection and refine your prompts.

But those activities should come after you’ve nailed your retrieval quality.

Today, LLMs treat any passed-in context (aka RAG) as gospel. If that context contains accurate info, your response will be accurate. If it’s not great, guess what? The LLM will confidently give you a bad answer (aka a hallucination).

So no matter how fancy or simple your retrieval pipeline is, defining evaluations are crucial. You must know if your system is genuinely improving the quality and accuracy of the model’s responses for your specific use cases.

Your mantra for better retrieval

Make sure you’ve defined and automated metrics like groundedness, faithfulness, NDCG / MRR, and more.

As advanced retrieval techniques evolve, there’s a temptation to rely solely on marketing claims or general hype about “agent-based retrieval” or “knowledge graphs.” The best guard against that is thorough, data-driven testing. If your system’s performance on your domain-specific tasks actually improves, then the new approach is worth adopting. If not, you might be adding complexity for little gain.

RAG Evolves, But It’s Not Going Away

So, is RAG truly dead? The short answer is no. What’s happening is more of an expansion or refinement of RAG. Techniques are emerging that more closely emulate how humans research or how code is structured. We’re adding iterative search steps, building knowledge graphs, or combining lexical and semantic signals. We’re also experimenting with extremely large context windows. Yet, all these approaches revolve around the same core principle that defines RAG: help a large language model by supplementing it with targeted, external context.

The real challenge is picking the right tool, or combination of tools, for each unique application.

Ultimately, the future of RAG is bright. As LLMs become more powerful, the burden on retrieval might lessen for some tasks (because the model’s baseline knowledge grows), but for domain-specific or time-sensitive tasks, retrieval remains paramount.

While AGI isn’t here yet, we have many large language models (LLMs) rapidly moving in that direction. However, a closer look at leaderboards such as Chatbot Arena and standard benchmarks reveals something interesting: most AI models aren’t generalists. They are actually specialists.

Most AI models aren’t generalists. They are specialists.

Some excel at real-world coding, others at mathematics, writing, science, research, competitive coding, or questions. Today, there are over 200,000 LLMs open-sourced on Hugging Face. And most are small and focused on specialized tasks.

So if you’re building an AI-powered product designed to handle diverse use cases, choosing just one LLM won’t cut it.

Enter LLM routers - technology that allows you to leverage multiple specialized models, routing each task to the model best suited to handle it. Why settle for one when you can pick exactly the right AI for every task?​

Let’s dive deeper into the world of LLM routing and explore how it works.

Understanding LLM Routing

LLM routing operates by evaluating incoming queries and assigning them to the optimal model. Simpler tasks can be handled by smaller, more efficient models. Complex tasks are directed to larger, more capable ones. And specialist tasks can be directed to the best model for the job. This approach ensures that resources are allocated effectively, maintaining high-quality responses without unnecessary expenditure.​

Benefits of LLM Routing

• Cost Reduction: By directing simpler queries to smaller models, you can significantly reduce operational costs. For instance, routers can cut inferencing costs by up to 85% by diverting a subset of queries to more efficient models.
• Improved Speed: Smaller models process queries faster, leading to quicker response times for less complex tasks. This enhances user experience by providing timely answers without compromising quality.​
• Maintained Accuracy: Complex queries are handled by larger models capable of delivering high-quality responses, ensuring that accuracy is preserved where it matters most.​ Simpler routers can maintain 90% of the accuracy, while more complex ones can push that closer to 98%.
• User Friendliness: Today’s chatbots require users to do a lot of work and know a lot of technical details (e.g. how to best prompt, what model is the best for my query). Routers allow you to hide that complexity or offer a “choose for me” option. Why force your users to make a choice while newer models are introduced every day? Keep things simple and hide those details behind a router.
• Benefit from Market Innovations. Related to user friendliness, companies that use just one model will be at a big disadvantage since new faster, cheaper, and more powerful models are launched every week. It’s more advantageous to have a router setup, allowing you to more simply swap in new models. Don’t put your eggs in one basket.

Routers can find a more optimal balance across quality and costs (RouteLLM)

Types of Routers

There’s many ways of introducing routing to your flows, ranging from simple to very complex. Let’s walk through each.

Simple Routing

This is the most basic version introduced in the early RouteLLM paper. Build a router to decide between a smaller LLM (like Llama3 8B, Phi-4-mini, or Claude Haiku) or a larger model (like, GPT 4.5, Claude Sonnet, or Grok 3). Again, the larger state of the art models can do almost anything well, but come at a very steep price.

Basic router for simple and complex tasks

Specialist Routing

This method involves training a routing algorithm on benchmark data to predict which model will best handle a given query. By analyzing the strengths and weaknesses of each model, the router can make informed decisions, enhancing both accuracy and cost-efficiency. Hugging Face lists 200,000 LLMs today that are lighter-weight and highly specialized.

Route to model tuned for specific task

An alternative approach to specialist routing was presented in the Prompt to leaderboard paper. They trained their router using data from Chatbot Arena which has humans decide what answer they prefer between two options. Normally the state of the art (SOTA) models dominate since they are the most capable across all categories. But by leveraging the fact that even the state of the art models differ in their strengths, their approach of routing to the optimal one led to them becoming #1.

Router to different company LLMs

Local Routing

A variant of simple routing, local routing can take advantage of more and more capable edge devices like your laptop or even mobile phone. With technology like llama.cpp and ollama, running LLMs on your laptop or server has never been easier.

The catch is that only the smallest models can reliably run on this hardware (for now). Google’s Gemma 3n is a great example of how these smaller models are innovating on compute and memory usage. For example, Gemma 3n uses a Matryoshka Transformer (MatFormer architecture) that contains nested, smaller models within a single, larger model.

So with local routing, simply send easier queries to your local models and harder queries to more capable cloud-based LLMs. The router itself can run on the device.

Leverage local compute to handle some tasks locally vs. paying cloud costs

The Hybrid LLM paper showed by training a router to handle easy queries locally, they were able to make 40% fewer calls to more expensive cloud-based LLMs.

Cascading Routing

In this approach, queries are first sent to smaller, less expensive models. If the response quality is insufficient, the query is escalated to more powerful models. This can then be repeated if necessary.

This tiered system ensures that only necessary resources are utilized for complex tasks. Though does come at the cost of latency through wasted compute on harder tasks.

A Unified Approach to Routing and Cascading for LLMs paper introduced an even more advanced version where both cascading and routing is done at each step. This leads to further improvements, cost efficiencies, and helps with overall latency.

Cascading routers

Decompose and Reassemble

This advanced approach uses a more powerful LLM to break down the user’s query into many, far simpler tasks. Then those tasks can be answered by smaller LLMs, either local or cloud-based, often in parallel. Then, a final remote call merges all the responses together (similar to MapReduce). While not exactly routing like the other techniques, this approach similarly makes use of a variety of models (likely smaller and more specialized) to accomplish the job.

Use more powerful model to break down tasks and route to those

The MinionS Protocol shows this approach achieves 5.7x reduction in remote costs, while maintaining 97.9% of cloud model performance.

How to Build a Router

In all our examples, we assumed there was an existing router that “just works” and sends your query to the right place. But how do you build that?

The process begins with data preparation, where a diverse set of user queries is collected and labeled according to the performance of different LLMs in addressing them. For example, queries can be rated on a scale from 1 to 5, indicating how effectively a particular model responds. This labeled dataset serves as the foundation for training a classifier, such as a causal LLM, to predict which model is best suited for a given query. Fine-tuning a classifier involves training it on this labeled data to enhance its ability to accurately route queries.

Techniques like Low-Rank Adaptation (LoRA) can be employed to efficiently fine-tune your model. After training, the router’s performance is evaluated using benchmark datasets to ensure it effectively balances response quality and cost by directing queries to the appropriate LLMs.

The team behind one of the earliest LLM routers, trained four routers using Chatbot Arena data:

1. A similarity-weighted (SW) ranking router that performs a “weighted Elo calculation” based on similarity
2. A matrix factorization model that learns a scoring function for how well a model can answer a prompt
3. A BERT classifier that predicts which model can provide a better response
4. A causal LLM classifier that also predicts which model can provide a better response

Connection with Mixture of Experts

Mixture of Experts (MoE) is a model architecture where one large language model contains multiple specialized sub-models, known as “experts.” The roots come from the 1991 paper Adaptive Mixture of Local Experts, where only a subset of parameters are active in a neural net. It was designed to retain the best quality of an output while improving the overall efficiency and scalability. So, in essence an MoE model is performing routing to generate output tokens within the model itself as compared to routing across very different models.

Traditional vs. Mixture of Experts model architecture

So while LLM routing chooses the best model for a task, MoE models have a gating network that selects the best “expert” for a given input token.

Conclusion

AGI may one day be the “master of all trades.” Today isn’t that day. And that’s fine. The winning pattern right now is a team of specialists with a great dispatcher. LLM routing is that dispatcher. It turns a messy, fast-changing model landscape into a system that is cheaper, faster, and just as accurate.

And because new models land weekly, a router isn’t just an optimization—it’s how you stay current without rewiring your product every month.

There also isn’t one “right” router. You can keep it simple (small vs. large), get smart with specialist routing, go hybrid with local models for easy wins, cascade when confidence drops, or decompose big problems and reassemble the answers. Mix and match. Let data, budget, and latency targets decide.

So I wanted to know how it works. And it’s surprisingly simple at its core.

Here’s what I learned in building a version of it in about 100 lines of code.

Of course, to build amazing products like Perplexity, Phind, and you.com, you’ll need major investments in a delightful user experience, plenty of extra features, fine-tuned models, custom web-crawling, and more. Ultimately, it’ll be much more than just 100 lines of code!

The Basics

To create your own, you need to make just two API calls:

1. Call a search engine API like Bing, Google, or Brave using the user’s query
2. Craft a prompt to an LLM like ChatGPT passing along the search results in the context

Using search results to augment and improve upon an LLM’s answer

With the basics out of the way, let’s get into the details.

Step 1: Get Search Results for a user query

The main challenge with LLMs like ChatGPT is that they have knowledge cutoffs (and they occasionally tend to hallucinate). It’s because they’re trained on data up to a specific date (eg Sep 2021). So if you want an answer to an up-to-date question or you simply want to research a topic in detail, you’ll need to augment the answer with relevant sources. This technique is known as RAG (retrieval augmented generation). And in our case we can simply supply the LLM up-to-date information from search engines like Google or Bing.

To build this yourself, you’ll want to first sign up for an API key from Bing, Google (via Serper), Brave, or others. Bing, Brave, and Serper all offer free usage to get started.

There’s plenty of ways to make a remote call, but here’s a basic way of using cURL / PHP to call Brave’s search API:

$params = array('q' => $query);
$ENDPOINT = "https://api.search.brave.com/res/v1/web/search";
$url = $ENDPOINT . '?' . http_build_query($params);
$headers = array(
    'X-Subscription-Token: ' . $BRAVE_KEY,
    'Accept: application/json'
);
$curl = curl_init();
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_ENCODING, 'gzip');
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec($curl);
curl_close($curl);

These APIs will return the URL link, URL title, a description or snippet of the page, and more. After parsing the results, you are ready to construct a prompt to ChatGPT or another LLM of your choice.

Wait, just the snippet? Is that enough data for this to work?

Yes! Search results will typically only show a few hundred characters worth of content related to the user’s query. But it turns out, search engines are highly optimized at finding the relevant part of the page that corresponds to the user’s query. They are tuned at query understanding and often perform query rewriting or expansion to increase the recall and quality of the search results.

There’s a surprising amount of relevant data just within these short snippets

So while you could scrape the entire page, simply leveraging the great ranking and retrieval those search engines are doing will save you a lot of work. Plus, you’ll be able to respond to the user very quickly by doing this. Scraping the full page content is error-prone and slow, so your users will just be waiting around. And speed matters in search. Users want their answer not only correct, but delivered quickly.

Step 2: Decide the LLM you want to use

Here, you’ll need to decide which LLM and inference provider to use. For LLMs, each model has unique strengths, different costs, and different speeds. For example, gpt-4 and Claude 3 Opus are very accurate but expensive and slow. If you want to track the best models, here’s a ranking leaderboard. When in doubt, I’d recommend using chatgpt-3.5-turbo from OpenAI. It’s good enough, cheap enough, and fast enough to test this out.

Next, you’ll need to sign up for an API key from an LLM provider. There’s a lot of inference providers to choose from right now. For example there’s OpenAI, Anthropic, Anyscale, Groq, Cloudflare, Perplexity, Lepton, or the big players like AWS, Azure, or Google Cloud. I’ve used many of these with success and they offer a subset of current and popular closed and open source models.

Fortunately, most of these LLM serving providers are compatible with OpenAI’s API format, so switching to another provider / model is only minimal work (or just ask a chatbot to write the code!).

Test any queries or view code samples on OpenAI’s playground

Step 3: Craft a prompt to pass along the search results in the context window

When you want to ask an LLM a question, you can provide a lot of additional context. Each model has its own unique limit and some of them are very large. For gpt-4-turbo, you could pass along the entirety of the 1st Harry Potter book with your question. Google’s super powerful Gemini 1.5 can support a context size of over a million tokens. That’s enough to pass along the entirety of the 7-book Harry Potter series!

Fortunately, passing along the snippets of 8-10 search results is far smaller, allowing you to use many of the faster (and much cheaper) models like gpt-3.5-turbo or mistral-7b.

In my experience, passing along the user question, custom prompt message, and search result snippets are usually under 1K tokens. This is well under even the most basic model’s limits so this should be no problem.

Here’s a sample prompt I’ve been playing around with you can use (Hat-tip to the folks at Lepton AI who open-sourced a similar project which helped me refine my prompt).

You are an assistant written by Josh Clemm. You will be given a question and you will respond with an answer to the question. It must be accurate, high-quality, and expertly written in a positive, interesting, and engaging manner. It must be informative and in the same language as the user question. You will be provided a set of citations for the question. Each will start with a reference number like [citation:x], where x is a number. Always use the related citations and cite the citation at the end of each sentence in the format [citation:x]. If a sentence comes from multiple citations, please list all applicable citations, like [citation:2][citation:3]. Here are the provided citations:

You will then need to dynamically add the sources / citations from the search results. For a user query like “best vegas hotels for families”, the citations would look a bit like this:

[citation:1] As for a hotel, I would echo the Mandalay Bay recommendation. There is a wide variety of restaurants, the pool complex is great, the Shark Reef is fun for families

[citation:2] Treasure Island is closest to the center strip and action as well as walking distance. Excalibur and Circus Circus for kids as 2nd floor has games galore.

The language model will then return the answer with citations like the following:

When planning a family trip to Las Vegas, there are excellent hotel options ideal for families. Mandalay Bay is a standout choice, offering a wide variety of amenities suitable for all ages. It features numerous offerings like pools, arcades, and a Shark Reef [citation:1]. Other great options include Treasure Island, Excalibur, and Circus Circus [citation:2].

And that’s it, you now have a working up-to-date answer engine with citations!

Step 4: Add Related or Follow Up Questions

One of the nice features of Perplexity and others is how they suggest follow up questions. Fortunately, this is easy to replicate.

To do this, you can make a second call to your LLM (in parallel) asking for related questions. And don’t forget to pass along those citations in the context again.

Or, you can attempt to construct a prompt so that the LLM answers the question AND comes up with related questions. This saves an API call and some tokens, but it’s a bit challenging getting these LLMs to always answer in a consistent and repeatable format.

Here’s an example prompt that has the LLM do both:

You will be given a question. And you will respond with two things.
First, respond with an answer to the question. It must be accurate, high-quality, and expertly written in a positive, interesting, and engaging manner. It must be informative and in the same language as the user question.
Second, respond with 3 related followup questions. First print "==== RELATED ====" verbatim. Then, write the 3 follow up questions in a JSON array format, so it's clear you've started to answer the second part. Do not use markdown. Each related question should be no longer than 15 words. They should be based on user's original question and the citations given in the context. Do not repeat the original question. Make sure to determine the main subject from the user's original question. That subject needs to be in any related question, so the user can ask it standalone.
For both the first and second response, you will be provided a set of citations for the question. Each will start with a reference number like [citation:x], where x is a number. Always use the related citations and cite the citation at the end of each sentence in the format [citation:x]. If a sentence comes from multiple citations, please list all applicable citations, like [citation:2][citation:3].
Here are the provided citations:

And like before, you then need to dynamically add the citations. And doing so still is less than our 100 lines of code.

I’ve open-sourced this more basic version in PHP. Give it a try!

Step 5: Make it look a lot better!

To make this a complete example, we need a usable UI. And of course we’ll now go well beyond our 100 lines of code. I’m not the best web / frontend engineer, so won’t go into these details. But I will share with you the UI I open sourced (I’m using Bootstrap, jquery, and some basic CSS / javascript to make this happen).

These questions are making me hungry

The key parts I added to replicate the Perplexity experience are:

• The answer streams back to the user (improving perception of speed)
• The citations are replaced by a nicer in-line UI with a clickable popup for the user to learn more
• The sources considered are included after the answer in case the user wants to explore further
• A second call in parallel is made to a search API for images to enhance and supplement results
• Markdown and code syntax highlighting are used if necessary

Markdown and code syntax highlighting support!

Advanced Ways to Make it even Better

Query Understanding

Search engines today use natural language processing to determine the type of query a user has typed or to identify the entity. And based on that often will show different results. For example, a question like “top news” will retrieve actual news content vs. just returning websites that have news results. A question about a known person will fetch a “knowledge card” for them to supplement any searched results. Queries that are more basic questions (eg starts with “what” or “why”) may trigger an actual answer at the top of the results.

AI search apps don't do well with queries related to up to date sports scores unless manual query understanding is done

So if we really want to improve upon search engines today, we need to replicate that query understanding and custom response behavior.

To get more specific, let’s consider searching for news. You could check if the user query says “top news”, and if so, instead of making a search API call, you make a news API call. Or you might be regularly fetching and storing news articles, and you just need to fetch those. Then, you simply pass along those results along with your prompt to the LLM. At which point, the LLM just functions to summarize the articles you supplied.

Follow ups and Threads

All AI search apps offer the ability to ask custom follow up questions. In order to be an actual followup, you’ll need to add the original question and answer into the follow up prompt. You would use a very similar technique to adding web sources as done above.

If the user continues to follow up on the followup (eg a full question thread), you may start getting close to the maximum token length of the context window. At this point, the solution gets a bit more complex and beyond the scope of this article. But you essentially have to start storing, chunking, and using your own vector search to pass along the most relevant parts of the past thread. You got this!

Pro Search

Let’s say a topic is particularly difficult to answer. Or is more research oriented. Right now, our current solution is optimized for speed (and cost). Within about a second, we can perform both our search and LLM calls and begin to stream back the answer. But users may have some queries that don’t need to be answered quickly.

There’s some ways of replicating that.

First, in your search call, simply fetch more than the default 8-10 results.

Second, you could start workers to parse and scrape each search result, pulling in way more content for your context. Keep in mind this can introduce significant work since you really should chunk the document into relevant summarizations. Search engines like Google use extractive summarization for their snippets so you are really trying to replicate that work and they have a pretty large head start on you!

Finally, you’ll want to use a more advanced LLM, like gpt-4 or Claude 3 Opus. Those are trained on significantly more data. And therefore are far more accurate at inference. Plus, they can take in far more context (like those fully scraped results). But they are considerably slower and cost a lot more, so you’ll want to only use this pro or deeper search when appropriate.

Improving Answer Quality

To further improve quality of your results, there’s a few things to consider. First, you can always utilize the current most powerful general models like Claude 3 Opus and GPT 4 Turbo. Take a look at the LLM ranking leaderboard to see who’s the current top model.

Second, you can ask the user to provide the type of question they are asking (or attempt to infer it). Perplexity has focus options like “Writing”, “Academic”, “Math”. And with that focus, you can then utilize a more appropriate LLM for that. Take a look at this leaderboard to determine the best models for specific tests. For example, some models are better at coding, some are better at creative writing, and some are better at general Q&A.

Finally, you can chain a number of LLMs together as Agents. Andrew Ng gave a talk on AI agentic workflows where instead of just using one powerful model, you can use a number of more focused and faster / cheaper models together. This technique significantly increases the quality of the final output. For our use case, you could have one agent perform the initial query and a second that checks and scores it for accuracy. This works particularly well for making sure out-of-date sources aren’t part of the final answer.

Test it out

First, feel free to explore my open source PHP and basic web code. It works well running on some of the most basic web servers (like on this website).

But to make this example a bit more robust, I ended up porting the PHP code over to Javascript and am using Cloudflare (with workers and a KV cache) to up the production level a bit.

Go check out that version at https://yaddleai.com

Final thoughts

These RAG-based AI search apps are fascinating. They are truly a blend of LLM inference and classic search retrieval and ranking. Is the model answering the question itself but using the search snippets? or is the model simply summarizing the already highly relevant snippets?

Daniel Tunkelang offers a similar take on RAG search apps: “the role that the generative AI model plays is usually secondary to that of retrieval. The name ‘retrieval-augmented generation’ is a bit misleading — it might be better to call it ‘generative-AI-enhanced retrieval’”.

In any case, combining search + LLMs proves to be a powerful and fairly straightforward way to improve upon the classic question / answer framework.

Let me know if you end up building something like this yourself and share any lessons you learned!

I wrote this post originally as an article published on LinkedIn. View the original there.

On a cold evening in Paris in 2008, Travis Kalanick and Garrett Camp couldn’t get a cab. That’s when the idea for Uber was born. How great would it be if you could “push a button and get a ride?”

Fast forward to today where Uber is the largest mobility platform in the world. It operates in over 70 countries and 10,500 cities. Uber Eats is the largest food delivery platform ex-China in the world. It operates across 45 countries. We connect millions of driver-partners and merchants with over 130 million customers. We offer dozens of services to go anywhere or get anything. We handle billions of database transactions and millions of concurrent users across dozens of apps and thousands of backend services.

So, how did we get there?

Back in 2009, Uber hired contractors to build the first version of Uber and launched it to friends in San Francisco soon after. The team used the classic LAMP stack to build the first version of Uber and the code was written in Spanish.

Original LAMP stack proved out the use case, but couldn’t scale.

The scaling challenges started as more people wanted to use Uber. There were often major concurrency issues, where we’d dispatch two cars to one person or one driver would be matched to two different riders. (Learn more about Uber’s earliest architecture from founding engineer Curtis Chambers).

But the product caught on. It was time to build the tech foundations from scratch.

Setting up for Global Scale

Circa 2011

To architect a better and more scalable solution, we needed to address those concurrency problems. Additionally, we needed a system that can process tons of real-time data. Not only are there requests from riders, but Uber needs to track driver real-time locations in order to match riders as efficiently as possible. Finally, the product was still early and would require a lot of testing and iteration. We needed a solution to solve all these scenarios.

Uber adopted Node.js for their real-time needs. And ended up being one of the first major adopters of Node.js in production. Node.js was ideal for a few reasons. First, Node.js handles requests asynchronously (using a non-blocking, single-threaded event loop) and so can process significant amounts of data quickly. Second, Node.js runs on the V8 JavaScript engine, so not only is it performant, but it’s excellent for quick iteration.

Uber then created a second service built in Python to handle business logic functions like authentication, promotions, and fare calculation.

The resulting architecture was two services. One built in Node.js (“dispatch”) connected to MongoDB (later Redis) and the other built in Python (“API”) connected to PostgreSQL.

Uber’s two monolith architecture allowed the engineering org to begin to scale

And to improve the resiliency of Uber’s core dispatching flow, a layer between dispatch and API known as “ON” or Object Node was built to withstand any disruptions within the API service. (Learn more about Uber’s earliest efforts to maintain service reliability in this video Designing for Failure: Scaling Uber by Breaking Everything).

This architecture started to resemble a service oriented architecture. Service oriented architectures can be very powerful. As you carve out services to handle more dedicated functionality, it has a side benefit of allowing easier separation of engineers into dedicated teams. Which then allows for more rapid team scaling.

But as the team and number of features grew, the API service was getting bigger and bigger. More and more features were conflicting with one another. Engineering productivity was slowing down. There were huge risks continuously deploying the codebase.

It was time to split out API into proper services.

From Monolith to Microservices

Circa 2013

To prepare for our next phase of growth, we decided to adopt a microservice architecture. This design pattern enforces the development of small services dedicated to specific, well-encapsulated domain areas (e.g. rider billing, driver payouts, fraud detection, analytics, city management). Each service can be written in its own language or framework, and can have its own database or lack thereof. However, many backend services utilized Python and many started to adopt Tornado to provide asynchronous response functionality. By 2014, we had roughly 100 services.

Uber’s API monolith to microservice migration

While microservices can solve many problems, it also introduces significant operational complexity. You must only adopt microservices after understanding the tradeoffs and potentially build or leverage tools to counteract those tradeoffs. And if you don’t consider the operational issues, you will simply create a distributed monolith.

Here’s some examples of the issues microservices create and what Uber did to address.

• To ensure all services use standardized service frameworks, we developed Clay. This was a Python wrapper on Flask to build restful backend services. It gave us consistent monitoring, logging, HTTP requests, consistent deployments, etc.
• To discover and talk to other services and provide service resilience (fault tolerance, rate limiting, circuit breaking), Uber built TChannel over Hyperbahn. TChannel as our bi-directional RPC protocol was built in-house mainly to gain better performance and forwarding for our Node and Python services, among other benefits.
• To ensure well-defined RPC interfaces and stronger contracts across services, Uber used Apache Thrift.
• To prevent cross-service capability issues, we use Flipr to feature flag code changes, control rollouts, and many other config-based use cases.
• To improve the observability of all service metrics, we built M3. M3 allows any engineer easy ways to observe the state of their service both offline or through Grafana dashboards. We also leverage Nagios for alerting at scale.
• For distributed tracing across services, Uber developed Merckx. This pulled data from a stream of instrumentation via Kafka. But as each service started to introduce asynchronous patterns, we needed to evolve our solution. We were inspired by Zipkin and ultimately developed Jaeger, which we still use today.

Over time, we’ve migrated to newer solutions like gRPC and Protobuf for interfaces. And many of our services utilize Golang and Java.

Scaling the Trip Database

Circa 2014

While Uber was creating many new backend services, they all continued to use one single PostgreSQL database.

The single PostgreSQL DB became a bottleneck

We were hitting some significant issues. First, the performance, scalability, and availability of this DB was struggling. There was only so much memory and CPUs you could throw at it. Second, it was getting very hard for engineers to be productive. Adding new rows, tables, or indices for new features became problematic.

And the problem was getting existential. By early 2014, Uber was 6 months away from Halloween night - one of the biggest nights of the year. We needed a more scalable solution and needed it fast.

When we looked into the data mix of this DB, the majority of storage was related to our trips, which was also growing the fastest.

The mix of data stored in our single PostgreSQL DB in early 2014

We use trip data in order to improve services like Uber Pool, provide rider and driver support, prevent fraud, and develop and test features like suggested pick-ups. So we embarked on developing Schemaless, our new trip data store. Schemaless is an append-only sparse three dimensional persistent hash map, similar to Google’s Bigtable, and built on top of MySQL. This model lended itself naturally to horizontal scaling by partitioning the rows across multiple shards and supported our rapid development culture.

And we successfully migrated all our services that access trip information in time to avert the Halloween peak traffic disaster. (Learn more with this video from lead engineer Rene Schmidt about our creation of and migration to Schemaless).

The Schemaless migration operational room - a common look into migrations at scale

While we used Schemaless for our trip data store, we started to use Cassandra as a replacement for our other data needs, including the database that we use for marketplace matching and dispatching.

Splitting up Dispatch

Circa 2014

Among Uber’s original two monoliths, we discussed the evolution of API into hundreds of microservices. But dispatch similarly was doing too much. Not only did it handle matching logic, it was the proxy that routed all other traffic to other microservices within Uber. So we embarked on an exercise to split up dispatch into two areas of cleaner separation.

Splitting the monolithic dispatch service into a real-time API gateway and an actual dispatch service

Extracting Uber’s Mobile Gateway from Dispatch

To better handle all the real-time requests from our mobile apps, we created a new API gateway layer named RTAPI (“Real-Time API”). And we continued to use Node.js for it. The service was a single repository that was broken up into multiple specialized deployment groups to support our growing businesses.

RTAPI provided a powerful new real-time layer that maintain high development velocity

The gateway provided a very flexible development space for writing new code and had access to the hundreds of services within the company. For instance, the first generation of Uber Eats was completely developed within the gateway. As the team's product matured, pieces were moved out of the gateway and into proper services.

Rewriting Dispatch for Uber’s Growing Size

The original dispatch service was designed for more simplistic transportation (one driver-partner and one rider). There were deep assumptions that Uber only needed to move people and not food or packages. Its state of available driver-partners was sharded by city. And some cities were seeing massive growth of the product.

So, dispatch was rewritten into a series of services. The new dispatch system needed to understand much more about the types of vehicles and rider needs.

It took on advanced matching optimizations, essentially to solve a version of the traveling salesman problem. Not only did it look at the ETAs of currently available driver-partners, but needed to understand which drivers could be available in the near future. So we had to build a geospatial index to capture this information. We used Google’s S2 library to segment cities into areas of consideration and used the S2 cell ID as the sharding key. (We’ve since updated to and open-sourced H3)

RTAPI provided a powerful new real-time layer that maintain high development velocity

Since these services were still running on Node.js and were stateful, we needed a way to scale as the business grew. So we developed Ringpop, a gossip-protocol based approach to share geospatial and supply positioning for efficient matching.

Learn more about the history of our dispatch stack here or watch this video on Scaling Uber’s Real-time Market Platform.

Mobile App Development at Scale

Circa 2016 to present

The flagship Uber product could only have existed due to the new mobile paradigm created by the iPhone and Android OS launches in 2007. These modern smartphones contained key capabilities like location-tracking, seamless mapping, payment experiences, on-device sensors, feature-rich user experiences, and so much more.

The Uber rider app was critical in defining a scalable mobile architecture

So Uber’s mobile apps were always a critical part of our scaling story.

As Uber scaled across the globe, there was a need for an ever-growing list of features. Many of these were specific to certain countries like localized payment types, different car product types, detailed airport information, and even some newer bets in the app like Uber Eats and Uber for Business.

The mobile app’s repositories slowly hit similar bottlenecks to a backend monolith. Many features and many engineers, all trying to work across a single releasable code base.

This led Uber to develop the RIB architecture for mobile, starting with the rewrite of the main Uber app.

RIB stands for Router, Interactor, Builder

Like the benefits of microservices, RIBs have clear separation of responsibilities. And since each RIB serves a single responsibility, it was easy to separate them and their dependencies into core and optional code. By demanding more stringent review for core code, we were more confident in the availability of our core flows. And this allows simple feature flagging to ensure the app continues to run reliably.

And like microservices, RIBs can be owned by different teams and engineers. This allowed our mobile codebases to easily scale to hundreds of engineers.

Today, all our apps have adopted RIBs or are migrating towards it. This includes our main Driver app, the Uber Eats apps, and Uber Freight.

The Rise of Uber Eats

Circa 2017

Uber had been experimenting with a number of “Uber for X on-demand” concepts since 2014. And all early signs pointed towards food. So in late 2015 Uber Eats launched in Toronto. And followed a similarly fast growth trajectory just like UberX.

Uber Eats business growth compared with Uber rides (Uber Q3 2020 Earnings)

To enable this rapid growth, Uber Eats leveraged as much of the existing Uber tech stack as possible, while creating new services and APIs that were unique to food delivery (e.g. e-commerce capabilities like carts, menus, search, browsing).

A simplified view into early Uber Eats architecture and how it leveraged systems built for original Uber

The operations team that needed to tune their cities’ marketplace often got creative and did things that didn’t scale (until the appropriate tech was built).

Uber Eats Canada ran scripts to manage active stores and their delivery radius. Learn more about this story .

Early Uber Eats was "simple" in that it supported a three-way marketplace of one consumer, one restaurant, and one driver-partner. Uber Eats today (130+ million users, dozens of countries) supports a variety of ordering modes and capabilities and can support 0-N consumers (eg. guest checkout, group ordering), N merchants (eg. multi-restaurant ordering), and 0-N driver partners (eg. large orders, restaurants which supply their own delivery fleet).

The history of how Uber Eats evolved probably deserves its own scaling story and I may one day get to it.

But for now, to learn more from the earliest days, I highly recommend listening to Uber Eats founder Jason Droege’s recount of “Building Uber Eats”.

Bring on the Standards - Project Ark

Circa 2018

No scaling story is complete without understanding the context and culture of the company. As Uber continued to expand city by city, local operations folks were hired to ensure their city launch would go successfully. They had tools to ensure their local marketplace would remain healthy and the flywheel would grow. As a result, Uber had a very distributed and decentralized culture. And that helped contribute to Uber’s success in getting to 600 cities by 2018.

That culture of decentralization continued within engineering, where one of our earliest cultural values was “Let Builders Build”. This resulted in rapid engineering development that complemented Uber’s success growing across the globe.

But after many years, it resulted in the proliferation of microservices (thousands by 2018), thousands of code repositories, multiple product solutions solving very similar problems, and multiple solutions to common engineering problems. For example, there were different messaging queue solutions, varying database options, communication protocols, and even many choices for programming languages..

"You've got five or six systems that do incentives that are 75 percent similar to one another."

Thuan Pham, Former Uber CTO

Developer productivity was hurting.

Engineering leadership recognized it was time for more standardization and consistency and formed Project Ark. Project Ark sought to address many aspects of engineering culture that contributes to scaling:

• Engineer productivity
• Engineer alignment across teams
• Duplication
• Unmaintained critical systems
• Knowledge access and documentation

As a result, we elevated Java and Go as official backend languages to gain type-safety and better performance. And deprecated the use of Python and Javascript for backend services. We embarked on reducing code repos from 12,000 down to just our main languages (Java, Go, iOS, Android, and web). We defined more standardized architectural layers where client, presentation, product, and business logic would have clear homes. We introduced abstractions where we could group a number of services together (service “domains”). And continued to standardize on a series of service libraries to handle tracing, logging, networking protocols, resiliency patterns, and more.

A Modern Gateway for the Modern Uber

Circa 2020

By 2019, Uber had many business lines with numerous new applications (Uber Eats, Freight, ATG, Elevate, and more). Within each line of business, the teams managed their backend systems and their app. We needed the systems to be vertically independent for fast product development.

And our current mobile gateway was showing its age. RTAPI had been built years ago and continued to use Node.js and Javascript, a deprecated language. We were also eager to make use of Uber’s newly defined architectural layers as the ad hoc code added to RTAPI over the years was getting messy with view generation and business logic.

So we built a new Edge Gateway to start standardizing on the following layers:

• Edge Layer: API lifecycle management layer. No extraneous logic can be added keeping it clean.
• Presentation Layer: microservices that build view generation and aggregation of data from many downstream services.
• Product Layer: microservices that provide functional and reusable APIs that describe their product. Can be reused by other teams to compose and build new product experiences.
• Domain Layer: microservices that are the leaf node that provides a single refined functionality for a product team.

This evolution set us up well to continue building new products with velocity, yet with the necessary structure to align our 1000s of engineers.

Next Generation Fulfillment

Circa 2021

Throughout the years, Uber has created a world-class platform for matching riders and driver-partners. So our dispatch and fulfillment tech stack is a critical part of Uber’s scaling story.

By 2021, Uber sought to power more and more delivery and mobility use cases. The fulfillment stack was showing its age and couldn’t easily support all these new scenarios. For example, we needed to support reservation flows where a driver is confirmed upfront, batching flows with multiple trips offered simultaneously, virtual queue mechanisms at Airports, the three-sided marketplace for Uber Eats, and delivering packages through Uber Direct.

Some example services the Fulfillment Platform needed to support

So we made a bold bet and embarked on a journey to rewrite the Fulfillment Platform from the ground up.

To satisfy the requirements of transactional consistency, horizontal scalability, and low operational overhead, we decided to leverage a NewSQL architecture. And opted to use Google Cloud Spanner as the primary storage engine.

As lead Uber engineer Ankit Srivastava puts it "as we scale and expand our global footprint, Spanner's scalability & low operational cost is invaluable. Prior to integrating Spanner, our data management framework demanded a lot of oversight and operational effort, escalating both complexity and expenditure. The adoption of Spanner streamlined operational costs, improved overall reliability, and strengthened overall value by providing better throughput and performance for the same price.”

The Uber-GCP network infrastructure

What else have we done?

Of course, our scaling story is never this simple. There’s a countless number of things we’ve done over the years across all engineering and operations teams, including some of these larger initiatives:

Many of our most critical systems have their own rich history and evolution to address scale over the years. This includes our API gateway, fulfillment stack, money stack, real-time data intelligence platform, geospatial data platform (where we open-sourced H3), and building machine learning at scale through Michelangelo.

We’ve introduced various layers of Redis caches. We’ve enabled powerful new frameworks to aid in scalable and reliable systems like Cadence (for writing fault-tolerant, long-running workflows).

We’ve built and leveraged data infrastructure that enables long term growth, like how we leveraged Presto or scaled Spark. Notably, we built Apache Hudi to power business critical data pipelines at low latency and high efficiency.

And finally, we continue to improve the performance of our servers with optimized hardware, advanced memory and system tuning, and utilizing newer runtimes.

What’s Next? Heading to the Cloud

Being a global company, Uber operated out of multiple on-prem data centers from the earliest days. But that introduced a number of challenges.

First, our server fleet had grown rapidly (over 250,000 servers) and the tooling and teams were always trying to keep up. Next, we have a large geographical footprint and need to regularly expand into more data centers and availability zones. Finally, with only on-prem machines, we were constantly needing to tune the size of our fleet.

We spent the last few years working towards making over 4000 of our stateless microservices portable. And to ensure our stack would work equally well across cloud and on-prem environments, we embarked on Project Crane to solve. This effort set Uber up well for the future. To learn more, watch lead Uber engineer Kurtis Nusbaum’s talk on how Crane solves our scaling problems.

We now have plans to migrate a larger portion of our online and offline server fleet to the Cloud over the next few years!

Even though I moved to engineering management in 2011, I still love to code and continue to work on side projects and even at work.

My first time “coding” was during the early days of the web using HTML and Javascript. I fondly remember putting together my personal website back in 1996, hosted by flash.net. That site is long gone, but the Internet Archive has a record of a site I created in 1999.

So I decided to create a homage to that past and make a 90s version of my home page.

Inspiration

I have nostalgia and fondness from those early days of the web. I love that the Space Jam movie website still exists and of course, loads super quickly. Modern javascript frameworks are powerful but often overkill.

Space Jam 1996

To inspire the 90s version of my homepage, I looked at the Web Design Museum, the Space Jam website, the Yahoo site from 1997, and my own website memories and on the wayback machine.

My personal links page from 1999

Gifs and Web Badges

No retro site is complete without blinking text, flashing gifs, and those tiny web badges to mark your personality.

Here’s a few resources I used for that.

• GifCities, the Animated GIF Search Engine hosted by the Internet Archive
• 99 Gif Shop, for even more inspiring animated gifs
• The 88x31 GIF Collection, an archive of those famous retro web badges
  
• Cameron’s World, an experience of the 90s in itself. Do yourself a favor and give it a click through!
• Free website hit counter
  

Pixelated Fonts

Things were looking good, except that the font rendering itself looked much too modern and smooth. “Old” fonts like Times New Roman render much nicer today using high resolution screens and antialiasing.

We're not tricking anyone with these super crisp and smooth fonts

We needed to actually create a retro-looking version of Times New Roman. For this, I found a great resource that looked into creating convincing-looking 90s fonts in modern browsers. After bringing over those font packages, the site really took on that retro feel.

Retro Looking Images

The final piece of my 90s homepage conversion was to make sure the pictures had that retro look. With today’s fast internet speeds and high resolution monitors, pictures look fantastic. But in the 90s, monitors were older and speed was a luxury. So images were a fraction of the size and often were pixelated.

So how should I pixelate my pictures?

We can emulate the older look by adding random noise, a technique known as Dithering. For this project, I found the Dither Me This tool using the “RBG Dither” option to really work nicely.

You then get photos that look like this:

We’re Done!

Now, go check out the 90s version of joshclemm.com. I’d love to see you create your own retro site one day too. Thanks!

I already had a ton of data about the upcoming season. I had player info, past statistics, team schedules, team news, player news, and depth charts. And I generate player projections using wisdom of the crowd, player tiers, automatic sleepers/busts, draft recommendations, and even use Markov chains to predict availability odds.

But I needed a compelling narrative to stitch that all together. So I went exploring large language models.

The language models I tried

I found OpenAI and their GPT-3 models to be pretty far along on the research and capability of large language models. We had explored a project at Uber Eats to use conversational AI to receive food orders from phone calls, so I was familiar with the technology and wanted to put it to use.

• text-davinci-002 Best long-form writer I tested. Good at following directions, keeping a consistent voice, and weaving facts into paragraphs. Slower and pricier, but the output was closest to “publishable.” (InstructGPT background)

• text-curie-001 Fast and cheap. Great for shorter tasks—headlines, bullet points, tier labels, and re-phrasing. It struggled with longer narratives without heavy scaffolding. (OpenAI API models overview)

• text-babbage-001 / text-ada-001 Useful workhorses for lightweight tasks: classifying news blurbs as relevant/irrelevant, tagging injury notes, quick sentiment on coach quotes. (API reference)

• Embeddings (text-similarity-* and text-search-* families) Critical for grounding. I used these to index player projections, depth charts, recent news, schedule difficulty, team tendencies, and my Markov availability odds so the generator could pull the right facts before it wrote anything. (Embeddings guide)

You’ll see later, I ended up finding generating embeddings and fetching relevant chunks were sometimes overkill. I could simply pass various player info straight into the language model and got pretty good results

What I tried in the APIs

I did nearly everything through the Completions API and Embeddings API. A few patterns emerged.

1. Retrieval-augmented generation (RAG) to keep it honest

LLMs are great at writing; they’re less great at being right by default. I built a tiny retrieval layer:

1. Chunk my data into bite-sized passages per player (projections deltas, role notes, schedule flags, depth chart context, news summaries).
2. Embed each chunk (text-similarity-ada-001 was the budget pick; *-curie-001 gave a small relevance bump).
3. At write time, embed the prompt/question, grab the top-k chunks via cosine similarity, and stuff those facts into the prompt before generating.

This kept the model focused on my numbers and today’s situation. If you want the deeper theory, start with Retrieval-Augmented Generation by Lewis et al. (paper).

I did find the effort generating vector embeddings was overkill at times. Later I found simply passing along the formatted text in the prompt template fit within the context window, generated as good results as before, and frankly simplified the setup for this project.

The original Retrieval-Augmented Generation paper seemed to suggest vectors were critical for this technique. But given we are working with text in the prompt (which then gets encoded in the language model), I tried it and it was pretty good. My input context size was small enough to work. So I could see how trying to apply RAG on a larger data set would need that pre-processing.

2. Prompt scaffolding (templates + facts)

I learned quickly that “write a blurb about DK Metcalf” yields creative fiction. The fix was structure and specifics.

Example Prompt template:

You are a fantasy football analyst writing a 180–220 word season outlook report for the 2022 NFL season.
Voice: clear, direct, helpful. Avoid cliches.

FACTS (verbatim, may include numbers):
- Team: SEA. Role: X. Depth chart: Y.
- Projection (PPR): 85-1,050-9, WR12 consensus (range WR8–WR16).
- Average Draft Position (ADP): 67
- Schedule: Weeks 1–4 tough vs perimeter CBs; weeks 14–16 favorable.
- News (last 14d): hamstring tightness, limited two practices, no setback.

Constraints:
- Do not invent facts. If unknown, say "unclear as of June 2022."
- Cite at least three of the FACTS directly.
- End with a concise draft plan: "Round/price | Risk | Ceiling | Team fit".

Write the report.

Completions params I leaned on:

• temperature: 0.3–0.5 (steadier, less flowery)
• top_p: 0.9 (aka nucleus sampling; see Holtzman et al. paper)
• max_tokens: 220–320 per section
• stop: ["\n\n--", "END"] to prevent run-ons
• presence_penalty small nudge to avoid repetition for multi-paragraph outputs
  For parameter details, see the OpenAI API reference.

3. Light post-processing and ranking

I occasionally generated two variants (e.g., one risk-forward, one opportunity-forward) and scored them against a checklist: number of facts used, coverage of projection range, mention of schedule, explicit draft guidance. The highest-scoring one shipped.

Combining generative text with pre-determined blocks

I was still struggling a bit controlling the quality and output of the generated reports. I did end up blending some pre-written sections with the AI response:

• “Usage & Role” – partially templated using depth chart and team tendency tags.
• “Injury/Availability” – templated sentence starters fed with any practice/injury notes.
• “Draft Plan” – templated rubric with round, risk, ceiling filled by the model but constrained to allowed phrases.

This hybrid kept tone consistent and let the model spend its creativity where it adds value: transitions, nuance, and connecting dots.

What worked, what didn’t

Worked

• Retrieval before generation: huge drop in hallucinations.
• Low temperature + explicit constraints: more concise, fewer cute metaphors.
• Shorter sections composed together: better than one giant 600-token ask.
• Curie/ada for classification and filtering tasks around the pipeline (cheap and fast).

Didn’t

• Letting the model summarize and opine and recommend in one pass.
• Overly clever prompts. Plain-language instructions beat flowery, lengthy prompts.
• Trusting the model with “latest news” without pinning a date (“as of June 2022”) and quoting the source snippet.

Lessons I’m taking forward

• Ground first, generate second. Retrieval isn’t optional if you care about accuracy.
• Shape the page, not just the paragraph. Break the problem into labeled sections with clear constraints.
• Use the right model for the job. Davinci for narrative, Curie/Ada for the plumbing.
• Keep it dated. “As of June 2022” prevents the model from time-traveling your facts.
• Expect to iterate. Prompt tuning is important. Small changes (tone, constraints, stop tokens) mattered more than fancy tricks.

I’m encouraged by how fast the field is moving. Even in my short tests I saw clear improvements across OpenAI’s releases. This technique will only get better and more accurate.

Give it a Try

Each player profile now blends my projections, schedule context, team/role signals, and fresh notes into a scouting report that reads like something you’d expect from a seasoned analyst.

If you want actionable, AI-assisted analysis with some of the most accurate projections in the industry, give Draft Punk a spin. Set your league settings, browse the player pages, and see how the reports line up with your gut. I think you’ll draft with more confidence. And have more fun doing it.

You don’t get better crops by yelling at the plants. So why do software engineering managers try and get better team velocity by “yelling” at their team?

It’s all too common to see managers take a page out of that “pressure” playbook. Any of these look familiar?

• Aggressive deadlines
• Tracking individual diffs or PR outputs
• Think adopting Agile is the silver bullet
• Assuming your team isn’t competent
• Blaming individuals or other teams
• Thinking one more coding tool is all that’s missing

The real secret is in how you prepare the ground.

A manager and team that obsesses with that preparation will win every time. And a lot of that work happens well before it’s time to “code.”

Applying Farming to Management

Let’s go through each aspect of “farming” and how it applies to engineering leadership

Don’t shout at the crops

Pressure tactics are among the least effective methods for gaining commitment. Your employees may do the task but they aren’t bought in. It can be used in emergencies for quick, short-term actions, but over-reliance will harm leadership effectiveness.

Are you the boss farmer or leader farmer?

Don’t jump to conclusions when progress stalls. Delays are rarely from lack of effort. Instead, it’s likely unclear requirements, shifting scope, or tech debt.

Don’t blame the crop

Getting instant results is rare. Velocity improves with clarity, tooling, and team trust.

Be deliberate about finding the reasons behind a problem

Often leaders expect instant dramatic results, but engineering / product development has ramp-up time: building team trust, setting up CI/CD tooling, cleaning up tech debt, improving flow.

Don’t uproot crops too early

Too much thrash kills overall velocity. Sprinting in a zig-zag will make less overall progress than clear focus on a solid plan.

So you need to pick a direction, give the team enough runway, then evaluate. Disciplined “pivot vs persevere” decisions rather than knee-jerk changes.

The research backs this up. Google’s long-term study on the best managers noted that great managers set a crisp direction, make hard calls, and then empower the team to execute without micromanaging.

Choose the best plants for the soil

Match the right team, tech stack, and process to the problem space. 0->1 startups and enterprise-scale infra need different solutions. One-size-fits-all tools rarely work everywhere. Consider what part of the s-curve your team, org, or company is one and tailor the team, talent, and process as appropriate.

Different phases of the s-curve have very different needs.

Other considerations? Growth teams need a lean process and rapid experimentation. Platform or Infra teams need more structured process, metrics around reliability. Product teams need a healthy mix of them all. You wouldn’t start a basketball team of all centers would you?

Irrigate and fertilize

Invest in your teams with mentorship, better tools, and clear strategy. Provide support and direction to keep your org healthy and productive. Focus on building a great team later vs. a good team today.

Top managers focus on upskilling their teams, focus on career development, and creating healthy working environments.

Remove weeds

Leaders unblock their team, get rid of bad processes, unclear roles, or unnecessary meetings. Weeds choke growth just like layers and layers of bureaucracy.

Even you as the manager might the biggest blocker

Remember you will have good seasons and bad seasons

Build a culture of resilience. Build slack into your plans, and stay flexible as markets change.

Keeping a team at high stress throughout the year leads to burnout and a reduced ability to react to unplanned events. At the same time, keeping a team too calm can lead to complacency and also leaves them ill-equipped to respond when something unexpected happens.

The key to fostering resilience is routine and predictability, paired with intentional doses of stress: moments of challenge where the team is pushed. When those moments are spaced and purposeful, the next challenge is absorbed better. This “sine-wave” approach keeps your team on their toes while avoiding burnout and sustained low performance.

Want Better Team Performance? Start with Yourself

So while you can focus on better coding tools, better sprint processes, and track team output, if you want to truly unlock better performance, start with yourself.

When you analyze the reasons for poor individual or team performance, you’ll often find you have most of the answers.

Are you…

• Obsessing about your business strategy?
• Understanding your customers?
• Thinking about org design?
• Ensuring your coaching and hiring the right people?
• Unblocking and aligning your team?
• Ensuring the team is creating a good extensible architecture?
• Thinking about practical processes for your team / phase / industry?

Nail those inputs and your team’s output will zip!