Latency Lessons From Building a ReAct AI Agent for Agentic Search

Egnyte AI surfaces insights from an organization's documents for regulated industries—life sciences, financial services, architecture, engineering, and construction—and does that within existing permissions and compliance controls. Our AI Assistant is the conversational front door. Ask a question about your documents in plain language, summarise a contract, find the latest version, pull a compliance clause, and get an answer grounded only in the files you're permitted to see.

Under the hood, it's a ReAct agent searching across content. It worked, but it was chatty in ways that compounded the problem: a typical answer took 5–7 tries, and because the system prompt was rebuilt at every turn, the prompt cache never hit. Each problem surfaced in production traces, and each fix revealed the next bottleneck. 

In the end, a handful of changes did most of the work as we improved AI Assistant, which included: 

• Stopping loading domain knowledge the agent didn't need
• Collapsing a nested search sub-agent into a single loop
• Trimming an overgrown prompt and teaching the agent to stop early instead of chasing every lead
• Renaming the tools after the bash commands every engineer already knows

What follows is a snapshot of our work, along with a look ahead at what comes next.

The Latency Model

A ReAct agent is a loop: 

• The model emits a tool call plus a rationale, or a final answer
• The runtime runs the tool
• The result is appended
• The model is re-invoked

In practice, this means every extra tool call or reasoning step increases the  wall-clock time to complete the process: 

(LLM round-trip + tool execution) × turns + final-answer streaming

Next, I’ll describe how we tuned the three knobs mentioned earlier—turns per session, tokens per turn, and wall-clock per tool call. The traces kept showing the same three habits: 

• Every request was treated like deep research
• All heavy retrieval was delegated to a search sub-agent running its own nested ReAct loop
• Broad keyword search—even when a precision filter was used—would have worked better

Clear the Structural Waste First

The first pass was cleanup, most of it not ReAct-specific. The sub-agent's prompt prefix was rebuilt every turn because four template variables (turn count, remaining turns, and the like) were spliced in each iteration, collapsing them into one stable placeholder, allowing the cache to cross the loop. 

The thought parameter (the reasoning field in tool call schemas) went from required to optional, since every required field is serialized and sent on every request, regardless of whether its content changes turn to turn. 

We deduplicated the tool registry and added a 30-second cap on external tool-server fetches, so a slow upstream can't tie up a worker. 

These changes didn’t directly move the headline numbers, but they’re needed to make the next big change easy. 

Load Domain Knowledge on Demand, Not up Front

Some queries hinge on metadata tags, such as the structured labels Egnyte applies to documents to classify them, like a contract's status or a project's phase. But the model doesn't know their shape: which fields exist, what enum values are legal, what operators apply. It discovered all of it at runtime, one call at a time, resulting in three round-trips before any retrieval, even though most queries needed no metadata at all.

So we introduced a capability, a SKILL.md file with YAML front-matter declaring a metadata namespace and an allowed-tool list, plus a body describing the domain protocol in plain language. One get_capability("<name>") call returns both halves:

The schema is fetched live, so the model always sees current definitions. Everything it needs to act in a domain—the schema and protocol—arrives in a single call instead of three. A capability loads only when its query type appears, meaning specialized instructions cost nothing on requests that don't need them. For example, in a system with projects and contracts, a customer whose users never ask about projects will never load that capability.

There is a compliance benefit here that’s as important as the latency one. Because a capability loads only on queries that need it, a life sciences customer's contract schema never surfaces in a project's query—that isolation is structural rather than enforced by policy. For customers in regulated industries, this distinction matters.

Underneath sat an earlier, simpler decision: prefer the precision tool and make the broad one an explicit fallback, not a co-equal option. For a query naming a document type like "contracts", "invoices", or "marketing plans", the agent assumes metadata tags exist and looks them up first. Keyword search is the fallback, scoped to content questions only. Structured search returns precise hits in one turn, and broad keyword search returns noise that the agent must then re-query to narrow. And once a precision search works, the SKILL.md hard-stops the model from second-guessing it:

"If any metadata-filter search returns relevant results - terminate. Do not add a broad keyword search on top."

Default to a Flat Loop

The search sub-agent existed for a reason: to isolate the search tool's context from the outer agent. The ReAct agent already juggles the caller's tools plus any MCP connectors (third-party tool servers a customer can plug in). An agent staring at 30 or more tools makes worse choices about which to call, and hiding the retrieval sub-stack behind one search_agent delegation kept the outer surface small.

But by the time we revisited it, the cost of not using the sub-agent had dropped. Capabilities had moved the domain protocol behind a single discovery call, and dynamic prompt assembly (next section) kept the system prompt scoped to loaded tools. The context bloat that the sub-agent guarded against was gone, and the delegation was pure overhead.

Before: Two loops, one delegation per retrieval

After: One flat loop

Every delegation cost dispatch overhead, a state round-trip through a synthetic HumanMessage, and an extra outer turn to interpret the summary. The outer agent never saw the sub-agent's intermediate retrievals, so it couldn't learn from them. The two prompts also described overlapping responsibilities and drifted apart. So we collapsed them: We dropped the delegation tool, loaded the retrieval tools directly into the React agent, and moved the sub-agent's prompt into a single SKILL.md appended when those tools are present. One prompt, one source of truth.

Turn count is the primary metric tracked here. Wall-clock response time improved proportionally across all three queries and specific timing data is omitted to avoid anchoring on infrastructure conditions that vary by deployment.

Flattening also lets us delete tools that were pure sub-agent bookkeeping. shortlist_results had two jobs: tracking "documents I've decided are relevant" across the sub-agent's turns, and curating results page-by-page during pagination. Both died: the first with the sub-agent, the second once "don't paginate unless asked" became the default.

Sub-agents have a real place—work that needs its own state, a different model, distinct responsibilities—but every delegation is dispatch overhead, a state round-trip, and a layer of invisibility. Default to a flat loop until you have a concrete reason to split.

And when you split, watch for bookkeeping tools that creep in like: shortlist this, remember that, return your final list. They aren't actions in the world—they exist to bridge agent layers and flattening the layers leaves them nothing to do.

Less Prompt, Sharper Directives

Here’s a close look at how we fine-tuned prompts for better results.

Cut the Prompt Down, Then Assemble It per Request

The default prompt opened with "You are an AI specialist driven by persistence and analytical rigor" and ran through six instruction blocks that fought each other. “Depth” wanted detail on every query, while “Research Methodology” budgeted 5 to 10 tool calls and said, "never stop after one failed search". The model treated every request like deep research and burned turns, exactly as instructed.

We cut it to one paragraph and four rules, opening with "You are a helpful AI assistant. Use available tools to answer questions accurately." Always-on tokens dropped ~30%, and the search budget fell to 2 to 3 calls for medium tasks, up to 5 for deep ones. The rest is assembled dynamically, appended only when the relevant tools load:

A search-only request no longer carries Excel instructions, while a request with no tools carries almost nothing.

Keep Cross-Tool Rules in the Prompt, Tool Specifics in the Tool

The cleanup surfaced a sharper rule: The system prompt carries only cross-tool instructions, and tool-specific guidance belongs in the tool's description.

Tool descriptions load with the tool, so if the tool isn't loaded, neither is its guidance, with no conditional logic to maintain. The 17-line response-framing block doesn’t need to be repeated into every tool's thought parameter—centralizing it once in the system prompt shrank every tool schema by those 17 lines.

When Classification Is Uncertain, Fan Out in Parallel

The original protocol asked the agent to discover metadata first, then search. This is sensible when the query needs metadata, but wasteful when it doesn't—and most don't. We tried teaching the model to classify upfront, but the line is fuzzy: "contracts about Acme" is keyword-only, "approved contracts about Acme" needs a status filter, and the model couldn't tell them apart without doing some of the work first. So we stopped classifying and told it to do both at once:

"On the first turn, issue all applicable search calls and metadata-discovery calls simultaneously. Since the calls run in parallel, there is no latency cost for issuing multiple at once."

If metadata mattered, the next turn used it. If not, the parallel call cost nothing. The lesson: When classification is uncertain and the alternatives are independent, parallel fan-out beats serial reasoning. A model predicting whether it needs X is slower and less reliable than just fetching X alongside everything else.

Tell the Agent When to Stop, Not to Be Exhaustive

This was the most directional change. The old prompt encoded exhaustiveness, "never stop after one failed search", a 5-to-10 call budget for "deeper research", and the model ground through all of it on queries that needed none. The new SKILL.md says the opposite: "Terminate early. Stop as soon as you have sufficient relevant results," and "Never ask for clarification. Infer the most likely interpretation, state your assumption, and proceed". You can't ask an agent to be both exhaustive and efficient, so we picked efficiency, since the model can always do more when asked.

The same instinct reshaped retrieval. We sized result sets to query intent: A QnA query that synthesises an answer pulls fewer results (15) than a "list everything" query (20), where one generic limit had been over-fetching for answer-synthesis. And pagination was capped; "do not paginate unless the user explicitly asks for exhaustive results". Pagination feels like serving the user, but it's 3 to 4 extra turns for results no one reads.

Name Tools After Bash Verbs. Compose, Don't Fan Out.

We also improved results by adjusting tool names. Here’s how that worked out.

Familiar Names Give the Model a Free Mental Model

The retrieval tools had organic names and awkward contracts, for example, a content-fetch tool gated on IDs from a prior search (every content question paid a two-turn tax), and a search tool whose docstring encouraged fan-out across keyword variations. We renamed the surface to bash verbs:

The model has internalized bash semantics from training and carries strong priors on argument shapes, composition patterns, and expected behavior for each verb. Naming tools after shell commands gives it a free mental model with no prompting cost, and that prior extends naturally to how the tools compose. It spends no reasoning relearning an in-house taxonomy. 

The small primitives like tree, stat, metadata, came from traces where the model walked a tree with repeated ls calls or fetched a whole file to read one tag. Each one collapsed several turns into one.

Fold Two-Turn Taxes Into a Single Tool

The biggest contract change: grep does the search and returns the content in one call. "What does this contract say about termination?" used to need two turns: find(query='termination') then cat(file_ids=[...]). Now it's one call returning the matching passages. Full-file reads still go through cat, but those are the cases where you actually want the whole file.

find changed the other way: a single composed query, not a fan-out.

"Compose, don't fan out (mandatory): combine keyword variations into a single call using boolean operators - query='contract OR agreement OR NDA'. Issuing one call per variation is prohibited and will be flagged as an anti-pattern."

The docstring names the failure mode to head it off: "Realizing mid-search that you forgot a variation and issuing a second call is a planning failure—equivalent to fanning out per synonym." One composed query usually replaces two to four parallel ones, and parallel calls aren't free in the conversation context even when they're free on the wire.

That discipline lives in the tool's description, not the system prompt, it instructs when to reach for the tool, when not to, what it costs:

Shape Tool I/O So Outputs Feed the Next Input

The model wires the outputs into inputs, because the parameter shapes mirror bash I/O, captures one output, and passes it as the next argument. find returns file IDs that drop into grep's file_ids, ls returns items stat can inspect and tree returns folder paths that scope a later find. Having seen find ... | xargs grep ... countless times in training, the model reaches for the same shape unprompted.

Mine What You Already Have Before Fetching More

The read tools form a granularity ladder—find's snippets (a couple hundred characters around the match), head (the start of a file), and grep (passages matching a query), cat (the whole file)—each rung costing more tokens. Pick the lowest rung that answers the question.

The traces showed the model ignoring it: find returned a snippet that already held the answer, and the model called cat anyway. So find's docstring now says "each result's snippet frequently contains the answer outright—a date, a name, a status; read it before escalating to grep, head, or cat", and grep's makes the cost explicit: "each extra file roughly multiplies the token cost". Naming the cost changes behaviour where vague guidance like "use sparingly" doesn't.

Carry Answers Forward, Not the Scratch Trace

ReAct sessions accumulate messages fast: each turn adds an AIMessage of tool calls plus ToolMessage results that can run to kilobytes. For follow-ups in the same conversation, we don't replay that - we keep only the HumanMessage and the final AIMessage of each prior turn, dropping everything between:

The follow-up turn sees the logical conversation: Q → A → Q → A, not the scratch trace, and dropping the 10–30 KB a search-heavy turn produces keeps the cache prefix stable. The trade-off: The model can't point back to a specific earlier tool result, but the final answer already captured what was salient, and it can re-fetch with find or grep if needed. Treat the visible conversation as the model's contract with the user. The tool trace is implementation detail and can be discarded.

Where It Stands Today

The agent is meaningfully faster, fewer turns, smaller prompts, a cache-stable prefix, a tighter tail. The compound effect beat any single change because the iterations stacked: Flattening the loop made the prompt cuts visible; the bash-verb tools composed freely once out from behind the delegation; the pagination curbs landed because the SKILL.md was finally the single source of truth; and snippet-mining worked because results sat in the outer agent's context instead of being summarized away inside a sub-agent.

Two threads next:

• A framework-managed agent loop: Summarization, native parallel tool calls, async streaming, and middleware instrumentation are awkward to bolt onto a hand-rolled graph and come free from a framework that owns the loop. Same flat-loop architecture, more pluggable surface.
• External skills over MCP: The capability pattern works well for protocols we control directly. We’re actively extending it to MCP-connected skills, applying the same load-only-when-needed model to third-party tool servers. The architecture is the same—MCP just widens the surface of what can plug in.

Almost every fix above came from sitting with one real trace and asking the same question over and over:

Why did the model take this turn, and what would have let it skip it?

The patterns are outcomes of that question. New traces keep raising new questions, so this list will look different in six months. It's a snapshot, not a conclusion.