AnnouncementPloy raises $27M seed to build the first website platform that grows your business.

Valuemaxxing your AI agent

Getting maximum value out of any LLM

Lorenzo Gentile

Lorenzo Gentile

AI Engineer @ Ploy

01 / KV Cache Crash Course

KV Caching 101

Without caching, every step reprocesses the full history. With caching, completed work becomes cheap input on the next step.

Without caching · every step cold

The full history is billed as uncached input again.

× Reprocess everything
Step 1context
Sys prompt + user msguncached input
Tool call 1output tokens
Step 2context
Sys prompt + user msguncached input
Tool call 1uncached input
Tool result 1tool result
Tool call 2output tokens
Step 3context
Sys prompt + user msguncached input
Tool call 1uncached input
Tool result 1tool result
Tool call 2uncached input
Tool result 2tool result
Responseoutput tokens

With caching · append-only prefix

Each completed step becomes the next step’s cache hit.

✓ Cache compounds
Step 1context
Sys prompt + user msguncached input
Tool call 1output tokens
Step 2context
Sys prompt + user msgcached input
Tool call 1cached input
Tool result 1tool result
Tool call 2output tokens
Step 3context
Sys prompt + user msgcached input
Tool call 1cached input
Tool result 1cached input
Tool call 2cached input
Tool result 2tool result
Responseoutput tokens

Cold cache · production sample

10.4s

Average TTFT at 100–200K input

Failure turn

$32.75

20.8% hit · ~9× cost

Warm cache · production sample

3.4s

Average TTFT at 100–200K input

Healthy turn

~$3.70

90–98% cache hit

Ploy production evidence: 91,485 spans measured warm TTFT at 3.4s vs 10.4s cold. In one image-cache failure, the worst turn cost $32.75 at 20.8% cache hit versus healthy turns around $3.70 at 90–98%. Pricing: OpenAI API, standard processing, per 1M tokens.

02 / Frozen means frozen

One changed token can invalidate everything

Trying to save tokens by dynamically loading tools can backfire.

Stable system prompt + tools

The provider tool catalog stays byte-identical.

✓ Cache grows
Step 1context
System prompt + toolsANALYTICS SEO ADSuncached input
User requestuncached input
Step 2context
System prompt + toolsANALYTICS SEO ADScached input
User requestcached input
Tool call 1 + result 1uncached input
Step 3context
System prompt + toolsANALYTICS SEO ADScached input
User requestcached input
Tool call 1 + result 1cached input
Tool call 2 + result 2uncached input

Changing tool set

Adding a provider tool rewrites block one.

× Always cold
Step 1context
System prompt + toolsANALYTICSuncached input
User requestuncached input
Step 2context
System prompt + toolsANALYTICS SEOuncached input
User requestuncached input
Tool call 1 + result 1uncached input
Step 3context
System prompt + toolsANALYTICS SEO ADSuncached input
User requestuncached input
Tool call 1 + result 1uncached input
Tool call 2 + result 2uncached input

× Changing system prompt

✓ Keep the system prompt byte-identical

× Dynamic system context (e.g. current timestamp)

✓ Fetch changing context only when needed

× Changing tool set

✓ Keep provider tools stable; load schemas at the tail

Ploy evidence: warm TTFT averaged 3.4s vs 10.4s cold across 91,485 production spans; one image-cache failure made the same work roughly 9× more expensive.

03 / Cache-safe tool loading

Load tools on demand without breaking the cache

Tool schemas are passed with every LLM call. Optimizing them creates real value—but only when done correctly.

Context

Stable prefix stays byte-identical

ALWAYS ON TOOLS + “ON-DEMAND TOOLS” TOOL01
SYSTEM PROMPT02
USER REQUEST + TOOL HISTORY03
append
Tool call04

onDemandTools({ action: "load", tool: "analytics" })

Tool result · analytics tool schema05
{
  action: "query" | "sql";
  queryConfig?: {
    sqlQuery?: string;
    metrics: Metric[];
    dimensions?: Dimension[];
    filters?: Filter[];
    limit: number;
  };
}

Agent uses the loaded schema

onDemandTools({
  action: "execute",
  tool: "analytics",
  input: {
    action: "query",
    queryConfig: {
      metrics: [{ fn: "count", alias: "views" }],
      limit: 2
    }
  }
})

Sample tool result

{
  rows: [
    { page: "/pricing", views: 1_284 },
    { page: "/demo", views: 932 }
  ]
}

Eval results

Pass rate was identical

Common tools must stay available and on-demand tools must be discoverable

−45%

schema tokens

−35%

total input

−33%

cost

04 / Freeze dynamic system context

Freeze dynamic context and leverage breakpoints

Snapshot workspace memory at chat start, tell the agent it is frozen, and fetch fresh state through tools only.

System-context example

<workspace_context frozen="true">
  workspace: "Ploy Decks"
  memory_version: 13
  rule: "Frozen for this chat. If reality moved,
         ask a tool—not your memory."
</workspace_context>

Two cache breakpoints isolate the stable prefix, memory snapshot, and user message

CachedUncached

Chat A

Initial snapshot

TOOLS + SYSTEM PROMPT

Cache write

breakpoint 1

WORKSPACE MEMORY · V12

First use · uncached

breakpoint 2

USER CHAT · BUILD WEEKLY REPORT

New tail · uncached · memory updated

Chat B

Memory changed

TOOLS + SYSTEM PROMPT

Cache hit

breakpoint 1

WORKSPACE MEMORY · V13

Changed tail · uncached

breakpoint 2

USER CHAT · CASUAL CHECK IN

New tail · uncached · no memory updates

Chat C

Memory unchanged

TOOLS + SYSTEM PROMPT

Cache hit

breakpoint 1

WORKSPACE MEMORY · V13

Same snapshot · cache hit

breakpoint 2

USER CHAT · UPDATE CHARTS

New tail · uncached

−89%

first-message cost

New chat

−5%

production token spend

Overall

GPT 5.6 implementation guide: ploy.ai/blog/migrating-a-production-ai-agent-to-gpt-5-6

05 / Steps are expensive

Batch tools to the max

Batch multiple tool calls into one step—and multiple actions into one tool call.

Before · 3 agent steps

Context accumulates between calls

Step 01context

Cached input

TOOLS + SYSTEM

Uncached input

USER REQUEST

Output tokens · high cost

HIDDEN THINKING 1

Output tokens · tool call

EDIT FILE A

Tool result

EDIT FILE A RESULT

Step 02context

Cached input

TOOLS + SYSTEM

Cached input

USER REQUEST

Cached input

THINKING 1

Cached input

EDIT FILE A CALL + RESULT

Output tokens · high cost

HIDDEN THINKING 2

Output tokens · tool call

READ FILE B

Tool result

READ FILE B RESULT

Step 03context

Cached input

TOOLS + SYSTEM

Cached input

USER REQUEST

Cached input

THINKING 1

Cached input

EDIT FILE A CALL + RESULT

Cached input

THINKING 2

Cached input

READ FILE B CALL + RESULT

Output tokens · high cost

HIDDEN THINKING 3

Output tokens · tool call

READ FILE C

Tool result

READ FILE C RESULT

After · 1 agent step · same outcome

2 batched tools

Cached input

TOOLS + SYSTEM

Uncached input

USER REQUEST

Output tokens · high cost

ONE HIDDEN THINKING BLOCK

Output tokens · tool call 1

EDIT FILE A

Output tokens · tool call 2

READ [FILE B, FILE C]

Result 1

EDIT FILE A RESULT

Result 2

READ [FILE B, FILE C] RESULTS

Real eval results

−32%

Tool calls

−20%

Agent steps

−14%

Cost

Same

Pass rate

33-case Ploy eval after batched reads/edits and save-owned smoke tests; code:read calls −71%. Parallelize only independent work.

06 / Slim the harness

Every byte a tool returns becomes future input

Return the minimum evidence needed for the next decision. Make deterministic work deterministic.

HARNESS

59%

Tool traffic

Calls, results, and screenshots. Results alone are 43%; user text is under 1%.

HARNESS

−70%

Exa highlights

42,552 → 12,809 chars; 30/30 pass both arms.

HARNESS

−54%

Efficient tool design

apply-theme tool avoided ~200-line CSS echo.

Exa estimate: ~$37K/year saved in input tokens.

Quality held constant before efficiency shipped.

07 / Closing

Quality first
Efficiency always

The playbook

  • Cache stable prefixes
  • Load tools deliberately
  • Batch independent work
  • Keep tool results lean
  • Measure the completed outcome
Lorenzo Gentile

Lorenzo Gentile

AI Engineer @ Ploy

linkedin.com/in/lorenzo-gentile-2292b281ploy.ai/blog/migrating-a-production-ai-agent-to-gpt-5-6