Building a Production LLM Agent in Rails

An LLM agent is a system that uses a language model not for a one-shot answer, but as a controlling “brain” that decides in a loop which actions to take to reach a goal — choosing each next step from the results of the previous ones.

The key difference from an ordinary LLM call: a plain call is “input → answer”. An agent is a loop: the model reasons, calls a tool, gets a result, reasons again, and so on until the task is solved.

Case

An insurance company wants to bring AI into its product to reduce the load on operations staff, lower staffing costs, and improve responsiveness — making the system more user-friendly and less error-prone.

AI agents should answer most user questions such as “What does my policy cover?”, “How much will my health insurance cost this year?”, or “I’d like to discuss my payouts from last month.”

Questions fall into two fundamentally different types:

• “What does my policy cover?” — the answer lives in the text of documents.

• “How much will my health insurance cost this year?” — the answer depends on the client’s personal data and requires a calculation.

This fork determines the whole architecture. The first type is handled by knowledge-base search; the second by calling code. The system must do both — and decide on its own which to apply.

Part 1. Core concepts

Model

We use “model” as shorthand for LLM (large language model) — the neural network that generates text. For example, Anthropic Claude Sonnet 4.6.

Prompt

A prompt is the input text fed to a language model. All an LLM does boils down to “given text → predict the continuation”; the prompt is exactly the text the model builds on.

A prompt is not just the user’s question. It usually consists of several parts:

• System prompt — sets the model’s role, rules, and behavior for the whole conversation.

• User message — the question itself.

• Context — supplied data: retrieved document fragments, conversation history, results of tool calls.

All of this is concatenated and sent to the model at every step. Answer quality depends directly on how the prompt is worded — hence the discipline of prompt engineering.

RAG

RAG (Retrieval-Augmented Generation) is an approach where the model first finds relevant information in an external source before answering, then generates an answer grounded in that material.

What this solves:

• Stale knowledge — an LLM only knows data up to its training cutoff. RAG gives access to current and private company documents.

• Hallucinations — the answer is tied to specific retrieved fragments, which makes it verifiable and citable.

• Cost and privacy — no need to fine-tune the model; data stays in your own database.

Two stages of RAG: retrieval (documents are pre-split into chunks and turned into embeddings; the query looks for the nearest ones) and generation (the retrieved content is placed into the prompt, and the model answers from it).

LLM agent

An LLM agent uses the model as a controlling brain that decides what to do next. A plain call is “input → answer”. An agent is a loop — “reason → act → observe” — that repeats until the task is solved.

Building blocks:

• LLM as orchestrator — decides what to do next.

• Tools — functions for acting on the outside world: a DB query, a search, a calculation. The model picks the tool and its arguments (function calling).

• Memory / context — conversation history and intermediate results.

• Control loop — until the task is solved or a step limit is reached.

Part 2. Architecture

At a high level the flow is simple:

Request routing

Through function calling, the agent classifies intent:

• Reference question (“what does policy X cover”, “how do I file a claim”) → RAG over the knowledge base.

• Personal calculation (“how much does MY insurance cost”, “when does my policy expire”) → DB functions and the premium calculator.

• Mixed → both, then synthesis.

A cost question almost always takes the second path, not RAG: the answer depends on the client’s personal data, not on document text.

Security principles (from day one)

Insurance is a financial and personal domain where mistakes are expensive. Four rules define the design:

1. Authentication before calculation. The model must not decide whose user_id to use — the identifier comes from the verified session, not from the request text. Otherwise a user could learn someone else’s policy cost.

2. Numbers are computed by code, not the LLM. The model phrases the answer, but the amount is always returned by a deterministic function. Premium arithmetic cannot be trusted to the model.

3. Current revision only. A tariff answer must be filtered by the document’s validity date — you can’t serve last year’s rate.

4. Audit. Logging every tool call is mandatory.

On the left is the agent’s decision zone. On the right is the deterministic code zone. Nothing on the right is exposed as a tool parameter — otherwise the model gains control over it.

Part 3. Implementing the agent

At CleverLabs we use the following stack: Rails, RubyLLM, PostgreSQL (pgvector + neighbor), Sidekiq, Falcon.

RubyLLM fits this design well: tool-calling and embeddings are supported natively; pgvector + neighbor for RAG; Sidekiq / Falcon for async handling of heavy requests and indexing.

Tools layer

The main security technique: user is injected into the tool’s constructor from the verified session, not passed by the model as a parameter. The model physically cannot substitute someone else’s ID.

# app/tools/calculate_premium.rb
module Tools
  class CalculatePremium < RubyLLM::Tool
    description "Calculates the cost of the client's insurance policy for a given year. " \
                "Returns the total amount and a breakdown. Use for questions about price."

    param :year, type: :integer, desc: "Calculation year (e.g., 2026)"
    param :plan, type: :string, required: false,
          desc: "Plan type: basic, standard, premium. If omitted — the client's current plan."

    # user from session, NOT from the model
    def initialize(user:)
      @user = user
    end

    def execute(year:, plan: nil)
      result = PremiumCalculator.new(user: @user, year: year, plan: plan).call
      {
        total: result.total,
        currency: result.currency,
        breakdown: result.breakdown, # array { name:, amount: }
        plan: result.plan_name
      }
    rescue PremiumCalculator::MissingDataError => e
      # the model sees the error and asks a clarifying question
      { error: e.message, missing: e.missing_fields }
    rescue => e
      Rails.logger.error("[CalculatePremium] #{e.message}")
      { error: "Could not complete the calculation. Please contact a manager." }
    end
  end
end

Note the error handling: the tool returns a structured error with a missing field instead of crashing. The model reads this and asks the client a clarifying question on its own — that’s a working agent loop.

We’ll come back to why description matters. For now, note that description is the agent’s routing logic: which code (tool) it should call.

Agent orchestrator

# app/agents/insurance_agent.rb
class InsuranceAgent < RubyLLM::Agent
  model "claude-sonnet-4-6"
  temperature 0.2 # low temperature — financial domain
  inputs :user    # verified client from the session

  instructions do
    <<~PROMPT
      You are an insurance company assistant. Respond politely and accurately.
      Rules:
      - ALWAYS obtain the policy cost via the calculate_premium tool.
        Never compute or invent amounts yourself.
      - Retrieve active policy data via get_policy.
      - Answer reference questions only from search_knowledge_base.
        If the information isn't there — say so honestly.
      - If a tool returns an error with missing fields — ask a clarifying question.
      - Accompany exact amounts with a note that the total is confirmed
        by an official offer.
      - For complex legal questions, offer to connect with a manager.
      - Never request or reveal another client's data.
    PROMPT
  end

  tools do
    [
      Tools::CalculatePremium.new(user: user),
      Tools::GetPolicy.new(user: user),
      Tools::SearchKnowledgeBase.new
    ]
  end
end

The system prompt here is not decorative text — it is a behavior-control layer. Each rule closes a specific risk: invented amounts, answers without a source, leaking someone else’s data.

Entry point

# app/controllers/chat_controller.rb
class ChatController < ApplicationController
  before_action :authenticate_user! # user comes ONLY from here

  def create
    ChatResponseJob.perform_async(current_user.id, params[:message], session_chat_id)
    head :accepted
  end
end

# app/jobs/chat_response_job.rb
class ChatResponseJob
  include Sidekiq::Job

  def perform(user_id, message, chat_id)
    user = User.find(user_id)
    agent = InsuranceAgent.new(user: user)
    response = agent.ask(message) do |chunk|
      broadcast_chunk(chat_id, chunk) # each fragment goes to the front immediately
    end
    AuditLog.create!(user: user, query: message, answer: response.content)
  end
end

Part 4. The RAG layer

RAG requires upfront preparation: documents must be turned into data the agent can search. Each document is split into chunks, each chunk becomes an embedding (a vector of numbers), and those embeddings are stored in a vector database — PostgreSQL with pgvector. At query time the user’s question is embedded the same way, and the most relevant chunks are retrieved to build the answer.

The top loop is offline indexing: parsing → chunking → enrichment → embeddings. The bottom loop is online retrieval: query → hybrid search (RRF) → reranking → LLM. They’re connected by the chunk store. The validity-date filter is applied during retrieval and rules out old revisions — of tariffs, for example.

Pipeline stages:

1. Structure-preserving parsing — text and tables are processed separately.

2. Structural chunking — by contract sections; tables are not split.

3. Enrichment — section context, version and plan metadata.

4. Embeddings → pgvector, plus tsvector for full-text search.

5. Hybrid search (vector + full-text, RRF) with a metadata filter.

6. Reranking via a cross-encoder.

7. Context assembly → LLM with source citation.

Building the knowledge base (pgvector + neighbor)

Indexing turns files into searchable chunks. Make this process asynchronous: a Sidekiq job processes submitted documents in the background.

Data schema

create_table :knowledge_chunks do |t|
  t.references :document, null: false, foreign_key: true
  t.text     :content            # original text shown to the user
  t.text     :embed_text         # enriched text that was embedded
  t.string   :heading
  t.string   :chunk_type         # "text" | "table"
  t.string   :plan               # filter: basic/standard/premium
  t.date     :valid_from
  t.date     :valid_until
  t.integer  :page
  t.vector   :embedding, limit: 1536 # depends on provider (OpenAI 1536, Voyage 1024, …)
  t.tsvector :content_tsv        # full-text search
  t.timestamps
end

add_index :knowledge_chunks, :embedding, using: :hnsw, opclass: :vector_cosine_ops
add_index :knowledge_chunks, :content_tsv, using: :gin

The valid_from / valid_until fields are not a formality: they keep the agent from serving a tariff from an old revision.

Indexing via an asynchronous Sidekiq job

# app/jobs/index_document_job.rb
class IndexDocumentJob
  include Sidekiq::Job

  def perform(document_id)
    doc = Document.find(document_id)
    sections = DocumentParser.new(doc.file_path).call
    chunks = StructuralChunker.new(sections).call
    ctx = ChunkContextualizer.new(doc.summary)

    chunks.each do |chunk|
      enriched  = ctx.call(chunk)
      embedding = RubyLLM.embed(enriched[:embed_text]).vectors

      KnowledgeChunk.create!(
        document:   doc,
        content:    enriched[:content],
        embed_text: enriched[:embed_text],
        heading:    enriched[:heading],
        chunk_type: enriched[:type],
        plan:       doc.plan,
        valid_from: doc.valid_from,
        valid_until: doc.valid_until,
        page:       enriched[:page],
        embedding:  embedding
      )
    end
  end
end

Parsing: text and tables are different worlds

The main mistake is to run a PDF through as flat text. A tariff table turns into a mush of numbers with no link to rows and columns, and the embedding of such a chunk is useless.

The fix is separate extraction. Tables are serialized into Markdown: both LLMs and embedding models understand Markdown tables better than “flattened” text.

For quality table extraction from PDFs in the Ruby ecosystem, it’s reasonable to call an external parser (camelot / pdfplumber) as a service or via a system call from a background job.

Chunking: structural, not “by N characters”

A fixed window cuts in the middle of a contract clause and loses meaning. The strategy depends on the type:

• Legal text — chunk by logical units (article / clause / subclause). One clause = one chunk. These are natural meaning boundaries.

• Tables — a chunk is the whole table. Never split by rows: the row “Premium — 5000” is meaningless without a header. If the table is huge, split by blocks of rows and duplicate the header into each chunk.

Enrichment: contextual retrieval

This technique gives the biggest quality boost on mixed documents. A bare chunk like | Premium | $5000 | won’t be found for the query “how much does extended coverage cost” — the words don’t match and the vector is weak.

The fix (Anthropic’s idea): before embedding, add one sentence to the chunk about its place in the document. Generate it cheaply via the same LLM at indexing time.

# app/services/chunk_contextualizer.rb
class ChunkContextualizer
  def initialize(document_summary)
    @doc_summary = document_summary # 2–3 sentences describing the whole document
  end

  def call(chunk)
    prompt = <<~PROMPT
      Document: #{@doc_summary}
      Section: #{chunk[:heading]}
      Fragment:
      #{chunk[:content]}
      Give ONE sentence of context: what this fragment relates to within the document. No filler.
    PROMPT

    context = RubyLLM.chat(model: "claude-haiku-4-5").ask(prompt).content
    # the prefix goes into the embedding; the user gets the original content
    chunk.merge(embed_text: "#{context}\n\n#{chunk[:content]}")
  end
end

Hybrid search

Pure vector search fails on exact terms (a clause number, a product name, a specific amount). Pure full-text fails on paraphrases. So use both, merging ranks via Reciprocal Rank Fusion (RRF).

# app/services/hybrid_search.rb
class HybridSearch
  K = 60 # RRF constant

  def initialize(query, plan: nil, on_date: Date.current, limit: 20)
    @query, @plan, @on_date, @limit = query, plan, on_date, limit
  end

  def call
    vec  = vector_results.map.with_index { |c, i| [c.id, 1.0 / (K + i)] }.to_h
    bm25 = fulltext_results.map.with_index { |c, i| [c.id, 1.0 / (K + i)] }.to_h

    fused = (vec.keys | bm25.keys).map do |id|
      [id, (vec[id] || 0) + (bm25[id] || 0)]
    end.sort_by { |_, score| -score }.first(@limit)

    KnowledgeChunk.where(id: fused.map(&:first))
                  .index_by(&:id)
                  .values_at(*fused.map(&:first))
  end

  private

  def base_scope
    scope = KnowledgeChunk.all
    scope = scope.where(plan: [@plan, nil]) if @plan
    # only the current version — critical for insurance
    scope.where(
      "valid_from <= ? AND (valid_until IS NULL OR valid_until >= ?)",
      @on_date, @on_date
    )
  end

  def vector_results
    embedding = RubyLLM.embed(@query).vectors
    base_scope.nearest_neighbors(:embedding, embedding, distance: "cosine").limit(@limit)
  end

  def fulltext_results
    base_scope
      .where("content_tsv @@ plainto_tsquery('english', ?)", @query)
      .order(Arel.sql(
        "ts_rank(content_tsv, plainto_tsquery('english', #{ActiveRecord::Base.connection.quote(@query)})) DESC"
      ))
      .limit(@limit)
  end
end

The date filter solves the “bot served last year’s tariff” problem. The plan filter narrows results to the client’s plan.

Reranking and context assembly with citation

Hybrid search returns about 20 candidates with a rough ranking. An external reranker (Cohere Rerank or Jina) can score relevance more precisely; at the start you can simply take the top 5 from RRF and add reranking once you hit a quality ceiling.

The RAG tool SearchKnowledgeBase returns the top chunks to the LLM with explicit source labels, so the model cites rather than invents.

# in the SearchKnowledgeBase tool
def execute(query:)
  candidates = HybridSearch.new(query, plan: @user&.plan).call
  # top = Reranker.new.call(query, candidates, top_n: 5)
  top = candidates.take(5)
  return "No answer found in the knowledge base." if top.empty?

  top.map do |c|
    {
      source: "#{c.document.title} — #{c.heading} (p. #{c.page})",
      type:   c.chunk_type,
      text:   c.content
    }
  end
end

Returning the source field alongside the text is what lets the model cite instead of inventing. The system prompt already requires answering only from what was retrieved and citing the source — that closes the citation loop.

Part 5. How it all works together

Walking through the query “How much will my health insurance cost this year?”:

1. The controller authenticates the user, takes current_user from the session, and enqueues a background job.

2. The agent receives the question, recognizes the “cost” intent → calls calculate_premium(year: 2026).

3. The tool returns { error: ..., missing: [:plan] } — the plan type is missing.

4. The agent reads the error and asks: “Which plan are you interested in — basic, standard, or premium?”

5. The client answers → the agent calls the tool again with the refined argument.

6. The code returns the amount and a breakdown → the model phrases the answer in plain language with a disclaimer.

7. The answer is streamed to the frontend (Turbo Streams, ActionCable / AnyCable), and everything is written to the audit log.

The route wasn’t hard-coded — the agent built it itself. At the same time, the client identifier and the arithmetic stayed outside its decision zone.

How does the agent know which code to run — RAG search or a calculation function?

The LLM receives a list of tools in the request — their names, descriptions, and parameter schemas. This is part of the prompt:

calculate_premium — Calculates the cost of the client's insurance policy
                    for a given year. Use for questions about price.
                    params: year (integer), plan (string, optional)

search_knowledge_base — Searches the knowledge base for answers to reference
                    questions: coverage terms, rules, procedures, exclusions.
                    Use for general questions that do NOT require the client's personal data.

Then the model generates a continuation — as always.

Tool description is the routing logic itself. You don’t write the router in code — you write it in the text of the descriptions.

Look at the wording: “Use for general questions that do NOT require the client’s personal data.” That is not documentation for a human. It is an instruction that separates two tools. Remove it — and the model will start searching for “how much does my insurance cost” in the knowledge base instead of calling the calculator. A bad description = broken routing, and you’ll debug it by editing text, not code.

That’s why the tool descriptions in this article are written with explicit “use for…” and “do NOT use when…” cues. It’s not a stylistic choice; it is a functional mechanism.

Part 6. System responsiveness: async and Falcon

A request to an LLM agent is slow: several seconds per call, and an agent loop with a tool call plus a follow-up easily takes 5–15 seconds. The user shouldn’t stare at a spinner the whole time, and the server shouldn’t choke on parallel requests. Both problems are solved by an asynchronous execution model — and it matters what exactly it cures.

The nature of the latency

A common misconception is that the latency comes from “slow Ruby”. It doesn’t. During an LLM request the process computes almost nothing — it waits for the API’s response over the network. This is I/O-bound load, not CPU-bound. Language speed has almost nothing to do with it: even in a faster language, a synchronous model would hit the same network wait.

The problem with the classic synchronous model is different: while one thread waits for the LLM’s response, it is blocked and idle. With a dozen concurrent clients the free workers run out and requests queue up — not because there isn’t enough compute, but because the workers are busy waiting.

What async gives

The asynchronous model breaks this link. While one request waits for the API, the process switches and serves others.

In Ruby this is provided by Falcon — an asynchronous web server built on fibers. Falcon spins up a lightweight fiber per request; when a fiber hits a network wait (an LLM call, a pgvector query), control automatically passes to other fibers. Thousands of concurrent “waiting” requests are cheap, because a waiting fiber consumes almost no resources.

For the agent this means: a hundred clients simultaneously waiting for the model’s response don’t exhaust the worker pool — they wait “in parallel” within a single process.

Putting async to work

Async solves server throughput, but the user still needs to see that the system is working. So the agent combines three techniques:

1. Streaming the answer. The model generates text incrementally — delivered via Turbo Streams / ActionCable / AnyCable rather than only once the whole answer is ready. The user sees the answer “typing” within a fraction of a second:

   response = agent.ask(message) do |chunk|
     broadcast_chunk(chat_id, chunk) # each fragment immediately goes to the front
   end

2. Background processing via Sidekiq. The agent request itself moves into a background job; the HTTP controller responds instantly (head :accepted). This decouples the web request’s lifetime from the agent’s runtime — the web worker isn’t held busy for all 15 seconds.

3. Async tool concurrency. If the agent needs several tools in one step, they can run concurrently — especially apt for I/O-bound tools on Falcon’s fibers:

   # config
   config.tool_concurrency = :fibers

How it looks to the user

Put together:

1. the client sends a question;

2. the controller instantly acknowledges receipt and enqueues a background job;

3. the agent starts working, while the server stays free for other clients;

4. as soon as the model generates the first tokens, they’re streamed into the chat.

Subjective waiting time is a fraction of a second until the answer begins, even if full generation takes seconds. And the server holds hundreds of such sessions at once.

This is exactly why the agent stack includes Falcon and Sidekiq from the start: they solve not “Ruby’s slowness” (there is none here) but correct operation under a load of many long I/O waits.

Part 7. Order of implementation

Don’t build everything at once. Order by effort / impact:

1. Skeleton. One agent, one tool (get_policy), strict authentication, an audit log. Confirm that the agent loop and tool calling work.

2. Calculations. Wire up calculate_premium and work through the clarifying-question scenario for incomplete data.

3. Basic RAG. Structural chunking with tables in Markdown, vector search, a validity-date filter. Without this foundation the rest is pointless.

4. Search quality. Hybrid search with RRF, then contextual retrieval, then reranking — in that order.

5. Production wiring. Tracing, an eval set (reference inputs and expected outputs), moderation, and a human fallback.

Production checklist

• Concurrency: parallel tool calls via fibers — relevant when running on Falcon.

• Tracing: OpenTelemetry → Langfuse / Datadog. In insurance, auditing calls is mandatory.

• Tests: stub model responses without real API calls so tests are deterministic.

• Eval layer: a golden set of “question → expected chunk / answer”, recall@k and accuracy. Without it, “it got better” stays a subjective impression.

• Moderation: filtering of incoming messages.

• Prompt injection: uploaded documents are an untrusted source; instructions inside them must not influence the agent’s behavior.

• PII in logs: filtering of personal data.

• Human fallback: when the model is uncertain and on legally significant questions.

• Reindexing: on a new document revision, set valid_until on old chunks instead of deleting them — history is needed for the audit.

Part 8. Everything costs money

The agent’s economics consist of two line items: one-time indexing of the knowledge base, and the cost of each client request. They’re counted separately — the first is paid once when documents are loaded; the second scales with traffic.

The pricing model

Take the Anthropic API as an example. It is priced per token: input (prompt, system instruction, context) and output (the model’s answer) are billed separately, per million tokens (MTok).

For every model in the table below, output tokens cost five times as much as input tokens, which simplifies planning: take the input price and multiply by five.

Current prices (per 1M tokens, input / output):

The table covers the models used in this design. Anthropic also offers Sonnet 5 (introductory $2 / $10 per MTok through 31 August 2026, then $3 / $15) and Fable 5 ($10 / $50), either of which would change the figures below.

There are two ways to reduce the bill, and each belongs to a different half of the system. Prompt caching cuts the price of repeated input tokens by up to 90% and is the only lever that helps live chat. The Batch API halves both input and output, but it is asynchronous: you submit up to 100,000 requests at once and collect results on a best-effort schedule with a 24-hour SLA, so it can never serve a client waiting in chat. It applies exclusively to offline work — chunk enrichment during indexing, reindexing after a document revision, and eval runs.

The cost of a single client request

A token is the smallest unit of text the model operates on. Frequent short words are a single token; long or rare words are split. For English it’s roughly 1.3 tokens per word.

Tokens add up across all steps of the loop:

• Agent system prompt: ~500 tokens

• Descriptions of the three tools: ~400 tokens

• User question: ~50 tokens

• The 5 retrieved chunks placed into context: ~2500 tokens

• Internal reasoning steps and the tool call: ~600 tokens

• Final answer: ~400 tokens

Roughly — about 4000 input and 400 output tokens per request (for an agent loop with a single tool call, the input counts the context being re-sent at the phrasing step).

On Sonnet 4.6 without optimizations:

• Input: 4000 / 1M × $3.00 = $0.012

• Output: 400 / 1M × $15.00 = $0.006

• Total ≈ $0.018 per request

That’s on the order of 1.8 cents per reference question. A cost-calculation request (calculate_premium) is usually cheaper — there’s no heavy RAG context, just a compact JSON result from the function.

Note the asymmetry: output tokens are five times dearer, but the agent sends roughly ten times more input than it receives back. Input dominates the bill — which is why caching the static part of the prompt is the optimization that matters here, and why a chattier answer costs far less than a larger retrieval context.

Where to optimize

The system prompt and tool descriptions are identical in every request — a perfect candidate for prompt caching. By caching ~900 tokens of the static part, you save up to 90% on that share of the input. With noticeable traffic this cuts the bill appreciably.

Estimated monthly cost at different volumes (Sonnet 4.6, ~$0.018/request, no cache):

With prompt caching of the static part, the real bill is noticeably lower. The Batch API does not help here: every one of these requests is a client waiting for an answer in the chat.

Indexing cost (one-time)

Enriching chunks via Haiku at document-load time. Say the base is 5000 chunks, each ~600 input and ~50 output tokens:

• Input: 5000 × 600 / 1M × $1.00 = $3.00

• Output: 5000 × 50 / 1M × $5.00 = $1.25

• Total ≈ $4.25 for a full indexing of the base

Plus embeddings: Anthropic doesn’t provide an embedding API, so they come from a third-party provider (Voyage, OpenAI) and are billed separately — pennies at this volume. Reindexing happens only on a new document revision, not per request, so this item barely affects operating costs.

Prices verified against the official Anthropic page (July 2026) and may change — check current rates at claude.com/pricing. All figures above are ballpark estimates for planning, not an exact bill forecast.

Takeaways

Five ideas that determine successful adoption of AI:

1. Separate reference questions from calculations. RAG handles document text; tools handle personal data and calculations. Don’t mix them: the model must not compute amounts.

2. Grant autonomy in orchestration, not in critical decisions. The agent picks the route itself, but client identification and arithmetic stay in deterministic code.

3. RAG is 80% data preparation. Structural chunking, preserving tables, and document-version metadata give more than any model swap.

4. LLM latency is a network wait, not slow code. Cure it with an async model (Falcon), streaming, and background processing — not by changing languages. The user sees the answer within a fraction of a second, and the server holds hundreds of sessions.

5. Every call to the model has a price. Cost is a design decision, not an invoice you discover later. At roughly $0.018 per request the bill scales linearly with traffic; caching the static prompt and tool descriptions is the single biggest saving available — and it has to be built in from the start, not retrofitted.