Saltar al contenido principal

AI Interactions & the Prompt Inspector

The AI Interactions page (under Monitoring in a domain) lists every turn the assistant handled and lets you open any one of them to see exactly what happened. When you open a turn, the detail is a single view: one header card leads with what was asked (the user query) and what the user saw (the final response), then shows the model input, the precise payload the model received (provider, model, endpoint, params). At the top of the card an outcome ribbon carries the turn verdict, an Export menu (Copy JSON / Copy cURL / Open in Playground), and the Turn details button. Below the card, System prompt, Messages, and Tools open in tabs.

Everything secondary is one click away, so the main view stays focused on what was asked and what the model read:

  • Turn details: a drawer opened from the Turn details button in the outcome ribbon. It holds the answer assembly, the runtime outcome, token usage and feedback.
  • Agent path: a collapsible section below the model input. It explains why the assistant moved through each iteration: which tool names were available, which tools it used, and the final answer at the end.
  • Model input: the spine of the view, including the Prompt Inspector, which shows the precise input the model received.

This page explains what belongs in each layer, how to read the system prompt and context messages in the Inspector, and how to use prompt versions to track quality over time.

What the system prompt is​

Every answer the assistant gives starts from a system prompt: the standing instructions that tell it who it is and how to behave. You don't write the whole thing. Appilot assembles it from two parts:

  • A template: the reusable skeleton of instructions, identical for every turn.
  • Variables (we call them literals): the handful of instruction values that change per turn: the page, the language, and the user's preferences.

The app-specific guidance your content team authored is not part of the system prompt. It is loaded from the Knowledge Base and shown in the Messages view as pre-loaded knowledge.

The Inspector shows both, plus the final result.

The three system-prompt views​

Open a turn, find the Prompt Inspector (the model-input panel that is the spine of the view), and expand System prompt. You'll see three tabs:

Composed​

The final system prompt, exactly as the model read it. This is the source of truth when you're checking the assistant's standing instructions. If a content article is missing, check the Messages view rather than this tab.

Literals​

The variables that filled the template on this turn, as a labelled list:

  • language: the response language.
  • appName / viewName / viewPath / url: where the user was.
  • userPreferences: anything the user set in My Account that shapes answers.

Each row has an ⓘ tooltip explaining what it is. This tab is the fastest way to confirm the assistant was grounded in the right page and language.

Template​

The reusable skeleton, with its slots highlighted. The Template tab is the same for every turn; it's here so you can see where your literals land in the larger set of instructions.

What the assistant read on the page (the Messages view)​

Below the system prompt, the Messages view shows the back-and-forth of the turn. The first message is the assistant's starting context for the turn, and it's broken into labelled, collapsible sections so you don't have to read one giant block:

  • User query: the user's actual question, verbatim.
  • Page context: where they were (URL, page title, current view).
  • Pre-loaded knowledge: the global and page-specific Knowledge entries that were automatically provided to the assistant. Global app guidance appears here, not inside the system prompt. This is often the biggest section; collapse it once you've checked it.
  • Available zones: the page regions your content team registered.
  • Live page snapshot: a lightweight outline of what was on screen.

Use a Sections / Raw toggle to switch between this structured view and the exact text. This is the fastest way to confirm the assistant was given the right knowledge for the page: if the answer was wrong and the Pre-loaded knowledge section is empty or off-topic, you've found a Knowledge gap to fill.

Tools, app tools, and injected context​

The inspector uses three different labels because they answer different questions:

  • Model-chosen tool: the assistant decided to call a real Appilot tool during the turn, such as Search knowledge base or List controls. These are counted in Agent path.
  • App tool: a capability your team registered for the host application, such as a file search or business lookup. The model can call it when it is declared for the turn.
  • Injected context: Appilot-generated context packets that help a specific model family read the same page and knowledge. These are not app tools, and the model did not choose to call them.

For example, Gemma-family models receive page context through synthetic messages named Injected app context (get_general_kb) and Injected page context (get_context). They can appear in the Model input wire payload because that is what the model received, but they are not listed as tools used in Agent path. They are generated by Appilot's provider-specific message assembly and cannot be clicked or configured as customer tools.

The Model input view shows the provider wire payload. The Messages panel groups the current turn by iteration so you can read each assistant tool call together with its matching tool result:

  • Model requested on an iteration group means the LLM returned a tool call in that iteration. For example, List controls is an Appilot registry tool declared to the model; if it appears inside a model-requested group, the model asked Appilot to run it.
  • Appilot injected on an iteration group means Appilot inserted context into the provider payload. The model did not choose this as an action.
  • Tool result is the response Appilot appended after running or injecting the corresponding entry.
  • call id is the provider's tool_call_id. It pairs a tool call row with its result row. It is not a control id, page id, or user-facing identifier.

Some internal tools are not customer app tools even though they are declared as tools. Emit plan (emit_plan) is Appilot's internal plan channel: the model can call it to propose guided steps, Appilot validates and assembles the plan, and the model still writes the final text response afterward.

Different model families receive context differently:

  • Gemma on-prem uses injected-context messages so knowledge and page state arrive as tool-result style context.
  • OpenAI-compatible models receive context as user-message items.
  • Claude models receive context through Anthropic's message shape. Only the supported Opus 4.8 family can use a mid-conversation system message, and Appilot uses that only for small trusted page metadata, not Knowledge Base or DOM content.

This matters when you compare turns across models: an injected context entry tells you how Appilot packaged the prompt, not that the assistant took an extra action.

Tool access​

The Tool access map separates several things that can otherwise look like one list:

  • Used this turn: the tools Appilot actually executed while producing this answer. This is the first list to check when the user asks why the assistant acted a certain way.
  • Core catalog tools: the tools Appilot configured as always visible in compact catalog mode. Internal tools such as Emit plan can appear here even when the captured tools[] array you are looking at is from a later call.
  • Discovered by Find tools: tools made visible after the model called the discovery tool.
  • Captured call tools[]: the exact tool schemas present on the provider call captured in this snapshot.
  • Provider-declared across iterations: the union of tool names Appilot actually declared in provider tools[] across the turn's internal model calls.

Use Agent path as the source of truth for tools actually executed during the turn. Use Tool access map to understand where access came from and why a tool such as Emit plan may be available even if it is not listed in the captured call's tools[] schemas.

Because compact catalog mode can change the tool list between internal model calls, Agent path shows Tool access by iteration before the executed tool calls. Each iteration card keeps the main path to names and counts. Open the card to see:

  • Provider-declared tools[]: the exact native schemas sent to the model provider on that internal call.
  • Requested by model: the tool calls the model actually returned.
  • Status: whether Appilot executed the tool, captured it as a plan/update channel, ran discovery, or rejected it.
  • Sources: whether the tool came from provider tools[], prompt guidance, compact core configuration, discovery, or Appilot's registry.

If a requested tool shows not in tools[] but registry fallback, the model asked for a registered Appilot tool that was not declared as a native provider schema on that specific iteration. Appilot could still run it because the server-side registry knew how to dispatch it. This is useful diagnostic evidence, especially for compact catalog mode and smaller models, but it is different from native provider availability.

Raw tool arguments, tool results and provider schemas are intentionally hidden from the main path. Use the Details button on a tool call, or Inspect on a schema row, to open a drawer with the raw payload when you need it.

Find tools and changing tool lists​

Some domains run in compact catalog mode. In that mode the model does not see every Appilot tool at the start of the turn. It first sees a small discovery tool called Find tools plus a few core tools.

The intended flow is:

  1. The model asks Find tools for a capability, for example “inspect controls on this page”.
  2. Appilot searches its trusted tool registry and records a Discovery decision.
  3. The next internal model call declares the selected tools in the provider's native tools[] list.
  4. The model can then call one of those selected tools.

When this works, AI Interactions shows both sides of the handoff:

  • The iteration where Find tools was requested.
  • The Discovery decision, including what Appilot considered and selected.
  • The next iteration where the selected tools were actually declared to the provider.

If the model calls a name that was never declared and Appilot does not recognize it, the ledger shows unknown tool. That means the model invented or misspelled a function name. For example, Find controls is not the same as List controls. The human-friendly label may look similar, so expand the row to see the raw function name.

If a selected tool does not appear in the next iteration's provider tools[], treat that as a runtime issue. The discovery step found something useful, but the provider call did not receive the schema needed for the model to call it reliably.

The separate KB chunks injected and Controls considered panels appear only when the legacy prompt snapshot stored non-empty summary data. On current provider-specific strategies, the same evidence is usually already visible in Messages, Tool results, or System prompt details, so empty summary panels are hidden.

Prompt versions​

The template evolves over time as Appilot improves the assistant. Each turn records the template version it ran on, shown as a Template v1 badge next to the system prompt.

Why it matters: when you notice a change in answer quality, the version tells you whether a prompt update lines up with it. The badge has a companion status chip:

  • Recomposed ✓: the prompt was reconstructed faithfully from its template and variables. Normal and healthy.
  • Stored in full: this turn was kept verbatim (errors and a small sample of turns are always kept in full).
  • Version drift: this turn ran on an older template than the one currently deployed. The Composed tab still shows the real text; the Template tab shows today's skeleton, which may differ.

Correlating reports with a prompt version​

On the AI Reports page (the turns users flagged with 👎), each row shows the Template version the flagged answer ran on, and there's a prompt version filter at the top.

Use it to answer questions like "did the spike in bad answers start when the prompt changed?". Filter the reports to a single version and compare the volume and the kinds of complaints across versions. If a cluster of poor answers is concentrated on one version, you've localised the regression to a prompt change rather than a content or configuration issue.

Replaying a turn in the Playground​

The Playground is a raw replay tool. Load a turn into it with Export → Open in Playground in the turn's outcome ribbon. It sends the messages and provider tools[] declarations you see, then streams the selected model's raw response. It does not rewrite the prompt for another model family and it does not run Appilot's retrieval, discovery, or tool dispatcher.

That means Playground output can show a tool call such as List controls, but the Playground will not execute it. To understand whether Appilot executed that call in the original turn, go back to AI Interactions and read Agent path plus the iteration's tool access drawer.

For compact catalog turns, replay the exact iteration you want to inspect. An early iteration may declare only Find tools. A later iteration may declare tools that were activated by a previous discovery decision. If you replay only the later provider call, the Playground does not rerun the earlier discovery step. If you replay only the earlier call, the Playground does not automatically synthesize the later tool list.

Before running, the Playground checks the payload against the selected provider family and shows Payload checks:

  • Errors stop the run before model tokens are spent. Examples: a tool result with no matching assistant tool call, a duplicate tool name, or declaring get_context as a real callable tool.
  • Warnings mean the run can continue, but you may be testing a provider-specific shape. For example, a Gemma injected-context snapshot replayed against OpenAI or Claude may not mean the same thing to that model.
  • Notes explain valid special cases, such as Gemma's Appilot injected-context messages.

Use these checks to decide whether you are doing a faithful replay of the captured model payload or an intentional compatibility experiment.