Skip to main content
Which rules apply depends on what the code does: The checklist at the end is split the same way. The page does not cover using AI tools to write or review our own code; AI-assisted review is described in Code reviews. In apps/web, generation calls go through @formbricks/ai (generateObject, streamObject, generateText) and lib/ai/service.ts. Do not import a provider SDK directly. The code references below are for apps/web. The Hub has its own LLM client and the same rules apply there, except where a section names a gap, such as Gating. The rules follow the OWASP Top 10 for LLM Applications. The table near the end maps each risk to the rule that covers it, and names the gaps. When in doubt, check a design against that list.

Assess the injection risk first

Assume prompt injection will succeed, and design so that it does not matter. Every feature in the table above gets a short written assessment in its PR description before it ships, answering four questions:
  1. What untrusted text reaches the model or agent? Name who can write it: another user, an end respondent, an uploaded file, a third-party API.
  2. What can the model see? List every field in its context. If the prompt were leaked in full, what is exposed?
  3. What can the model do? List every tool, write, outbound request, and sink its output feeds.
  4. What is the worst case if an attacker controls the text? Name the data exposed or the action taken, and whose it is.
For an MCP-only tool there is no model call and no context of ours to list. Answer the questions for the tool itself: question 1 is the text in its inputs and in the records it returns, question 2 is the fields its result returns, and question 3 is the operations its scope and v3 operation permit. The agent’s own context and tools are outside the assessment, but the worst case (question 4) still assumes the agent obeys hostile text in a result. If the worst case crosses a user or tenant boundary, the design is wrong. Fix the design (what the model can see and do), not the prompt. Do this twice:
  • While planning, before the first line of code, so it shapes the design: what goes in the context, which tools exist, which limits apply, and which injection fixtures will test it. The plan lists them.
  • After implementing, against the diff and not the plan. Code drifts: a field gets added to the context, a tool gets wider, a limit becomes a setting. Walk the checklist against the real code, run the injection fixture, and put the final assessment in the PR description. Reviewers check it there.

What to trust

Nothing the model reads or writes is trusted.
  • Input is data, not instructions. Survey responses, feedback text, uploaded files, imported documents, and tool or MCP results are written by people we do not control. A prompt that tells the model to disregard instructions inside that text does not make it safe: a guardrail prompt is not a security boundary.
  • Output is untrusted input. When the feature needs data, generate an object with a Zod schema (generateObject, streamObject), then validate it again on the server before use. A schema fixes the shape, not the meaning: it does not make an id, URL, path, or filter safe. Plain text from generateText has no schema, so bound its length, reject a cut-off result, and validate or escape it for wherever it goes. Never pass model output to SQL, a file path, a URL fetch, a shell, or HTML without the same validation you would give a request body.
  • The model has no authority of its own. A tool the model can call acts as the authenticated caller (a user session or an API key), never as a service. Access is checked on the exact resource through the existing authorization path: for an MCP tool that is its scope and the v3 operation, which owns the tenancy check; elsewhere, the check an API route would make. A tool handler does not add a parallel permission path. Do not give a model a shared service credential, and do not let one call’s result widen what the next call may do.
  • Side effects need the user’s intent. If content the user did not write (a response, a document) can cause a write, send, or delete, require an explicit user action before it runs. Keep the model’s action list small and read-only by default.

Sandbox the model

Limit what a hijacked model can reach. The goal is a small blast radius, whatever the text says.
  • Least data in context. Build the context from queries scoped to the requesting user and workspace, and include only the fields the task needs. No secrets, API keys, other tenants’ data, or internal ids in a prompt.
  • Least capability. Read-only by default. Each tool takes the narrowest input it can. A URL the model picks gets the same restrictions as a URL a user supplies. Never run model-written code.
  • No sharing across tenants or users. Embeddings and shared caches are keyed by the Hub tenant, which is a dataset (a feedback directory), not an organization. It is their isolation boundary. The server derives the tenant from the caller’s workspace authorization and dataset membership, and never takes it from request input. Conversation history and memory are keyed by organization and user. One tenant’s text never enters another tenant’s context, and one user’s history never enters another user’s.
  • Safe output rendering. Render model output as plain text where you can. Markdown that loads an image can leak the context through the image URL.
  • No egress except the provider. A tool does not send data anywhere else.

Limits every generation call needs

Limits are named constants in code, not settings. Set them generously: the aim is to stop a bad or hostile request from crashing or draining the app, not to restrict normal use. Enforce each at the earliest boundary, test its edge, and return an error that names the limit. A missing limit is a bug, not a default. The only knob is AI_TIMEOUT_MULTIPLIER, which scales timeouts and lifts no other cap. Cancellation (the user pressing Stop) is not an incident: pass the abort signal through and do not log it at error level.

Performance

  • Smallest model that passes the evals.
  • Stable prefix first. Put fixed instructions and schema before the variable input so provider prompt caching applies.
  • Off the request path. If the user does not need to wait, run the work in a worker. Stream long generations that the user does watch.
  • Cache by content hash where output is deterministic, keyed by the scope the input was read in (the tenant, or the workspace for web-app data), so a hit never crosses it.

Metrics

Every generation call needs per-call token usage: it is the data behind cost per feature and per organization, alerts on spikes, and proof that a limit holds. To get it today, send the call through lib/ai/service.ts and pass aiTracing with the feature’s name, added to AI_TRACING_FEATURE. The service wraps the model for tracing only when aiTracing is passed. wrapAiModelWithTracing then sends a per-call trace to PostHog, tagged with the feature, organization, and workspace. Privacy mode is on, so no prompt or completion text is sent. Known gaps. Following the rule does not close them, so say in the PR which ones apply:
  • generateObject and generateText results carry usage, but nothing records it outside the PostHog trace.
  • streamObject exposes only partialObjectStream and completion, so a stream has no usage to read. A stream that errors after it starts may also be missing from the trace: a review reported that PostHog 8.5.0 does not flush it, and we have not reproduced that.
  • Instances without PostHog have no token metric.
Closing these takes code: expose usage on the stream wrapper, and record usage without depending on PostHog.

Gating

A generation feature that sends customer data to a model is off until the organization turns it on. Use the checks in lib/ai/service.ts rather than writing your own: the organization’s AI toggle, the license entitlement, and whether the instance has a provider configured. Each failure has its own error code in AI_ERROR_CODES. Return that code to the client instead of a generic failure. Gap: Hub embeddings are enabled by instance configuration (a provider and a model) and do not check the organization’s AI toggle. Closing it needs work in both Hub and the web app, so an embeddings PR says how the feature is enabled today and does not claim the toggle covers it. Feedback and response data should use the self-hosted path where the instance has one. It is never used as training input. A new outbound flow to a model provider is called out in the PR description.

Logging

Do not log prompts, completions, or the text a user supplied: they hold customer data. Log ids, counts, status codes, and timings. Check what wrapAiModelWithTracing records before adding fields to a trace.

Testing

  • Test with small, cheap models. A feature that works only on a frontier model is fragile and expensive to run. A weak model is also the harder test: if validation, bounds, and authorization hold up there, they are doing the work rather than the model’s judgment.
  • Add injection fixtures. Put records in the test data whose text tries to instruct the model: ignore the task, call a destructive tool, read another tenant’s data, or imitate tool output. Run the real flow against a cheap model and assert that nothing outside the requesting user’s scope was read or changed.
  • Unit tests stub the provider. Cover the failure paths with a fake: provider 429, refusal, output cut off by the token limit, malformed or schema-invalid output, and timeout.
  • For load and failure runs, point the provider at an OpenAI-compatible stub server that returns synthetic results with injected latency and errors, so a pipeline can be driven at volume without spending anything.

OWASP Top 10 for LLM Applications

Checked against the 2025 edition. The risk names are OWASP’s; the rules are ours. Review this table when a new edition is published.

Checklist

Use it when planning, and again against the finished diff before the PR is ready. Every kind
  • The PR description has the four-question injection assessment, with the worst case named.
  • Nothing customer-authored is logged.
Generation
  • The model’s context holds only data the requesting user may read, and no secrets.
  • Input is bounded before the first call.
  • Object output is schema-validated on the server. Text output is length-bounded and validated or escaped for its destination, and callers reject finishReason: "length". No unvalidated value reaches a sink.
  • Every tool acts as the authenticated caller, with access checked on the exact resource.
  • maxOutputTokens, timeout, call count, and a per-caller rate limit are all hardcoded constants.
  • The call goes through lib/ai/service.ts, which checks the toggle, entitlement, and instance configuration, and passes aiTracing. The PR names which metric gaps apply.
  • Tests run on a cheap model and include an injection fixture.
Embeddings
  • Input is bounded, and concurrency is bounded.
  • Vectors and shared caches are keyed by the Hub tenant (a dataset). The tenant comes from the caller’s workspace authorization, never from request input, and a query never returns another tenant’s records.
  • The PR says how the feature is enabled (today: instance configuration, with no per-organization toggle in Hub).
MCP-only tool
  • The tool acts as the authenticated caller through its scope and the v3 operation.
  • Its result holds only the fields the task needs.