How to Keep LLM API Keys Out of Application Code

Provider Keys in Application Code Create Sprawl and Slow Rotation

Each copy of a provider key needs storage, access control, and rotation. Copies live in Secrets, CI variables, developer laptops, agent config files, and vendor tools. Each store has an owner, and some stores have no clear owner.

Access grows with each copy. A provider key often opens the full provider account for the key. A developer who needs one model gets a key that can call every model on the account. A tool that needs read access to one project gets a key with account-wide spend power.

Rotation hurts. Your team must find every copy, update every store, and redeploy every app. Teams delay rotation because the work touches many owners. A delayed rotation leaves a leaked key valid for weeks.

The Tetrate docs cover the human side in how to onboard developers without API key sprawl. Agent workloads add the same problem at machine scale. Each agent deployment can carry a key, and agents run without a person watching each call.

The Gateway Keeps Provider Keys and Adds the Keys on the Way Out

Your platform team creates a provider record and stores the provider credential in Agent Router Enterprise. Your team grants the provider and models to a project. Your team issues a project-scoped API key for the app or agent. The docs walk through provisioning models via API and issuing a project-scoped API key.

The app changes the base URL and the key. The gateway receives the call, checks project policy, picks the route, and sends the call to the provider with the stored credential. The app never sees the provider key.

Two limits. The gateway key is still a secret. Store the gateway key in your secret manager and scope the key to one project. A gateway cannot protect a provider key that teams keep using outside the gateway. Remove direct provider keys from apps as you move each app to the gateway path.

A Kubernetes Config Example Moves the Secret to the Gateway

The example shows the app side after the move. The app gets the gateway URL and a gateway key from a Secret your platform team issues for one project. The example uses placeholder values.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: support-agent
spec:
  template:
    spec:
      containers:
        - name: support-agent
          env:
            - name: LLM_BASE_URL
              value: "https://gateway.example.com/v1"
            - name: LLM_API_KEY
              valueFrom:
                secretKeyRef:
                  name: support-agent-gateway-key
                  key: api-key

The provider credential lives in the gateway catalog. Your platform team sets the credential in the Admin Console or through the management API. The app deployment has no provider credential to rotate.

Keep one gateway key per agent or per app. A shared key across agents gives you shared attribution and a shared blast radius. The docs on containing a leaked key show how to bound damage with a rate limit, revoke the key, and pause the workload.

Rotation and Revocation Happen in One Place

Rotate a provider credential at the gateway. New calls use the new credential. Apps keep the gateway key and keep running. Your team can rotate on a schedule and during an incident with the same steps.

Revoke a gateway key when an app or agent misbehaves. The gateway rejects new calls with the revoked key. Other teams keep their keys and their access. Pair revocation with token limits and budgets, so a leaked key has a cap before a person reacts. AI ingress vs. AI egress covers the full traffic model, and MCP egress control covers tool credentials on the same outbound path.

Agent Router Enterprise also supports passthrough in set cases, where a caller brings a provider credential or a subscription credential. Passthrough fits personal subscription tools such as Claude Code plans. Production agents should use gateway-held credentials, so your team keeps custody and attribution. The docs on using your own provider credentials cover bring-your-own-credential setups for Agent Router Service.

The AI gateway glossary defines gateway key and provider credential terms. AI gateway vs. API gateway explains how AI key controls differ from web API key controls. The AI gateway performance benchmarks publish gateway overhead, best enterprise AI gateway compares current options, and who created Envoy AI Gateway covers the open source data plane project.

Availability and How to Get Started

Tetrate Agent Router Enterprise is available now with provider credentials stored at the gateway and project-scoped gateway keys for apps and agents. Start with the docs on provisioning models via API and issuing a project-scoped API key. Book a demo to move one app and count the provider key copies you remove.

Frequently Asked Questions

How Do I Keep LLM API Keys Out of Application Code?

Store provider credentials at the gateway. Give each app a project-scoped gateway key and point the app at the gateway URL. The gateway adds the provider credential on the outbound call.

What Key Does My App Send to the Gateway?

Your app sends a gateway API key issued for one project. The key names the project on each call and gives the gateway an identity for model access, limits, and cost records.

How Does Rotation Work With Keys at the Gateway?

Your platform team updates the provider credential at the gateway. New outbound calls use the new credential. Apps keep the gateway key and need no redeploy for a provider rotation.

What Happens When a Gateway Key Leaks?

Revoke the gateway key in the console or through the management API. The gateway rejects new calls with the key. Rate limits and token limits bound the damage before revocation.

Should Agents Use Passthrough Provider Credentials?

Passthrough fits personal subscription tools. Production agents should use gateway-held provider credentials, so your team keeps credential custody, model policy, and per-agent cost records.

Can I Keep Provider Keys in Kubernetes Secrets and Still Use a Gateway?

Yes for the gateway key. Store the gateway key in a Kubernetes Secret for the workload. Keep the provider credential at the gateway, so the cluster stores no provider key for the app.

Sources

• Provision models via API, Tetrate docs
• Issue a project-scoped API key, Tetrate docs
• Contain leaked keys, Tetrate docs

Related: How to onboard developers to AI models without API key sprawl, on identity-based access for people and teams.