Stop Asking Developers to Import Governance Libraries

The Import-and-Hope Pattern

The typical enterprise approach to AI governance goes something like this:

1. Create a central governance team
2. They write a Python library with all the compliance logic
3. Publish it to the internal package repository
4. Tell developers to import it and use it
5. Hope

Step 5 is doing a lot of work here.

Some teams will import the library and use it correctly. Some will import it and use it incorrectly (calling the wrong functions, passing the wrong parameters, not handling errors). Some will forget to import it. Some will never hear about it because they’re contractors or joined after the all-hands announcement.

Now your compliance posture depends on dozens of developers across multiple teams all correctly implementing the same logic in their services. If even one team doesn’t do it right, you have a gap.

Good luck auditing that.

Why Developers Don’t Follow Governance Guidelines

It’s not (usually) malice. It’s competing priorities.

Developers are being asked to:

• Ship new features faster
• Fix bugs in existing features
• Reduce latency
• Cut costs
• Improve observability
• Handle on-call rotations
• Attend meetings about quarterly planning
• Oh and also please import this governance library and make sure you call it correctly

Governance doesn’t make the sprint review demo more impressive. It doesn’t close customer tickets. It’s the thing you know you should do but can probably defer until next sprint.

And if the governance library adds latency or complexity to the code, there’s pressure to skip it or find workarounds.

The Coupling Problem

When governance logic lives in application code, it’s coupled to application deployments.

Want to update your PII filtering rules? You need to update the library, release a new version, and wait for all 12 services to upgrade. That could take weeks or months depending on each team’s release cadence.

Need to add a new compliance check for a regulatory change? Same problem. You’re coordinating updates across multiple teams with different priorities and schedules.

Meanwhile, your auditor is asking why the new rule isn’t being enforced yet, and your answer is “we’re waiting for the payments team to finish their sprint so they can upgrade the library.”

The Decoupling Alternative

What if compliance checks didn’t live in application code at all?

What if they lived in the infrastructure that all AI requests flow through?

Every AI system in your organization makes network calls: prompts go out to LLM APIs, responses come back. That traffic flows through infrastructure—load balancers, API gateways, service mesh, whatever your architecture uses.

That infrastructure is your enforcement point.

PII filtering? Happens at the gateway for every request, regardless of which service sent it. Topic blocking? Enforced at the gateway, centrally configured. Rate limiting? Gateway handles it. Audit logging? Gateway sees everything.

Developers don’t import a library. Developers don’t call functions. Developers don’t have to remember anything. The governance happens automatically because the infrastructure enforces it.

The Developer Experience Improvement

From a developer’s perspective, this is dramatically simpler.

With governance libraries:

from enterprise_ai_governance import PIIFilter, TopicChecker, AuditLogger

def call_llm(prompt, user_context):
    # Did I import the right modules?
    # Am I calling them in the right order?
    # Am I handling errors correctly?
    # Is this the latest version of the library?

    filtered_prompt = PIIFilter.strip_pii(prompt)
    if not TopicChecker.is_allowed(filtered_prompt):
        raise ForbiddenTopicError()

    response = llm_api.call(filtered_prompt)

    AuditLogger.log(user_context, prompt, filtered_prompt, response)
    return response

With infrastructure-layer governance:

def call_llm(prompt, user_context):
    # Just call the API
    # Governance happens automatically at the gateway
    response = llm_api.call(prompt)
    return response

Which one is more likely to be implemented correctly across 15 teams?

The Policy Update Advantage

When governance is centralized at the infrastructure layer, policy updates are instant and universal.

New regulation requires blocking additional topics? Update the gateway policy. It applies to all services immediately.

Need to change PII filtering rules? One config change, affects every AI request in the organization.

Security incident requires temporary rate limiting on certain model endpoints? Update the gateway, no application deployments needed.

The difference is stark. Rolling out a governance library update across a microservices portfolio can take months, coordinating releases across teams. The same change as a gateway policy update: minutes.

What About Custom Logic?

The obvious objection: “But our services have different governance needs! We can’t use the same policies for everything!”

Fair point. Your customer-facing chatbot probably needs stricter content filtering than your internal code documentation tool.

This is where policy configuration comes in. The governance logic lives at the gateway, but it’s configurable per service, per environment, or per request context.

You’re not implementing the logic 12 different times in 12 different services. You’re configuring centrally-managed policies to apply differently based on context.

Same enforcement mechanism, different parameters.

The Trust Boundary

There’s a deeper architectural principle here: governance should happen at trust boundaries, not within trusted components.

Once a request is inside your application code, you’re in trusted territory. You’re hoping developers wrote secure code, hoping they called the right functions, hoping nothing went wrong.

The infrastructure layer is the trust boundary. It’s where requests from outside meet your internal services. It’s where you should enforce rules, because that’s the choke point where you can see and control everything.

This is how network security works (firewalls at the perimeter, not “please configure iptables correctly on every server”). It’s how authentication works (gateway handles it, applications trust the authenticated context). It should be how AI governance works.

The Enforcement Guarantee

The difference between asking developers to do something and making it automatic:

Governance library approach: “87% of our services have imported the latest governance library according to our package scan.”

Infrastructure approach: “100% of AI requests are evaluated against current governance policies because they all flow through the gateway.”

Which statement would you rather make to your auditor?

The Migration Path

You don’t have to rip out existing governance libraries overnight.

Start by implementing infrastructure-layer policies for new requirements. Leave existing application-level controls in place as defense in depth.

Over time, as you gain confidence in the infrastructure approach, you can deprecate the application-level libraries. Fewer dependencies for developers, less code to maintain, simpler architecture.

Eventually you get to a steady state where governance is just infrastructure, and developers focus on building features instead of importing compliance libraries.

When Application-Layer Governance Makes Sense

Infrastructure-layer governance is great for policies that apply broadly (PII filtering, topic blocking, rate limiting, audit logging).

There are still cases where application-level logic makes sense:

• Business-specific rules that vary significantly by service
• Complex decision logic that needs deep application context
• Optimizations that benefit from application-specific caching

The key is using application-level governance for truly application-specific needs, not for organization-wide policies that should be consistent everywhere.

The Real Goal

Governance shouldn’t be a developer task. It should be an infrastructure property.

When you stop asking developers to remember to import libraries and call functions, you get:

• Consistent enforcement across all services
• Instant policy updates without code deployments
• Simpler application code
• Happier developers
• Better compliance

And you can finally delete that 47-page PDF that nobody reads.

Tetrate believes AI governance should be infrastructure, not application code. Our Agent Router Service enforces governance policies at the gateway layer automatically, removing the burden from development teams while ensuring consistent compliance across all AI services. Learn more here ›