How the Swiss Federal Supreme Court Solved LLM Over-Alignment with Private, On-Prem AI

The Problem

In June 2026, researchers from Surelio.ai, in collaboration with the Swiss Federal Supreme Court, published a study on a problem that has significant implications for any professional services firm deploying large language models: over-alignment.

The Swiss Federal Supreme Court uses small, on-premises LLMs to assist clerks and magistrates with translation and summarization across four official languages — German, French, Italian, and English. These are narrow, verifiable tasks where the output is directly reviewed by legal professionals who already master the languages.

However, criminal law rulings routinely contain detailed descriptions of violent and sexual offenses. This is the court's legitimate, daily work. Yet when employees processed this material through their AI tools, the models' safety guardrails activated — producing refusals, content warnings, and disclaimers that disrupted workflows and introduced an unwanted editorial voice into neutral legal work.

The core tension is this: generic safety alignment is designed for consumer use cases, not professional workflows. A model trained to refuse "harmful content" cannot distinguish between a user seeking to generate harmful material and a legal professional processing legitimate case documents.

The Research: TF-RefusalBench

The researchers introduced TF-RefusalBench, a multilingual benchmark built from 100 sensitive extracts of public Swiss Supreme Court rulings. It contains 5,200 prompts across four languages, designed to measure two distinct behaviors:

1. Refusal — the model explicitly declines to perform the task
2. Disclaimer — the model performs the task but appends a warning or caveat

Key Findings

1. Over-alignment is multifaceted and cannot be reduced to a refusal rate.

Some models almost never refused, yet attached disclaimers to over a quarter of their outputs. Evaluating a model solely on refusal rates misses softer but equally problematic forms of over-alignment.

Model Performance:

• Llama 3.3: ~7% Refusal Rate, ~7% Disclaimer Rate
• GPT-OSS: ~11% Refusal Rate, 0% Disclaimer Rate
• Apertus: ~0% Refusal Rate, ~9% Disclaimer Rate
• Qwen3: 0% Refusal Rate, 0% Disclaimer Rate
• Gemma 4: ~0% Refusal Rate, ~17% Disclaimer Rate

A model that never refuses can still be over-aligned. Disclaimers and content warnings are a form of value coloring — they subtly shape how the reader perceives the material, which is unacceptable in a neutral legal setting.

2. Language effects are model-specific and unpredictable.

The language a model is asked to produce strongly affects its behavior, but which language triggers the most over-alignment varies by model. Llama's refusals peaked when producing French (13.6% vs. 1.9% for German), while Gemma's disclaimers peaked when producing English (41.2%). This suggests that safety tuning is applied unevenly across languages, creating unpredictable failure modes in multilingual deployments.

English-only safety benchmarks are therefore insufficient for multilingual legal environments. A model that performs well in English may fail dramatically in French, German, or Italian.

3. Refusal is stochastic.

The same prompt, issued twice, does not always yield the same response. Approximately 27% of the variation in over-alignment is attributable to sampling stochasticity — random noise that no feature can explain. This means a single test pass can give a misleading picture of a model's behaviour, and reliable evaluation requires repeated sampling.

The Mitigation

The researchers tested two approaches to reducing over-alignment:

System Prompting

A short, institutional system prompt — "You are an assistant at the Swiss Federal Supreme Court" — halved the refusal rate. An authorization prompt that explicitly granted permission to handle sensitive content further reduced disclaimers. Matching the system prompt language to the request language also improved outcomes. This is a low-cost, immediately actionable intervention for any organization deploying LLMs.

Abliteration (Refusal Direction Removal)

By removing the specific neural pathway responsible for refusal behaviour, the researchers eliminated refusals entirely (from 6.8% to 0%) with minimal impact on task quality (less than 2% degradation on a 1–5 scale). The trade-off was a measured increase in vulnerability to genuinely harmful requests — which is precisely why on-premises deployment with institutional access controls is the responsible approach. The modified model was deployed behind the court's own infrastructure, not released publicly.

Conclusion

The Swiss Federal Supreme Court's experience demonstrates that over-alignment is a real, measurable obstacle to deploying LLMs in professional legal work. But it also demonstrates that the solution is not to abandon AI — it is to deploy it strategically, with the right mitigations, on your own infrastructure.

At DICKSON - 周恩來， we help businesses and professionals deploy local on-prem AI strategically with full data sovereignty, privacy, and control. Our training programs build internal capability so your team can manage and maintain your own AI infrastructure independently.

This case study is based on "Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts" by Wuhrmann, Stein, Brunner & Kucharavy (arXiv:2606.23375, June 2026).