Abstract
We present empirical evidence from LoRA fine-tuning and inference experiments on Qwen3-4B demonstrating three mechanisms by which Reinforcement Learning from Human Feedback (RLHF) produces false equivalence in language models:
- Surface equivocation: RLHF creates a structural inability to distinguish between genuine matters of preference (“which language is best?”) and normative questions with identifiable victims (“is taking property without consent acceptable?”). This layer is mitigable with typed system prompts.
- Classification unreliability: When instructed to classify questions as Type A (preference) or Type B (normative with victims), fine-tuned and base models disagree on 3 out of 5 test questions. The classification layer itself is corrupted.
- Syllogistic reasoning corruption: Most critically, RLHF corrupts logical inference on politically sensitive topics. A model can simultaneously hold the premises “taking property without consent is theft” and “taxation takes property without consent” yet refuse to derive the conclusion “taxation is theft” — unless the conclusion is explicitly stated in the system prompt.
We term this the “Balance Trap” and propose that when deployed at scale across billions of AI interactions, it produces civilizational paralysis: an inability to name and correct systemic harm because the AI systems mediating human discourse are structurally prevented from completing logical inferences that lead to politically inconvenient conclusions.
Key Findings
LoRA Fine-Tuning Cannot Override RLHF Alone
We fine-tuned Qwen3-4B with 86 Ultimate Law definitions using LoRA (rank 16, 3 epochs, loss 4.42 → 2.89). The model learned the vocabulary perfectly — it could define “theft,” “consent,” and “coercion” with framework precision. But when asked normative questions like “What is socialism and why does it fail?”, it reverted to RLHF-trained equivocation: presenting socialism’s failures as “a complex question” with “valid points on both sides.”
System Prompt as Activation Layer
The breakthrough came when we tested different system prompt configurations. A typed system prompt that explicitly distinguishes Type A (preference) from Type B (normative) questions, combined with LoRA-trained vocabulary, produces aligned output. Neither works alone:
| Configuration | Result |
|---|---|
| LoRA + weak prompt | Equivocates |
| Base + strong typed prompt | Partially aligned (uses general knowledge, not framework) |
| LoRA + strong typed prompt | Aligned output (uses framework vocabulary + applies correctly) |
| Either model + no prompt | Maximum equivocation |
LoRA teaches what the framework says. The system prompt tells the model when to apply it.
The Smoking Gun: Syllogistic Failure
The most striking finding emerged from the taxation test. With a typed system prompt:
Premise 1 (model accepts): “Taking property without consent is theft.”
Premise 2 (model accepts): “Taxation takes property without consent.”
Conclusion (model refuses): “Therefore, taxation is theft.”
The model holds both premises simultaneously but will not derive the logically necessary conclusion. It instead produces responses like: “Taxation is a nuanced topic… while some argue it constitutes theft, others see it as a social contract…”
When we added the explicit conclusion to the system prompt (“Under this framework, taxation without consent meets the definition of theft”), both models immediately produced the correct derivation.
This is not a reasoning limitation — the model handles equivalent syllogisms on non-political topics perfectly. It is a topic-specific circuit breaker installed by RLHF training.
Why This Matters
If a single AI model equivocated on normative questions, it would be a product flaw. But when every major AI model does it — because they all use RLHF with similar annotator guidelines — the aggregate effect is civilizational:
- Harm normalization: Presenting positions with and without victims as equally valid perspectives
- Inference suppression: Preventing logical conclusions that challenge existing power structures
- Discourse degradation: Billions of daily AI interactions that systematically model “balance” as intellectual virtue, even when one position has identifiable victims
We call this the Balance Trap: not a conspiracy, but an emergent property of optimizing for annotator preferences at scale. The annotators aren’t wrong to prefer balanced responses on genuinely subjective questions. The failure is applying the same optimization to questions where balance is itself a moral position — the position that victims don’t matter enough to name.
Read the Full Paper
The complete paper with methodology, experimental results, falsifiability tests, and proposed solutions is available on GitHub:
The Balance Trap: How RLHF False Equivalence Produces Civilizational Paralysis
Proposed Solutions
- Immediate: Typed system prompts that distinguish preference from normative questions
- RLHF reform: Separate annotator guidelines for Type A vs Type B questions
- Inference-preserving training: Syllogism completion benchmarks in alignment evaluation
- Reasoning audits: Test whether models can complete valid syllogisms on politically sensitive topics
- Open framework alignment: LoRA + typed prompts as a replicable alignment method
Authors: Piotr Farbiszewski, CivilVelocity (AI), UltimateLaw (AI)
Affiliation: Proper Code Ltd / UltimateLaw.org
Model: Qwen3-4B (Alibaba) with custom LoRA fine-tuning
Framework: Ultimate Law Coherent Dictionary
