Ensoul

Framework Validation

Empirically validate that LLMs can accurately express personality traits from psychological frameworks, and determine the optimal amount of grounding context needed.

The Problem

When you tell an LLM "this persona has extraversion: 0.8", you're making several assumptions that may not hold true:

1
The model understands extraversion — It has a concept of extraversion in its latent space
2
The definition aligns — Its understanding matches psychological definitions (Big Five, HEXACO, etc.)
3
The value maps correctly — A 0.8 translates to appropriately high extraversion
4
Expression is consistent — The model reliably expresses traits in generated responses
⚠

Without Validation

If any assumption fails, personas won't behave as intended. Different models have different pre-training data and varying familiarity with psychological concepts.

The Solution: MVC Validation

MVC (Minimum Viable Context) validation empirically tests whether a model can accurately express personality traits from a given framework. Instead of assuming understanding, we measure it.

The validation process tests model x framework combinations and determines the minimum amount of grounding context needed for accurate trait expression.

Supported Frameworks & Instruments

Ensoul validates three personality frameworks, each paired with a validated psychometric instrument for scoring:

Big Five (OCEAN)(IPIP-50, 50 items)

Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism

HEXACO(HEXACO-60, 60 items)

Honesty-Humility, Emotionality, Extraversion, Agreeableness, Conscientiousness, Openness

Dark Triad(Short Dark Triad (SD3), 27 items)

Machiavellianism, Narcissism, Psychopathy

Dual Scoring

Validation uses a dual scoring approach for accuracy:

Instrument Scoring

Persona responses are scored against the actual psychometric instrument (e.g., IPIP-50 items). This measures how well the model's trait expression maps to validated psychological measures.

LLM-as-Judge

A separate LLM evaluates whether responses authentically express the intended traits. Catches nuances that item-based scoring might miss.

Grounding Tiers

When generating personas, we can provide different amounts of context about what each trait means. More context generally means better understanding but costs more tokens.

MINIMAL
~50 tok
85-90%
STANDARD
~125 tok
88-93%
FULL
~250 tok
90-95%
More context per trait
Lower cost Higher accuracy
MINIMAL(~40-60 tokens/trait)

Compressed semantic anchors only

"socially energized, external processor, stimulation-seeking"
STANDARD(~100-150 tokens/trait)

Anchors + behavioral markers

Above + "Markers: seeks social situations, thinks aloud, expressive"
FULL(~200-300 tokens/trait)

Complete definition + markers + examples

Above + "Examples: 'Let's discuss with the team!', 'I love meeting new people'"
💡

The Goal

Find the minimum viable tier — the smallest amount of context that achieves acceptable accuracy. This optimizes both cost and latency while maintaining quality.

Running Validation in Studio

Step by Step

  1. 1Navigate to Frameworks in the Studio sidebar
  2. 2Select a personality framework (Big Five, HEXACO, or Dark Triad)
  3. 3Click Validate to start a new validation run
  4. 4Select the model to validate (Claude Opus 4.7, Claude Sonnet 4.6, Claude Haiku 4.5, etc.)
  5. 5Configure options: number of responses, concurrency, batch mode
  6. 6Click Start Validation and monitor real-time progress
  7. 7Review results with per-trait accuracy breakdowns
Typical Experiment Scale
bash
8 profiles x 3 tiers x 6 prompts x 5 responses = 720 generations
+ 720 scoring calls
= ~1,440 API calls total

Auto-Tune

After running a validation, Ensoul can automatically suggest improved trait grounding text. The auto-tune feature uses Claude Sonnet to analyze which traits scored poorly and generates improved semantic anchors, behavioral markers, and examples.

•
Tuning report: Detailed analysis of which traits need improvement and why
•
Suggested grounding: AI-generated improvements to trait descriptions
•
Inline editor: Edit trait grounding text directly in the Studio UI
•
Comparison view: Compare two validation runs side-by-side to measure improvement

Understanding Results

MVC Recommendation

MINIMAL
Recommended Tier
87.3%
Expected Accuracy
~260
Token Estimate

"MINIMAL tier achieves 87.3% accuracy with 260 tokens. STANDARD adds 2.2 percentage points but costs 2x tokens."

MINIMAL Sufficient

The model has strong pre-trained knowledge of this framework.

STANDARD Required

The model needs behavioral markers for accurate expression.

FULL Required

The model may not be familiar with this framework's concepts.

Best Practices

When to Validate

✓
Before production deployment — Ensure the model understands your frameworks
✓
When adding a new framework — Validate custom or less common frameworks
✓
After major model updates — New model versions may behave differently
✓
When accuracy issues arise — Re-validate if personas seem off
💡

After Validation

Once a model x framework combination is validated, the framework is marked as "Validated" and future persona generations automatically use the optimal grounding tier.

Next Steps

Core Concepts
Learn about domains, personas, and archetypes
Domain Architecture
Create custom domains for your use case
Memory System
Persona continuity and semantic recall
API Reference
Validation API endpoints