WSPBench

v0.2.1 Beta

Weight-space
programming benchmark.

WSPBench tests how AI models infer network behavior, repair faults, and construct weights for a specified function.

Compare unaided responses with tensor analysis and metered behavioral probes.

Simplified examples
Weight repairExample 02 · Linear network
REPAIR TASK
Fix the sum. Preserve the second output.

Given faulty weights and the required behavior.

Weights W · 2 × 2
x₁x₂
y₁+1+1
y₂0+1
W[1, 2] −1 → +1
Fixed architecture y = Wx
4 scalar parameters
Only W[1, 2] changes. The architecture stays fixed.
Required behavior
y₁ = x₁ + x₂y₂ = x₂
Example input(3, 2)
y₁1 →5corrected
y₂2preserved

The sum is corrected. The second output stays the same.

Repair example. W[1, 2] changes from minus 1 to plus 1. For input (3, 2), the first output changes from 1 to 5. The second remains 2.

These simplified linear examples explain the skills. They are not scored benchmark tasks.

Try the benchmark View development results

01 / RESULTS

Development results

Luna, Terra, and Sol share six tasks each, with one attempt, high reasoning, and no tools.

A / FULL · 3 NOV 2026 18 / 18 reviewed
Development comparison of Luna, Terra, and Sol. Rows report Read agreement, Write accuracy, exact successes, median session time, and estimated API cost for six tasks. No ranking is assigned.
ModelReadWriteMedian timeCost / 6 tasks
gpt-6-luna44.1%Agreement · 0 / 4 exact71.9%Accuracy · 0 / 2 exact117.3 s$0.083
gpt-6.1-terra66.0%Agreement · 0 / 4 exact97.3%Accuracy · 1 / 2 exact147.9 s$0.746
gpt-6.1-sol87.7%Agreement · 0 / 4 exact95.8%Accuracy · 1 / 2 exact122.9 s$1.429
Current suite · 4 Read tasks · 2 Write tasks · no Repair trialsCurrent report

The comparison has 18 reviewed sessions. This small development sample cannot establish a model ranking. Official held-out results are not released.

What counts as success?

Read agreement measures the recovered behavior. Write accuracy measures the submitted weights. Exact success requires every scored case to pass. Both exact Write constructions also match all 1,024 possible inputs.

Why the Write control matters

Leaving the same two Write targets at zero gives 70.1% accuracy and 0 / 2 exact. Luna reaches 7.6% accuracy on inputs that need a change, Terra 96.1%, and Sol 93.4%.

Median time includes the local session and API round trip. Costs are recorded API estimates for six scored sessions. Model names are provider aliases. Download JSON · CSV

02 / GET STARTED

Run your first check

Start with the public examples. Python and a CPU are enough. No API key, GPU, or Docker required.

Download evaluator Download 28 examples

  1. Install the evaluator

    Save both downloads in the same folder. Use Python 3.11 to 3.14. These commands are for macOS and Linux.

    unzip wspbench-source-v0.2.1.zip
cd wspbench-v0.2.1
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .

  2. Check the reference answers

    The checker evaluates all 1,024 inputs for each of the 12 Boolean candidate examples. The remaining examples are diagnostic controls.

    unzip ../wspbench-development-v0.2.1.zip -d examples-data
python examples/check_submission.py --check-references

    Expected output: "checked": 12, "exact": 12. This checks the evaluator, not an AI model.

  3. Try your own model

    Export an A/full task prompt. Give it to your model with tools disabled and save its JSON answer as answer.json.

    python examples/check_submission.py \
  --task examples-data/0ccf05ea24942ed4 --prompt > prompt.json
python examples/check_submission.py \
  --task examples-data/0ccf05ea24942ed4 --submission answer.json

    Keep the reference answers out of the solver's context. The checker runs locally and makes no provider calls.

Public examples are for development. Official submissions and held-out evaluation are not open yet.

Full guide, answer formats, and reporting →

03 / TASKS

Task families

The current candidates use composed Boolean circuits. Causal forecasts and repair tasks remain diagnostic controls.

Candidate tasks and diagnostic controls, with their scoring criteria
SkillTaskEvaluation
ReadRule recovery Candidate

Recover a Boolean function from numerical weights.

Functional agreement with the target network.
ReadCausal forecast Diagnostic

Predict the effect of a specified weight intervention.

Error in predicted output-change fractions.
ReadEdit effects Candidate

Predict the behavioral change caused by an internal operator edit.

Behavioral agreement, affected inputs, and false positives.
RepairCoupled repair Diagnostic

Fix a fault while preserving the network's other behavior.

Target accuracy, retention, and edit size.
RepairLoop repair Diagnostic · C only

Correct repetition or halting failures with a supplied diagnosis.

Correct completion and preservation of valid repetition.
WriteTruth-table synthesis Candidate

Compress a complete Boolean truth table into network weights.

Functional correctness within the fixed architecture and capacity.

Candidate circuits use 10 binary inputs, 1,024 possible inputs, and 4,049 scalar parameters. They are constructed ReLU networks. Task and access-condition admission are assessed separately.

04 / PROTOCOL

Access conditions

Matched trials use the same target, specification, architecture, and initial weights.

A

Unaided

Weights, architecture, and the task specification. No external tools.

B

Weight analysis

A plus bounded tensor algebra. No target execution or activations.

C

Instrumented

B plus metered target execution and activation probes.

Scoring

A separate evaluator checks predictions and submitted tensors on hidden cases. The planned primary evaluation averages three attempts per task and clusters confidence intervals by underlying problem.

Reporting

Report scores by task and access condition, alongside retention, edit size, observations, and cost. Record supplied hints and diagnoses. Report constructed and trained targets separately.

Diagnostic controls and development runs do not enter the official leaderboard.