Jev / Luna

success

valid=true with a finite probability and valid target in [0,1]; label targets must be 0 or 1

accuracy

Fraction correct among valid responses, predicting yes when p_yes >= 0.5

accuracy_all_requests

Correct valid responses divided by every measured request; invalid requests count as failures

brier

Mean (p_yes - observed_label)^2; lower is better

log_loss

Mean binary negative log likelihood in natural-log units; p clipped to [1e-15, 1-1e-15]

mae

Mean absolute difference between reported p_yes and exact reference probability

rmse

Square root of mean squared error against exact reference probabilities

excess_brier

Mean (p_yes - true_probability)^2: expected Brier above the optimal probability forecast

expected_brier

Mean [(p_yes - true_probability)^2 + true_probability*(1-true_probability)]

latency

End-to-end request time through receipt of a complete response; main table includes valid responses only

cost

Recorded billed USD across all requests in the stated phase, including failed/invalid requests when billed

cost_per_1000

Measured total billed cost / measured request count * 1000; withheld if any bill is missing

cost_per_correct

All measured billed cost / number of correct valid label responses; withheld for missing bills or zero correct

bootstrap

Percentile 95% paired bootstrap by case identity; quality repeats averaged within case; latency repeats use median; no CI for fewer than 2 pairs

{
  "models": {
    "Jev": "typesafe/jev-1.13",
    "Luna": "openai/gpt-6-luna"
  },
  "created_at": "2026-10-02T08:42:23.244948+00:00",
  "seed": 20261002,
  "cases": 180,
  "calls_planned": 360,
  "budget_usd": 0.999912658,
  "luna_reasoning": "none",
  "luna_provider": "OpenAI",
  "protocol": "One non-streaming request per model per case; sequential paired random order; one shared persistent HTTPS connection; no automatic retries; two warmups per model excluded from measured results.",
  "cases_sha256": "60e6db60c72869936548c3c4620aa3b070cf12e516f3be42a61a306d0f592660",
  "runner_sha256": "3a05cb20816fbc5b7c30a065429a7365afa9583f273d683bf69784f4af3e9ad7",
  "status": "complete",
  "charged_usd": 0.011550069999999994,
  "calls_completed": 364,
  "finished_at": "2026-10-02T08:46:47.575786+00:00"
}