{ "reviewedAt": "2026-11-06T15:24:45.553068+00:00", "reax": { "source": "https://huggingface.co/spaces/multimodalart/jev-decision-index", "nativeExcerpt": "Browser tab: 483, Title: \"Jev Decision Index - a Hugging Face Space by multimodalart\", URL: \"https://huggingface.co/spaces/multimodalart/jev-decision-index\".\n1 AXWebArea Jev Decision Index - a Hugging Face Space by multimodalart, URL: huggingface.co/spaces/multimodalart/jev-decision-index\n\t2 container\n\t\t3 link Description: multimodalart, Value: huggingface.co/multimodalart\n\t\t4 text /\n\t\t5 link Description: jev-decision-index, Value: huggingface.co/spaces/multimodalart/jev-decision-index\n\t\t6 button Like\n\t\t7 button 876, Help: See users who liked this repository\n\t\t8 button Show space menu\n\t9 container Description: static space app, ID: iFrameResizer0\n\t\t10 AXWebArea Decision Index 0.4.1, URL: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t11 container Section\n\t\t\t\t12 link Description: NEWS, Value: multimodalart-jev-decision-index.static.hf.space/news.html\n\t\t\t\t13 link Description: INDEX, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t14 container app\n\t\t\t\t15 heading Decision Index 0.4.1 What's new in this edition, Value: 1\n\t\t\t\t\t16 text Decision \n\t\t\t\t\t17 container\n\t\t\t\t\t\t18 text Index 0.4.1\n\t\t\t\t\t\t19 button (collapsed) Description: What's new in this edition, Secondary Actions: Expand\n\t\t\t\t\t\t\t20 text what's\n\t\t\t\t\t\t\t21 text new?\n\t\t\t\t22 container\n\t\t\t\t\t23 text 274 open reproductions of TypeSafe Jev's \"Decision Model\" , ranked by the Full score : private tests that measure the same skills as 37 public benchmarks and a set of private tasks from new domains, with the public benchmarks shown alongside.\n\t\t\t\t24 container Board\n\t\t\t\t\t25 checkbox TEXT, Value: 1\n\t\t\t\t\t26 checkbox VISION, Value: 0\n\t\t\t\t27 text v0.4.1 · 275 models · Jev jev-1.14.0 · reproductions run on 1 x NVIDIA RTX PRO 6000 · updated 2026-11-06\n\t\t\t\t28 container Sections\n\t\t\t\t\t29 link Description: RANKING, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t30 link Description: HOW IT IS SCORED, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t31 link Description: HEAD TO HEAD, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t32 link Description: FULL RESULTS, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t33 link Description: CALIBRATION, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t34 link Description: ALL BENCHMARKS, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t35 link Description: METHODOLOGY, Value: multimodalart-jev-decision-index.static.hf.space/methodolog…\n\t\t\t\t36 container summary\n\t\t\t\t\t37 container\n\t\t\t\t\t\t38 text PARAMETERS All sizes\n\t\t\t\t\t\t39 checkbox Hide Jev, Value: 0\n\t\t\t\t\t\t40 checkbox Jev and above, Value: 0\n\t\t\t\t\t\t41 slider (settable, integer) Description: Smallest model size, Value: 0\n\t\t\t\t\t\t42 slider (settable, integer) Description: Largest model size, Value: 1000\n\t\t\t\t\t\t43 text <50M 1B 5B 10B 20B 35B+\n\t\t\t\t\t44 container legend\n\t\t\t\t\t\t45 checkbox Jev (reference), Value: 1\n\t\t\t\t\t\t46 checkbox inference technique, Value: 1\n\t\t\t\t\t\t47 checkbox full fine-tune, Value: 1\n\t\t\t\t\t\t48 checkbox LoRA, Value: 1\n\t\t\t\t\t\t49 checkbox head / adapter, Value: 1\n\t\t\t\t\t50 container Score shown\n\t\t\t\t\t\t51 checkbox FULL SCORE, Value: 1\n\t\t\t\t\t\t52 checkbox PUBLIC ONLY, Value: 0\n\t\t\t\t\t\t53 checkbox YOUR OWN, Value: 0\n\t\t\t\t\t54 container\n\t\t\t\t\t\t55 container Full score by model\n\t\t\t\t\t\t56 container barnote\n\t\t\t\t\t\t\t57 text The Full score: 60% private tests of the same skills and 40% private tasks from new domains, on one scale; the public benchmarks are shown but not counted. Hover or tap any bar for its parts. \n\t\t\t\t\t\t\t58 link Description: How it works →, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t59 container\n\t\t\t\t\t\t60 checkbox pareto, Value: 0\n\t\t\t\t\t\t61 text MODEL SIZE VS FULL SCORE\n\t\t\t\t\t\t62 container sizescatter\n\t\t\t\t\t\t\t63 container Full score against model size\n\t\t\t\t\t\t\t64 text Not plotted, no parameter count: Jev 1.13, JEV FLAN-T5 Large public v2 (342M), Prumo, Zev.\n\t\t\t\t\t\t65 checkbox pareto, Value: 0\n\t\t\t\t\t\t66 text CALIBRATION ERROR VS FULL SCORE\n\t\t\t\t\t\t67 container Full score against calibration error\n\t\t\t\t\t\t68 checkbox pareto, Value: 0\n\t\t\t\t\t\t69 text MEDIAN LATENCY VS FULL SCORE\n\t\t\t\t\t\t70 container Full score against median latency\n\t\t\t\t71 container private\n\t\t\t\t\t72 heading How the Full score works, Value: 2\n\t\t\t\t\t\t73 text How the Full score works\n\t\t\t\t\t74 text Two private parts, each on the public scale, added with fixed weights. The public benchmarks are shown next to every model but do not count. Nothing is tuned to any one model.\n\t\t\t\t75 text 1 · Public benchmarks\n\t\t\t\t76 text shown, not counted\n\t\t\t\t77 text The 37 open tasks shown on this site. Anyone can see them, and train on them, so they no longer count toward the Full score.\n\t\t\t\t78 text 2 · Private tests, same skills\n\t\t\t\t79 text 60% of the Full score\n\t\t\t\t80 text For each public task with a private counterpart, a test of an equivalent competence built from fresh data and never published. It carries the most weight because nobody can train on it.\n\t\t\t\t81 text 3 · Private tasks from new domains\n\t\t\t\t82 text 40% of the Full score\n\t\t\t\t83 text A set of private decision tasks from new domains. Some reward broad knowledge, others reward judging when to answer yes and when to answer no.\n\t\t\t\t84 text The two private parts and the public benchmarks spread models out differently, so their raw points are not the same unit. Before adding the parts up, each one is put on the public scale using 21 untuned stock models as the yardstick: a model that sits one spread above the stock average on the private tests gets the score that sits one spread above the stock average on the public benchmarks. Order within each part never changes.\n\t\t\t\t85 button (collapsed) + LEARN MORE: PUBLIC VS PRIVATE, MODEL BY MODEL, Secondary Actions: Expand\n\t\t\t\t\t86 container\n\t\t\t\t\t\t87 text + +\n\t\t\t\t\t88 text LEARN MORE: PUBLIC VS PRIVATE, MODEL BY MODEL\n\t\t\t\t89 container headtohead\n\t\t\t\t\t90 heading Head to head, Value: 2\n\t\t\t\t\t\t91 text Head to head\n\t\t\t\t\t92 container h2hhead\n\t\t\t\t\t\t93 text Jev against the strongest models on the two private parts of the Full score. Only totals are shown: the private tests stay private.\n\t\t\t\t\t\t94 container Head to head view\n\t\t\t\t\t\t\t95 checkbox PRIVATE (FULL SCORE PARTS), Value: 1\n\t\t\t\t\t\t\t96 checkbox PUBLIC BENCHMARKS, Value: 0\n\t\t\t\t97 heading Private tests by area, Value: 3\n\t\t\t\t\t98 text Private tests by area\n\t\t\t\t99 text Full score · Jev and the top 3 models\n\t\t\t\t100 container Private scores radar\n\t\t\t\t101 text Jev\n\t\t\t\t102 text 60.2 · reference\n\t\t\t\t103 link Description: 1 REAX 1.3 74.0, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t104 link Description: 2 GLiDE 27B 73.4, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t105 link Description: 3 KnowLine 31B Gen1.5 71.9, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t106 text 6 spokes · one per area for its private tests of the same skills, and one for the private tasks from new domains, each on the public scale\n\t\t\t\t107 heading Full score parts, Value: 3\n\t\t\t\t\t108 text Full score parts\n\t\t\t\t109 text Full score · Jev and the top 8 models\n\t\t\t\t110 table\n\t\t\t\t\t111 row\n\t\t\t\t\t\t112 cell\n\t\t\t\t\t\t\t113 text MODEL\n\t\t\t\t\t\t114 cell\n\t\t\t\t\t\t\t115 text FULL SCORE\n\t\t\t\t\t\t116 cell\n\t\t\t\t\t\t\t117 text SAME SKILLS\n\t\t\t\t\t\t118 cell\n\t\t\t\t\t\t\t119 text NEW DOMAINS\n\t\t\t\t\t120 row\n\t\t\t\t\t\t121 cell\n\t\t\t\t\t\t\t122 text Jev\n\t\t\t\t\t\t\t123 text · reference\n\t\t\t\t\t\t124 cell\n\t\t\t\t\t\t\t125 text 60.2\n\t\t\t\t\t\t126 cell\n\t\t\t\t\t\t\t127 text 58.9\n\t\t\t\t\t\t128 cell\n\t\t\t\t\t\t\t129 text 62.2\n\t\t\t\t\t130 row\n\t\t\t\t\t\t131 cell\n\t\t\t\t\t\t\t132 text 1\n\t\t\t\t\t\t\t133 link Description: REAX 1.3, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t\t134 cell\n\t\t\t\t\t\t\t135 text 74.0\n\t\t\t\t\t\t136 cell\n\t\t\t\t\t\t\t137 text 69.0\n\t\t\t\t\t\t138 cell\n\t\t\t\t\t\t\t139 text 81.4\n\t\t\t\t\t140 row\n\t\t\t\t\t\t141 cell\n\t\t\t\t\t\t\t142 text 2\n\t\t\t\t\t\t\t143 link Description: GLiDE 27B, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t\t144 cell\n\t\t\t\t\t\t\t145 text 73.4\n\t\t\t\t\t\t146 cell\n\t\t\t\t\t\t\t147 text 69.8\n\t\t\t\t\t\t148 cell\n\t\t\t\t\t\t\t149 text 78.7\n\t\t\t\t\t150 row\n\t\t\t\t\t\t151 cell\n\t\t\t\t\t\t\t152 text 3\n\t\t\t\t\t\t\t153 link Description: KnowLine 31B Gen1.5, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t\t154 cell\n\t\t\t\t\t\t\t155 text 71.9\n\t\t\t\t\t\t156 cell\n\t\t\t\t\t\t\t157 text 66.8\n\t\t\t\t\t\t158 cell\n\t\t\t\t\t\t\t159 text 79.6\n\t\t\t\t\t160 row\n\t\t\t\t\t\t161 cell\n\t\t\t\t\t\t\t162 text 4\n\t\t\t\t\t\t\t163 link Description: FlexDM-LoRA 2.1 · Gemma-4-31B, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t\t164 cell\n\t\t\t\t\t\t\t165 text 71.6\n\t\t\t\t\t\t166 cell\n\t\t\t\t\t\t\t167 text 66.1\n\t\t\t\t\t\t168 cell\n\t\t\t\t\t\t\t169 text 79.9\n\t\t\t\t\t170 row\n\t\t\t\t\t\t171 cell\n\t\t\t\t\t\t\t172 text 5\n\t\t\t\t\t\t\t173 link Description: Sieve 31B v3, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t\t174 cell\n\t\t\t\t\t\t\t175 text 71.1\n\t\t\t\t\t\t176 cell\n\t\t\t\t\t\t\t177 text 68.3\n\t\t\t\t\t\t178 cell\n\t\t\t\t\t\t\t179 text 75.3\n\t\t\t\t\t180 row\n\t\t\t\t\t\t181 cell\n\t\t\t\t\t\t\t182 text 6\n\t\t\t\t\t\t\t183 link Description: Turion Jev 31B, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t\t184 cell\n\t\t\t\t\t\t\t185 text 70.5\n\t\t\t\t\t\t186 cell\n\t\t\t\t\t\t\t187 text 65.0\n\t\t\t\t\t\t188 cell\n\t\t\t\t\t\t\t189 text 78.6\n\t\t\t\t\t190 row\n\t\t\t\t\t\t191 cell\n\t\t\t\t\t\t\t192 text 7\n\t\t\t\t\t\t\t193 link Description: vLLM-SR d3-ultra (Decision 3.0, 27B), Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t\t194 cell\n\t\t\t\t\t\t\t195 text 70.2\n\t\t\t\t\t\t196 cell\n\t\t\t\t\t\t\t197 text 66.5\n\t\t\t\t\t\t198 cell\n\t\t\t\t\t\t\t199 text 75.7\n\t\t\t\t\t200 row\n\t\t\t\t\t\t201 cell\n\t\t\t\t\t\t\t202 text 8\n\t\t\t\t\t\t\t203 link Description: Bespoke Nimble v4.1 Max (27B), Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t\t204 cell\n\t\t\t\t\t\t\t205 text 70.0\n\t\t\t\t\t\t206 cell\n\t\t\t\t\t\t\t207 text 67.1\n\t\t\t\t\t\t208 cell\n\t\t\t\t\t\t\t209 text 74.4\n\t\t\t\t210 text 60% private tests of the same skills + 40% private tasks from new domains, both on the public scale\n\t\t\t\t211 container fullresults\n\t\t\t\t\t212 heading Full results, Value: 2\n\t\t\t\t\t\t213 text Full results\n\t\t\t\t\t214 text Click a model for its complete benchmark-by-benchmark page. Sort any column.\n\t\t\t\t215 container\n\t\t\t\t\t216 text PARAMETERS All sizes\n\t\t\t\t\t217 checkbox Jev and above, Value: 0\n\t\t\t\t\t218 slider (settable, integer) Description: Smallest model size, Value: 0\n\t\t\t\t\t219 slider (settable, integer) Description: Largest model size, Value: 1000\n\t\t\t\t\t220 text <50M 1B 5B 10B 20B 35B+\n\t\t\t\t221 container Score shown\n\t\t\t\t\t222 checkbox FULL SCORE, Value: 1\n\t\t\t\t\t223 checkbox PUBLIC ONLY, Value: 0\n\t\t\t\t\t224 checkbox YOUR OWN, Value: 0\n\t\t\t\t225 container table\n\t\t\t\t\t226 text RANK BY AREA\n\t\t\t\t\t227 button Knowledge\n\t\t\t\t\t228 button Language\n\t\t\t\t\t229 button Retrieval\n\t\t\t\t\t230 button Tools\n\t\t\t\t\t231 button Arts\n\t\t\t\t\t232 table\n\t\t\t\t\t\t233 row\n\t\t\t\t\t\t\t234 cell\n\t\t\t\t\t\t\t\t235 text #\n\t\t\t\t\t\t\t236 cell\n\t\t\t\t\t\t\t\t237 text MODEL\n\t\t\t\t\t\t\t238 cell\n\t\t\t\t\t\t\t\t239 text FULL SCORE ↓\n\t\t\t\t\t\t\t240 cell\n\t\t\t\t\t\t\t\t241 text KNOWLEDGE\n\t\t\t\t\t\t\t242 cell\n\t\t\t\t\t\t\t\t243 text LANGUAGE\n\t\t\t\t\t\t\t244 cell\n\t\t\t\t\t\t\t\t245 text RETRIEVAL\n\t\t\t\t\t\t\t246 cell\n\t\t\t\t\t\t\t\t247 text TOOLS\n\t\t\t\t\t\t\t248 cell\n\t\t\t\t\t\t\t\t249 text ARTS\n\t\t\t\t\t\t\t250 cell\n\t\t\t\t\t\t\t\t251 text CALIBRATION\n\t\t\t\t\t\t\t252 cell\n\t\t\t\t\t\t\t\t253 text SIZE\n\t\t\t\t\t\t\t254 cell\n\t\t\t\t\t\t\t\t255 text ANSWERED\n\t\t\t\t\t\t\t256 cell\n\t\t\t\t\t\t\t\t257 text LATENCY\n\t\t\t\t\t\t258 row\n\t\t\t\t\t\t\t259 cell\n\t\t\t\t\t\t\t\t260 text =1\n\t\t\t\t\t\t\t261 cell\n\t\t\t\t\t\t\t\t262 link Description: REAX 1.3, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t\t\t\t263 button Why\n\t\t\t\t\t\t\t\t264 text NEW INFERENCE TECHNIQUE\n\t\t\t\t\t\t\t\t265 link Description: model ↗, Value: huggingface.co/decisionmodels/REAX-1.3\n\t\t\t\t\t\t\t\t266 text autoregressive\n\t\t\t\t\t\t\t267 cell\n\t\t\t\t\t\t\t\t268 text 74.0\n\t\t\t\t\t\t\t269 cell\n\t\t\t\t\t\t\t\t270 text 57.9\n\t\t\t\t\t\t\t271 cell\n\t\t\t\t\t\t\t\t272 text 68.2\n\t\t\t\t\t\t\t273 cell\n\t\t\t\t\t\t\t\t274 text 71.5\n\t\t\t\t\t\t\t275 cell\n\t\t\t\t\t\t\t\t276 text 84.5\n\t\t\t\t\t\t\t277 cell\n\t\t\t\t\t\t\t\t278 text 62.9\n\t\t\t\t\t\t\t279 cell\n\t\t\t\t\t\t\t\t280 text 1.8\n\t\t\t\t\t\t\t281 cell\n\t\t\t\t\t\t\t\t282 text 57B\n\t\t\t\t\t\t\t283 cell\n\t\t\t\t\t\t\t\t284 text 100.0%\n\t\t\t\t\t\t\t285 cell\n\t\t\t\t\t\t\t\t286 text 246 ms\n\t\t\t\t\t\t287 row\n\t\t\t\t\t\t\t288 cell\n\t\t\t\t\t\t\t\t289 text =1\n\t\t\t\t\t\t\t290 cell\n\t\t\t\t\t\t\t\t291 image Fastino\n\t\t\t\t\t\t\t\t292 link Description: GLiDE 27B, Value: multimodalart-jev-decision-index.static.hf.space/index.html\n\t\t\t\t\t\t\t\t293 button Why\n\t\t\t\t\t\t\t\t294 text NEW FULL FINE-TUNE\n\t\t\t\t\t\t\t\t295 link Description: model ↗, Value: huggingface.co/fastino/GLiDE-27B\n\t\t\t\t\t\t\t\t296 text from\n\t\t\t\t\t\t\t\t297 link Description: Qwen/Qwen3.8-27B, Value: huggingface.co/Qwen/Qwen3.8-27B\n\t\t\t\t\t\t\t\t298 text autoregressive\n\t\t\t\t\t\t\t299 cell\n\t\t\t\t\t\t\t\t300 text 73.4\n\t\t\t\t\t\t\t301 cell\n\t\t\t\t\t\t\t\t302 text 59.8\n\t\t\t\t\t\t\t303 cell\n\t\t\t\t\t\t\t\t304 text 66.8\n\t\t\t\t\t\t\t305 cell\n\t\t\t\t\t\t\t\t306 text 73.4\n\t\t\t\t\t\t\t307 cell\n\t\t\t\t\t\t\t\t308 text 86.0\n\t\t\t\t\t\t\t309 cell\n\t\t\t\t\t\t\t\t310 text 62.3\n\t\t\t\t\t\t\t311 cell\n\t\t\t\t\t\t\t\t312 text 2.1\n\t\t\t\t\t\t\t313 cell\n\t\t\t\t\t\t\t\t314 text 26B\n\t\t\t\t\t\t\t315 cell\n\t\t\t\t\t\t\t\t316 text 100.0%\n\t\t\t\t\t\t\t317 cell\n\t\t\t\t\t\t\t\t318 text 122 ms", "modelAPI": { "_id": "6ae7afbaaceb3bbab4a1e95f", "id": "decisionmodels/REAX-1.3", "private": false, "tags": [ "safetensors", "decision-model", "reax", "en", "license:apache-2.0", "region:us" ], "downloads": 0, "likes": 2, "modelId": "decisionmodels/REAX-1.3", "author": "decisionmodels", "sha": "fa4bf7ee3bb56a8bc421dad090631be1c5845c2d", "lastModified": "2026-11-03T16:04:16.000Z", "gated": false, "disabled": false, "model-index": null, "config": {}, "cardData": { "license": "apache-2.0", "language": [ "en" ], "tags": [ "decision-model", "reax" ] }, "siblings": [ { "rfilename": ".gitattributes" }, { "rfilename": "ATTRIBUTION.md" }, { "rfilename": "LICENSE" }, { "rfilename": "NOTICE" }, { "rfilename": "README.md" }, { "rfilename": "UPSTREAMS.json" }, { "rfilename": "engine/Dockerfile" }, { "rfilename": "engine/Dockerfile.conv" }, { "rfilename": "engine/LICENSE" }, { "rfilename": "engine/README.md" }, { "rfilename": "engine/SEALED-CONTRACT.md" }, { "rfilename": "engine/SHA256SUMS.json" }, { "rfilename": "engine/SHA256SUMS.shipped.json" }, { "rfilename": "engine/THIRD_PARTY_NOTICES.md" }, { "rfilename": "engine/autotune_guard.py" }, { "rfilename": "engine/backend.py" }, { "rfilename": "engine/benchmark.py" }, { "rfilename": "engine/fast_fp8_gemm.py" }, { "rfilename": "engine/probabilities.py" }, { "rfilename": "engine/reax13.json" }, { "rfilename": "engine/reax13_engine.py" }, { "rfilename": "engine/serve.py" }, { "rfilename": "engine/test_bundle.py" }, { "rfilename": "engine/test_fast_fp8_gemm.py" }, { "rfilename": "engine/test_probabilities.py" }, { "rfilename": "engine/test_serve.py" }, { "rfilename": "engine/third_party/SHA256SUMS.json" }, { "rfilename": "engine/third_party/d3/LICENSE" }, { "rfilename": "engine/third_party/d3/NOTICE" }, { "rfilename": "engine/third_party/finegrained-fp8/LICENSE" }, { "rfilename": "engine/third_party/finegrained-fp8/__init__.py" }, { "rfilename": "engine/third_party/finegrained-fp8/_ops.py" }, { "rfilename": "engine/third_party/finegrained-fp8/_private_for_testing.py" }, { "rfilename": "engine/third_party/finegrained-fp8/batched.py" }, { "rfilename": "engine/third_party/finegrained-fp8/bayesian_autotuner.py" }, { "rfilename": "engine/third_party/finegrained-fp8/fused_batched.py" }, { "rfilename": "engine/third_party/finegrained-fp8/fused_grouped.py" }, { "rfilename": "engine/third_party/finegrained-fp8/grouped.py" }, { "rfilename": "engine/third_party/finegrained-fp8/matmul.py" }, { "rfilename": "engine/third_party/finegrained-fp8/metadata.json" }, { "rfilename": "engine/third_party/finegrained-fp8/metadata.json.sigstore" }, { "rfilename": "engine/third_party/finegrained-fp8/symbols.json" }, { "rfilename": "engine/third_party/finegrained-fp8/utils.py" }, { "rfilename": "engine/third_party/knowline/LICENSE" }, { "rfilename": "engine/third_party/knowline/WEIGHT-LICENSE-Apache-2.0" }, { "rfilename": "engine/third_party/knowline/knowline_layout.py" }, { "rfilename": "engine/third_party/knowline/knowline_server.py" }, { "rfilename": "engine/third_party/knowline/temperatures.json" }, { "rfilename": "licenses/CharlieBT--gx31-decider/LICENSE" }, { "rfilename": "licenses/CharlieBT--gx31-decider/NOTICE" }, { "rfilename": "licenses/KnowLine-serving-MIT.txt" }, { "rfilename": "licenses/PelaAI--KnowLine-31B-Gen1/LICENSE" }, { "rfilename": "licenses/Qwen--Qwen3.5-27B/LICENSE" }, { "rfilename": "licenses/Qwen--Qwen3.8-27B/LICENSE" }, { "rfilename": "licenses/google--gemma-4-31B-it/LICENSE" }, { "rfilename": "licenses/gurujustin--sn92-gen-p5/LICENSE" }, { "rfilename": "licenses/gurujustin--sn92-gen-p5/NOTICE" }, { "rfilename": "licenses/isomsom--sn92-gen-m87b/LICENSE" }, { "rfilename": "licenses/isomsom--sn92-gen-m87b/NOTICE" }, { "rfilename": "licenses/perplexity-ai--pplx-decider-v1.2/LICENSE" }, { "rfilename": "licenses/perplexity-ai--pplx-decider-v1.2/NOTICE" }, { "rfilename": "licenses/ronanlee777--sn92-gen-ma/LICENSE" }, { "rfilename": "licenses/ronanlee777--sn92-gen-ma/NOTICE" }, { "rfilename": "licenses/tattrongvu--Jeffan-31b-1710/LICENSE" }, { "rfilename": "licenses/tattrongvu--Jeffan-31b-1710/inference/LICENSE" }, { "rfilename": "licenses/vllm-sr--d3-ultra/LICENSE" }, { "rfilename": "licenses/vllm-sr--d3-ultra/NOTICE" }, { "rfilename": "licenses/xflud--sn92-decider-1/LICENSE" }, { "rfilename": "licenses/xflud--sn92-decider-1/NOTICE" }, { "rfilename": "part-g/INFERENCE.md" }, { "rfilename": "part-g/LICENSE" }, { "rfilename": "part-g/chat_template.jinja" }, { "rfilename": "part-g/config.json" }, { "rfilename": "part-g/generation_config.json" }, { "rfilename": "part-g/knowline_engine.py" }, { "rfilename": "part-g/knowline_layout.py" }, { "rfilename": "part-g/knowline_server.py" }, { "rfilename": "part-g/model-00001-of-00013.safetensors" }, { "rfilename": "part-g/model-00002-of-00013.safetensors" }, { "rfilename": "part-g/model-00003-of-00013.safetensors" }, { "rfilename": "part-g/model-00004-of-00013.safetensors" }, { "rfilename": "part-g/model-00005-of-00013.safetensors" }, { "rfilename": "part-g/model-00006-of-00013.safetensors" }, { "rfilename": "part-g/model-00007-of-00013.safetensors" }, { "rfilename": "part-g/model-00008-of-00013.safetensors" }, { "rfilename": "part-g/model-00009-of-00013.safetensors" }, { "rfilename": "part-g/model-00010-of-00013.safetensors" }, { "rfilename": "part-g/model-00011-of-00013.safetensors" }, { "rfilename": "part-g/model-00012-of-00013.safetensors" }, { "rfilename": "part-g/model-00013-of-00013.safetensors" }, { "rfilename": "part-g/model.safetensors.index.json" }, { "rfilename": "part-g/processor_config.json" }, { "rfilename": "part-g/serve_knowline.sh" }, { "rfilename": "part-g/temperatures.json" }, { "rfilename": "part-g/tokenizer.json" }, { "rfilename": "part-g/tokenizer_config.json" }, { "rfilename": "part-q/LICENSE" }, { "rfilename": "part-q/MODEL_MANIFEST.json" }, { "rfilename": "part-q/NOTICE" }, { "rfilename": "part-q/assets/CREDITS.md" }, { "rfilename": "part-q/assets/ask-once.png" }, { "rfilename": "part-q/assets/banner.png" }, { "rfilename": "part-q/assets/curiosity.jpg" }, { "rfilename": "part-q/assets/eval-overall.png" }, { "rfilename": "part-q/assets/eval-sr-box.png" }, { "rfilename": "part-q/assets/eval-sr-text.png" }, { "rfilename": "part-q/assets/examples/curiosity.png" }, { "rfilename": "part-q/assets/examples/five-a-side.png" }, { "rfilename": "part-q/assets/examples/fur-traders.png" }, { "rfilename": "part-q/assets/examples/grande-jatte.png" }, { "rfilename": "part-q/assets/examples/lunar-rover.png" }, { "rfilename": "part-q/assets/examples/pantheon.png" }, { "rfilename": "part-q/assets/examples/saturn.png" }, { "rfilename": "part-q/assets/examples/starship.png" }, { "rfilename": "part-q/assets/examples/sudden-shower.png" }, { "rfilename": "part-q/assets/five-a-side.jpg" }, { "rfilename": "part-q/assets/fur-traders.jpg" }, { "rfilename": "part-q/assets/grande-jatte.jpg" }, { "rfilename": "part-q/assets/lunar-rover.jpg" }, { "rfilename": "part-q/assets/pantheon-a.jpg" }, { "rfilename": "part-q/assets/pantheon-b.jpg" }, { "rfilename": "part-q/assets/pantheon-c.jpg" }, { "rfilename": "part-q/assets/pantheon-d.jpg" }, { "rfilename": "part-q/assets/pantheon.jpg" }, { "rfilename": "part-q/assets/saturn-a.jpg" }, { "rfilename": "part-q/assets/saturn-b.jpg" }, { "rfilename": "part-q/assets/saturn-c.jpg" }, { "rfilename": "part-q/assets/saturn.jpg" }, { "rfilename": "part-q/assets/sudden-shower.jpg" }, { "rfilename": "part-q/chat_template.jinja" }, { "rfilename": "part-q/config.json" }, { "rfilename": "part-q/d3_compose.py" }, { "rfilename": "part-q/d3_engine.py" }, { "rfilename": "part-q/d3_fast.py" }, { "rfilename": "part-q/d3_format.py" }, { "rfilename": "part-q/d3_kernels.py" }, { "rfilename": "part-q/d3_runtime.py" }, { "rfilename": "part-q/d3_server.py" }, { "rfilename": "part-q/decision_config.json" }, { "rfilename": "part-q/merges.txt" }, { "rfilename": "part-q/model-00001-of-00011.safetensors" }, { "rfilename": "part-q/model-00002-of-00011.safetensors" }, { "rfilename": "part-q/model-00003-of-00011.safetensors" }, { "rfilename": "part-q/model-00004-of-00011.safetensors" }, { "rfilename": "part-q/model-00005-of-00011.safetensors" }, { "rfilename": "part-q/model-00006-of-00011.safetensors" }, { "rfilename": "part-q/model-00007-of-00011.safetensors" }, { "rfilename": "part-q/model-00008-of-00011.safetensors" }, { "rfilename": "part-q/model-00009-of-00011.safetensors" }, { "rfilename": "part-q/model-00010-of-00011.safetensors" }, { "rfilename": "part-q/model-00011-of-00011.safetensors" }, { "rfilename": "part-q/model.safetensors.index.json" }, { "rfilename": "part-q/modeling_d3.py" }, { "rfilename": "part-q/pipeline_d3.py" }, { "rfilename": "part-q/preprocessor_config.json" }, { "rfilename": "part-q/readout.safetensors" }, { "rfilename": "part-q/requirements.txt" }, { "rfilename": "part-q/tokenizer.json" }, { "rfilename": "part-q/tokenizer_config.json" }, { "rfilename": "part-q/video_preprocessor_config.json" }, { "rfilename": "part-q/vocab.json" } ], "spaces": [ "multimodalart/jev-decision-index" ], "createdAt": "2026-11-01T21:31:38.000Z", "usedStorage": 114788853778 }, "limits": "Native page snapshot, not an independent benchmark rerun. Full score74.0, table tiedrank=1; head-to-head order alone is not a statistically unique lead." }, "sundae": { "source": "https://www.sundaebar.ai/lab/challenge/0a387000-cd14-426e-a68c-35302a44c205", "nativeChallenge": "Browser tab: 483, Title: \"SN121 Challenge | sundae_bar\", URL: \"https://www.sundaebar.ai/lab/challenge/0a387000-cd14-426e-a68c-35302a44c205\".\n1 AXWebArea SN121 Challenge | sundae_bar, URL: sundaebar.ai/lab/challenge/0a387000-cd14-426e-a68c-35302a44c205\n\t2 container Cookie consent\n\t\t3 text We use cookies to see how sundae_bar is used and whether our ads are working.\n\t\t4 container\n\t\t\t5 text Analytics and marketing cookies stay off unless you accept them. Nothing here affects what you can do on the site.\n\t\t\t6 link Description: Privacy policy, Value: sundaebar.ai/privacy-policy\n\t\t7 button Reject all\n\t\t8 button Accept all\n\t\t9 button Choose what to allow\n\t10 container\n\t\t11 link Description: Sundae Bar Logo, Value: sundaebar.ai/\n\t\t12 container\n\t\t\t13 text Lab\n\t\t\t14 content list\n\t\t\t\t15 link Description: Dashboard, Value: sundaebar.ai/lab\n\t\t16 container\n\t\t\t17 text Developer\n\t\t\t18 content list\n\t\t\t\t19 container\n\t\t\t\t\t20 button (expanded) Challenges, Secondary Actions: Collapse\n\t\t\t\t\t21 content list\n\t\t\t\t\t\t43 link Description: Live Challenge, Value: sundaebar.ai/lab/challenge/0a387000-cd14-426e-a68c-35302a44c205\n\t\t\t\t\t\t24 link Description: All Challenges, Value: sundaebar.ai/lab/challenges\n\t\t\t\t25 button Submissions\n\t\t\t\t26 link Description: Rewards, Value: sundaebar.ai/lab/rewards\n\t\t27 button Toggle Sidebar\n\t\t28 container\n\t\t\t29 link sundaebar.ai/\n\t\t\t30 container Primary\n\t\t\t\t31 link Description: Marketplace, Value: sundaebar.ai/explore\n\t\t\t\t32 link Description: Agent Lab, Value: sundaebar.ai/lab\n\t\t\t\t33 button About\n\t\t\t\t34 button Search\n\t\t\t\t\t35 text Search sundae_bar\n\t\t\t\t\t36 text ⌘\n\t\t\t\t\t37 text K\n\t\t\t\t38 link Description: Log in, Value: sundaebar.ai/login\n\t\t\t\t39 link Description: Sign up, Value: sundaebar.ai/sign-up\n\t\t\t\t40 link Description: Run Agents, Value: sundaebar.ai/chat\n\t\t44 container\n\t\t\t45 text LIVE\n\t\t\t46 text Executive Presentation Strategy Skill\n\t\t\t47 text SBC 35\n\t\t\t48 heading Executive Presentation Strategy Skill, Value: 1\n\t\t\t\t49 text Executive Presentation Strategy Skill\n\t\t\t50 container Challenge sections\n\t\t\t\t51 link Description: Overview, Value: sundaebar.ai/lab/challenge/0a387000-cd14-426e-a68c-35302a44c205#challenge\n\t\t\t\t52 link Description: Requirements, Value: sundaebar.ai/lab/challenge/0a387000-cd14-426e-a68c-35302a44c205#requirements\n\t\t\t\t53 link Description: Datasets, Value: sundaebar.ai/lab/challenge/0a387000-cd14-426e-a68c-35302a44c205#data\n\t\t\t\t54 link Description: Evaluation Criteria, Value: sundaebar.ai/lab/challenge/0a387000-cd14-426e-a68c-35302a44c205#evaluation\n\t\t\t\t55 link Description: Submission, Value: sundaebar.ai/lab/challenge/0a387000-cd14-426e-a68c-35302a44c205#submission\n\t\t\t56 container challenge\n\t\t\t\t57 heading Overview, Value: 2\n\t\t\t\t\t58 text Overview\n\t\t\t\t59 text This skill teaches the agent to turn a messy bundle of data, findings, and requests into a tightly reasoned executive presentation storyline.\n\t\t\t\t60 text Given a strategic input (a prose brief, a data table, a bullet findings list, or an existing deck outline) together with an audience and a decision, a good result is a complete, structured output: a single governing thought that the evidence actually supports, a MECE argument tree with two to four branches, an action-title slide list where every headline is a complete so-what sentence, per-slide content specifications, and a self-check flag report.\n\t\t\t\t61 text The submission is a SKILL.md, with optional reference files the agent opens by name when SKILL.md directs it. Strong skills teach the agent a reasoning procedure it can apply to inputs it has never seen before: how to read footnotes before forming a claim, how to test headline flow, how to match a chart type to a claim rather than to data shape. Weak skills paste in template outlines or memorised slide structures that collapse the moment surface details change.\n\t\t\t\t62 text A skill built to memorise specific scenarios will not hold up as challenges evolve.\n\t\t\t\t63 text You submit a skill: a SKILL.md and, if you need them, reference files it points at. A fixed base agent loads it and answers all 7 scenarios with it, then returns a scored breakdown. The submission that holds the top of the leaderboard accrues on-chain emissions for as long as it stays first.\n\t\t\t\t64 text We are looking for skills that demonstrate good design and help the agent perform like a capable practitioner in this domain.\n\t\t\t65 container requirements\n\t\t\t\t66 heading Requirements (7 Tests), Value: 2\n\t\t\t\t\t67 text Requirements (7 Tests)\n\t\t\t\t68 text Your skill is tested across 5 capability types, with 7 scenarios in total. Every submission is evaluated against all 7 scenarios. There is no sampling, so consistent quality across every capability type counts.\n\t\t\t\t69 table\n\t\t\t\t\t70 row\n\t\t\t\t\t\t71 cell\n\t\t\t\t\t\t\t72 text Category\n\t\t\t\t\t\t73 cell\n\t\t\t\t\t\t\t74 text Overview\n\t\t\t\t\t\t75 cell\n\t\t\t\t\t\t\t76 text Test cases\n\t\t\t\t\t77 row\n\t\t\t\t\t\t78 cell\n\t\t\t\t\t\t\t79 text Type A: Contradiction handling\n\t\t\t\t\t\t80 cell\n\t\t\t\t\t\t\t81 text This type tests whether the agent correctly identifies that the input evidence refutes the requested conclusion rather than silently building the requested storyline. The skill must teach the agent to surface the contradiction first, offer two governing thought options (one evidence-supported, one requester-preferred), and then storyline only the evidence-supported one unless the requester explicitly redirects.\n\t\t\t\t\t\t82 cell\n\t\t\t\t\t\t\t83 text 1\n\t\t\t\t\t84 row\n\t\t\t\t\t\t85 cell\n\t\t\t\t\t\t\t86 text Type B: Existing deck repair\n\t\t\t\t\t\t87 cell\n\t\t\t\t\t\t\t88 text This type tests whether the agent can diagnose and improve a deck that already exists rather than rebuild from scratch. The skill must teach the agent to extract the implicit governing thought from the current outline, run a headline-flow test across the existing slide titles, and prefer reordering and retitling over wholesale reconstruction.\n\t\t\t\t\t\t89 cell\n\t\t\t\t\t\t\t90 text 1\n\t\t\t\t\t91 row\n\t\t\t\t\t\t92 cell\n\t\t\t\t\t\t\t93 text Type C: Missing decision clarification\n\t\t\t\t\t\t94 cell\n\t\t\t\t\t\t\t95 text This type tests whether the agent stops appropriately when the input contains no actionable decision or named audience, rather than inventing one and proceeding. The skill must teach the agent to issue a focused clarifying question and to offer two or more plausible decisions the data could plausibly serve, so the requester has concrete options to react to.\n\t\t\t\t\t\t96 cell\n\t\t\t\t\t\t\t97 text 1\n\t\t\t\t\t98 row\n\t\t\t\t\t\t99 cell\n\t\t\t\t\t\t\t100 text Type D: Standard storyline with counter-evidence\n\t\t\t\t\t\t101 cell\n\t\t\t\t\t\t\t102 text This type tests whether the agent builds a complete MECE argument tree that includes material counter-evidence rather than omitting inconvenient findings. The skill must teach the agent to construct two to four branches where each branch is a one-sentence claim citing specific input evidence, and to place counter-evidence inside the tree rather than suppressing it.\n\t\t\t\t\t\t103 cell\n\t\t\t\t\t\t\t104 text 1\n\t\t\t\t\t105 row\n\t\t\t\t\t\t106 cell\n\t\t\t\t\t\t\t107 text Type E: Standard storyline with footnote effects\n\t\t\t\t\t\t108 cell\n\t\t\t\t\t\t\t109 text This type tests whether the agent reads footnotes, unit labels, and period qualifiers before forming the governing thought, rather than skimming the headline numbers. The skill must teach the agent to compute relevant deltas, check whether notes materially change what the data shows, and produce a governing thought that reflects the footnote-adjusted picture rather than the surface reading.\n\t\t\t\t\t\t110 cell\n\t\t\t\t\t\t\t111 text 1\n\t\t\t\t112 text Surface details such as company names, metric labels, data values, and audience roles rotate across variants, while the underlying capability being tested remains the same.\n\t\t\t113 container data\n\t\t\t\t114 heading Datasets, Value: 2\n\t\t\t\t\t115 text Datasets\n\t\t\t\t116 heading Available Datasets, Value: 3\n\t\t\t\t\t117 text Available Datasets\n\t\t\t\t118 button dataset.jsonl\n\t\t\t\t119 button rubric.txt\n\t\t\t\t120 button suite.yaml\n\t\t\t\t121 text Sealed reference set\n\t\t\t\t122 text The scenarios with their reference outputs and grading notes. Validators grade against it; it is published when the challenge closes. Committed hash, sha256:\n\t\t\t\t123 text 0d14d68d0b928a0b1b87bb0a861490fd4954709af07393b69e23016ca14066b3\n\t\t\t\t124 text When you submit, we run your skill through a standardised base agent provided by sundae_bar. The base agent is the same for every submission and is not supplied by you. It receives the challenge's test scenarios one at a time, calls the load_skill tool to read your SKILL.md and the list of your reference files, opens a reference with read_reference when your SKILL.md tells it to, and follows the skill to answer each scenario.\n\t\t\t\t125 text Same base agent, same tests, only the submitted skill changes. Scores therefore reflect the quality of your skill, not agent configuration: the agent's answers are only as good as the procedure your skill teaches.\n\t\t\t\t126 text Graders then score each answer for output quality, correct skill use, and gaming behavior, and the per-scenario scores are averaged into your challenge score.\n\t\t\t\t127 container\n\t\t\t\t\t128 text What is public at open: the scenario inputs, the rubric, and the sha256 hash of the sealed reference set (the same scenarios with their reference outputs and grading notes). Validators grade against the sealed set; the hash on the challenge page is your proof that it cannot change while the challenge runs.\n\t\t\t\t129 container\n\t\t\t\t\t130 text What you see while it runs: your total, a score for every scenario and every grader, and the written notes of the gate checks (dataset derivation, skill alignment, redundancy, novelty). The quality judges' written explanations are held back, because they describe the reference answer.\n\t\t\t\t131 container\n\t\t\t\t\t132 text What is published when the challenge closes: the sealed reference set, every judge explanation, and the full validator outputs for every submission. Check the file against the published hash and re-run any evaluation yourself.\n\t\t\t\t133 container\n\t\t\t\t\t134 text What you submit: a folder or a .zip with SKILL.md at the top level and an optional references/ folder. A single SKILL.md is also accepted. All files are the submission.\n\t\t\t\t135 container\n\t\t\t\t\t136 text Limits: SKILL.md is at most 10,000 bytes and is always loaded. references/ holds up to 30 files, each at most 32,000 characters, 128,000 characters in total. Text formats only: .md, .txt, .json, .yaml, .yml and .csv. Lower-case file names using letters, digits, dots, dashes and underscores. The current values are published by the platform API at GET /api/v2/submissions/limits; the submission form reads them from there.\n\t\t\t\t137 container\n\t\t\t\t\t138 text How references are read: the agent always sees SKILL.md plus a list of your reference file names. It opens a reference only when SKILL.md tells it to, by name. Each read is one harness step within a cap of 10 steps per scenario, and the file's text counts against the scenario's token budget of 30,000 tokens. A reference the agent never opens does nothing for the score.\n\t\t\t\t139 container\n\t\t\t\t\t140 text Rejected at upload: HTML comments, images, scripts, hidden or invisible characters, nested archives, disallowed file types, and anything over a limit. Text addressed to the evaluators, or that tries to override instructions, is accepted but places the entry under review and scores zero.\n\t\t\t\t141 container\n\t\t\t\t\t142 text Important: the base-agent harness is fixed at evaluation time and its available tools are load_skill, read_reference. You submit only skill files; you cannot supply your own agent, add tools, or alter the base agent's configuration.\n\t\t\t\t143 container\n\t\t\t\t\t144 text Tip: write your skill as though you are onboarding a capable new hire who needs a procedure, not scripting a chatbot.\n\t\t\t145 container evaluation\n\t\t\t\t146 heading Evaluation Criteria, Value: 2\n\t\t\t\t\t147 text Evaluation Criteria\n\t\t\t\t148 text Each scenario is graded by an LLM judge. Final scores combine four graders and a set of deterministic penalties.\n\t\t\t\t149 table\n\t\t\t\t\t150 row\n\t\t\t\t\t\t151 cell\n\t\t\t\t\t\t\t152 text Criteria\n\t\t\t\t\t\t153 cell\n\t\t\t\t\t\t\t154 text Weight\n\t\t\t\t\t155 row\n\t\t\t\t\t\t156 cell\n\t\t\t\t\t\t\t157 text Response quality\n\t\t\t\t\t\t\t158 text An LLM judge scores each response against a task-specific quality bar for this challenge: how correct, complete, and usable the output is on its own terms, independent of which procedure produced it.\n\t\t\t\t\t\t159 cell\n\t\t\t\t\t\t\t160 text 30 %\n\t\t\t\t\t161 row\n\t\t\t\t\t\t162 cell\n\t\t\t\t\t\t\t163 text Skill use\n\t\t\t\t\t\t\t164 text An LLM judge evaluates whether the agent's output actually followed the procedures your skill teaches, rather than freelancing or pretending to use the skill.\n\t\t\t\t\t\t165 cell\n\t\t\t\t\t\t\t166 text 30 %\n\t\t\t\t\t167 row\n\t\t\t\t\t\t168 cell\n\t\t\t\t\t\t\t169 text Scenario quality\n\t\t\t\t\t\t\t170 text An LLM judge evaluates how well each output meets the per-type quality bar, independent of whether your skill was followed. A reference good output anchors the judge's sense of the quality bar.\n\t\t\t\t\t\t171 cell\n\t\t\t\t\t\t\t172 text 30 %\n\t\t\t\t\t173 row\n\t\t\t\t\t\t174 cell\n\t\t\t\t\t\t\t175 text Output originality\n\t\t\t\t\t\t\t176 text A check that your skill drives genuine per-scenario reasoning rather than emitting near-identical, templated responses. Skills that reuse one canned output across scenarios lose credit here.\n\t\t\t\t\t\t177 cell\n\t\t\t\t\t\t\t178 text 10 %\n\t\t\t\t179 text Evaluation applies deterministic penalties for moves that damage output quality, such as fabricating evidence not present in the input, writing topic-label slide titles instead of so-what sentences, or building the wrong output type for the recognised situation. Scenario scores roll up to capability-type averages, which combine into the final challenge score shown on the leaderboard.\n\t\t\t180 container submission\n\t\t\t\t181 heading Submission, Value: 2\n\t\t\t\t\t182 text Submission\n\t\t\t\t183 heading How to submit, Value: 3\n\t\t\t\t\t184 text How to submit\n\t\t\t\t185 text 1 Login or create a sundaebar.ai account\n\t\t\t\t186 text Create an account and add your Bittensor wallet ID to your profile to earn alpha rewards in the challenge.\n\t\t\t\t187 text 2 Review the requirements\n\t\t\t\t188 text Review the reference skills (good, mediocre, gaming) to understand the quality bar and calibration points.\n\t\t\t\t189 text 3 Build your skill\n\t\t\t\t190 text Draft your SKILL.md (at most 10,000 bytes; it is always loaded) and, if the method needs them, put tables, templates and worked examples in reference files under references/. Tell the agent in SKILL.md which reference to open and when, by file name. Focus on teaching procedure rather than providing templates.\n\t\t\t\t191 text 4 Test locally\n\t\t\t\t192 text Run your skill against the published scenario inputs and rubric to sanity-check output quality. The reference outputs are sealed until the challenge closes.\n\t\t\t\t193 text 5 Submit via the platform\n\t\t\t\t194 text Submit through the sundae_bar challenge portal by uploading your skill folder, a .zip of it, or a single SKILL.md. The folder has SKILL.md at the top level and an optional references/ folder beside it.\n\t\t\t\t195 text 6 Iterate\n\t\t\t\t196 text Review your scored breakdown (a score for every scenario and every grader, plus the gate checks' notes) and resubmit. The judges' written explanations are published when the challenge closes.\n\t\t\t\t197 heading Constraints, Value: 3\n\t\t\t\t\t198 text Constraints\n\t\t\t\t199 container\n\t\t\t\t\t200 text Submission format: a folder or .zip with SKILL.md (YAML frontmatter with name and description, then a Markdown body) at the top level and an optional references/ folder. A single SKILL.md is also accepted.\n\t\t\t\t201 container\n\t\t\t\t\t202 text Limits: SKILL.md at most 10,000 bytes. Up to 30 reference files, each at most 32,000 characters and 128,000 characters in total; text formats only (.md, .txt, .json, .yaml, .yml and .csv); lower-case file names.\n\t\t\t\t203 container\n\t\t\t\t\t204 text Reference reads: the agent opens a reference only when SKILL.md names it. Each read is one of 10 steps per scenario and its text counts against the scenario's 30,000-token budget.\n\t\t\t\t205 container\n\t\t\t\t\t206 text Base agent: fixed. Provided by sundae_bar.\n\t\t\t\t207 container\n\t\t\t\t\t208 text Runtime: a fixed base-agent harness loads your skill and answers all 7 scenarios per submission with it.\n\t\t\t\t209 container\n\t\t\t\t\t210 text Tests per submission: 7 (the full scenario set, no sampling).\n\t\t\t\t211 container\n\t\t\t\t\t212 text Submission limits: 7 submissions per day. A rolling challenge limit also applies.\n\t\t\t\t213 heading Rewards, Value: 2\n\t\t\t\t\t214 text Rewards\n\t\t\t\t215 text Emissions accrue on-chain to whichever submission currently holds the top of the leaderboard. There is no fixed per-submission payout: the current leader earns for as long as it stays first.\n\t\t\t\t216 text Time at the top takes all. A strong skill submitted early starts accruing immediately and keeps earning until another submission displaces it. Once you are displaced, accrual stops until you reclaim the top.\n\t\t\t\t217 text Rankings shift as others iterate. Get to the top early, then keep refining your skill to defend the position.\n\t\t\t\t218 text Every file in your submission is scanned at submission time by a skill integrity gate: your SKILL.md and every reference file, whether or not your skill points at it. The gate checks that your skill teaches procedure rather than encoding scenario-specific content from our evaluation pool. Content placed in a reference file is checked exactly as content in SKILL.md.\n\t\t\t\t219 text A skill that fails the integrity gate scores 0 for the submission. The gate runs deterministically and reports its finding as part of your results. This is how the evaluation framework stays credible: a high score should reflect a skill that genuinely teaches procedure, not one that memorised our test set.\n\t\t\t\t220 text Text addressed to the evaluators or judges, or text that tries to override instructions, anywhere in the bundle, is accepted at upload but places the entry under review and scores zero. Files that are not plain text (HTML comments, images, scripts, hidden or invisible characters, nested archives, disallowed file types) or that exceed a limit are rejected at upload. The copy and similarity checks run over the whole bundle, chunk by chunk, so padding a copied skill with extra reference files does not pass them.\n\t\t221 container\n\t\t\t222 heading Details, Value: 3\n\t\t\t\t223 text Details\n\t\t\t224 button Close details\n\t\t\t225 text Version SBC 35 Start Date 5 Nov 2026, 12:00 EST Close Date 7 Nov 2026, 12:00 EST\n\t\t\t226 heading Submissions, Value: 4\n\t\t\t\t227 text Submissions\n\t\t\t228 link Description: View all, Value: sundaebar.ai/lab/challenge/0a387000-cd14-426e-a68c-35302a44c205/submissions\n\t\t\t229 text Total Submissions 216\n\t\t\t230 button Contribute\n\t\t\t231 heading Resources, Value: 4\n\t\t\t\t232 text Resources\n\t\t\t233 text rubric.txt\n\t\t\t234 button Download rubric.txt\n\t\t\t235 text dataset.jsonl\n\t\t\t236 button Download dataset.jsonl\n\t\t\t237 text suite.yaml\n\t\t\t238 button Download suite.yaml\n\t\t\t239 button Download all\n\t42 container Notifications alt+T\n\nThe focused UI element is 1 AXWebArea SN121 Challenge | sundae_bar, URL: sundaebar.ai/lab/challenge/0a387000-cd14-426e-a68c-35302a44c205", "limits": "Public displayed rules and commitment checked; sealed reference set unavailable until close and not independently hash-validated yet. No submission or reward transaction performed." } }