Recently released benchmarks, ranked by the attention they are receiving now: GitHub stars, Hugging Face paper upvotes and Hugging Face dataset downloads over the last 30 days.

1. 01openclaw 2026.8.34released Oct 2, 202655low

   This is a gateway-only extended-stable release, which is our current equivalent to LTS. This release is OpenClaw from the end of August 2026, plus critical security updates, reliability and performance fixes, and features like new model support. The latest version of OpenClaw at the time of this release is 2026.9.7 2026.8.34 Highlights Extended-stable correctness rollup: backport 113 audit-selected fix units across upgrades, Doctor, authentication, sessions, channels, plugins, sandboxing, filesystem safety, model runtimes, and release packaging. Complete rescan: re-evaluate the full 2026.8.33 discovery range, large mixed-purpose pull requests, and the 2026.7.35 lineage instead of advancing only from the previous backport cursor. Upgrade and recovery hardening: preserve credentials, agent state, plugin inventory, transcripts, schedules, service ownership, and runtime links through upgrades, restarts, and Doctor repair. Boundary fixes: tighten remote filesystem mutation, plugin and skill scanning, browser authentication, channel reply identity, private skill ingress, and scoped runtime ownership. Changes Extended-stable release preparation: align OpenClaw, publishable plugins, native version metadata, generated channel catalogs, and npm package inventories to 2026.8.34. Fixes Plugins: Recover orphan installs without weakening ownership. Doctor: Repair keyed multi-agent rosters without ownership (#134706) Prevent device pairing migration crash on contaminated records. Doctor: Preserve per-agent memory search during upgrades. Doctor: Detect shared auth migration by provenance (#134808) Cron: Refuse stale doctor store rewrites. Gateway: Refresh model catalog after auth changes (#134361) Openai: Repair stale doctor route pins.

   GitHub stars

   391,752normalized 1.00source ↗weight 55%

   Hugging Face paper upvotes

   not observednormalized not scoredweight 30%

   Hugging Face dataset downloads, last 30 days

   not observednormalized not scoredweight 15%

   Coverage 0.55 · confidence low

2. 02OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialoguereleased Sep 29, 202651high

   We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, good replies often depend on multimodal context and can be phrased in many ways, making keyword matching unreliable for evaluation. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. Replies are judged by a large language model based on explicit scoring criteria. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.

   GitHub stars

   213normalized 0.42source ↗weight 55%

   Hugging Face paper upvotes

   153normalized 0.93source ↗weight 30%

   Hugging Face dataset downloads, last 30 days

   not observednormalized not scoredweight 15%

   Coverage 0.85 · confidence high

3. 03Rethinking Latent Visual Reasoning: Grounding Latent Reasoning in Visual Evidencereleased Sep 28, 202647high

   Latent visual reasoning (LVR) enables multimodal large language models (MLLMs) to perform intermediate computation in continuous latent tokens rather than expressing every reasoning step in words. However, unlike textual CoT, latent reasoning is not directly observable, making it difficult to supervise what latent tokens learn. In this work, we first conduct a thorough analysis of latent-token behavior and identify a latent evidence-credit gap: latent tokens respond only weakly to image perturbations that alter the correct answer. We hypothesize that this issue stems from the lack of explicit supervision during GRPO training. These findings suggest that a final-answer reward provides too little guidance on what visual evidence to preserve or how credit should be assigned across latent tokens. To bridge this gap, we propose ReaLVR, which brings visual-evidence supervision to the model's own free-running latent trajectories. ReaLVR contrasts correct and model-generated wrong answers to determine where stronger supervision is needed, and relevant and mismatched visual evidence to specify what to preserve. Across three model families, ReaLVR consistently outperforms evaluated LVR baselines, achieving the highest five-task average of 63.7% on Qwen2.5-VL-7B. Crucially, we are the first to scale visual reasoning in latent space, showing that our framework continues to deliver robust improvements at frontier model scales up to 235B. Further analyses show more question-sensitive latent-token positions, stronger alignment with relevant visual regions, and greater fixed-context dependence on the most attended latent tokens.

   GitHub stars

   47normalized 0.30source ↗weight 55%

   Hugging Face paper upvotes

   220normalized 1.00source ↗weight 30%

   Hugging Face dataset downloads, last 30 days

   not observednormalized not scoredweight 15%

   Coverage 0.85 · confidence high

4. 04UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generationreleased Oct 8, 202645high

   Dense visual text requires image generators to reproduce long strings across multiple regions with correct placement and legibility. As short-string rendering improves, evaluation must test sustained performance across more demanding scenes. We introduce UltraText Bench, a bilingual benchmark for prompt-only generation of dense visual text. It contains 432 prompts spanning 24 real-world scene categories and three difficulty levels, split equally between English and Chinese. Each human-reviewed prompt supplies exact strings for four to twelve text regions, paired with structured references for their content, placement, and visual attributes. We use the Q-Judger vision-language model to assess each image against the complete reference, reporting text fidelity, text clarity, spatial quality, and scene quality. Across 24 model configurations, these dimensions reveal different strengths: Z-Image-Turbo gains 3.81 clarity points over Z-Image-Base while losing 14.76 fidelity points under the reported settings. Performance also varies with workload; Qwen-Image-2512's English composite falls from 86.50 at L1 to 42.86 at L3. Ten participants took part in human evaluation of the automatic scores. Repository: https://github.com/LINs-lab/UltraText_Bench.

   GitHub stars

   41normalized 0.29source ↗weight 55%

   Hugging Face paper upvotes

   182normalized 0.97source ↗weight 30%

   Hugging Face dataset downloads, last 30 days

   not observednormalized not scoredweight 15%

   Coverage 0.85 · confidence high

5. 05From Traces to Agentic Worlds: Agentic Language World Models for Interactive Environment Simulationreleased Oct 5, 202644high

   Realistic environment replicas are increasingly valuable for training and evaluating LLM agents, yet the original systems may be inaccessible or impractical to reproduce. We explore agentic language world modeling: rather than rebuilding an executable environment, a world model agent serves as the environment for a task agent and supports faithful and stateful simulation. We instantiate this paradigm with Trace2Env, a learning-free framework for settings where the original system is unavailable but historical interaction traces remain accessible. Trace2Env reconstructs these traces into a reusable environment worldbook containing environment schemas, grounded evidence, and induced behavioral knowledge. At runtime, the world model agent actively consults the worldbook together with persistent episodic state to infer each action's observation and lasting state effects. Across nine environments, Trace2Env improves both next-observation fidelity and long-horizon interaction consistency over conventional prompt-based LWMs. In multi-turn interaction, task agent actions generated against Trace2Env remain valid more often when replayed in the real environment, indicating that its simulated dynamics better preserve the consequences of earlier actions across successive turns. These results establish agentic language world modeling as an alternative direction for building realistic environment replicas without reconstructing the original executable system.

   GitHub stars

   51normalized 0.31source ↗weight 55%

   Hugging Face paper upvotes

   129normalized 0.90source ↗weight 30%

   Hugging Face dataset downloads, last 30 days

   not observednormalized not scoredweight 15%

   Coverage 0.85 · confidence high

6. 06UniWAM: Unified World-Action Modelreleased Oct 1, 202644high

   Vision-language-action models benefit from the understanding and reasoning capabilities of pretrained vision-language models, but action-only supervision provides limited grounding in world dynamics. Conversely, world-action models inherit spatiotemporal priors from video generation models, yet remain limited in semantic understanding and reasoning under distribution shifts. We introduce UniWAM, a unified architecture that integrates a physical reasoner, a world generator, and an action predictor to jointly learn semantic understanding of the physical world, visual generation, and action prediction. To ensure the quality of the training data, we developed a rigorous data cleaning and annotation pipeline for both human egocentric data and robot data. To adapt the vision-language component to embodied tasks while preserving its inherited language capabilities, we represent low-level actions in natural language and introduce a pre-training recipe that assigns complementary supervision from visual question answering (VQA) data, human egocentric data, and robot demonstrations to the appropriate model components. During post-training, future visual noise augmentation reduces reliance on precise future predictions, while history-conditioned flow matching uses encoded action history to initialize action generation. Together, these designs significantly reduce denoising steps while maintaining performance. UniWAM achieves state-of-the-art (SOTA) performance across multiple evaluations, including in-distribution performance, robustness, generalization, instruction following, and long-horizon task execution. Furthermore, we uncover a log-linear scaling law of unified human-robot co-training, demonstrating the effectiveness of large-scale pre-training on a mixture of human and robot data.

   GitHub stars

   116normalized 0.37source ↗weight 55%

   Hugging Face paper upvotes

   66normalized 0.78source ↗weight 30%

   Hugging Face dataset downloads, last 30 days

   not observednormalized not scoredweight 15%

   Coverage 0.85 · confidence high

7. 07Introducing Jev in DeepEvalreleased Sep 22, 202642low

   New to deepeval ? Get started here. 🧠 Introducing Jev: System One for LLM Evals Every LLM-as-a-judge metric has had the same awkward secret: somewhere inside the thing measuring your LLM is another LLM generating text. Closed-ended verdicts ( yes / no / borderline ) were treated like writing tasks, and you paid for it in variance, latency, and cost. deepeval 4.2.2 introduces support for Jev , TypeSafe AI's System One model. Jev is not a language model. It does not generate text. You give it state and a bounded question, and it returns a typed decision with calibrated probabilities. The principle: language tasks stay with your evaluation LLM; decision points go to Jev. What changes when Jev is on Less flaky — verdicts become probabilities mapped through fixed thresholds. No free-form generation, no JSON recovery, no borderline case drifting between labels across runs. Cheaper — $0.042 per million input tokens, no output token charge. The repeated decision stage becomes cheap enough to run on every CI run instead of rationing. Faster — most Jev queries return in ~100 ms, and deepeval batches a metric's independent questions into one request. Where it plugs in | Metric | LLM still handles | Jev now handles | | --- | --- | --- | | FaithfulnessMetric (and other QAG metrics) | Extracting claims and truths | One Noul ( P(yes) ) per claim | | GEval | Generating evaluation steps | Noul in strict / rubric-free mode, Score when a rubric is provided | | DAGMetric | Task node text | Noul for binary nodes, Choice for non-binary nodes | | Classifiers | — | Choice over the closed label set | Score calculations are unchanged. Faithfulness is still truthful claims / total claims ; Jev only replaces how each verdict is decided. Try it (opt-in, experimental) Nothing changes for existing users. Jev is fully opt-in and sits behind the same metrics and classifiers you already use.

   GitHub stars

   18,765normalized 0.76source ↗weight 55%

   Hugging Face paper upvotes

   not observednormalized not scoredweight 30%

   Hugging Face dataset downloads, last 30 days

   not observednormalized not scoredweight 15%

   Coverage 0.55 · confidence low

8. 08Does Learning Protein Folding Generalize to Broader Reasoning?released Sep 30, 202642high

   Large language models rely heavily on human text, which often conveys surface answers rather than the spatial and structural logic behind them. Protein folding is a natural testbed, because one solved structure yields thousands of exactly checkable spatial and topological statements. We ask: can learning to fold proteins teach general models reusable reasoning capabilities? To answer this, we build FoldingCorpus, a protein-derived question-answer dataset, and Fold2Reason, a recipe that post-trains on it through two complementary signals: discrete structural answers predicted via the model's native language head, and continuous 3D geometry decoded from the same shared representations. On FoldBench, Fold2Reason achieves structure prediction scores 2.7 to 3.5 times those of Qwen3.5-9B. Beyond protein structure prediction, it improves performance on all 10 benchmarks spanning spatial, graph, scientific, and general reasoning, raising macro-average accuracy from 45.09% to 48.33% (+3.23 pp), with positive gains on all 10 benchmarks, while matched controls built from random, synthetic, and shuffled structure yield substantially smaller or negative gains. Our work shows that non-linguistic, structure-dense scientific data can systematically improve broad reasoning in language models, making a solved scientific problem a practical source of post-training supervision.

   GitHub stars

   78normalized 0.34source ↗weight 55%

   Hugging Face paper upvotes

   62normalized 0.77source ↗weight 30%

   Hugging Face dataset downloads, last 30 days

   not observednormalized not scoredweight 15%

   Coverage 0.85 · confidence high

9. 09Sharpening Tax in Post-Trainingreleased Oct 2, 202640high

   An emerging hypothesis about reinforcement learning (RL) post-training of large language models (LLMs) is that it merely sharpens existing behaviors of a base model, improving single-shot accuracy at the cost of solution coverage. Although this trade-off has been observed in math and coding tasks, it need not extend to agentic tasks, where multi-turn tool use and interaction may require capabilities newly acquired during post-training. Our surprising finding is that pre-trained LLMs, equipped with a light inference harness, can serve as capable agents. Despite far lower accuracy (pass@1), they often surpass their post-trained counterparts in solution coverage (pass@K) given a sufficient test-time budget. We further analyze the underlying mechanism and show that post-training pushes tasks toward two extremes, always solved or never solved, and thereby improves sampling efficiency and consistency at the cost of solution coverage. To measure this cost, we propose Sharpening Tax, a diagnostic metric that quantifies the loss in test-time scalability after post-training. Across 14 base/post-trained model pairs from four families and three agentic benchmarks (42 cases in total), the tax is prevalent in most settings, can be estimated from a few rollouts, and correlates well with other metrics. Finally, we present posterior-tempered group sampling (PTGS), a simple plug-and-play Bayesian sampler that adapts the sampling temperature per prompt to its estimated difficulty. Applied during RL training in two agentic environments, PTGS pays a smaller tax than the fixed-temperature baseline, solving more tasks under repeated sampling while also improving single-shot accuracy.

   GitHub stars

   26normalized 0.26source ↗weight 55%

   Hugging Face paper upvotes

   111normalized 0.87source ↗weight 30%

   Hugging Face dataset downloads, last 30 days

   not observednormalized not scoredweight 15%

   Coverage 0.85 · confidence high

10. 10Reasoning-Informed Visual Editingreleased Oct 9, 202639high

    Large Multi-modality Models (LMMs) have made significant progress in visual understanding and generation, but still face challenges in visual editing, particularly in following complex instructions, preserving appearance consistency, and supporting flexible input formats. To study this gap, we introduce RISEBench, the first benchmark for evaluating Reasoning-Informed viSual Editing (RISE), and extend it to RISEBench++, a more comprehensive and fine-grained benchmark for this emerging task. RISEBench++ extends the taxonomy into a hierarchical scheme spanning six reasoning dimensions: Temporal, Causal, Spatial, Logical, and Counterfactual Reasoning, together with Hybrid Reasoning integrating multiple reasoning types across multi-turn edits. These dimensions are further decomposed into 12 subcategories and 65 fine-grained task types. We expand input formats to include multi-image conditioning and scale the benchmark to 1000 human-annotated test cases, released in English and Chinese. We also improve our evaluation framework, assessing Instruction Reasoning, Appearance Consistency, and Visual Plausibility with human judges and an LMM-as-a-judge approach for more reliable and calibrated judgements. Beyond benchmarking, we introduce RISE-Agent, a training-free agentic framework integrating reasoning-driven planning, tool-augmented execution, and verifier-guided refinement, outperforming most strong existing approaches across diverse RISE tasks. We evaluate 58 visual editing approaches, including 34 open-source models, 19 closed-source models, and 5 agentic methods. The results reveal substantial challenges in reasoning-based visual editing, with even the strongest evaluated approach, GPT-Image-2.5 Sunburst, achieving only 56.6% accuracy. RISEBench++ highlights the limitations of contemporary editing models, provides insights, and indicates future directions for reasoning-aware visual editing.

    GitHub stars

    162normalized 0.40source ↗weight 55%

    Hugging Face paper upvotes

    21normalized 0.57source ↗weight 30%

    Hugging Face dataset downloads, last 30 days

    not observednormalized not scoredweight 15%

    Coverage 0.85 · confidence high

378 of 1,052 releases in this window are ranked; the rest are listed with limited signals. Data through Oct 12, 2026 · average signal coverage 32%.