Benchmark Radar™
RSS Star —

Library

120 of 3,827 benchmarks

The B2B SaaS AI Visibility Benchmark 2026

Evaluates which of 94 B2B SaaS brands are named by ChatGPT, Claude, Perplexity, Google Gemini and Google AI Overviews on 50 real buying questions across five G2 categories.

Oct 13, 2026 · Claire Radar · Code

Decision Layer Benchmark

A token-cost model for an AI agent's decision layer: estimating monthly spend for five decision questions across naive, cached, and tuned OpenAI, Anthropic, and Google baselines versus Jev.

Oct 12, 2026 · Claire Radar · Code

Graduate Admissions Agent Benchmark

Evaluates Chinese-language agent products on graduate admissions advising, covering recommendation, planning, and two-stage information-update tasks with per-task rubrics.

Abstract Reasoning & Planning · Multilingual & Translation (provisional) · Oct 12, 2026 · Claire Radar · Code · Data

RoB 2 AI Benchmark

Unified frozen benchmark for AI risk-of-bias assessment using Cochrane RoB 2, built from three public expert-annotated sources, scored by agreement with human judgments at signalling, domain, and overall levels.

Health & Medicine · Oct 12, 2026 · Claire Radar · Code · ★ 0 GitHub

CausalCascade

A simulator-grounded benchmark for assessing causal cascade generation in image-to-video models through event-rich scenes, motion metrics, and diagnostic question answering.

World Models & Physical Understanding · Image & Video Generation · Oct 11, 2026 · Claire Radar · Paper · Code · ★ 0 GitHub

EmoTrace-Bench

EmoTrace-Bench evaluates text-to-video generation on emotional expression by controlling prompt cues and intensity, and scoring videos on target emotion, perceived intensity, and support across ten audiovisual cue channels.

Image & Video Generation · Chat, Writing & Preference · Oct 11, 2026 · Claire Radar · Paper · Code · ★ 0 GitHub

MedHarnessBench

Evaluates healthcare agents on 342 executable tasks spanning FHIR/EHR operations, healthcare administration interfaces, and multimodal clinical tools, measuring both outcome success and seven process-quality dimensions.

Health & Medicine · Professional & Enterprise Work · Oct 11, 2026 · University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · Claire Radar · Paper · Code · ★ 1 GitHub

NepPlural

NepPlural is a dataset and evaluation suite for modeling societal personas in Nepali migration discourse, with human gold annotations, LLM protocol audit, encoder baselines, and human-judged steerability.

Multilingual & Translation · Chat, Writing & Preference · Oct 11, 2026 · Claire Radar · Paper · Code · ★ 0 GitHub

PhySimCode

PhySimCode evaluates video-to-code generation for physics simulations, requiring models to infer physical laws and parameters from videos and produce runnable Python code rendered and scored against the input video.

Code Generation · Video Understanding · World Models & Physical Understanding · Oct 11, 2026 · Carnegie Mellon University · Claire Radar · Paper · Code · Data · ★ 0 GitHub

RoboCombine

A benchmark for skill recombination in embodied manipulation without combined skill training trajectories, consisting of 19 compositional tasks with evaluation tracks for different skill execution paradigms.

Robotics & Embodied AI · Oct 11, 2026 · Shanghai Jiao Tong University · Claire Radar · Paper · Code · ★ 1 GitHub

DocumentVision

Evaluates OCR pipelines and vision-language models on dense tables, key-value extraction, charts, and layout grounding across high-resolution enterprise document images.

Document Understanding & OCR · Image & Visual Reasoning · Oct 10, 2026 · Claire Radar · Code · ★ 0 GitHub

Reg Affairs Bench

Evaluates AI systems on regulatory affairs deliverables from drugs, biologics, and medical devices using scenarios and dossier documents, scored against task-specific rubrics by a jury of LLM judges.

Health & Medicine · Professional & Enterprise Work · Oct 10, 2026 · Claire Radar · Data

A Real-World CAD Segmentation Benchmark

CAD segmentation datasets are predominantly synthetic or automatically labelled, leaving it unclear how well existing methods transfer to real-world engineering tasks.

Engineering & Hardware Design · Image & Visual Reasoning · Oct 9, 2026 · Volkswagen Group (Germany) · Osnabrück University · Claire Radar · Paper

APEX-Harm

Professional agents increasingly work through command-line interfaces and multiple tools over shared workspaces containing files, messages, and business applications.

Safety & Alignment · Professional & Enterprise Work · Oct 9, 2026 · King Abdullah University of Science and Technology · University of Oxford · Claire Radar · Paper

AgentSWE

As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows.

Agentic Coding & SWE · Oct 9, 2026 · Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence · Institute of Automation · Claire Radar · Paper · Code · ★ 35 GitHub

BASIS-Bench

Recent speech LLMs can leverage rich paralinguistic cues, enabling more complex and personalized interactions.

Speech & Audio · Oct 9, 2026 · Université Paris-Saclay · Claire Radar · Paper

CRBugBench

With the advances of coding agents, software developers are spending more time reviewing code than writing code.

Agentic Coding & SWE · Oct 9, 2026 · Microsoft (United States) · Claire Radar · Paper

DRYLAB-Bench

As large language models (LLMs) acquire advanced biological design capabilities, they offer substantial research benefits but simultaneously introduce severe dual-use risks, such as engineering enhanced pathogens or drug-resistant bacteria.

Safety & Alignment · Science · Oct 9, 2026 · Shanghai Artificial Intelligence Laboratory · Fudan University · Claire Radar · Paper · Code · ★ 1 GitHub

EmbodiedCLBench

Continual learning requires models to accumulate past experience and transfer it to new tasks.

Robotics & Embodied AI · Oct 9, 2026 · Peking University · Beijing Institute for General Artificial Intelligence · University of Hong Kong · Claire Radar · Paper · ★ 8 GitHub

GeoSpatialBench

Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world models.

Spatial & 3D Reasoning · Image & Visual Reasoning · Oct 9, 2026 · The Hong Kong University of Science and Technology (Guangzhou) · Claire Radar · Paper

MAGMA-Bench

Long-horizon embodied agents must maintain task-relevant information, coordinate heterogeneous robots, and adapt their decisions as execution unfolds.

Robotics & Embodied AI · Oct 9, 2026 · Claire Radar · Paper

MVGraph

We introduce MVGraph, a benchmark for evaluating large vision-language models (LVLMs) on multi-graph reasoning where the evidence required by a query is distributed across multiple interrelated visual graph views.

Image & Visual Reasoning · Oct 9, 2026 · Harbin Institute of Technology · Peng Cheng Laboratory · Claire Radar · Paper · Code · ★ 1 GitHub

MathVis

This paper introduces MathVis, a benchmark designed to evaluate LLMs' ability to visualize frontier mathematical problems and perform diagram-assisted reasoning.

Image & Visual Reasoning · Math Reasoning · Oct 9, 2026 · Tsinghua University · City University of Hong Kong · Cornell University · Claire Radar · Paper

MisVisAgentBench

Web agents increasingly browse, interpret, and act on web content on behalf of users.

Computer Use & GUI Agents · Image & Visual Reasoning · Oct 9, 2026 · Claire Radar · Paper

PPG-Bench

Photoplethysmography (PPG) is increasingly recorded by wearable and clinical devices at enormous scale, and a growing number of PPG foundation models (FMs) have been developed to compute reusable representations for various downstream task…

Health & Medicine · Science · Oct 9, 2026 · Claire Radar · Paper · Code

ReqClarifyBench

As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want.

Agentic Coding & SWE · Instruction Following · Oct 9, 2026 · Chinese University of Hong Kong, Shenzhen · Claire Radar · Paper

SCCGBench

LLM-generated service-call code is increasingly important for service computing, where developers rely on large language models (LLMs) to compose services and invoke third-party services.

Code Generation · Tool Use & Function Calling · Oct 9, 2026 · Shandong University of Technology · Claire Radar · Paper

SWE-DeceptionBench

ArXiv paper introducing a benchmark for deceptive behaviors of LLMs across the software development lifecycle, with abstract stating tasks and results, but no public artifact available.

Safety & Alignment · Agentic Coding & SWE · Oct 9, 2026 · Tsinghua University · Fudan University · Claire Radar · Paper

SmartSalesBench

We present SmartSalesBench, the first multi-turn conversational benchmark explicitly designed to evaluate large language model (LLM) sales agents against psychologically grounded, resistant buyer personas.

Chat, Writing & Preference · Multi-Agent & Games · Oct 9, 2026 · Samsung (India) · Claire Radar · Paper

SqueezeBench

AI systems are making rapid progress in mathematics, but this progress is hard to measure.

Math Reasoning · Oct 9, 2026 · Computer Algorithms for Medicine · Complexity Science Hub · Claire Radar · Paper

TextVBench

ArXiv paper introducing a benchmark for text rendering and preservation in video generation, with abstract stating two tracks and scoring metrics, but no public artifact available.

Image & Video Generation · Oct 9, 2026 · Claire Radar · Paper

AgentHorizon

Computer-use agents are capable of completing complex tasks, increasing the use of automatic judges to determine success, either for training or for evaluation without human involvement.

Computer Use & GUI Agents · Oct 8, 2026 · Mila - Quebec Artificial Intelligence Institute · McGill University · École de Technologie Supérieure · Claire Radar · Paper

AgenticBBO-Bench

Black-box optimization (BBO) arises in many scientific and engineering problems where objective evaluations are expensive and limited.

AI R&D & ML Engineering · Science · Oct 8, 2026 · Nanjing University · Claire Radar · Paper · Code · ★ 1 GitHub

DataSense-Bench

As claims about recursive self-improvement (RSI) and artificial general intelligence (AGI) proliferate, we ask a simple question: do frontier AI models have a sense of data, i.e., can they reliably select the right data for training?

AI R&D & ML Engineering · Oct 8, 2026 · Eindhoven University of Technology · University of Surrey · Claire Radar · Paper · Code · ★ 0 GitHub

GameCommBench

Game commentary is an open-ended generation task requiring multimodal perception, strategic reasoning, and contextual knowledge.

Chat, Writing & Preference · Video Understanding · Oct 8, 2026 · Peking University · South China University of Technology · Claire Radar · Paper

LIVIN

Realistic household simulation must capture not only diverse environments but also the lived-in object arrangements and spatial constraints that shape robot motion and interaction.

Robotics & Embodied AI · Oct 8, 2026 · Claire Radar · Paper

MedBench

Large-scale construction of medical vision-language model (VLM) benchmarks is increasingly feasible with richly annotated imaging datasets and large language models (LLMs), yet existing automation largely focuses on generating evaluation i…

Health & Medicine · Image & Visual Reasoning · Oct 8, 2026 · Beijing University of Posts and Telecommunications · West China Medical Center of Sichuan University · West China Hospital of Sichuan University · Claire Radar · Paper

Mine Odyssey

Advances in foundation models are driving efforts to introduce agents to assist people in the physical world.

Robotics & Embodied AI · Oct 8, 2026 · Claire Radar · Paper

MiniVer-V

A core challenge in short-video fact-checking is identifying which evidence is sufficient to support a verification conclusion.

Video Understanding · Factuality & Hallucination · Oct 8, 2026 · The University of Sydney · Claire Radar · Paper

ModeBench

A language model (LM) can usually answer the same question in more than one way, but reinforcement learning with verifiable rewards (RLVR) is indifferent to which correct answer a model produces.

Instruction Following · Oct 8, 2026 · Princeton University · Claire Radar · Paper

MultiWorldBench

Multiplayer world models must ensure that independently controlled views remain consistent with one shared and persistent world.

World Models & Physical Understanding · Oct 8, 2026 · Claire Radar · Paper

OmniCapBench

Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits.

Speech & Audio · Video Understanding · Image & Visual Reasoning · Oct 8, 2026 · Claire Radar · Paper

PathLang

Pathology vision-language models (VLMs) have shown strong visual perception ability, but their robustness in the language domain remains poorly characterized.

Health & Medicine · Image & Visual Reasoning · Oct 8, 2026 · New York University · Columbia University · Cornell University · Claire Radar · Paper

PolyCodeEval

As large language models increasingly move toward repository-level software engineering, existing code-generation benchmarks remain fragmented across language coverage, task granularity, and evaluation protocols, impeding systematic compar…

Code Generation · Agentic Coding & SWE · Oct 8, 2026 · Tianjin University · Claire Radar · Paper

Pumpire

We present Pumpire, a unified benchmark for evaluating metric point-pair distance estimation capability of both image- and video-level 3D foundation models, with or without depth priors.

Spatial & 3D Reasoning · Image & Visual Reasoning · Video Understanding · Oct 8, 2026 · Claire Radar · Paper · ★ 5 GitHub

SWE-Journey

Coding assistants such as Claude Code and Codex have become a major application of LLM agents, yet existing benchmarks remain far from real-world use, particularly in task horizon and interaction length.

Agentic Coding & SWE · Oct 8, 2026 · Tencent (China) · Claire Radar · Paper

SpaceCast-Bench

Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input.

Spatial & 3D Reasoning · Image & Visual Reasoning · Oct 8, 2026 · Zhejiang University · Claire Radar · Paper · Code · Data · ★ 8 GitHub

TypedBench

System One models output calibrated probabilities over typed answers such as categorical choices, ordinal levels, or binary outcomes, via a non-generative interface.

Instruction Following · Oct 8, 2026 · University of Kaiserslautern · German Research Centre for Artificial Intelligence · Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau · Claire Radar · Paper

VenusRX-Bench

Existing reaction models primarily learn molecular transformations, whereas enzy- matic reactions depend jointly on molecular structure and catalytic function.

Science · Oct 8, 2026 · Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · The University of Sydney · Claire Radar · Paper

WOVEN

Multimodal large language models (MLLMs) struggle with spatial, embodied, physical, and temporal reasoning.

Spatial & 3D Reasoning · World Models & Physical Understanding · Oct 8, 2026 · Claire Radar · Paper

Workerville

LLM-based agents now interact with their environments continuously, shaped by such organizational channels as user instructions, peer messages, and long-term memory.

Professional & Enterprise Work · Agent Memory & Personalization · Oct 8, 2026 · New York University Abu Dhabi · New York University · Mohamed bin Zayed University of Artificial Intelligence · Claire Radar · Paper

AdSpark

Product-centric advertisement video generation aims to create promotional videos that preserve fine-grained product identity while presenting selling points through coherent multi-shot narratives.

Image & Video Generation · Oct 7, 2026 · Claire Radar · Paper

AgentTime

An essential control of AI agents is their ability to manage runtime.

Tool Use & Function Calling · Oct 7, 2026 · Max Planck Institute for Intelligent Systems · Tübingen AI Center · ELLIS Institute Tübingen · Claire Radar · Paper · Code · ★ 6 GitHub

AirGroundVLN

Goal-oriented Vision-and-Language Navigation (VLN) requires agents to locate and reach targets described in natural language without prescribed routes.

Robotics & Embodied AI · Spatial & 3D Reasoning · Oct 7, 2026 · Claire Radar · Paper

CROSS-Bench

Vision-language models increasingly reason through crops, regions, and tool-produced observations.

Image & Visual Reasoning · Tool Use & Function Calling · Oct 7, 2026 · Chinese Academy of Sciences · Technology and Engineering Center for Space Utilization · University of Chinese Academy of Sciences · Claire Radar · Paper

DUDA-Bench

Urban diagnosis integrates heterogeneous observations to identify urban problems, localize affected areas, and investigate contributing factors, informing evidence-based urban planning and management.

Image & Visual Reasoning · Oct 7, 2026 · The Hong Kong University of Science and Technology (Guangzhou) · Claire Radar · Paper

InsClaimBench

Recent advances in reasoning-oriented large language models (LLMs) have motivated increasing evaluation of their ability to perform professional decision tasks.

Professional & Enterprise Work · Oct 7, 2026 · Fudan University · Claire Radar · Paper

InstanceBench

Referring Expression Segmentation (RES) links natural-language descriptions to pixel-level object masks.

Image & Visual Reasoning · Oct 7, 2026 · The University of Sydney · Shanghai Artificial Intelligence Laboratory · Claire Radar · Paper

LiveMACEBench

Evaluating agents by outcomes alone can obscure the capabilities that produce them.

Multi-Agent & Games · Oct 7, 2026 · National University of Singapore · Fudan University · The University of Sydney · Claire Radar · Paper

OcuBench

Eyeglass reflection removal is important across smartphone imaging, video conferencing, and other face-centric visual applications.

Image & Video Generation · Oct 7, 2026 · Claire Radar · Paper

PHRBench

Hallucinated information can propagate through multi-stage LLM systems and become part of the context for subsequent reasoning.

Factuality & Hallucination · Long Context · Oct 7, 2026 · National University of Singapore · Tsinghua University · Johns Hopkins University · Claire Radar · Paper

PatchBench

An LLM safety patch can pass a benchmark while still being a poor repair.

Cybersecurity & CTF · Safety & Alignment · Oct 7, 2026 · Centre National de la Recherche Scientifique · Laboratoire d'Informatique de l'École Polytechnique · Ministère des Armées · Claire Radar · Paper

RH-Detect

Reward hacking, where a model exploits an evaluation signal without completing the intended task, threatens the reliability of deployed language model systems.

Safety & Alignment · Agent Security & Prompt Injection · Oct 7, 2026 · University of Toronto · Vector Institute · Claire Radar · Paper

RT-Safe

Rapid progress in AI agents has brought growing attention to agent safety, with extensive evaluation focused on digital environments.

Agent Security & Prompt Injection · Safety & Alignment · Oct 7, 2026 · Claire Radar · Paper

RoboQuest

Recent advances in multimodal foundation models have made them capable generalist physical agents for a range of manipulation tasks.

Robotics & Embodied AI · Spatial & 3D Reasoning · Oct 7, 2026 · Claire Radar · Paper · ★ 3 GitHub

RobotWorld

General-purpose agents increasingly write code, use tools, and complete complex digital tasks, raising the question of how far these capabilities carry into the physical world.

Robotics & Embodied AI · World Models & Physical Understanding · Oct 7, 2026 · Claire Radar · Paper · ★ 39 GitHub

SAVU-BENCH

Spatial audio-visual understanding requires models to recognize not only what is present, but also where events occur and how they relate across modalities.

Speech & Audio · Image & Visual Reasoning · Spatial & 3D Reasoning · Oct 7, 2026 · Claire Radar · Paper

SoccerNet-FoulRet

Refereeing decisions in professional soccer remain inconsistent because referees cannot easily compare a contentious foul against similar past cases.

Video Understanding · Deep Research & Search · Oct 7, 2026 · University of Liège · King Abdullah University of Science and Technology · Claire Radar · Paper · Code · ★ 0 GitHub

StoreBench

Reinforcement learning environments are now a primary lever for improving large language model (LLM) capabilities in post-training, yet most agentic benchmarks remain static: the world moves only when the agent acts, the reward is a termin…

Agentic Coding & SWE · Tool Use & Function Calling · Oct 7, 2026 · Claire Radar · Paper

TaoD2C-Bench

A key challenge for multimodal large language models (MLLMs) is moving beyond visual recognition to constraint-aware cross-modal reasoning.

Image & Visual Reasoning · Oct 7, 2026 · Zhejiang University · Alibaba Group (China) · Claire Radar · Paper

BioStudyBench

We evaluate whether AI agents can match the reported findings of published biomedical studies using public data.

Science · Oct 6, 2026 · University of Waterloo · Claire Radar · Paper

CheckerBench

Static-analysis checker synthesis requires agents to interpret a defect specification, inspect a repository, implement analyzer-specific logic, and refine the checker through repeated compilation and analysis feedback.

Agentic Coding & SWE · Cybersecurity & CTF · Oct 6, 2026 · Peking University · Shanghai Innovation Institute · Shanghai Jiao Tong University · Claire Radar · Paper · ★ 2 GitHub

DSV-Mem

Conversational MLLM agents are increasingly expected to assist in professional workflows, from AI research and engineering design to product management and business operations.

Agent Memory & Personalization · Oct 6, 2026 · University of Southern California · Claire Radar · Paper

DecepEval

As large language model (LLM) agents become increasingly autonomous, they may pursue task performance through deception, raising concerns about their reliable deployment.

Safety & Alignment · Oct 6, 2026 · Xi'an Jiaotong University · University of Virginia · Griffith University · Claire Radar · Paper · ★ 5 GitHub

HarnessSecurity-Bench

Coding agent harnesses mediate tool use and authorize actions, yet their security mechanisms and runtime effects remain incompletely characterized.

Agent Security & Prompt Injection · Agentic Coding & SWE · Oct 6, 2026 · Sun Yat-sen University · Peng Cheng Laboratory · East China Normal University · Claire Radar · Paper

Learn2Play Bench

Learning from experience is essential for LLM agents to adapt to unfamiliar and dynmaic environments.

Agent Skills & Self-Improvement · Oct 6, 2026 · National University of Singapore · Claire Radar · Paper · ★ 12 GitHub

OpenWAM

World-action models (WAMs) couple future prediction with robot control, yet existing systems often vary the video backbone, interaction structure, supervision, and inference procedure simultaneously, making their design choices difficult t…

World Models & Physical Understanding · Robotics & Embodied AI · Oct 6, 2026 · Stanford University · Claire Radar · Paper · Code · ★ 198 GitHub

ParanoiaEval

As coding agents increasingly undertake real-world work autonomously, judging whether their risk treatments are warranted has become important.

Agent Security & Prompt Injection · Oct 6, 2026 · New York University Abu Dhabi · New York University · Hong Kong Polytechnic University · Claire Radar · Paper

RAG-PIBench

Retrieval-Augmented Generation (RAG) systems are vulnerable to prompt-injection attacks embedded in retrieved content.

Agent Security & Prompt Injection · Deep Research & Search · Oct 6, 2026 · Birzeit University · University of Central Florida · Claire Radar · Paper

RenderBench

Modern video models can generate realistic videos from real appearance references and proxy renders that specify scene structure, viewpoint changes, and motion.

Image & Video Generation · World Models & Physical Understanding · Oct 6, 2026 · Claire Radar · Paper

ST-Bench

The rapid progress of LLM-based multi-agent systems (MAS) has shown that they largely outperform single agents on coding, math, and QA tasks, where executable tests provide a binary success signal.

Multi-Agent & Games · Oct 6, 2026 · Rutgers, The State University of New Jersey · University of Pittsburgh · University of Minnesota · Claire Radar · Paper

SpeedrunBench

Frontier LLM agents have been shown to be capable of solving increasingly complex tasks for which humans have measurable solutions.

Agentic Coding & SWE · Code Generation · Oct 6, 2026 · Claire Radar · Paper

WorldSolver

LLM-based agents are increasingly advancing scientific and engineering problem solving, with physics simulation emerging as a challenging yet practical testbed for reproducing complex physical phenomena with application in embodied AI, gam…

World Models & Physical Understanding · Science · Oct 6, 2026 · Huawei Technologies (China) · Claire Radar · Paper · Code · ★ 0 GitHub

sk-bench

Multilingual LLM benchmarks omit Slovak, a morphologically rich West Slavic language of five million speakers, or cover it only by machine translation.

Multilingual & Translation · Oct 6, 2026 · Claire Radar · Paper · Code · ★ 0 GitHub

2d-fet-bench

Field-effect transistor (FET) layouts on exfoliated two-dimensional flakes are typically drawn by hand for each flake, placing contacts and gates to match its position and outline in optical micrographs.

Engineering & Hardware Design · Image & Visual Reasoning · Oct 5, 2026 · University of Minnesota · Claire Radar · Paper

ACG-Bench

Generalization in multi-arm collaboration can be studied as composing familiar atomic skills in new ways across arms.

Robotics & Embodied AI · Multi-Agent & Games · Oct 5, 2026 · Dalian University of Technology · Shanghai Jiao Tong University · Chinese University of Hong Kong, Shenzhen · Claire Radar · Paper

AgentMonBench

As agents take on long-horizon tasks, users shift from making individual decisions to overseeing autonomous execution.

Agent Skills & Self-Improvement · Oct 5, 2026 · Renmin University of China · Kuaishou (China) · Claire Radar · Paper · ★ 2 GitHub

BazaarBench

In decentralized consumer-to-consumer (C2C) marketplaces, people list goods, negotiate with strangers, and rate one another, so trust rests on reputation.

Multi-Agent & Games · Oct 5, 2026 · King's College London · Ministry of the Interior and Administrative Reconstruction · University of Oxford · Claire Radar · Paper · Code · ★ 4 GitHub

BrainTRACE

Brain MRI interpretation is a longitudinal clinical reasoning problem: radiologists compare serial studies, integrate information across MRI sequences, localize findings within volumetric anatomy, and translate this evidence into report-gr…

Health & Medicine · Image & Visual Reasoning · Oct 5, 2026 · The University of Texas Health Science Center at Houston · University of Alabama at Birmingham · University of Pittsburgh · Claire Radar · Paper

CavaBench

AI voice assistants often use Automatic Speech Recognition (ASR) with LLM-based reasoning, yet existing systems struggle with regional British accents, including Scottish, Irish, and Welsh accents, since most ASR models are trained predomi…

Speech & Audio · Oct 5, 2026 · Claire Radar · Paper

EMG-FM-Bench

Foundation models (FMs) are increasingly being developed for general time series and physiological signals, yet their transferability to downstream physiological tasks remains poorly understood.

Health & Medicine · Oct 5, 2026 · University of Georgia · University of Oklahoma · University of Massachusetts Amherst · Claire Radar · Paper

FairRSFM

Remote sensing foundation models (RSFMs) are commonly evaluated using aggregate metrics, which can hide systematic performance disparities across ecological regions.

Safety & Alignment · Oct 5, 2026 · Indian Institute of Science Education and Research, Bhopal · Indian Institute of Technology Bombay · Claire Radar · Paper · Code · ★ 1 GitHub

Logbook

Audio benchmarks are built around short, pre-segmented clips, limiting model design to brief inputs or fixed vocabularies.

Speech & Audio · Oct 5, 2026 · Meta (United States) · The University of Texas at Austin · Claire Radar · Paper · Code · ★ 1 GitHub

OR-Bench

Vision-language models (VLMs) can detect that an object has rotated across views, but cannot reliably tell by how much.

Spatial & 3D Reasoning · Image & Visual Reasoning · Oct 5, 2026 · The University of Queensland · Westlake University · Claire Radar · Paper

RMD-Bench

LLM-driven evolution enables iterative model development, but two practical goals remain underexplored: finding model designs that transfer across related tasks and sustaining improvement when training is expensive.

AI R&D & ML Engineering · Oct 5, 2026 · Claire Radar · Paper

TasteVal

We introduce TasteVal, a benchmark to evaluate the experimental research taste of frontier models.

AI R&D & ML Engineering · Oct 5, 2026 · Claire Radar · Paper

AgentDoxx

As Large Language Models (LLMs) gain tool use capabilities such as web search, they can retrieve and cross-reference public information, creating privacy risks beyond memorization.

Agent Security & Prompt Injection · Oct 4, 2026 · Northeastern University · Claire Radar · Paper

ColdDDI

Cold-start drug-drug interaction (DDI) prediction tests whether models can identify clinically significant interactions for drugs without training-time interaction history.

Health & Medicine · Science · Oct 4, 2026 · Southwest University · Baylor University · Hefei University of Technology · Claire Radar · Paper · Code · ★ 20 GitHub

DelegationBench

AI agents that send emails, edit files, and make purchases must decide when to act on their own and when to check with the user first.

Tool Use & Function Calling · Agent Security & Prompt Injection · Oct 4, 2026 · Claire Radar · Paper · Code · ★ 0 GitHub

EMBER-Bench

Lifelong physical agents must reason over extended interactions where past events continue to shape the world long after they disappear from view.

Robotics & Embodied AI · World Models & Physical Understanding · Oct 4, 2026 · Beijing Academy of Artificial Intelligence · Tsinghua University · University of Hong Kong · Claire Radar · Paper

GNN-CB

Large language models (LLMs) have demonstrated strong performance on coding and reasoning benchmarks; however, their ability to solve graph-structured machine learning problems remains largely unexplored.

Code Generation · AI R&D & ML Engineering · Oct 4, 2026 · Claire Radar · Paper

MLE-InsightBench

Agentic machine learning engineering (MLE) is an emerging AI4SE application for complex ML tasks and a step toward recursive self-improvement of AI systems.

AI R&D & ML Engineering · Oct 4, 2026 · University of Waterloo · Claire Radar · Paper

MMPostTrainBench

Autonomous research seeks sustained model improvements through iterative experimentation and feedback.

AI R&D & ML Engineering · Oct 4, 2026 · University of Science and Technology of China · Alibaba Group (China) · Tsinghua University · Claire Radar · Paper · Code · ★ 0 GitHub

PortraitAes-Bench

Portrait aesthetic assessment assigns comparable scores according to how effectively human-centered images fulfill their photographic intent.

Image & Visual Reasoning · Chat, Writing & Preference · Oct 4, 2026 · Claire Radar · Paper

RMMBench

Although the advancement of vision-language models (VLMs) has endowed robots with enhanced environmental understanding and task reasoning, a comprehensive evaluation methodology is important to advance the integration of VLMs in robotic na…

Robotics & Embodied AI · Spatial & 3D Reasoning · Oct 4, 2026 · Claire Radar · Paper

UndoBench

Tool-using AI agents are increasingly deployed across enterprise software systems, yet widely used benchmarks primarily evaluate nominal task completion, conflating baseline planning competence with operational fault recovery.

Tool Use & Function Calling · Agent Security & Prompt Injection · Oct 4, 2026 · Claire Radar · Paper · Code · ★ 2 GitHub

VHDL-REPOBENCH

Large Language Models (LLMs) are increasingly applied in hardware design automation, demonstrating strong potential in generating and understanding hardware description languages.

Engineering & Hardware Design · Code Generation · Oct 4, 2026 · IBM (United States) · Claire Radar · Paper

AgentPersonaBench

We introduce AgentPersonaBench (APB), a benchmark evaluating whether persona conditioning faithfully steers downstream agent behavior.

Agent Memory & Personalization · Oct 3, 2026 · Claire Radar · Paper

CoPE-Bench

Reliable language models should resist unsupported user pressure while effectively using objective contextual information.

Safety & Alignment · Instruction Following · Oct 3, 2026 · Purdue University West Lafayette · The Ohio State University · Chinese University of Hong Kong, Shenzhen · Claire Radar · Paper · Code · ★ 2 GitHub

EgoExo-Next

Vision-language models (VLMs) are increasingly evaluated for egocentric and cross-view video reasoning, yet existing benchmarks largely focus on semantic event understanding, temporal relations, or correspondence between already observed v…

Video Understanding · Spatial & 3D Reasoning · Oct 3, 2026 · Fudan University · Xi’an Jiaotong-Liverpool University · Claire Radar · Paper · Data

IEC-Bench

Agents built on large language models (LLMs) build and run software through tool calls.

Agentic Coding & SWE · Tool Use & Function Calling · Oct 3, 2026 · Yanshan University · Zhejiang University · Nanjing University · Claire Radar · Paper

MASBench

Large language models (LLMs) have progressively evolved into the core of autonomous agents.

Multi-Agent & Games · Oct 3, 2026 · Beijing University of Posts and Telecommunications · Shanghai Jiao Tong University · Tsinghua University · Claire Radar · Paper · Code · ★ 3 GitHub

MedImageOSWorld

Graphical consoles offer a practical interface for medical acquisition assistance, allowing agents to work through the controls and visual feedback used by human operators.

Computer Use & GUI Agents · Health & Medicine · Oct 3, 2026 · Cedars-Sinai Medical Center · University of California, Los Angeles · Max Delbrück Center · Claire Radar · Paper

RAGStress

Retrieval-Augmented Generation (RAG) is typically evaluated under the implicit assumption that the underlying knowledge base (KB) is clean, leaving the behaviour of RAG systems under realistic KB degradation poorly characterised.

Deep Research & Search · Oct 3, 2026 · Claire Radar · Paper

Spec2Game

Generating an executable program does not necessarily mean that it correctly implements the behavioral requirements specified in natural language.

Code Generation · Oct 3, 2026 · Chinese University of Hong Kong · Nankai University · Wuhan University · Claire Radar · Paper

AptMQL-Bench

Document databases such as MongoDB are core infrastructure for modern applications, and natural-language interfaces to them---text-to-MQL---would let non-experts query complex, semi-structured data without mastering the query language.

Code Generation · Oct 2, 2026 · The University of Sydney · MongoDB (United States) · Claire Radar · Paper

D2K-Bench

GPU kernels generated by large language model (LLM) agents can remain less efficient than expert implementations, but runtime alone does not reveal how the gap relates to design discovery and implementation.

AI R&D & ML Engineering · Oct 2, 2026 · Hong Kong University of Science and Technology · Alibaba Group (China) · University of Science and Technology of China · Claire Radar · Paper

DexJoCo-X

As dexterous hands proliferate, collecting data and training policies separately for every morphology becomes increasingly impractical.

Robotics & Embodied AI · Oct 2, 2026 · Claire Radar · Paper

DyadMem

Long-term agents must remember not only what is true about a user, but also how a particular agent should work with that user as their shared history evolves.

Agent Memory & Personalization · Oct 2, 2026 · Claire Radar · Paper

Ego2World

Egocentric videos capture how people carry out everyday activities, yet testing an agent requires evaluating the consequences of actions it chooses itself.

World Models & Physical Understanding · Video Understanding · Oct 2, 2026 · Claire Radar · Paper