Ten Tasks, No Solves: Reproducing Agents' Last Exam with Qwen3-VL

September 22, 2026 — Agent Evaluation Reproducibility Qwen3-VL

This post's focus: what I learned by running Qwen3-VL on ten Agents' Last Exam tasks and digging into the failed attempts.

I ran ten tasks from Agents' Last Exam with Qwen3-VL-30B-A3B-Instruct through ALE-Claw. The agent solved none of them. Nine primary attempts scored 0.0; the tenth hit a grading failure and had no original numeric score. Looking through the saved attempts revealed unfinished submissions, a pipeline that discarded its own results, repeated conversation compaction, and a grader that overlooked a valid response.

The useful part was figuring out what happened behind those scores. The saved files and action history show how far the agent got, where it stopped, and what we still can't explain.

One genomic task makes this concrete. The agent repaired permissions, wrote a script, and produced a nonempty intermediate file. It then filtered on the wrong column, left the final result empty, and never created a required summary. Real progress and a specific mistake, all hidden behind another zero.

Here's what held up across the evidence:

• Every primary and included additional attempt lacked at least one required output.
• All 15 comparable regrades matched, while repeated agent attempts varied in duration and behavior.
• Reviewing the saved attempts exposed a coding error and separate problems in ALE-Claw and the grader.

The repository contains the result tables, figures, settings, and case-study evidence. I describe the study as a partial reproduction of the evaluation workflow: all ten environments ran, but only nine primary attempts received official numeric scores.

What I tested

Agents' Last Exam evaluates agents on professional workflows with verifiable outputs. Its evaluation framework provisions task environments, runs an agent, collects its work, and grades the result. This makes completing and delivering files part of the task itself.

I selected ten Linux tasks that required no licensed applications, covering data engineering, mathematical optimization, energy engineering, forecasting, and genomics. They aren't a representative sample of the full benchmark.

ALE-Claw supplies the tools and manages the interaction; Qwen3-VL chooses the actions. I needed a vision-language model because ALE-Claw supplies screenshots, so I used the 30B-A3B Instruct checkpoint from the Qwen3-VL family, served on one GPU with the same settings across tasks.

I set the context limit to 32,768 tokens; the model family supports larger windows.

I pinned ALE to commit 75a3f866535946b67f9a57e4f158eb30ad50be8a and the model to revision 9c4b90e1e4ba969fd3b5378b57d966d725f1b86c. Container digests, task-data revisions, grader fingerprints, and the remaining settings are recorded in the scientific settings. No settings were tuned for individual tasks.

Results

Here's how the ten primary attempts ended, with elapsed times rounded from the saved records.

Source: task results and trajectory review. A dash means the original score is missing. A later independent check of the MPC outputs returned 0.0; that result is recorded separately.

Six attempts reached the step limit, and two ended when the agent signaled completion. All eight were missing required files. ALE marked them as completed because the run had ended—even though the work wasn't done.

The power-reliability attempt spent most of its time shortening the conversation history: ALE-Claw performed 1,232 context compactions before the run timed out after just over two hours. The MPC attempt hit a grading bug, discussed below.

Every primary attempt lacked at least one required output. Missing files tell us the work was incomplete, but they don't explain why. Poor planning, incorrect code, environment problems, and lost progress can all lead there.

Inside one failure

The genomic task required processing three BED files, which describe genomic regions using chromosome, start, and end coordinates. The requested deliverables were sorted, non-overlapping regions in union_peaks.bed, the commands in commands.sh, and counts in summary.json.

I selected this case because it reached actual processing work. Two required files were present, but one was empty. The public case study links each finding to the retained evidence.

The agent initially struggled to read instructions and run software. Three script executions failed to launch software, and a fourth failed to read the inputs. After fixing permissions, a fifth run reported no errors and produced 15,296 three-column rows in an intermediate file. The agent was now processing data.

The pipeline then made a specific mistake. Both processing stages used bedtools intersect -c, tested column 7 for a positive overlap count, and kept columns 1–3. The BEDTools documentation specifies that -c appends the count after the input columns. The second stage received three-column input, so its count was in column 4. Testing the nonexistent seventh column discarded every row.

I traced this error through the saved script and files. I didn't rerun a corrected solution or compare it with hidden answers, so fixing this bug alone wouldn't establish that the whole solution was correct.

The script also never created summary.json, and no later action supplied it. The repeated grader flagged that omission. The zero records an incomplete submission; it doesn't tell us how a corrected algorithm would score.

Two details made this harder to diagnose. All four failed script runs returned exit code zero: later commands masked earlier failures. I had to check error messages and saved files to see what had actually run.

The agent also inspected the empty output without fixing it before a conversation compaction. The summary left out the implementation and debugging work, describing implementation as still to do. The agent then returned to setup and hit its 100-step limit. Useful progress information may have been lost, but this sequence doesn't prove that compaction caused the failure or tell us what the model retained.

Evaluation bugs

ALE-Claw passed the model server's endpoint and credentials to its main calls, but omitted those connection settings from helper calls used for conversation compaction and memory saving. The primary-v2 experiment supplied the settings through environment variables as a workaround. I proposed an upstream fix in ALE pull request #91; that patch was not part of the evaluated code. This credential issue is separate from the power task's repeated-compaction loop.

The MPC grader had another problem. It read the checking program's response from output, falling back to stdout only if output was absent. An empty output field therefore hid valid text in stdout, leaving ALE unable to parse the response or record a score.

Running the checker independently against the collected empty output returned 0.0 for missing deliverables. I kept both records: the missing original score and the later zero. Otherwise, the grading failure would disappear from the results. The outcome report documents both bugs.

Repeatability

Regrading unchanged files gave 15 out of 15 matching results: nine primary attempts and six repeats. MPC had no original score to compare. Since all comparable grades were zero, this only shows stability for these incomplete submissions. The regrading table records the comparisons and confirms the files stayed unchanged.

The agent's behavior varied on reruns. I gave three preselected tasks—recommendations, ETL, and game equilibrium—two extra attempts each. All six still scored zero and missed required outputs. Three other attempts were interrupted when the inference VM stopped; they're retained but excluded from performance statistics.

The same scores hid different amounts of work. Across three included attempts, recommendations took 61.95–527.37 seconds, while game equilibrium generated 4,612–32,450 recorded output tokens. These small-sample ranges describe the runs, not the model's broader variability. The repeated-run table has the details.

Input-token totals include conversation context sent repeatedly. Helper calls may be missing, so the chart isn't a complete account of inference work. I didn't estimate GPU cost.

Limits

This study covers one model, one agent program, ten selected tasks, and one fixed configuration. I didn't compare models, increase the context window, or tune individual tasks. The results don't reproduce ALE's full leaderboard or isolate model ability from the surrounding software and environment. The genomic case was chosen for closer inspection, and some of its causes remain uncertain.

“Partial reproduction” describes the evaluation process: ten environments ran, nine primary scores were available, and all comparable regrades agreed. None of the tasks earned partial credit.

What made this worth investigating was the evidence behind each zero: stopping without deliverables, emptying a result after real processing, or getting stuck in conversation compaction. Keeping the files and action history made those differences visible.

The public repository provides aggregate results, figures, settings, and evidence references. Raw trajectories, task inputs, and deployment records remain private because they can contain restricted data and operational details. The provenance guide explains that boundary.

Figures come from the reproduction repository, source snapshot 852eaac, under its CC BY 4.0 content license. Benchmark tasks are attributed to the Agents' Last Exam authors and contributors; model work is attributed to the Qwen team. See the companion's attribution record.