Overnight Telemetry

Python Model Report

TL;DR

This saved report records Qwen2.5 Coder 3B on 10 primary Python prompts. The recorded average first-answer latency is 31.1 s. Marker coverage counts expected text in the response; it is not a code execution or correctness score. First-versus-later prompt differences also do not establish a controlled cold-start effect.

This report records response times, output speed and expected text markers for the listed Python prompts. First and later requests are shown separately; request order alone does not prove a cold-start effect.

Host: Finished: 2026-04-11T10:30:28Z Source: python_task_suite.json Avg primary latency: 31.1 s Avg follow-up throughput: 14.34 tok/s

Run 2026-04-11T10:23:41Z Overnight Python suite start time

Total Wall Time 0.11 h End-to-end runtime across all models

Models 1 Large and small models included

Questions 10 Primary prompts per model plus follow-up summaries

Avg Primary Marker Coverage 78% Average marker coverage across the suite

Measurements in this report

These cards summarize only the models and prompts present in this saved report.

Fastest Primary

Qwen2.5 Coder 3B

31.1 s

Small model with the quickest average first-answer pass.

Fastest Follow-up

Qwen2.5 Coder 3B

9.6 s

Quickest summary-request average once the model already had context.

Primary Marker Coverage

Qwen2.5 Coder 3B

78%

Average expected text-marker coverage for the primary responses.

Follow-up Marker Coverage

Qwen2.5 Coder 3B

Expected text-marker coverage in the follow-up responses; this does not measure code execution.

First vs Later Latency

Qwen2.5 Coder 3B

-16.3 s

Latency difference between the first primary request and the later requests in this saved sequence.

First vs Later Throughput

Qwen2.5 Coder 3B

-1.79 tok/s

Output-speed difference between the first primary request and later requests in this saved sequence.

Primary Prompt Latency

Average time to answer the main Python task prompt. Lower is better.

Qwen2.5 Coder 3B Small

31.1 s

Follow-up Prompt Latency

Average time to answer the follow-up summary request. Lower is better.

Qwen2.5 Coder 3B Small

9.6 s

Primary Prompt Throughput

Average tokens per second while answering the main Python tasks. Higher is better.

Qwen2.5 Coder 3B Small

14.01 tok/s

Follow-up Prompt Throughput

Average tokens per second on the summary requests. Higher is better.

Qwen2.5 Coder 3B Small

14.34 tok/s

Primary Marker Coverage

Average marker coverage on the full Python task prompts. Higher is better.

Qwen2.5 Coder 3B Small

78%

Follow-up Marker Coverage

Average marker coverage on the follow-up summary prompts. Higher is better.

Qwen2.5 Coder 3B Small

First vs Later Latency Difference

Positive values mean the first primary prompt was slower than the later-request average. Negative values mean the first prompt was faster. Prompt length and task difficulty may also differ.

Later prompts got faster Later prompts got slower

Qwen2.5 Coder 3B Small

-16.3 s Later prompts got slower

First vs Later Throughput Difference

Positive values mean token generation sped up after the first primary prompt. Negative values mean throughput dropped on later prompts.

Later prompts got faster Later prompts got slower

Qwen2.5 Coder 3B Small

-1.79 tok/s Later prompts got slower

Model Overview

This table separates speed, output rate, primary marker coverage, usable-response checks and first-versus-later latency for each model.

Model	Primary Avg	Follow-up Avg	Primary tok/s	Follow-up tok/s	Primary Marker Coverage	Usable	First/Later Difference
Qwen2.5 Coder 3B Small	31.1 s	9.6 s	14.01 tok/s	14.34 tok/s	78%	10/10	-16.3 s

Python Task-Type Heatmap

Values show primary-prompt marker coverage by task type. Green means the model hit more of the requested Python requirements for that category.

Model	cross_repo_debugging	debugging	forensics	planning	review	tests
Qwen2.5 Coder 3B Small	83%	77%	100%	100%	100%	61%