Overnight Telemetry

Python Model Report

TL;DR

This saved report records Qwen2.5 Coder 3B on 20 primary Python prompts. The recorded average first-answer latency is 38.1 s. Marker coverage counts expected text in the response; it is not a code execution or correctness score. First-versus-later prompt differences also do not establish a controlled cold-start effect.

This report records response times, output speed and expected text markers for the listed Python prompts. First and later requests are shown separately; request order alone does not prove a cold-start effect.

Host: vmi3206382 Finished: 2026-04-06T07:57:51Z Source: 2026-04-11-qwen3b-python-task-suite-v1-20q.json Avg primary latency: 38.1 s Avg follow-up throughput: 3.91 tok/s

Run 2026-04-05T23:17:56Z Overnight Python suite start time

Total Wall Time 0.35 h End-to-end runtime across all models

Models 1 Large and small models included

Questions 20 Primary prompts per model plus follow-up summaries

Avg Primary Marker Coverage 89% Average marker coverage across the suite

Measurements in this report

These cards summarize only the models and prompts present in this saved report.

Fastest Primary

Qwen2.5 Coder 3B

38.1 s

Small model with the quickest average first-answer pass.

Fastest Follow-up

Qwen2.5 Coder 3B

25.3 s

Quickest summary-request average once the model already had context.

Primary Marker Coverage

Qwen2.5 Coder 3B

89%

Average expected text-marker coverage for the primary responses.

Follow-up Marker Coverage

Qwen2.5 Coder 3B

Expected text-marker coverage in the follow-up responses; this does not measure code execution.

First vs Later Latency

Qwen2.5 Coder 3B

29.7 s

Latency difference between the first primary request and the later requests in this saved sequence.

First vs Later Throughput

Qwen2.5 Coder 3B

1.91 tok/s

Output-speed difference between the first primary request and later requests in this saved sequence.

Primary Prompt Latency

Average time to answer the main Python task prompt. Lower is better.

Qwen2.5 Coder 3B Small

38.1 s

Follow-up Prompt Latency

Average time to answer the follow-up summary request. Lower is better.

Qwen2.5 Coder 3B Small

25.3 s

Primary Prompt Throughput

Average tokens per second while answering the main Python tasks. Higher is better.

Qwen2.5 Coder 3B Small

4.16 tok/s

Follow-up Prompt Throughput

Average tokens per second on the summary requests. Higher is better.

Qwen2.5 Coder 3B Small

3.91 tok/s

Primary Marker Coverage

Average marker coverage on the full Python task prompts. Higher is better.

Qwen2.5 Coder 3B Small

89%

Follow-up Marker Coverage

Average marker coverage on the follow-up summary prompts. Higher is better.

Qwen2.5 Coder 3B Small

First vs Later Latency Difference

Positive values mean the first primary prompt was slower than the later-request average. Negative values mean the first prompt was faster. Prompt length and task difficulty may also differ.

Later prompts got faster Later prompts got slower

Qwen2.5 Coder 3B Small

29.7 s Later prompts got faster

First vs Later Throughput Difference

Positive values mean token generation sped up after the first primary prompt. Negative values mean throughput dropped on later prompts.

Later prompts got faster Later prompts got slower

Qwen2.5 Coder 3B Small

1.91 tok/s Later prompts got faster

Model Overview

This table separates speed, output rate, primary marker coverage, usable-response checks and first-versus-later latency for each model.

Model	Primary Avg	Follow-up Avg	Primary tok/s	Follow-up tok/s	Primary Marker Coverage	Usable	First/Later Difference
Qwen2.5 Coder 3B Small	38.1 s	25.3 s	4.16 tok/s	3.91 tok/s	89%	20/20	29.7 s

Python Task-Type Heatmap

Values show primary-prompt marker coverage by task type. Green means the model hit more of the requested Python requirements for that category.

Model	analysis	async	cli	concurrency	config	debugging	file_io	filesystem	http	logging	package	parsing	refactor	sqlite	tests	typing	validation	web
Qwen2.5 Coder 3B Small	75%	100%	100%	100%	50%	100%	100%	75%	100%	50%	100%	88%	100%	100%	100%	100%	75%	100%