Overnight Telemetry

Python Model Report

This mini page turns the overnight suite into a quick visual read: which models were fastest, which ones improved after the cold first prompt, and which Python task types each model handled best.

Host: Finished: 2026-04-11T09:54:49Z Source: python_task_suite.json Avg primary latency: 11.4 s Avg follow-up throughput: 16.56 tok/s

Run 2026-04-11T09:48:31Z Overnight Python suite start time

Total Wall Time 0.11 h End-to-end runtime across all models

Models 1 Large and small models included

Questions 20 Primary prompts per model plus follow-up summaries

Avg Primary Quality 88% Average marker coverage across the suite

Headline Winners

These are the standout models for the metrics that matter most in the overnight run.

Fastest Primary

Qwen2.5 Coder 1.5B

11.4 s

Small model with the quickest average first-answer pass.

Fastest Follow-up

Qwen2.5 Coder 1.5B

7.5 s

Quickest summary-request average once the model already had context.

Best Primary Quality

Qwen2.5 Coder 1.5B

88%

Highest average hit-rate on the full Python task requirements.

Best Follow-up Quality

Qwen2.5 Coder 1.5B

Strongest performance on the follow-up summary prompts.

Biggest Warmup Gain

Qwen2.5 Coder 1.5B

-4.6 s

How much faster the later primaries got after the first cold request.

Biggest Warmup Throughput Gain

Qwen2.5 Coder 1.5B

-1.20 tok/s

How much token generation speed improved after the first primary prompt.

Primary Prompt Latency

Average time to answer the main Python task prompt. Lower is better.

Qwen2.5 Coder 1.5B Small

11.4 s

Follow-up Prompt Latency

Average time to answer the follow-up summary request. Lower is better.

Qwen2.5 Coder 1.5B Small

7.5 s

Primary Prompt Throughput

Average tokens per second while answering the main Python tasks. Higher is better.

Qwen2.5 Coder 1.5B Small

16.89 tok/s

Follow-up Prompt Throughput

Average tokens per second on the summary requests. Higher is better.

Qwen2.5 Coder 1.5B Small

16.56 tok/s

Primary Quality

Average marker coverage on the full Python task prompts. Higher is better.

Qwen2.5 Coder 1.5B Small

88%

Follow-up Quality

Average marker coverage on the follow-up summary prompts. Higher is better.

Qwen2.5 Coder 1.5B Small

Cold-Start Latency Delta

Positive values mean the first primary prompt was slower than the warmed-up average. Negative values mean the first prompt was actually faster.

Later prompts got faster Later prompts got slower

Qwen2.5 Coder 1.5B Small

-4.6 s Later prompts got slower

Cold-Start Throughput Delta

Positive values mean token generation sped up after the first primary prompt. Negative values mean throughput dropped on later prompts.

Later prompts got faster Later prompts got slower

Qwen2.5 Coder 1.5B Small

-1.20 tok/s Later prompts got slower

Model Overview

This is the quick scan: speed, throughput, primary quality, usable answers, and the cold-start latency gap for each model.

Model	Primary Avg	Follow-up Avg	Primary tok/s	Follow-up tok/s	Primary Quality	Usable	Cold Gain
Qwen2.5 Coder 1.5B Small	11.4 s	7.5 s	16.89 tok/s	16.56 tok/s	88%	20/20	-4.6 s

Python Task-Type Heatmap

Values show primary-prompt marker coverage by task type. Green means the model hit more of the requested Python requirements for that category.

Model	analysis	async	cli	concurrency	config	debugging	file_io	filesystem	http	logging	package	parsing	refactor	sqlite	tests	typing	validation	web
Qwen2.5 Coder 1.5B Small	75%	100%	100%	100%	50%	100%	100%	75%	75%	75%	100%	88%	75%	100%	100%	100%	75%	100%