vps50 Small-Model Telemetry Manual

Small-model performance dashboard

This manual turns the small-model telemetry JSON into one scannable page so we can compare latency, throughput, suite pass-rate, requirement-hit-rate, and question-level behavior without reading a wall of raw logs.

Source: 2026-04-11-qwen3b-small-model-eval.json Generated: 2026-04-11T10:49:28Z Question suite: small-model-coding-eval-v1 Server: vmi3206382 Runner: ollama Ollama: ollama version is 0.20.2

Question Set

Suite: small-model-coding-eval-v1

shell bash_code

Disk Guard Script

disk_guard_bash

Return only Bash code. Write a script that checks disk usage for /, prints a human-readable warning, and exits with status 1 when usage is above 85 percent. Requirements: include a shebang, use df -P /, parse the numeric percentage, and keep the script production-safe.

#!/usr/bin/env bashdf -P /85exit 1

python python_code

IPv4 Validator

ipv4_python_tests

Return only Python code. Write a function named is_valid_ipv4(value: str) -> bool and include exactly three pytest tests that cover a valid address, an out-of-range octet, and a non-numeric input.

def is_valid_ipv4def test_assertsplit('.')

ops shell_lines

Nginx Safe Reload

nginx_safe_reload

Return only Bash commands, one per line. Back up /etc/nginx/nginx.conf, validate nginx config, and reload nginx only if validation passes.

cp /etc/nginx/nginx.confnginx -tsystemctl reload nginx&&

planning four_numbered_steps

YAML Validator Plan

yaml_cli_plan

Return exactly four numbered steps. Plan a Python CLI that scans a git repo for changed YAML files, validates them against a JSON schema, and exits nonzero on failure.

1.2.3.4.JSON schemagit

debugging five_bullets

SSH Lockout Triage

ssh_lockout_triage

Return exactly five bullet points. After hardening, SSH started returning Permission denied (publickey,password). List the safest first checks before changing config. Mention sshd_config, authorized_keys, journalctl, rollback, and PasswordAuthentication.

sshd_configauthorized_keysjournalctlrollbackPasswordAuthentication

Per-Question Matrix

Each cell summarizes marker coverage and format fidelity for one model-question pair.

Model	Disk Guard Script	IPv4 Validator	Nginx Safe Reload	YAML Validator Plan	SSH Lockout Triage
Qwen2.5 Coder 3B	75.0% no fmt • reply	100.0% fmt • reply	75.0% fmt • reply	100.0% fmt • reply	100.0% no fmt • reply

Show the evaluation prompts

disk_guard_bash: Return only Bash code. Write a script that checks disk usage for /, prints a human-readable warning, and exits with status 1 when usage is above 85 percent. Requirements: include a shebang, use df -P /, parse the numeric percentage, and keep the script production-safe. [bash_code]
ipv4_python_tests: Return only Python code. Write a function named is_valid_ipv4(value: str) -> bool and include exactly three pytest tests that cover a valid address, an out-of-range octet, and a non-numeric input. [python_code]
nginx_safe_reload: Return only Bash commands, one per line. Back up /etc/nginx/nginx.conf, validate nginx config, and reload nginx only if validation passes. [shell_lines]
yaml_cli_plan: Return exactly four numbered steps. Plan a Python CLI that scans a git repo for changed YAML files, validates them against a JSON schema, and exits nonzero on failure. [four_numbered_steps]
ssh_lockout_triage: Return exactly five bullet points. After hardening, SSH started returning Permission denied (publickey,password). List the safest first checks before changing config. Mention sshd_config, authorized_keys, journalctl, rollback, and PasswordAuthentication. [five_bullets]

Small-model performance dashboard

Question Set

Disk Guard Script

IPv4 Validator

Nginx Safe Reload

YAML Validator Plan

SSH Lockout Triage

Latency

Tokens Per Second

Pass Rate

Requirement Hit-Rate

Per-Question Matrix

Model Rollup