This manual turns the small-model telemetry JSON into one scannable page so we can compare latency, throughput, suite pass-rate, requirement-hit-rate, and question-level behavior without reading a wall of raw logs.
Suite: small-model-coding-eval-v1
disk_guard_bash
Return only Bash code. Write a script that checks disk usage for /, prints a human-readable warning, and exits with status 1 when usage is above 85 percent. Requirements: include a shebang, use df -P /, parse the numeric percentage, and keep the script production-safe.
ipv4_python_tests
Return only Python code. Write a function named is_valid_ipv4(value: str) -> bool and include exactly three pytest tests that cover a valid address, an out-of-range octet, and a non-numeric input.
nginx_safe_reload
Return only Bash commands, one per line. Back up /etc/nginx/nginx.conf, validate nginx config, and reload nginx only if validation passes.
yaml_cli_plan
Return exactly four numbered steps. Plan a Python CLI that scans a git repo for changed YAML files, validates them against a JSON schema, and exits nonzero on failure.
ssh_lockout_triage
Return exactly five bullet points. After hardening, SSH started returning Permission denied (publickey,password). List the safest first checks before changing config. Mention sshd_config, authorized_keys, journalctl, rollback, and PasswordAuthentication.
Lower is better. Sorted fastest to slowest.
Higher is better. Sorted fastest to slowest.
Higher is better. Derived from the small-model evaluation suite.
Higher is better. Measures how closely the answer matches the requested shape.
Each cell summarizes marker coverage and format fidelity for one model-question pair.
| Model | Disk Guard Script | IPv4 Validator | Nginx Safe Reload | YAML Validator Plan | SSH Lockout Triage |
|---|---|---|---|---|---|
| Qwen2.5 Coder 3B | 75.0% | 100.0% | 75.0% | 100.0% | 100.0% |
| Qwen2.5 Coder 1.5B | 75.0% | 100.0% | 75.0% | 100.0% | 100.0% |
| Qwen2.5 3B | 50.0% | 100.0% | 75.0% | 100.0% | 80.0% |
| Llama 3.2 3B | 75.0% | 75.0% | 75.0% | 33.3% | 80.0% |
| Phi-3 Mini | 50.0% | 75.0% | 100.0% | 100.0% | 80.0% |
Prompt: hi can you help me?
This keeps the earlier six-model smoke test visible inside the same dashboard.
Size: 32b
Latency: 175.0 s
Characters: 255 chars
Of course! I'd be happy to help you. Could you please provide more details about what you need assistance with? Are you facing an issue with a specific piece of software, a bug, or something else? The more details you provide, the better I can assist you.
Size: 14b
Latency: 54.0 s
Characters: 44 chars
Of course! What do you need assistance with?
Size: 22b
Latency: 101.0 s
Characters: 229 chars
Of course! I'd be happy to help you with your coding questions or problems. Could you please provide more details about the specific issue or task you're facing? This will allow me to provide a more accurate and helpful response.
Size: 34b
Latency: 182.0 s
Characters: 267 chars
Hello! I'd be happy to help you with any questions or problems you have. Could you please provide more information about what you need help with, such as the code you're working on or the issue you're facing? The more details you provide, the better I can assist you.
Size: 34b
Latency: 113.0 s
Characters: 186 chars
Of course, I'd be happy to help. However, you haven't mentioned what issue or problem you're facing. Please provide more details about your query or the problem you need assistance with.
Size: 14b
Latency: 66.0 s
Characters: 241 chars
Of course! How can I assist you today? Do you need help with creating an execution plan, identifying risks, defining acceptance criteria, or something else related to software delivery? Please provide more details so I can better assist you.
Compact index of the captured model rows for quick cross-checking.