Files
ollama37/tests/testcases/runtime/TC-RUNTIME-001.yml
Shang Chieh Tseng 1a185f7926 Add comprehensive Ollama log checking and configurable LLM judge mode
Test case enhancements:
- TC-RUNTIME-001: Add startup log error checking (CUDA, CUBLAS, CPU fallback)
- TC-RUNTIME-002: Add GPU detection verification, CUDA init checks, error detection
- TC-RUNTIME-003: Add server listening verification, runtime error checks
- TC-INFERENCE-001: Add model loading logs, layer offload verification
- TC-INFERENCE-002: Add inference error checking (CUBLAS/CUDA errors)
- TC-INFERENCE-003: Add API request log verification, response time display

Workflow enhancements:
- Add judge_mode input (simple/llm/dual) to all workflows
- Add judge_model input to specify LLM model for judging
- Configurable via GitHub Actions UI without code changes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-16 23:27:57 +08:00

58 lines
1.6 KiB
YAML

id: TC-RUNTIME-001
name: Container Startup
suite: runtime
priority: 1
timeout: 120000
dependencies:
- TC-BUILD-002
steps:
- name: Stop existing container
command: cd docker && docker compose down 2>/dev/null || true
- name: Start container with GPU
command: cd docker && docker compose up -d
- name: Wait for startup
command: sleep 15
- name: Check container status
command: cd docker && docker compose ps
- name: Capture startup logs
command: |
cd docker && docker compose logs 2>&1 | head -100
- name: Check for startup errors in logs
command: |
cd docker
LOGS=$(docker compose logs 2>&1)
# Check for critical errors
if echo "$LOGS" | grep -qE "(level=ERROR|CUBLAS_STATUS_|CUDA error|cudaMalloc failed)"; then
echo "CRITICAL ERRORS FOUND IN STARTUP LOGS:"
echo "$LOGS" | grep -E "(level=ERROR|CUBLAS_STATUS_|CUDA error|cudaMalloc failed)"
exit 1
fi
# Check for CPU-only fallback (GPU not detected)
if echo "$LOGS" | grep -q "id=cpu library=cpu"; then
echo "ERROR: Ollama fell back to CPU-only mode"
exit 1
fi
echo "SUCCESS: No critical errors in startup logs"
criteria: |
The ollama37 container should start successfully with GPU access.
Expected:
- Container starts without errors
- docker compose ps shows container in "Up" state
- No "Exited" or "Restarting" status
- No critical errors in logs (level=ERROR, CUBLAS_STATUS_, CUDA error)
- No CPU-only fallback (id=cpu library=cpu)
Accept startup warnings (flash attention not supported is OK). Container should be running.