matt/ollama37
1
0
Fork 0
mirror of https://github.com/dogkeeper886/ollama37.git synced 2025-12-09 23:37:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix Claude validation response format parsing

Browse Source 
The Claude AI validator was receiving detailed explanations with markdown
formatting (e.g., '**PASS**') instead of the expected simple format.

Updated the validation prompt to explicitly require responses to start
with either 'PASS' or 'FAIL: <reason>' without any additional formatting,
explanations, or markdown before the verdict.

This fixes the 'Warning: Unexpected Claude response format' error that
was causing valid test results to be incorrectly marked as unclear.
This commit is contained in:
Shang Chieh Tseng
2025-10-30 12:34:02 +08:00
parent c8b7015a2c
commit 46f1038724
1 changed files with 7 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
cmd/test-runner/validate.go
View File

@@ -228,11 +228,16 @@ func (v *Validator) validateWithClaude(prompt *PromptTest, simpleCheckPassed boo
4. Appears to be from a working LLM model (not system errors or failures)
5. Has reasonable quality for a 4B parameter model
Respond with ONLY one of these formats:
CRITICAL: Your response MUST start with exactly one of these two words:
- "PASS" if the response is valid and acceptable
- "FAIL: <brief reason>" if the response has issues
Be concise. One line only.`)
Example valid responses:
- "PASS"
- "FAIL: Response is gibberish"
- "FAIL: Contains error messages instead of proper response"
Do NOT include explanations, markdown formatting, or additional text before the verdict. Start your response with PASS or FAIL: only.`)
// Write to temp file
promptFile := filepath.Join(v.claudeTempDir, fmt.Sprintf("prompt_%d.txt", os.Getpid()))