automatically set num_keep if num_keep < 0

num_keep defines how many tokens to keep in the context when truncating inputs. if left to its default value of -1, the server will calculate num_keep to be the left of the system instructions
2025-12-11 00:07:07 +00:00 · 2023-08-07 14:51:31 -07:00
parent 089d03bc8d
commit 4dc5b117dd
3 changed files with 28 additions and 14 deletions
--- a/api/types.go
+++ b/api/types.go
@@ -264,6 +264,7 @@ func DefaultOptions() Options {
 		UseNUMA: false,

 		NumCtx:             2048,
+		NumKeep:            -1,
 		NumBatch:           512,
 		NumGPU:             1,
 		NumGQA:             1,