mirror of https://github.com/dogkeeper886/ollama37.git synced 2025-12-10 15:57:04 +00:00

Files

Jesse Gross 7121dfa309 runner.go: Retry decoding after defragmentation if needed

Fragmentation of the KV cache can occur due to cache shifting or
different sequences getting processed. Decode uses a heuristic to
decide if it should defrag. However, this heuristic isn't 100%
accurate, so decoding can sometimes fail by surprise.

For these cases, if decode indicates that there is no KV cache space,
we should defrag and then try again.

2024-11-20 12:49:24 -08:00

cache_test.go

runner.go: Better abstract vision model integration

2024-10-30 14:53:43 -07:00

cache.go

runner.go: Make KV entry accounting more robust

2024-11-11 20:23:03 -08:00

image_test.go

runner.go: Better abstract vision model integration

2024-10-30 14:53:43 -07:00

image.go

runner.go: Check for zero length images

2024-11-08 09:39:32 -08:00

README.md

Re-introduce the llama package (#5034 )

2024-10-08 08:53:54 -07:00

requirements.go

Re-introduce the llama package (#5034 )

2024-10-08 08:53:54 -07:00

runner.go

runner.go: Retry decoding after defragmentation if needed

2024-11-20 12:49:24 -08:00

stop_test.go

runner.go: Handle truncation of tokens for stop sequences

2024-10-09 20:39:04 -07:00

stop.go

runner.go: Handle truncation of tokens for stop sequences

2024-10-09 20:39:04 -07:00

README.md

`runner`

Note: this is a work in progress

A minimial runner for loading a model and running inference via a http web server.

./runner -model <model binary>

Completion

curl -X POST -H "Content-Type: application/json" -d '{"prompt": "hi"}' http://localhost:8080/completion

Embeddings

curl -X POST -H "Content-Type: application/json" -d '{"prompt": "turn me into an embedding"}' http://localhost:8080/embeddings