Jesse Gross 17b386a891 runner.go: Enforce NUM_PARALLEL directly in the runner ...

NUM_PARALEL is currently enforced by the Ollama server process - it
will only issue requests to the runner if the maximum number of
concurrent requests has not been exceeded. Although this should
be sufficient, it is good for the runner to protect its own data
structures. Currently, if too many requests get through to the
runner, they will just get stuck and never return.

This may help with reports of Ollama hanging, though it is unclear
how it would actually occur.

Bug #7573

2024-11-14 11:21:59 -08:00

24 KiB

Raw Blame History

View Raw