Jesse Gross 65973ceb64 runner.go: Make KV entry accounting more robust ...

The structure of the accounting for KV cache shifting was carried
over from the old runner but it now doesn't feel natural with the new
runner. There are a number of invariants that should hold true but
are difficult to reason about. There is at least one bug report
that would imply that the invariants are not holding.

This reduces the number of implicit assumptions and is more forgiving
of unexpected situations. It also improves behavior around which input
tokens are kept when truncation occurs.

Bug #7545

2024-11-11 20:23:03 -08:00

5.6 KiB

Raw Blame History

View Raw