Skip to content

perf(server): reduce HTTP and PD streaming overhead - #1605

Open
WANDY666 wants to merge 1 commit into
mainfrom
perf/http-pd-streaming
Open

WANDY666 wants to merge 1 commit into
mainfrom
perf/http-pd-streaming

Conversation

@WANDY666

Copy link
Copy Markdown
Contributor

Serialize common chat/completion SSE deltas directly, coalesce events at PD batch boundaries, and cache per-stream tool metadata. Keep event ordering, tool arguments, logprobs/echo and usage unchanged; flush buffered output before stream errors.

Also avoid per-message receive tasks and allocating a fake token list solely for length validation. These remove CPU work on the streaming path without waiting for the next model token to flush a completed batch.

This covers only the remaining optimizations; compact forwarding stays in #1486 and #1487 (the latter needs rebasing for main's updated ObjType values). Tests: streaming/API and PD regressions (79 passed).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants