Add TTFT & ITL Metrics for LLM Inference (#28)
CI / CI (push) Successful in 4m7s

Time-to-First-Token (TTFT) and Inter-Token Latency (ITL) metrics for LLM inference observability

Metrics: llm_ttft_seconds, llm_itl_seconds, llm_tokens_total

Closes #31 #32 #33

---------

Co-authored-by: poimen <[email protected]>
Reviewed-on: #28
This commit was merged in pull request #28.
This commit is contained in:
2026-09-15 08:53:58 +00:00
co-authored by poimen
parent 30e0a83a50
commit d439536ca9
22 changed files with 2224 additions and 199 deletions
+9 -5
View File
@@ -1,6 +1,7 @@
package serviceadapter
import (
"net/http"
"time"
)
@@ -49,11 +50,14 @@ type Status struct {
}
// ServiceAdapter is a gateway service adapter.
// When Handler is set, the dispatcher routes directly to the internal handler
// instead of reverse-proxying to Spec.Upstream.URL.
type ServiceAdapter struct {
Name string // namespace/name
Namespace string
Name string // namespace/name
Namespace string
ServiceName string
Spec Spec
Status Status
CreatedAt time.Time
Spec Spec
Status Status
CreatedAt time.Time
Handler http.Handler `json:"-" yaml:"-"` // internal handler (skip serialization)
}