New model's weight footprint (35B total MoE at on-the-fly bnb-4bit) leaves less confirmed KV-cache headroom on the 32GB card than the old one had -- reducing concurrent-sequence worst case until real memory use is verified.
New model's weight footprint (35B total MoE at on-the-fly bnb-4bit) leaves less confirmed KV-cache headroom on the 32GB card than the old one had -- reducing concurrent-sequence worst case until real memory use is verified.