rock
41dcda819e
The Clef runtime was running from hand-applied objects that existed only in the cluster.
A rebuild would have lost it.
Three things move into git:
- clef-serve.yaml: the Deployment and the Service the gateway routes to
- configMapGenerator for clef-serve/app.py, so an edit to the code rolls the pod
instead of leaving a stale copy mounted
- clef-flash.yaml is deleted: it described an SGLang predictor that cannot run on a
V100, and its slot is now clef-serve.yaml
Recorded in the manifest, because both cost hours: SGLang needs CUDA 13 for Clef and
CUDA 13 has no sm_70 kernels, and device_map over two cards breaks the forward pass with
'indices should be either on cpu or on the same device as the indexed tensor'.
Still open and written down in the file: the image cannot be pushed, because Cloudflare
answers 413 for layers above 100 MB. So each start pip-installs and needs PyPI.
23 lines
787 B
Docker
23 lines
787 B
Docker
# Base image: torch 2.8.0+cu128 still lists sm_70 in torch.cuda.get_arch_list().
|
|
# The CUDA 13 builds of torch drop sm_70. worker-1 holds Tesla V100 cards with
|
|
# compute capability 7.0, so a newer base kills the kernel launch at load time.
|
|
# Keep this pin until the model moves to newer hardware.
|
|
FROM pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime
|
|
|
|
# Unbuffered output lets the pod log show the startup lines in order.
|
|
ENV PYTHONUNBUFFERED=1 \
|
|
PIP_NO_CACHE_DIR=1
|
|
|
|
WORKDIR /app
|
|
|
|
# torch comes from the base image. Never reinstall it here, or pip pulls a
|
|
# wheel built for another CUDA version.
|
|
COPY requirements.txt .
|
|
RUN pip install --no-cache-dir -r requirements.txt
|
|
|
|
COPY app.py .
|
|
|
|
EXPOSE 8080
|
|
|
|
CMD ["uvicorn", "app:create_app", "--factory", "--host", "0.0.0.0", "--port", "8080"]
|