Ray Serve — Model Serving

The Ray → Model Serving page lets you deploy language models as scalable, load-balanced HTTP endpoints using Ray Serve. Serve deployments run on your Ray clusters and can be called from Actors, external applications, or directly via HTTP.

What is Ray Serve?

Ray Serve is Ray's distributed model-serving library. It creates a long-running HTTP server backed by a pool of worker replicas. Requests are load-balanced across replicas. When replicas are unhealthy, Ray Serve restarts them automatically.

CleverThis uses the vLLM serving backend by default, which provides:

• Continuous batching — requests are batched together dynamically for maximum GPU throughput
• PagedAttention — memory-efficient KV cache management that allows more concurrent requests
• OpenAI-compatible API — any OpenAI SDK client works without code changes

Creating a deployment

1. Navigate to Ray → Model Serving.

2. Click Create Deployment.

3. Fill in the form:

   Field	Default	Description
   Name	—	Deployment name (e.g. llama-3-prod, mistral-7b-v1)
   Ray Cluster	—	Which Ray cluster to run the deployment on
   Model Name	meta-llama/Llama-2-7b-hf	HuggingFace model ID or local model path
   Model Path	/models/llama-2-7b	Path on compute nodes where the model weights are stored
   Model Class	vllm	Serving backend — currently vllm
   Replicas	1	Number of parallel serving processes
   GPUs per Replica	1	Number of GPUs to allocate to each replica
   CPUs per Replica	4	Number of CPU cores to allocate to each replica

4. Click Create. Ray Serve starts the deployment on the selected cluster. The endpoint URL appears once the first replica is healthy.

Deployment states

Status	Meaning
deploying	Replicas are starting; model weights are loading into VRAM
running	All replicas healthy; endpoint accepting requests
degraded	Some replicas unhealthy; endpoint still available with reduced capacity
unhealthy	All replicas failed; endpoint not available
deleting	Deployment is being torn down

Endpoint URL

Once the deployment is running, the detail panel shows a public HTTPS endpoint URL. This URL is OpenAI-compatible — you can point any OpenAI SDK client at it:

from openai import OpenAI

client = OpenAI(
    api_key="ct_live_YOUR_KEY",   # or any string if auth is disabled on the endpoint
    base_url="https://serve.cleverthis.com/d/YOUR_DEPLOYMENT_ID/v1",
)

response = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-hf",
    messages=[{"role": "user", "content": "Hello!"}]
)

You can also call it directly with curl:

curl https://serve.cleverthis.com/d/YOUR_DEPLOYMENT_ID/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-2-7b-hf",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Streaming responses

Ray Serve 2.9 supports Server-Sent Events (SSE) streaming via the stream: true parameter:

stream = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-hf",
    messages=[{"role": "user", "content": "Write a poem about the moon."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Or with curl:

curl https://serve.cleverthis.com/d/YOUR_DEPLOYMENT_ID/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-2-7b-hf","messages":[{"role":"user","content":"Hello!"}],"stream":true}'

Each SSE event is a data: {...} line; the stream ends with data: [DONE].

Scaling replicas

To scale up (more concurrent requests) or scale down (save GPU cost), update the deployment:

1. Click the edit icon on the deployment row.
2. Adjust Replicas.
3. Click Update.

Ray Serve rolls the change gradually — new replicas start before old ones are removed, so there is no downtime during scaling.

With replicas = 1 and gpus_per_replica = 1, one GPU is consumed. With replicas = 4 and gpus_per_replica = 1, four GPUs are consumed. Ensure your compute cluster has enough GPU nodes to accommodate the total GPU count.

When to add replicas

Signal	Recommended action
Latency above 5 seconds (P99)	Add 1–2 replicas
GPU utilization consistently > 90%	Add replicas
Requests queuing (queue depth > 0 in Ray Dashboard)	Add replicas
Single user workload, < 5 req/min	Keep 1 replica

GPU consumption by replica count

Replicas	GPUs per replica	Total GPUs	Suitable for
1	1	1	Dev/staging, single-user loads
2	1	2	Small production, ~10 concurrent users
4	1	4	Medium production, ~40 concurrent users
1	4	4	70B models requiring tensor parallelism

Model path vs model name

Field	Used for
model_name	The model identifier sent in API requests and used to look up configuration (tokenizer, architecture, context length)
model_path	Where vLLM loads the actual weights from on the worker node's filesystem

For HuggingFace models, both typically match:

model_name: meta-llama/Llama-2-7b-hf
model_path: /models/meta-llama/Llama-2-7b-hf  # path on compute node

For a fine-tuned model saved locally:

model_name: my-company/llama-finetune-v2       # used in API requests
model_path: /models/finetune-checkpoints/v2    # actual weights path on node

Pre-download model weights to your compute nodes using the model.download command before creating a deployment. Large models (13B+) can take 5–30 minutes to download; having them pre-cached means deployments start in seconds rather than waiting for the download.

GPU memory requirements

Model size	Min VRAM (fp16)	Min VRAM (int4 quantized)
7B	14 GB	4–5 GB
13B	26 GB	7–8 GB
34B	68 GB	18 GB
70B	140 GB	35 GB

The A100 40GB GPU has 40 GB VRAM — suitable for 7B and 13B models in fp16, or up to 34B in int4 quantization. For 70B models in fp16, use tensor parallelism across multiple A100-80GB GPUs.

Quantization

Quantization reduces model VRAM by 2–4× at a small quality cost. vLLM supports AWQ and GPTQ quantization natively.

AWQ (recommended)

AWQ (Activation-aware Weight Quantization) is the recommended 4-bit quantization format. It preserves more quality than GPTQ at the same bit width.

Many popular models have pre-quantized AWQ variants on HuggingFace Hub:

TheBloke/Llama-2-7B-Chat-AWQ
TheBloke/Mistral-7B-Instruct-v0.1-AWQ
TheBloke/Llama-2-13B-chat-AWQ

To use a pre-quantized AWQ model, set the model name to the AWQ variant and no additional configuration is needed — vLLM auto-detects AWQ from the model config.

GPTQ

GPTQ is another 4-bit format with wide availability:

TheBloke/Llama-2-7B-Chat-GPTQ
TheBloke/Mistral-7B-Instruct-v0.1-GPTQ

vLLM configuration for quantization

If you want to apply quantization at serve-time (rather than using a pre-quantized model), add a vllm_config.json file to the model path:

{
  "quantization": "awq",
  "gpu_memory_utilization": 0.90
}

Parameter	Default	Description
quantization	null	"awq", "gptq", or null (no quantization)
gpu_memory_utilization	0.90	Fraction of GPU VRAM to allocate to vLLM (0.0–1.0)
max_model_len	(model max)	Maximum context length in tokens; reduce to save VRAM
max_num_batched_tokens	4096	Max tokens across all requests in one batch
dtype	"auto"	Weight dtype: "auto", "float16", "bfloat16", "float32"
tensor_parallel_size	1	Number of GPUs to shard the model across (tensor parallelism)
swap_space	4	CPU swap space in GB for KV cache overflow

Tensor parallelism (large models)

For models that don't fit on a single GPU, distribute the model across multiple GPUs using tensor parallelism. Set gpus_per_replica in the deployment form to the number of GPUs to use per replica, and add tensor_parallel_size to vllm_config.json:

{
  "tensor_parallel_size": 4
}

Model	Recommended setup
7B fp16	1 GPU per replica
13B fp16	1 A100-80GB or 2 A100-40GB
70B fp16	4 A100-40GB (tensor parallel)
70B int4	2 A100-40GB (tensor parallel)

When gpus_per_replica = 4, the deployment uses 4 GPUs per replica. With replicas = 2, 8 GPUs total are consumed. Ensure your compute cluster has that many GPU nodes available.

LoRA multi-adapter serving

vLLM supports serving multiple LoRA adapters on the same base model, hot-swapping between them per request. This is efficient for serving many fine-tuned variants of the same base.

To enable LoRA adapter serving, add to vllm_config.json:

{
  "enable_lora": true,
  "max_loras": 4,
  "max_lora_rank": 64,
  "lora_extra_vocab_size": 256
}

Then specify the adapter in the API request:

response = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-hf",   # base model
    extra_body={"lora_name": "customer-support-v1"},  # adapter to apply
    messages=[{"role": "user", "content": "Hello!"}],
)

Register adapters by copying them to paths on the compute node and referencing them in vllm_config.json:

{
  "enable_lora": true,
  "lora_modules": [
    {"name": "customer-support-v1", "path": "/adapters/customer-support-v1"},
    {"name": "coding-assistant-v2", "path": "/adapters/coding-assistant-v2"}
  ]
}

Monitoring deployments

Deployment detail panel

Click any deployment row to open the detail panel, which shows:

• Status — current deployment status and last state change time
• Replicas — each replica's status, GPU utilization, and request rate
• Endpoint URL — copy button for the HTTP endpoint
• Throughput — requests per second (last 5 minutes)
• Latency — P50 and P99 time-to-first-token and end-to-end latency
• Error rate — 4xx and 5xx response percentage

Per-replica health

Each replica is listed with:

• healthy — processing requests normally
• starting — loading model weights
• unhealthy — failed to start or crashed; Ray Serve is restarting it

A deployment in degraded state has some unhealthy replicas but is still serving on the healthy ones. Ray Serve continuously attempts to restart unhealthy replicas.

Ray Dashboard integration

Click Open Ray Dashboard on the cluster detail panel to see per-task metrics for the serving deployment, including:

• GPU memory used per replica
• Object store usage (for KV cache shared across requests)
• Task queue depth

High task queue depth means replicas are overloaded — scale up replicas or reduce the number of concurrent users.

Health checks

Ray Serve performs health checks every 10 seconds on each replica. A replica is considered unhealthy after 3 consecutive failed health checks. Failed replicas are restarted automatically.

A health check failure can be caused by:

• GPU OOM (out of memory) — the vLLM process crashes
• CUDA error — GPU driver issue
• Python exception in the server process — misconfiguration or bug in a custom backend

The first replica restart is immediate. Subsequent restarts use exponential backoff (up to 5 minutes between attempts) to avoid thrashing.

Load balancing

Ray Serve load balances requests across replicas using a least-pending-requests algorithm — each new request goes to the replica with the fewest in-flight requests.

With continuous batching enabled (the default in vLLM), multiple requests are combined into a single GPU inference pass, maximizing throughput. You do not need to batch requests yourself.

Rolling updates

When you update a deployment (change replicas, model path, or vLLM config), Ray Serve performs a rolling update:

1. Starts new replicas with the updated configuration
2. Waits for each new replica to become healthy
3. Removes old replicas one at a time

Traffic is only routed to healthy replicas throughout the update, so there is no downtime. The update may take 2–5 minutes for large models while new replicas load the model weights.

Using a deployment in an Actor

Reference the deployment endpoint URL as a tool in an Actor YAML:

agents:
  - id: llama_inference
    type: model
    model: "https://serve.cleverthis.com/d/YOUR_DEPLOYMENT_ID/v1"
    api_key_env: SERVE_KEY
    messages:
      - role: user
        content: "{{ input }}"

Or as an HTTP tool for raw access:

agents:
  - id: llama_raw
    type: tool
    tool: http
    url: "https://serve.cleverthis.com/d/YOUR_DEPLOYMENT_ID/v1/chat/completions"
    method: POST
    headers:
      Content-Type: application/json
    body_template: |
      {
        "model": "meta-llama/Llama-2-7b-hf",
        "messages": {{ messages | tojson }}
      }

Store the deployment URL as a namespace secret and reference it with {{ secrets.SERVE_ENDPOINT_URL }} to avoid hard-coding deployment IDs in Actor YAML.

Deleting a deployment

Click Delete on the deployment row. Ray Serve gracefully shuts down all replicas. In-flight requests are allowed to complete (up to a 30-second timeout) before the replicas are terminated.

Troubleshooting

Deployment stuck in deploying for more than 5 minutes — The replica(s) are taking a long time to load model weights. For 70B models, this is normal. For 7B models, check that the model weights exist at the model_path on the compute node. Use model.download in Compute Fleet to pre-cache the model.

All replicas unhealthy — The model failed to load. Common causes:

• Insufficient VRAM: model requires more VRAM than available. Check gpu_memory_utilization in vllm_config.json and compare against node VRAM.
• Wrong model path: the weights don't exist at model_path on the worker node.
• Unsupported model architecture: some newer architectures require a newer vLLM version.

High latency (> 10 seconds per request) — The replica is processing one request at a time without batching, or the model is too large for the GPU to run efficiently. Try: reduce max_model_len in vllm_config.json, or add more replicas.

413 Request Entity Too Large — The prompt + expected completion exceeds max_model_len. Set max_model_len to a higher value in vllm_config.json or truncate your input prompt.

model_name mismatch error — The model field in the API request must exactly match the Model Name field set when creating the deployment. Check the deployment detail panel for the expected model name.

GPU utilization near 0% — vLLM is idle between requests. This is normal for low-traffic deployments. If you're sending requests and GPU utilization is still 0%, the request may not be reaching the replica — check the endpoint URL and model name.