Ray Clusters

A Ray cluster is a group of worker processes coordinated by a Ray head node. In CleverThis, Ray clusters are provisioned on top of your Compute Fleet — the head node and workers run as containers on your registered compute nodes.

How it works

When you create a Ray cluster in CleverThis:

1. The head node container starts on one of your compute cluster's nodes. It runs:

   • The Ray scheduler (GCS — Global Control Store)
   • The Ray Dashboard on port 8265 (tunneled through CleverThis)
   • The Ray Job API server on port 8265/jobs
   • The Ray Serve HTTP server on port 8000 (proxied by CleverThis)
   • The autoscaler process that monitors job queue depth

2. Worker nodes are started as additional containers on remaining compute nodes. Each worker connects to the head via Ray's internal protocol (port 6379 for Redis, ports 10001–19999 for object store transfers).

3. Ray's autoscaler adds and removes workers based on task queue depth — scaling down to min_workers when idle and up to max_workers under load.

4. Training jobs, Serve deployments, and Tune experiments all submit their tasks to the head node's job API, which distributes them to workers.

Creating a Ray cluster

1. Navigate to Ray → Clusters.

2. Click Create Cluster.

3. Fill in the form:

   Field	Default	Description
   Name	—	Unique name for this cluster (e.g. fine-tune-cluster, serve-prod)
   Compute Cluster	—	The Compute Fleet cluster to run on — select from the dropdown
   Ray Version	2.9.0	Ray runtime version installed on head and worker containers
   Min Workers	0	Minimum number of worker nodes to keep running when idle
   Max Workers	10	Maximum number of worker nodes the autoscaler can start

4. Click Create. The head node starts within 30–60 seconds on a fast machine; longer if the rayproject/ray:2.9.0-gpu image needs to be pulled for the first time.

Cluster states

Status	Meaning
pending	Head node container is being created and Ray processes are starting
running	Head node is online; Ray scheduler is accepting work
failed	Head node failed to start — check compute cluster node availability
stopped	Cluster is halted; no compute is allocated; state is preserved
deleting	Cluster teardown in progress

A cluster in running state will have min_workers workers always online. Workers beyond that are added on demand and removed when idle for more than 60 seconds.

Autoscaling behavior

With min_workers = 0 (the default), all worker nodes shut down when the cluster is idle. The first job submitted to an idle cluster triggers autoscaling; workers spin up within 1–3 minutes depending on container pull time and node startup speed.

min_workers setting	Trade-off
0	Zero cost when idle; cold start of 1–3 minutes for the first job
1	Low baseline cost; single-worker jobs start instantly
= max_workers	No autoscaling; fully warm; pays for idle capacity 24/7

Autoscaling triggers

The autoscaler starts a new worker when:

• There are pending Ray tasks with CPU/GPU requirements that exceed current worker capacity
• A Ray Serve deployment needs more replicas than current workers can provide
• A Tune experiment has more concurrent trials than current workers

The autoscaler removes a worker when:

• The worker has been idle (no tasks running) for more than 60 seconds
• The total worker count would remain above min_workers after removal

Autoscaling limits

max_workers limits the total worker count. Set it to the number of compute nodes available in your compute cluster minus 1 (for the head node). For example, if your compute cluster has 5 GPU nodes, set max_workers = 4.

If max_workers is reached and there are still pending tasks, they queue until a worker becomes available. Tasks do not fail — they wait.

Viewing cluster details

Click any cluster row to open the detail panel. It shows:

Summary tab

• Head node — hostname, container ID, IP address on the compute node's network
• Status — current cluster status with a timestamp of the last state change
• Ray version — the Ray runtime version running on this cluster
• Uptime — how long the cluster has been running

Nodes tab

• All worker nodes — list of worker containers with:
  • Node hostname and status (running, stopped, failed)
  • CPU utilization (%)
  • Memory utilization (%)
  • GPU utilization and VRAM used (if GPU node)
  • Number of Ray tasks currently running on this worker
• Head node resource usage — head node should use minimal CPU/memory; high usage indicates too many pending object store transfers or GCS pressure

Jobs tab

• Active training jobs linked to this cluster
• Active Serve deployments linked to this cluster
• Active Tune experiments linked to this cluster

Logs tab

• Head node logs — Ray GCS and autoscaler logs; useful for diagnosing why workers are not starting or why jobs are not being scheduled

Ray Dashboard

Click Open Ray Dashboard to access the full Ray Dashboard UI running on the head node. This shows real-time per-task metrics, object store usage, cluster resource utilization, and individual actor states. The dashboard is tunneled through CleverThis so no direct network access to the head node is required.

Stopping and restarting a cluster

Click Stop on the cluster row (or in the detail panel) to halt the head node and all worker containers. Stopped clusters cost nothing. Running jobs are interrupted:

• Training jobs are cancelled (a checkpoint is written first, if enabled)
• Serve deployments become unreachable
• Tune trials are interrupted; completed trial results are preserved

Click Start to restart a stopped cluster. The head node restarts; workers scale back up to min_workers within 1–3 minutes. You will need to re-submit training jobs and restart Serve deployments.

Deleting a cluster

Click Delete on the cluster row. This permanently removes the head node and all worker containers. Associated training jobs, Serve deployments, and Tune experiments that reference this cluster are also removed. This cannot be undone.

Head node resource consumption

The head node consumes a small but non-zero amount of resources even when the cluster is idle:

Resource	Typical usage (idle)
CPU	0.5–1.0 cores
Memory	2–4 GB RAM
GPU	None (head node does not run model workloads)

The head node is scheduled on one of your compute cluster's nodes. For pure GPU training, use a CPU node for the head (e.g. cpu-4-ram-16) and GPU nodes for workers to avoid wasting GPU capacity on the coordinator.

Multiple clusters

You can create multiple Ray clusters for different purposes:

• fine-tune-cluster — on a GPU compute cluster; stopped when not fine-tuning
• serve-prod — on a multi-GPU compute cluster; always running for production inference
• dev-cluster — on a single cheap node for testing scripts before submitting to prod

This lets you allocate GPU resources to training when needed and free them up without affecting production inference. A Serve deployment on serve-prod is unaffected when you stop fine-tune-cluster.

Resource requirements

Workload	Minimum worker node spec
Head node only	cpu-2-ram-8 — minimum for coordinator
CPU training / data processing	cpu-8-ram-32
Single-GPU training (7B model)	gpu-a100-40gb
Multi-GPU training (70B model)	Multiple gpu-a100-40gb nodes (tensor parallelism)
GPU inference serving (7B, fp16)	gpu-a100-40gb (14 GB VRAM needed)
Hyperparameter tuning (parallel)	One GPU node per concurrent trial

If your compute cluster has no nodes with sufficient GPU capacity, Ray schedules the job but it sits in pending state until resources become available. The job does not fail — it queues.

Troubleshooting

Cluster stuck in pending

The head node container failed to start. Most common causes:

1. No nodes available — all compute nodes are offline. Check Compute Fleet → Nodes tab.
2. Insufficient memory — head node requires 4 GB free RAM minimum; a node running another workload may not have enough available.
3. Port conflict — another container is using port 6379 or 8265 on the target node. Drain the node and retry.

Workers not scaling up

1. No available compute nodes — max_workers is set higher than available nodes. The autoscaler can only create workers on online nodes in the compute cluster.
2. Node resource exhaustion — the worker container's resource request (CPU/GPU) cannot be satisfied by any available node. Cordon/drain nodes to free up capacity.
3. Image pull failure — the Ray worker image cannot be pulled (network issue or registry auth). Check the head node logs in the cluster detail panel.

Ray Dashboard unreachable

The CleverThis tunnel to the head node's port 8265 has dropped. Click Restart on the cluster. The tunnel reconnects within 30 seconds of the head node restarting.

Jobs immediately fail

The training script or Serve deployment configuration has an error that prevents the Ray job from starting. Check the job detail panel for the failure reason — it usually shows the Python traceback from the job entry point.

High object store memory usage

If the head node shows high memory (>3 GB), the Ray object store (Plasma) may be accumulating unreferenced objects. This is normally handled by Ray's garbage collector but can build up if workers crash mid-task. Restarting the cluster clears the object store.

Worker keeps disconnecting

A worker container that repeatedly connects and disconnects usually means:

• The node it's on is running out of memory (OOM killer restarting the container)
• The node has a GPU driver issue that causes the container to crash

Check journalctl -u cleverthis-agent -n 100 on the affected compute node for crash evidence.