Ray

The Ray section gives you a managed interface for running distributed AI workloads using Ray — the open-source unified compute framework for distributed machine learning. CleverThis integrates Ray so you can fine-tune models, serve them at scale, and optimize hyperparameters without managing Ray infrastructure by hand.

What is Ray?

Ray is a general-purpose distributed computing framework built on a task-based execution model. Every Python function or class can be annotated as a Ray remote task or actor, and Ray's scheduler distributes those tasks across a cluster of workers transparently.

In the context of AI and ML, the Ray ecosystem includes four libraries that CleverThis integrates:

Library	Purpose	CleverThis UI
Ray Core	Distributed task scheduling; the foundation everything runs on	N/A (internal)
Ray Train	Distributed model training (PyTorch DDP, FSDP, DeepSpeed, TF)	Ray → Training
Ray Serve	Scalable model deployment with load balancing and replica management	Ray → Model Serving
Ray Tune	Hyperparameter optimization with parallel trials and smart search algorithms	Ray → Tune

CleverThis wraps these Ray primitives in a UI and API so you can use them without writing cluster provisioning code. Your Ray clusters run on Compute Fleet nodes — the workers are your own hardware.

Architecture overview

CleverThis Control Plane
        │
        ▼ schedules
  Ray Head Node (container on one of your compute nodes)
        │
        ├── Ray Train Job Executor
        ├── Ray Serve Router + Replicas
        └── Ray Tune Trial Manager
              │
              ├── Worker 1 (container on compute node A)
              ├── Worker 2 (container on compute node B)
              └── Worker N (container on compute node C)

The head node is the Ray coordinator. It does not do heavy computation — it schedules work onto workers, manages the distributed object store (Plasma), and hosts the Ray Dashboard UI.

Worker nodes are where training, inference, and tuning computation actually happen. Each worker is a container on one of your registered compute nodes. Workers communicate with the head node over the internal cluster network.

CleverThis routes requests to Ray Serve's built-in HTTP server running on the head node. For training and tuning, jobs are submitted via Ray's job API, also hosted on the head node.

End-to-end workflow: train → tune → serve → integrate

A complete AI pipeline on CleverThis looks like this:

1. Prepare a compute cluster — register one or more GPU servers as a Compute Fleet cluster.

2. Create a Ray cluster — a Ray head node and autoscaling workers on that compute cluster (Ray → Clusters).

3. Run a hyperparameter search — find the best learning rate, batch size, and other configuration for your fine-tuning task (Ray → Tune).

4. Fine-tune a model — submit a training job with the optimal hyperparameters from Tune, producing a checkpoint (Ray → Training).

5. Deploy for inference — create a Ray Serve deployment with the checkpoint (Ray → Model Serving).

6. Integrate into an Actor — reference the Serve deployment's endpoint URL in an Actor YAML and deploy it as a callable namespace endpoint (Actors).

Steps 3–5 iterate: run Tune to find better config, retrain, redeploy.

Ray Dashboard

Navigate to Ray in the sidebar to open the Ray Dashboard. The dashboard shows four summary cards:

Card	What it shows
Clusters	Total clusters across all states; hover to see running vs stopped
Deployments	Active Ray Serve model deployments (status: running, degraded, unhealthy)
Training Jobs	Running, completed, and failed training/fine-tuning jobs
Tune Experiments	Running experiments with trial progress indicators

The dashboard auto-refreshes every 30 seconds. Each card links to its detail page. Click any card's number to navigate directly to that sub-section.

Sub-sections

Page	Description
Clusters	Create and manage Ray clusters backed by your compute nodes
Training	Distributed training and fine-tuning with PyTorch, FSDP, DeepSpeed, LoRA
Model Serving	Deploy models as Ray Serve vLLM endpoints with load balancing
Tune	Hyperparameter optimization with parallel trials and early stopping

Prerequisites

Before using any Ray features:

1. Compute Cluster online — at least one node registered in Compute Fleet with status online
2. Ray Cluster running — a cluster created in Ray → Clusters with status running
3. Model weights available (for serving) — model downloaded to compute node via the model.download command in Compute Fleet, or accessible via network path

Training jobs and model serving deployments both require an existing running Ray cluster. Hyperparameter tuning also runs inside a Ray cluster.

For GPU workloads (fine-tuning, GPU inference), at least one compute node must have:

• NVIDIA driver 525+
• nvidia-container-toolkit installed
• Reported GPU visible to the CleverThis compute agent

Ray version

CleverThis runs Ray 2.9.0 by default. The version is selected at cluster creation time and cannot be changed on a running cluster. All sub-systems (Train, Serve, Tune) are included in this version.

Key features in Ray 2.9:

• Unified streaming in Ray Serve (long-lived connections, SSE)
• Ray Train V2 API with simplified distributed scaling
• Tune ASHA and PBT schedulers for early stopping
• Native vLLM integration in Ray Serve

Relation to Containers

Both Containers (the containers page) and Ray clusters use containerized compute workloads, but they serve different purposes:

	Containers page	Ray (on Compute Fleet)
Hardware	CleverThis managed cloud	Your own registered nodes
Orchestration	Direct container management	Ray's distributed scheduler
Best for	One-off tasks, simple services, quick experiments	Multi-GPU training, parallel HPO, production model serving
Billing	Hourly at CleverThis rates	Your own compute costs

For long-running training on your own GPU hardware, use Ray. For quick one-off scripts or services on managed hardware, use Containers.

Troubleshooting overview

Symptom	Likely cause	Where to look
Ray cluster stays pending	Compute nodes offline or no available capacity	Compute Fleet → Nodes tab
Training job stays queued	All cluster workers are full	Ray → Clusters → worker utilization
Ray Serve deployment unhealthy	Model failed to load (OOM, missing weights)	Deployment detail → replica logs
Tune trials all failed	Training script crash; metric not reported	Tune experiment → trial logs
High latency on Serve endpoint	Single replica overloaded	Scale replicas up
Cannot access Ray Dashboard URL	Network tunnel from head node disconnected	Restart the Ray cluster