Training Jobs

The Ray → Training page lets you launch distributed model training and fine-tuning jobs on your Ray clusters. Jobs run on the compute nodes in the cluster, using PyTorch or TensorFlow as the deep learning backend, and are monitored from the CleverThis UI.

Training vs fine-tuning

CleverThis surfaces two flavors of training jobs from a single page:

Type	What it does	When to use
Training	Train a model from scratch or continue pre-training from a checkpoint	Building domain-specific models; continued pre-training on domain data
Fine-tuning	Adapt a pretrained base model on a smaller labeled dataset	Instruction following, task-specific behavior, style adaptation

In practice, fine-tuning is the far more common operation — you take a powerful open- source base model like Llama 2 and teach it to behave differently on your data. Both types use the same Ray distributed training API internally.

Creating a training job

1. Navigate to Ray → Training.
2. Click Create Job.
3. The dialog has two tabs — Training and Fine-tuning. Select the appropriate tab.

Job configuration fields

Field	Default	Description
Name	—	Unique job name (e.g. llama2-customer-support-v1)
Ray Cluster	—	Which Ray cluster to submit the job to; must be running
Base Model	meta-llama/Llama-2-7b-hf	HuggingFace model ID or local path
Dataset Path	/datasets/training	Path on compute nodes where training data lives
Job Type	—	train (from scratch) or finetune
Framework	torch	Training framework: torch (PyTorch) or tensorflow
Epochs	3	Number of passes over the full training dataset
Batch Size	32	Per-worker micro-batch size

4. Click Create. The job is submitted to the Ray cluster's job API and begins running once workers have capacity.

Job lifecycle

Status	Meaning
queued	Job submitted; waiting for Ray cluster workers to become available
initializing	Workers are loading the base model into memory/VRAM
running	Active training; loss is decreasing (or should be)
succeeded	Training completed; final checkpoint is written to Dataset Path/../checkpoints/
failed	Training crashed — see job detail panel for the Python traceback
cancelled	User stopped the job; checkpoint written at the most recent epoch boundary

Click a job row to open the detail panel, which shows the current status, configuration, elapsed time, and a link to the live Ray job log stream on the head node.

Base model selection

The Base Model field accepts any HuggingFace Hub model ID:

meta-llama/Llama-2-7b-hf         # Llama 2 7B base — best starting point for most fine-tunes
meta-llama/Llama-2-13b-hf        # Llama 2 13B — better quality, needs 28 GB VRAM
meta-llama/Meta-Llama-3-8B        # Llama 3 8B — stronger base than Llama 2 7B
mistralai/Mistral-7B-v0.1        # Mistral 7B — strong alternative to Llama 2 7B
mistralai/Mixtral-8x7B-v0.1      # MoE 7B×8 — 45B params, runs like ~12B; needs large VRAM
google/gemma-7b                  # Gemma 7B — Google's open model
tiiuae/falcon-40b                 # Falcon 40B — requires multi-GPU node

For fine-tuning, the model is downloaded to the compute node's model cache before training begins. Use the model.download command in Compute Fleet to pre-fetch large models. Pre-fetching saves time: a 7B model download takes 10–20 minutes; if the weights are already cached the job starts in seconds.

You can also pass a local path (e.g. /models/my-checkpoint) if you have copied weights to the compute node's filesystem.

Dataset preparation

Training data must be on the compute node at the path in Dataset Path. All worker nodes must be able to read the data from the same path.

JSONL format (recommended)

JSONL with one JSON object per line:

{"instruction": "Translate to French", "input": "Good morning", "output": "Bonjour"}
{"instruction": "Summarize", "input": "The cat sat on the mat.", "output": "A cat was on a mat."}

Or using the ChatML/conversation format for instruction fine-tuning:

{"messages": [{"role": "user", "content": "What is 2+2?"}, {"role": "assistant", "content": "4"}]}
{"messages": [{"role": "system", "content": "Be concise."}, {"role": "user", "content": "Summarize AI."}, {"role": "assistant", "content": "AI is machine intelligence."}]}

The CleverThis training runtime auto-detects which format you are using based on the presence of messages vs instruction/output keys.

HuggingFace Arrow format

Pre-tokenized HuggingFace datasets.Dataset saved in Arrow format are also supported:

from datasets import Dataset
data = Dataset.from_list(my_examples)
data.save_to_disk("/datasets/training/my-dataset")

Dataset access for distributed training

With multiple worker nodes, every worker needs to read the dataset. Options:

Approach	Setup	Pros/Cons
Shared NFS/EFS mount	Mount the same NFS at the same path on all nodes	Works transparently; requires NFS server
Copy to each node	Pre-copy with rsync or scp before the job	Simple but manual; data drift risk
Object storage (s3://)	Set dataset path to s3://bucket/dataset/	Scales to any size; requires IAM/credentials on nodes
gs:// (Google Cloud Storage)	Same as S3	Requires service account credentials on nodes

For development, copying to each node is the easiest. For production, mount a shared NFS or use object storage.

Dataset size recommendations

Fine-tuning task	Recommended dataset size
Instruction following (general)	10,000–100,000 examples
Domain adaptation	1,000–10,000 examples (quality > quantity)
Task-specific (classification, extraction)	500–5,000 labeled examples
Full language model pre-training	Billions of tokens

Fine-tuning is surprisingly data-efficient. 1,000 high-quality examples often outperform 10,000 low-quality ones. Focus on example quality first.

Distributed training strategies

Ray Train distributes work across workers using several parallelism strategies. The CleverThis training runtime selects the appropriate strategy based on model size and available workers:

Data Parallel (DDP / FSDP)

The most common strategy. Each worker holds a copy of the model; the batch is split across workers; gradients are averaged after each step.

Worker 1: batch_slice_1 → gradient_1 ─┐
Worker 2: batch_slice_2 → gradient_2 ─┤─► averaged gradient → all workers update
Worker N: batch_slice_N → gradient_N ─┘

• DDP (DistributedDataParallel) — each worker has a full model copy in VRAM. Works for models that fit in a single GPU (7B in fp16 = 14 GB, fits A100-40GB).
• FSDP (Fully Sharded Data Parallel) — shards model parameters across workers. Use this for models too large for a single GPU (e.g. 70B). Ray Train uses PyTorch FSDP automatically when the model doesn't fit on one GPU.

Pipeline Parallel

For very large models, layers are split across workers (not shards of the same layers). Worker 1 runs layers 1–8, Worker 2 runs layers 9–16, etc. Currently configured via advanced training script options; not exposed as a UI field.

Tensor Parallel

For the largest models (70B+), individual weight matrices are split across GPUs. Typically combined with FSDP. Requires specific model support (Llama and Falcon support this via HuggingFace Accelerate).

Fine-tuning methods

Full fine-tuning

All model parameters are updated during training. Most accurate but requires the most VRAM (same as serving the model in fp16 plus optimizer states — typically 3–4× serving VRAM).

Model	VRAM for full fine-tune
7B (fp16)	~60 GB (model + optimizer + gradients)
13B (fp16)	~120 GB
70B (fp16)	~600 GB (requires multiple A100-80GB)

LoRA / QLoRA (recommended)

LoRA (Low-Rank Adaptation) freezes the base model and trains only a small set of adapter parameters (typically 1–5% of total parameters). This massively reduces VRAM and compute requirements with competitive quality.

QLoRA combines LoRA with 4-bit quantization of the frozen base model, further reducing VRAM by ~75% compared to full fine-tuning.

Method	VRAM for 7B	Quality
Full fine-tune (fp16)	~60 GB	Best
LoRA (fp16 base)	~18 GB	Excellent
QLoRA (4-bit base + LoRA)	~8 GB	Very good

The CleverThis training runtime applies LoRA automatically when the available VRAM is insufficient for full fine-tuning. To explicitly request LoRA or QLoRA, include a training_config.json in your dataset path:

{
  "method": "qlora",
  "lora_r": 16,
  "lora_alpha": 32,
  "lora_dropout": 0.05,
  "target_modules": ["q_proj", "v_proj"],
  "quantization": "4bit"
}

Config key	Description
method	full, lora, or qlora
lora_r	Rank of the LoRA matrices (4–64; higher = more capacity but more VRAM)
lora_alpha	Scaling factor (typically 2 × lora_r)
lora_dropout	Dropout on LoRA layers (0.0–0.1)
target_modules	Which attention projections to adapt (q_proj, v_proj is standard)
quantization	4bit or 8bit for the frozen base model (QLoRA only)

Mixed precision and DeepSpeed

Mixed precision (fp16/bf16)

Training in 16-bit precision instead of 32-bit halves VRAM usage with minimal quality loss. The CleverThis runtime defaults to bf16 on A100s (which natively support bf16) and fp16 on older GPUs.

To explicitly configure mixed precision, add to training_config.json:

{
  "mixed_precision": "bf16"
}

bf16 is generally preferred for stability (wider dynamic range than fp16).

DeepSpeed integration

For large-scale training across many GPUs, the CleverThis runtime integrates DeepSpeed, which provides ZeRO (Zero Redundancy Optimizer) stages:

ZeRO Stage	What is sharded	When to use
Stage 1	Optimizer states	2–4 GPUs; mild memory savings
Stage 2	Optimizer states + gradients	4–8 GPUs; significant savings
Stage 3	Optimizer states + gradients + model params	8+ GPUs; maximum sharding

To enable DeepSpeed, add to training_config.json:

{
  "deepspeed": {
    "zero_stage": 2,
    "offload_optimizer": false,
    "fp16": true
  }
}

DeepSpeed Stage 3 with CPU offloading can fit a 70B model on as few as 2 A100 GPUs by offloading parameters to CPU memory when not in use. This trades throughput for accessibility.

Gradient checkpointing

Gradient checkpointing reduces VRAM usage during backpropagation by recomputing activations during the backward pass instead of storing them. This roughly halves activation memory at the cost of ~20% slower training.

Enable it in training_config.json:

{
  "gradient_checkpointing": true
}

Gradient checkpointing is automatically enabled when the runtime detects available VRAM is within 20% of the model's weight size.

Monitoring training progress

Live logs

Click the job row → View Logs. Logs stream from the Ray head node in real time, including:

• Training loss per batch
• Validation loss per epoch (if validation split configured)
• GPU utilization per worker
• Tokens per second (throughput)
• Estimated time remaining

Metrics tracking

The CleverThis training runtime reports metrics to the Ray job dashboard on the head node. Click Open Ray Dashboard in the cluster detail panel to see:

• Per-worker resource utilization (CPU, GPU, memory)
• Task graph (which operations are running on which workers)
• Object store usage (how much data is in Ray's shared memory)

Loss curve interpretation

Pattern	Meaning
Loss decreasing smoothly	Training is working correctly
Loss oscillating without decrease	Learning rate too high; reduce by 10×
Loss decreasing then spiking	Gradient instability; enable gradient clipping
Loss near zero very quickly	Overfitting or data leakage; check dataset quality
Loss not decreasing at all	Learning rate too low, or model not receiving gradients

Checkpoint strategy

The CleverThis training runtime writes checkpoints at the end of each epoch into a checkpoints/ subdirectory of the dataset path. Checkpoint filenames are:

/datasets/training/checkpoints/
  epoch-1/       # After epoch 1
  epoch-2/       # After epoch 2
  epoch-final/   # After the last epoch (symlinked to the last epoch folder)
  best/          # Checkpoint with lowest validation loss (if validation set configured)

Each checkpoint directory contains the model weights in HuggingFace safetensors format plus the tokenizer configuration, making it directly loadable:

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("/datasets/training/checkpoints/epoch-final")
tokenizer = AutoTokenizer.from_pretrained("/datasets/training/checkpoints/epoch-final")

Best checkpoint selection: if you configure a validation set and the runtime tracks validation loss per epoch, the checkpoint with the lowest validation loss is tagged as best/ in addition to its epoch folder.

To save only the LoRA adapter weights (much smaller than full model), the runtime writes them to:

/datasets/training/checkpoints/epoch-final/adapter_model/

Load a LoRA adapter separately:

from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
model = PeftModel.from_pretrained(base, "/datasets/training/checkpoints/epoch-final/adapter_model")

Using a checkpoint for inference

After a successful training job, use the checkpoint in a Ray Serve deployment:

1. Note the checkpoint path from the job detail panel.
2. Navigate to Ray → Model Serving → Create Deployment.
3. Set Model Path to the checkpoint path (e.g. /datasets/training/checkpoints/epoch-final).
4. Set Model Name to a descriptive identifier (e.g. my-company/customer-support-v1).
5. vLLM will load the HuggingFace checkpoint from that path.

For LoRA adapters, vLLM supports loading the base model + adapter:

model_path: /models/meta-llama/Llama-2-7b-hf
lora_adapter_path: /datasets/training/checkpoints/epoch-final/adapter_model

(Configure via training_config.json in the dataset path.)

Framework support

Framework	Notes
torch	PyTorch with Ray Train TorchTrainer. Supports DDP, FSDP, DeepSpeed, LoRA, QLoRA. Default and recommended.
tensorflow	TensorFlow with Ray Train TensorflowTrainer. Suitable for TF-based models. Does not support LoRA or DeepSpeed.

Cancelling and resuming jobs

Cancelling

Click Cancel on a running job row. The job receives a stop signal; the runtime writes a checkpoint at the current epoch boundary before terminating. The checkpoint is intact and can be used for inference or as the starting point for a new job.

Resuming from checkpoint

To resume a cancelled or failed job from its last checkpoint:

1. Create a new job.
2. Set Base Model to the checkpoint path (e.g. /datasets/training/checkpoints/epoch-2).
3. Reduce Epochs to the remaining number.
4. Click Create. The runtime detects the checkpoint format and resumes from it rather than downloading the base model again.

Troubleshooting

Job stays queued indefinitely — No workers have the required GPU capacity. Check the cluster's Nodes tab; if all workers are occupied, wait for them to finish or increase max_workers.

Job fails with CUDA out of memory — The model + optimizer + activations exceed available VRAM. Try:

1. Enable QLoRA (method: qlora in training_config.json)
2. Reduce batch size
3. Enable gradient checkpointing
4. Use DeepSpeed ZeRO Stage 2 or 3

Slow training throughput — Common causes:

• Data loading bottleneck: dataset path is on slow storage (NFS with high latency)
• Single-GPU: not utilizing data parallelism; ensure cluster has multiple workers
• Batch size too small: increase batch size to keep GPU utilization above 80%

Loss not decreasing — Common causes:

• Learning rate too low (try 1e-4 to 2e-5 for fine-tuning)
• Dataset format not parsed correctly (check logs for tokenization warnings)
• Base model frozen unintentionally (check method in training config)

Job shows initializing for more than 10 minutes — The base model is still being downloaded or loaded into VRAM. Very large models (70B) can take 10+ minutes to load. For repeated jobs on the same cluster, pre-download using the model.download Compute Fleet command.