Hyperparameter Tuning

The Ray → Tune page lets you run automated hyperparameter optimization (HPO) experiments using Ray Tune. Tune searches across a parameter space in parallel, using algorithms like Bayesian optimization and early stopping schedulers to find configurations that maximize a target metric efficiently.

What is Ray Tune?

Ray Tune automates the trial-and-error process of finding good hyperparameters. Instead of manually testing learning rates and batch sizes one at a time, Tune runs many trials in parallel — each trial trains with a different configuration, measures a metric (e.g. validation loss), and reports back. The search algorithm uses those results to propose better values for the next batch of trials.

A trial is one training run with a specific set of hyperparameter values. Tune manages the lifecycle of all trials: submission, monitoring, early termination of underperforming trials, and result collection.

This is especially valuable when:

• Training runs are expensive (30+ minutes per trial on GPU)
• You're adapting a model to a new domain and don't have good hyperparameter intuition
• You want to maximize model quality within a GPU-budget constraint

Typical HPO workflow

1. Start with a small Ray cluster (max_workers = 2) and few samples (num_samples = 5) to validate that your training script works and reports metrics correctly.
2. Expand to more workers and more samples once you have confirmed the script is healthy.
3. Use the best trial's parameters in a full Training Job.
4. Deploy the resulting checkpoint as a Serve deployment.

Creating a tuning experiment

1. Navigate to Ray → Tune.

2. Click Create Experiment.

3. Fill in the form:

   Field	Default	Description
   Name	—	Experiment name (e.g. llama-ft-lr-search-v1)
   Ray Cluster	—	Which Ray cluster to submit trials to; must be running
   Search Algorithm	bayesian	Algorithm for proposing new parameter combinations
   Scheduler	none	Early stopping strategy (see schedulers below)
   Metric	accuracy	The metric each trial reports; used to compare trials
   Mode	max	Whether to maximize (max) or minimize (min) the metric
   Samples	10	Total number of trials to run
   Max concurrent trials	(cluster workers)	How many trials run simultaneously
   Search Space	JSON	Parameter names and their distributions (see below)

4. Click Create. The experiment begins submitting trials to the cluster.

Search algorithms

Algorithm	When to use
bayesian	Default. Gaussian Process surrogate model. Converges faster than random for expensive trials. Best when you have 20+ budget and most parameters are continuous.
random	Pure random sampling. Use as a baseline; also good when budget < 10 trials (Bayesian can't learn enough).
grid	Exhaustive grid search over all choice values. Only practical for tiny spaces (< 1,000 combinations). Ignores num_samples.
hyperopt	Tree-structured Parzen Estimator (TPE). Often outperforms Bayesian on mixed discrete/continuous spaces. Good default alternative to Bayesian.
optuna	Multi-objective optimization (NSGA-II). Use when you want to jointly optimize accuracy AND latency, or quality AND training cost.
ax	Adaptive experimentation (Bayesian with bandit optimization). Best for multi-fidelity searches (early stopping + full runs).

Schedulers (early stopping)

A scheduler observes all running trials and can terminate trials that are clearly not going to reach the top performance — freeing those resources for better-looking trials. Schedulers are orthogonal to search algorithms; you can combine any scheduler with any search algorithm.

Scheduler	How it works	When to use
none	No early stopping. Every trial runs for the full number of epochs.	When trials are short (< 10 min) or you can't afford to miss any configuration
asha	ASHA (Asynchronous Successive Halving Algorithm). At each rung, the bottom 50% of trials by metric are stopped.	Most common choice. GPU-intensive training where early stopping saves significant cost
pbt	Population-Based Training. Running trials periodically copy parameters from better-performing trials (exploitation) and add noise (exploration).	Long training runs (hours per trial); especially effective for fine-tuning
median_stopping	Stop a trial if its metric is below the median of all completed trials at the same step.	Simple, interpretable baseline early stopping

ASHA details

ASHA is the recommended scheduler for most HPO tasks. It works in "rungs":

All trials start training
    │
    After 1/R of training:
    ├── Keep top 1/η by metric
    └── Stop bottom (1 - 1/η)
         │
         After 2/R of training:
         ├── Keep top 1/η²
         └── Stop more...
              │
              After full training:
              └── Champion trial determined

With the default η = 4 and 10 samples, only ~2 trials complete the full run; the rest are stopped early. This saves ~75% of compute compared to training all trials to completion.

PBT details

PBT maintains a "population" of trials that run concurrently. Every N steps, the bottom 25% of trials by metric:

1. Copy the model checkpoint from a top-25% trial (exploitation)
2. Perturb some hyperparameters by ±20% (exploration)
3. Continue training from the copied checkpoint

This allows hyperparameters to evolve during a single experiment rather than being fixed at the start. PBT is especially powerful for fine-tuning tasks where the optimal learning rate schedule changes as training progresses.

Defining the search space

The Search Space field accepts a JSON object where each key is a hyperparameter name and each value defines the sampling distribution.

Basic example (learning rate + batch size)

{
  "lr": {
    "type": "loguniform",
    "min": 1e-5,
    "max": 1e-3
  },
  "batch_size": {
    "type": "choice",
    "values": [8, 16, 32, 64]
  }
}

All supported distribution types

Type	JSON shape	When to use
loguniform	{"type":"loguniform","min":X,"max":Y}	Learning rates, weight decay, regularization (spans orders of magnitude)
uniform	{"type":"uniform","min":X,"max":Y}	Dropout rate, gradient clipping, warmup fraction (linear range)
choice	{"type":"choice","values":[...]}	Batch size, optimizer type, architecture variants (discrete options)
randint	{"type":"randint","min":X,"max":Y}	Number of layers, attention heads (integer range)
quniform	{"type":"quniform","min":X,"max":Y,"q":Z}	Any continuous value that must be rounded to a step (e.g. q=64 for hidden dimensions)
qloguniform	{"type":"qloguniform","min":X,"max":Y,"q":Z}	Quantized log-uniform (e.g. context lengths that must be powers of 2)
normal	{"type":"normal","mean":X,"std":Y}	When you have a prior belief about the optimal value (Gaussian centered there)

Comprehensive fine-tuning search space

{
  "lr": {
    "type": "loguniform",
    "min": 5e-6,
    "max": 5e-4
  },
  "weight_decay": {
    "type": "loguniform",
    "min": 1e-5,
    "max": 1e-1
  },
  "warmup_ratio": {
    "type": "uniform",
    "min": 0.0,
    "max": 0.2
  },
  "batch_size": {
    "type": "choice",
    "values": [8, 16, 32]
  },
  "lora_r": {
    "type": "choice",
    "values": [4, 8, 16, 32]
  },
  "lora_alpha": {
    "type": "choice",
    "values": [16, 32, 64]
  },
  "gradient_clip": {
    "type": "uniform",
    "min": 0.5,
    "max": 2.0
  }
}

Conditional search spaces

Some hyperparameters only matter when another takes a specific value. Use a training_config.json in your dataset path to handle these; the search space JSON passes values through to the training script, which can implement conditional logic:

# In your training script (called by CleverThis training runtime):
import ray.train
config = ray.train.get_context().get_trial_config()

lora_r = config["lora_r"]
# lora_alpha should typically be 2x lora_r
lora_alpha = lora_r * 2   # conditional: derived from lora_r

Metric and mode

The Metric field must exactly match the key your training script reports to Tune. If your script uses ray.train.report({"val_loss": 0.45}), set Metric to val_loss.

Common metric/mode combinations:

Task	Metric	Mode	Notes
Classification fine-tuning	accuracy	max
Regression / LM	val_loss	min	Lower is better
Text generation quality	bleu or rouge_l	max	Needs eval set
Token efficiency	perplexity	min	Lower perplexity = better language model
Retrieval	ndcg@10	max

Multi-metric experiments

When using optuna scheduler, specify multiple metrics:

{
  "metric": ["accuracy", "latency_ms"],
  "mode": ["max", "min"]
}

The result shows a Pareto frontier — the set of trials where no other trial is better on all metrics simultaneously. You choose the trade-off point based on your requirements.

Reporting metrics from your training script

The training script must call ray.train.report() at each reporting point (epoch end or evaluation interval):

import ray.train

def train_func(config):
    model = load_model(config["model_path"])
    optimizer = AdamW(model.parameters(), lr=config["lr"],
                      weight_decay=config["weight_decay"])

    for epoch in range(config["epochs"]):
        train_loss = train_one_epoch(model, optimizer, train_data, config)
        val_loss, accuracy = evaluate(model, val_data)

        # Report metrics — Tune reads these to compare trials
        ray.train.report({
            "epoch": epoch,
            "train_loss": train_loss,
            "val_loss": val_loss,
            "accuracy": accuracy,
        })

Metrics are reported at every epoch; Tune can early-stop based on trends across epochs, not just the final value.

Number of samples

Samples is the total trial budget. Choose based on:

• How expensive each trial is (compute × time)
• How large the search space is
• Which search algorithm you're using

Budget guidance	Samples
Quick exploration (validate script + range)	5–10
Reasonable search (Bayesian, ~4 concurrent)	20–50
Thorough search (large space, PBT)	50–200
Production HPO (many parameters, high stakes)	100+

With asha or pbt schedulers, effective coverage is much higher than num_samples suggests, because poorly-performing trials are stopped early.

Concurrent trials

By default, Tune runs as many concurrent trials as there are available worker slots in the Ray cluster. With a cluster of max_workers = 4 GPU nodes and each trial needing 1 GPU, up to 4 trials run simultaneously.

To limit concurrent trials (e.g. to leave capacity for other workloads), set Max concurrent trials in the experiment form.

Throughput examples

Setup	Concurrent trials	Time for 20 trials (2 hr each, ASHA ~75% savings)
1 GPU node	1	~40 hrs (20 trials × 2 hrs × 25% survival rate)
4 GPU nodes	4	~10 hrs
8 GPU nodes	8	~5 hrs

With ASHA at 75% savings rate, many trials terminate after 30 minutes rather than 2 hours, so effective throughput is much higher than the above suggests.

Viewing results

Click an experiment row to open the detail panel:

Summary tab

• Best trial — the trial configuration that achieved the best metric value
• Progress — X/Y trials completed, currently running count, stopped early count
• Status — running, completed, or stopped

Trials table

Each row represents one trial:

Column	Description
Trial ID	Unique identifier for this trial
Status	running, completed, failed, stopped_early
Hyperparameters	The specific values Tune assigned to this trial
Best metric	The best metric value this trial achieved at any reporting step
Final metric	The metric at the last reporting step
Duration	How long this trial ran

Click a trial row to expand it and see:

• The full metric history across all reporting steps (as a sparkline chart)
• The config values that were assigned
• A link to the trial's training logs

Best configuration

The trial with the highest (or lowest) metric is marked Best in the trials table. The full configuration is displayed in the Summary tab under "Best trial config":

{
  "lr": 0.000048,
  "weight_decay": 0.0023,
  "batch_size": 16,
  "lora_r": 16,
  "lora_alpha": 32
}

Use these values in a Training Job for the full production run.

Ray Tune Dashboard

Click Open Ray Tune Dashboard to access the live Tune dashboard on the Ray head node. It shows:

• Real-time metric plots for all running trials
• Resource utilization per trial
• Trial status timeline (when each trial started and what happened)

Resuming a stopped experiment

If an experiment is stopped mid-run (cluster restart, manual stop), you can resume it:

1. Click the stopped experiment in the list.
2. Click Resume.
3. Tune reloads the checkpoints of completed trials and continues scheduling new trials from where it left off.

Resumption requires that trial checkpoints are available on the compute nodes. Checkpoints are written at each ray.train.report() call, so trials are resumable from any epoch boundary.

If the compute cluster was deleted (nodes gone), checkpoints may be lost. In that case, restart the experiment — Bayesian and Hyperopt algorithms re-use their surrogate model from previous observations even if trial checkpoints are lost (they re-run from scratch with better-informed proposals).

Early stopping in practice

With ASHA enabled (scheduler = asha) and num_samples = 20:

Trial   lr       batch   val_loss(ep1)  val_loss(ep2)  val_loss(final)  Status
T1      1e-4     16      0.62           0.51           0.44             ✓ completed
T2      5e-6     32      0.88           0.87           —                ✗ stopped (ep2)
T3      2e-4     8       0.71           0.65           0.59             ✓ completed
T4      1e-3     64      1.2            —              —                ✗ stopped (ep1)
T5      3e-5     16      0.65           0.58           0.52             ✓ completed
...

After 20 trials, ~5 completed to the end; the rest stopped early. The savings: ~15 trials × average 1.5 hr saved = 22.5 GPU-hours saved vs running all to completion.

Cost estimation

total_cost = num_samples × avg_trial_duration × gpu_cost_per_hour × (1 - asha_savings)

Example: 20 trials, 2 hr average, A100 at $5/hr, ASHA (75% savings):

= 20 × 2 × $5 × 0.25 = $50

Without ASHA: 20 × 2 × $5 = $200.

Cost management tips:

• Start with 5 trials and 1 epoch each to validate the pipeline before scaling
• Use min_workers = 0 in the cluster so idle workers shut down between runs
• Use QLoRA instead of full fine-tuning to reduce per-trial GPU cost
• Set conservative max_concurrent when sharing the cluster with production Serve deployments

Troubleshooting

All trials immediately failed — The training script has a crash in its first step. Check the trial logs for a Python traceback. Common causes: missing imports, wrong dataset path, or syntax error in the script.

Metric not found / trials show N/A — The training script is not calling ray.train.report() with the correct metric key. Verify the key in report() exactly matches the Metric field.

Trials queue but don't start — All cluster workers are busy with other trials or other workloads. Either wait, reduce max_concurrent, or scale up the cluster's max_workers.

ASHA not stopping underperforming trials — ASHA needs at least 4+ trials at each rung to make stopping decisions. With num_samples < 10 and ASHA, stopping may not happen. Use none or median_stopping for small budgets.

PBT producing worse results than random — PBT works best with many long trials (> 30 minutes each) and moderate concurrency. For short trials or small budgets, bayesian or hyperopt with asha performs better.