Optimization Experiments
Optimize Your Model
A key component of the SigOpt Platform is the ability to go from tracking your model with SigOpt Runs to optimizing that very same model with minimal changes to your code.
At a high level, a SigOpt Experiment is a grouping of SigOpt Runs and is defined by user-defined parameter and metric spaces. A SigOpt Experiment has a budget that is used to determine the number of hyperparameter tuning loops to conduct. Each hyperparameter loop produces a SigOpt Run with suggested assignments for each parameter. Different sets of hyperparameter values are suggested by either SigOpt algorithms and/or the user with the goal of finding the optimal set(s) of hyperparameters. Overtime, when using the SigOpt Optimizer, you can expect your model to perform better on your metrics.
The Optimization Loop
There are 3 core steps in the optimization loop:
Create a SigOpt Experiment
1
experiment = sigopt.create_experiment(
2
name="Keras Model Optimization (Python)",
3
type="offline",
4
parameters=[
5
dict(name="hidden_layer_size", type="int", bounds=dict(min=32, max=128)),
6
dict(name="activation_function", type="categorical", categorical_values=["relu", "tanh"]),
7
],
8
metrics=[dict(name="holdout_accuracy", objective="maximize")],
9
parallel_bandwidth=1,
10
budget=30,
11
)
Copied!
Iterate over your Experiment
You can iterate over your Experiment in 2 ways. Each optimization loop produces a SigOpt Run with suggested assignments for each parameter.
1
for run in experiment.loop():
2
with run:
3
# execute model
4
# evaluate model
5
# report metric values to SigOpt
Copied!
1
while not experiment.is_finished():
2
with experiment.create_run() as run:
3
# execute model
4
# evaluate model
5
# report metric values to SigOpt
Copied!
Report metric values to SigOpt
1
run.log_metric("holdout_accuracy", holdout_accuracy)
Copied!
Putting It All Together
1
experiment = sigopt.create_experiment(
2
name="Keras Model Optimization (Python)",
3
type="offline",
4
parameters=[
5
dict(name="hidden_layer_size", type="int", bounds=dict(min=32, max=128)),
6
dict(name="activation_function", type="categorical", categorical_values=["relu", "tanh"]),
7
],
8
metrics=[dict(name="holdout_accuracy", objective="maximize")],
9
parallel_bandwidth=1,
10
budget=30,
11
)
12
​
13
for run in experiment.loop():
14
with run:
15
holdout_accuracy = execute_keras_model(run)
16
run.log_metric("holdout_accuracy", holdout_accuracy)
17
​
18
# get the best Runs for the Experiment
19
best_runs = experiment.get_best_runs()
Copied!
The rest of the Experiment docs will walk through how to set up your SigOpt Experiment, how you can bring your own optimizer, and advanced features you can leverage for your optimization.
Last modified 17d ago