User Case
Once you have installed the
sigopt.hyperopt client and have configured your environment for SigOpt, you are ready to seamlessly track your Hyperopt executions as training runs. The tracking is done using Hyperopt’s concept of Trials. Shown below are three examples of how to do this.This mode uploads all Trials to SigOpt in a single API request after the optimization loop completes. There are two main use cases:
- 1.When iterations are evaluated quickly and the time to complete API requests is non-negligible.
- 2.When evaluations must be done offline, and an API request cannot be made within the optimization loop. In this case the
upload_trialsfunction will need to be called from a machine with the ability to execute HTTPS requests.
import time
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials
from sigopt.hyperopt import upload_trials
def objective(args):
x, = args
return {
"loss": x ** 2,
"status": STATUS_OK,
"eval_time": time.time(),
"other_stuff": {"type": None, "value": [0, 1, 2]},
}
trials = Trials()
space = (hp.uniform("x", -10, 10),)
best = fmin(objective, space=space, algo=tpe.suggest, max_evals=1, trials=trials)
upload_trials(project="project-name", trials=trials)
This mode uploads each trial immediately after the objective evaluation. SigOptTrials replaces the Hyperopt Trials object in this integration.
import time
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials
from sigopt.hyperopt import SigOptTrials
def objective(args):
x, = args
return {
"loss": x ** 2,
"status": STATUS_OK,
"eval_time": time.time(),
"other_stuff": {"type": None, "value": [0, 1, 2]},
}
space = (hp.uniform("x", -10, 10),)
trials = SigOptTrials(project="project-name")
best = fmin(objective, space=space, algo=tpe.suggest, max_evals=1, trials=trials)
All Hyperopt algorithms can be parallelized using either MongoTrials or SparkTrials. Both alternatives to Hyperopt’s base Trials can be wrapped with the SigOptTrials object. Here is the example from Hyperopt’s documentation reworked to log results contained in MongoTrials to SigOpt.
import time
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials
from sigopt.hyperopt import SigOptTrial
from hyperopt.mongoexp import MongoTrials
...
trials = MongoTrials("mongo://localhost:1234/foo_db/jobs", exp_key="exp1")
trials = SigOptTrials(project="project-name", trials=trials)
best = fmin(objective, space=space, algo=tpe.suggest, max_evals=1, trials=trials)
Limitations: Only trial status, numerical values in trial result, and parameters of trial are saved in SigOpt.
Last modified 7mo ago