User Case

Use Cases: Batch Upload, Real Time Upload and Parallel Trials

Once you have installed the sigopt.hyperopt client and have configured your environment for SigOpt, you are ready to seamlessly track your Hyperopt executions as training runs. The tracking is done using Hyperopt’s concept of Trials. Shown below are three examples of how to do this.

Batch Upload Trials

This mode uploads all Trials to SigOpt in a single API request after the optimization loop completes. There are two main use cases:

1. When iterations are evaluated quickly and the time to complete API requests is non-negligible.

2. When evaluations must be done offline, and an API request cannot be made within the optimization loop. In this case the upload_trials function will need to be called from a machine with the ability to execute HTTPS requests.

import time
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials
from sigopt.hyperopt import upload_trials


def objective(args):
  x, = args
  return {
    "loss": x ** 2,
    "status": STATUS_OK,
    "eval_time": time.time(),
    "other_stuff": {"type": None, "value": [0, 1, 2]},
  }


trials = Trials()
space = (hp.uniform("x", -10, 10),)
best = fmin(objective, space=space, algo=tpe.suggest, max_evals=1, trials=trials)
upload_trials(project="project-name", trials=trials)

Realtime Upload Trials

This mode uploads each trial immediately after the objective evaluation. SigOptTrials replaces the Hyperopt Trials object in this integration.

import time
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials
from sigopt.hyperopt import SigOptTrials


def objective(args):
  x, = args
  return {
    "loss": x ** 2,
    "status": STATUS_OK,
    "eval_time": time.time(),
    "other_stuff": {"type": None, "value": [0, 1, 2]},
  }


space = (hp.uniform("x", -10, 10),)
trials = SigOptTrials(project="project-name")
best = fmin(objective, space=space, algo=tpe.suggest, max_evals=1, trials=trials)

Wrapped Trials

All Hyperopt algorithms can be parallelized using either MongoTrials or SparkTrials. Both alternatives to Hyperopt’s base Trials can be wrapped with the SigOptTrials object. Here is the example from Hyperopt’s documentation reworked to log results contained in MongoTrials to SigOpt.

import time
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials
from sigopt.hyperopt import SigOptTrial
from hyperopt.mongoexp import MongoTrials

...
trials = MongoTrials("mongo://localhost:1234/foo_db/jobs", exp_key="exp1")
trials = SigOptTrials(project="project-name", trials=trials)
best = fmin(objective, space=space, algo=tpe.suggest, max_evals=1, trials=trials)

Limitations: Only trial status, numerical values in trial result, and parameters of trial are saved in SigOpt.