Running an Experiment on several machines at once is easy and natural with the SigOpt API. Before you start running experiments in parallel, make sure you know how to create and run an experiment.
Create an Experiment
Create your experiment on a controller machine; you only need to perform this step once. Be sure to make a note of your experiment's id because you'll need it in subsequent steps.
Then set the parallel_bandwidth field as the number of parallel workers you are using in your experiment. This field ensures SigOpt can factor your level of parallelism into account and provide better suggestions.
Initialize the Workers
Initialize each of your workers with the EXPERIMENT_ID from the experiment that you just created. All workers, whether individual threads or machines, will receive the same experiment ID.
Run SigOpt Optimization Experiments in Parallel
Now, start the optimization loop on each worker machine. Workers will individually communicate with SigOpt's API, creating suggestions, and evaluating your metric.
Why This Works
A large benefit of SigOpt's parallelization is that each worker communicates asynchronously with the SigOpt API, so you do not need to worry about task management.
SigOpt acts as a distributed scheduler for your SigOpt suggestions, ensuring that each worker machine receives parameter assignments at the moment it asks for a new parameter configuration. SigOpt tracks which Suggestions are currently open, so machines independently creating Suggestions will not receive duplicates.
Example Setups
In these examples each machine needs to be configured with API Token.
Core Module
These code snippets provide an example combine suggested controller/worker division of labor, as well as incorporating metadata to track which machines have reported Observations.
Controller: Create Experiment, Spin Up Workers
from sigopt import Connection
def controller(api_token, num_workers=1):
# Create the SigOpt connection
conn = Connection(client_token=api_token)
# Create the experiment on master
experiment = conn.experiments().create(
name="Classifier Accuracy",
parameters=[
{
'bounds': {
'max': 1.0,
'min': 0.001
},
'name': 'gamma',
'type': 'double'
}
],
metrics=[dict(name='Accuracy', objective='maximize')],
observation_budget=20,
parallel_bandwidth=num_workers,
)
for _ in range(num_workers):
# Launch a worker and run the run_worker
# function (below) on the worker machine
# You implement this function
run_worker(
api_token=api_token,
experiment_id=experiment.id,
)
Worker: Run Optimization Loop with Metadata
import socket
from sigopt import Connection
# Each worker runs the same optimization loop
# for the experiment created on master
def run_worker(api_token, experiment_id):
# Create the SigOpt connection
conn = Connection(client_token=args.api_token)
# Keep track of the hostname for logging purposes
hostname = socket.gethostname()
experiment = conn.experiments(experiment_id).fetch()
while experiment.progress.observation_count < experiment.observation_budget:
# Receive a Suggestion
suggestion = conn.experiments(experiment.id).suggestions().create()
# Evaluate Your Metric
# You implement this function
value = evaluate_metric(suggestion.assignments)
# Report an Observation
# Include the hostname so that you can track
# progress on the web interface
conn.experiments(experiment.id).observations().create(
suggestion=suggestion.id,
value=value,
metadata=dict(hostname=hostname),
)
Recovering Open Suggestions
suggestions = conn.experiments(experiment_id).suggestions().fetch(state="open")
for suggestion in suggestions.iterate_pages():
value = evaluate_metric(suggestion.assignments) # implement this
conn.experiments(experiment_id).observations().create(
suggestion=suggestion.id,
value=value,
)
