Comment on page
Metadata
Use metadata to store information alongside SigOpt objects. For example, training/evaluation time, IP addresses, and tags. Metadata is a collection of user-provided key-value pairs that SigOpt stores on your behalf under the
metadata
field. Think of metadata as your annotation for a SigOpt object. You can include metadata when you create an object:Python
Bash
Java
from sigopt import Connection
conn = Connection(client_token="USER_TOKEN")
experiment = conn.experiments().create(
name="Sample experiment",
parameters=[
dict(
name="x",
bounds=dict(
min=0,
max=1
),
type="double"
)
],
metrics=[
dict(
name="f",
objective="maximize",
strategy="optimize"
)
],
metadata=dict(
ip="127.0.0.1"
)
)
print("Created experiment: https://app.sigopt.com/experiment/" + experiment.id)
EXPERIMENT=curl -s -X POST https://api.sigopt.com/v1/experiments -u "$SIGOPT_API_TOKEN": \
-H "Content-Type: application/json" -d "`cat experiment_meta.json`"
JSON file defining the Experiment
{
"name": "Sample experiment",
"parameters": [
{
"name": "x",
"bounds": {
"min": 0,
"max": 1
},
"type": "double"
}
],
"metrics": [
{
"name": "f",
"objective": "maximize",
"strategy": "optimize"
}
],
"metadata": {
"ip": "127.0.0.1"
}
}
Experiment experiment = Experiment.create()
.data(
new Experiment.Builder()
.name("Classifier Accuracy")
.parameters(java.util.Arrays.asList(
new Parameter.Builder()
.name("x")
.bounds(new Bounds.Builder()
.min(0)
.max(1)
.build())
.type("double")
.build()
))
.metrics(java.util.Arrays.asList(
new Metric.Builder()
.name("f")
.objective("maximize")
.strategy("optimize")
.build()
))
.metadata(new Metadata.Builder()
.set("ip", "127.0.0.1")
.build()
)
.build()
)
.call();
You can also update an object's metadata later:
Python
Bash
Java
from sigopt import Connection
conn = Connection(client_token="USER_TOKEN")
experiment = conn.experiments(experiment.id).update(
metadata=dict(
ip="172.31.255.255"
)
)
EXPERIMENT=`curl -s -X PUT https://api.sigopt.com/v1/experiments/EXPERIMENT_ID -u "$SIGOPT_API_TOKEN": \
-H 'Content-Type: application/json' \
-d "{\"metadata\":{\"ip\":\"172.31.255.255\"}}"`
import com.sigopt.SigOpt;
import com.sigopt.exception.SigoptException;
import com.sigopt.model.*;
import java.util.Arrays;
public class YourSigoptExperiment {
public static Experiment createExperiment() throws SigoptException {
Experiment experiment = Experiment.update(experiment.id)
.data(
new Experiment.Builder()
.metadata(new Metadata.Builder()
.set("ip", "172.31.255.255")
.build())
.build()
)
.call();
return experiment;
}
The Analysis page in the Experiment Dashboard allows you to plot metadata values of observations as axes in the Experiment History visualizations. This feature enables visualizing the relationship between metadata values and the metric value, as well as metadata values and other parameter values.
Metadata is a series of up to 100 key/value pairs. Keys can have a maximum length of 100 characters. Values must be non-null and can be numbers or strings, and string values can have a maximum length of 500 characters.
If the value of a Metadata item is
None
, SigOpt will ignore this item in the metadata. For example: {"ip": None}
will be treated as an empty dictionary.Python
Bash
Java
from sigopt import Connection
conn = Connection(client_token="USER_TOKEN")
suggestion = conn.experiments(experiment.id).suggestions().create(
metadata=dict(
ip="127.0.0.1"
)
)
SUGGESTION=`curl -s -X POST https://api.sigopt.com/v1/experiments/EXPERIMENT_ID/suggestions -u "$SIGOPT_API_TOKEN": \
-H 'Content-Type: application/json' \
-d "{\"metadata\":{\"ip\":\"127.0.0.1\"}}"`
import com.sigopt.SigOpt;
import com.sigopt.exception.SigoptException;
import com.sigopt.model.*;
import java.util.Arrays;
public class YourSigoptExperiment {
public static Experiment createExperiment() throws SigoptException {
Suggestion suggestion = new Experiment(experiment.id).suggestions().create()
.data(
new Suggestion.Builder()
.metadata(new Metadata.Builder()
.set("ip", "127.0.0.1")
.build())
.build()
)
.call();
return experiment;
}
Python
Bash
Java
from sigopt import Connection
conn = Connection(client_token="USER_TOKEN")
observation = conn.experiments(experiment.id).observations().create(
assignments=dict(
x=0.5
),
metadata=dict(
ip="127.0.0.1"
),
values=[{"name": "f", "value": 1}]
)
OBSERVATION=`curl -s -X POST https://api.sigopt.com/v1/experiments/EXPERIMENT_ID/observations -u "$SIGOPT_API_TOKEN": \
-H 'Content-Type: application/json' \
-d "{\"assignments\":{\"x\":0.5},\"values\":[{\"name\":\"f\",\"value\":1}], \"metadata\":{\"ip\":\"127.0.0.1\"}}"`
import com.sigopt.SigOpt;
import com.sigopt.exception.SigoptException;
import com.sigopt.model.*;
import java.util.Arrays;
public class YourSigoptExperiment {
public static Experiment createExperiment() throws SigoptException {
Observation observation = new Experiment(experiment.id).observations().create()
.data(
new Observation.Builder()
.assignments(new Assignments.Builder()
.set("x", 0.5)
.build())
.metadata(new Metadata.Builder()
.set("ip", "127.0.0.1")
.build())
.values(java.util.Arrays.asList(
new Value.Builder()
.name("f")
.value(1)
.build(),
))
)
.call();
return experiment;
}
Metadata is extremely flexible and there are many different ways to use it.
- Track information such as error codes and hostnames, which can be helpful in diagnosing issues within distributed systems.
- Store IP information for suggestions while completing parallel evaluations of a model. Examining open suggestions will tell you at a glance which machines are currently tuning models, and can give a rough time estimate for how long a set of assignments has currently run for.
- Tag experiments that are all tuning the same type of model.
- Annotate observations with timing data. Use the Analysis page to see how long each model took to evaluate versus the value of the metric.
When running SigOpt in parallel, one may be interested in tracking which worker (e.g., an EC2 instance on AWS) a Suggestion or Observation is running on. This can be useful, for instance, when a failed observation appears and you want to identify the corresponding failed worker.
Including
metadata
at Suggestion creation can be done viaconn.experiments(experiment.id).suggestions().create(
metadata=dict(instance_id=instance_id),
)
or at Observation creation via
conn.experiments(experiment.id).observations().create(
suggestion=suggestion.id,
value=value,
metadata=dict(instance_id=instance_id),
)
Information stored in
metadata
will be displayed on the SigOpt Experiment webpage. If instance_id
is passed in, the Suggestion tab in each Suggestion entry will include the associated instance_id
. Likewise, under the History tab, the user can see all metadata
passed in with any Observation. In particular if the user clicks on entries with failed Observations, they can see which instance_id
is associated with that result(see example below).
To find the
instance_id
of an EC2 instance, the UNIX command isec2metadata --instance-id
or in Python
import boto.utils
boto.utils.get_instance_metadata()["instance-id"]
If an instance crashes during a Suggestion computation, the user can reconnect to that instance and run the following commands to restart the open Suggestion(s) associated with that instance:
suggestions = conn.experiments(experiment_id).suggestion().fetch(state="open")
for suggestion in suggestions.iterate_pages():
if suggestion.metadata["instance_id"] == current_instance_id:
value_dict = evaluate_metric(suggestion.assignments)
conn.experiments(experiment_id).observations().create(
suggestion=suggestion.id,
values=value_dict,
)
When generating plots on the SigOpt webpage, the user can choose fields from
metadata
as coordinates for plots. In particular, Experiment History, Full Experiment History, and Parallel Coordinates plots all include fields from metadata
as possible choices for coordinates. The user can select and combine them with other coordinates to generate custom visualizations. 
Example where the metadata entry
instance_id
is included as one coordinate. Note that plotting metric value against instance_id
is unlikely to show any correlation since Suggestions are served out independently of the instance_id
.Last modified 1yr ago