Metadata

Use metadata to store information alongside SigOpt objects. For example, training/evaluation time, IP addresses, and tags. Metadata is a collection of user-provided key-value pairs that SigOpt stores on your behalf under the metadata field. Think of metadata as your annotation for a SigOpt object. You can include metadata when you create an object:

Python

from sigopt import Connection

conn = Connection(client_token="USER_TOKEN")
experiment = conn.experiments().create(
  name="Sample experiment",
  parameters=[
    dict(
      name="x",
      bounds=dict(
        min=0,
        max=1
        ),
      type="double"
      )
    ],
  metrics=[
    dict(
      name="f",
      objective="maximize",
      strategy="optimize"
      )
    ],
  metadata=dict(
    ip="127.0.0.1"
    )
  )
print("Created experiment: https://app.sigopt.com/experiment/" + experiment.id)

Bash

EXPERIMENT=curl -s -X POST https://api.sigopt.com/v1/experiments -u "$SIGOPT_API_TOKEN": \
    -H "Content-Type: application/json" -d "`cat experiment_meta.json`"

JSON file defining the Experiment

{
  "name": "Sample experiment",
  "parameters": [
    {
      "name": "x",
      "bounds": {
        "min": 0,
        "max": 1
      },
      "type": "double"
    }
  ],
  "metrics": [
    {
      "name": "f",
      "objective": "maximize",
      "strategy": "optimize"
    }
  ],
  "metadata": {
    "ip": "127.0.0.1"
  }
}

Java

Experiment experiment = Experiment.create()
  .data(
    new Experiment.Builder()
      .name("Classifier Accuracy")
      .parameters(java.util.Arrays.asList(
        new Parameter.Builder()
          .name("x")
          .bounds(new Bounds.Builder()
            .min(0)
            .max(1)
            .build())
          .type("double")
          .build()
        ))
      .metrics(java.util.Arrays.asList(
        new Metric.Builder()
          .name("f")
          .objective("maximize")
          .strategy("optimize")
          .build()
        ))
      .metadata(new Metadata.Builder()
            .set("ip", "127.0.0.1")
            .build()
        )
      .build()
    )
  .call();

You can also update an object's metadata later:

Python

from sigopt import Connection

conn = Connection(client_token="USER_TOKEN")
experiment = conn.experiments(experiment.id).update(
  metadata=dict(
    ip="172.31.255.255"
    )
  )

Bash

EXPERIMENT=`curl -s -X PUT https://api.sigopt.com/v1/experiments/EXPERIMENT_ID -u "$SIGOPT_API_TOKEN": \
  -H 'Content-Type: application/json' \
  -d "{\"metadata\":{\"ip\":\"172.31.255.255\"}}"`

Java

import com.sigopt.SigOpt;
import com.sigopt.exception.SigoptException;
import com.sigopt.model.*;

import java.util.Arrays;

public class YourSigoptExperiment {
  public static Experiment createExperiment() throws SigoptException {
    Experiment experiment = Experiment.update(experiment.id)
      .data(
        new Experiment.Builder()
          .metadata(new Metadata.Builder()
            .set("ip", "172.31.255.255")
            .build())
          .build()
        )
      .call();
    return experiment;
  }

The Analysis page in the Experiment Dashboard allows you to plot metadata values of observations as axes in the Experiment History visualizations. This feature enables visualizing the relationship between metadata values and the metric value, as well as metadata values and other parameter values.

Specs

Metadata is a series of up to 100 key/value pairs. Keys can have a maximum length of 100 characters. Values must be non-null and can be numbers or strings, and string values can have a maximum length of 500 characters.

If the value of a Metadata item is None, SigOpt will ignore this item in the metadata. For example: {"ip": None} will be treated as an empty dictionary.

Supported Objects

Suggestion, Observation, and Experiment objects all support metadata.

Suggestion Metadata

Python

from sigopt import Connection

conn = Connection(client_token="USER_TOKEN")
suggestion = conn.experiments(experiment.id).suggestions().create(
  metadata=dict(
    ip="127.0.0.1"
    )
  )

Bash

SUGGESTION=`curl -s -X POST https://api.sigopt.com/v1/experiments/EXPERIMENT_ID/suggestions -u "$SIGOPT_API_TOKEN": \
  -H 'Content-Type: application/json' \
  -d "{\"metadata\":{\"ip\":\"127.0.0.1\"}}"`

Java

import com.sigopt.SigOpt;
import com.sigopt.exception.SigoptException;
import com.sigopt.model.*;

import java.util.Arrays;

public class YourSigoptExperiment {
  public static Experiment createExperiment() throws SigoptException {
    Suggestion suggestion = new Experiment(experiment.id).suggestions().create()
      .data(
        new Suggestion.Builder()
          .metadata(new Metadata.Builder()
            .set("ip", "127.0.0.1")
            .build())
          .build()
        )
      .call();
    return experiment;
  }

Observation Metadata

Python

from sigopt import Connection

conn = Connection(client_token="USER_TOKEN")
observation = conn.experiments(experiment.id).observations().create(
  assignments=dict(
    x=0.5
    ),
  metadata=dict(
    ip="127.0.0.1"
    ),
  values=[{"name": "f", "value": 1}]
  )

Bash

OBSERVATION=`curl -s -X POST https://api.sigopt.com/v1/experiments/EXPERIMENT_ID/observations -u "$SIGOPT_API_TOKEN": \
  -H 'Content-Type: application/json' \
  -d "{\"assignments\":{\"x\":0.5},\"values\":[{\"name\":\"f\",\"value\":1}], \"metadata\":{\"ip\":\"127.0.0.1\"}}"`

Java

import com.sigopt.SigOpt;
import com.sigopt.exception.SigoptException;
import com.sigopt.model.*;

import java.util.Arrays;

public class YourSigoptExperiment {
  public static Experiment createExperiment() throws SigoptException {
    Observation observation = new Experiment(experiment.id).observations().create()
      .data(
        new Observation.Builder()
          .assignments(new Assignments.Builder()
            .set("x", 0.5)
            .build())
          .metadata(new Metadata.Builder()
            .set("ip", "127.0.0.1")
            .build())
          .values(java.util.Arrays.asList(
        new Value.Builder()
          .name("f")
          .value(1)
          .build(),
        ))
        )
      .call();
    return experiment;
  }

Example Use Cases

Metadata is extremely flexible and there are many different ways to use it.

• Track information such as error codes and hostnames, which can be helpful in diagnosing issues within distributed systems.

• Store IP information for suggestions while completing parallel evaluations of a model. Examining open suggestions will tell you at a glance which machines are currently tuning models, and can give a rough time estimate for how long a set of assignments has currently run for.

• Tag experiments that are all tuning the same type of model.

• Annotate observations with timing data. Use the Analysis page to see how long each model took to evaluate versus the value of the metric.

Example: Tracking Suggestions and Observations Using Metadata

When running SigOpt in parallel, one may be interested in tracking which worker (e.g., an EC2 instance on AWS) a Suggestion or Observation is running on. This can be useful, for instance, when a failed observation appears and you want to identify the corresponding failed worker.

Including metadata at Suggestion creation can be done via

conn.experiments(experiment.id).suggestions().create(
  metadata=dict(instance_id=instance_id),
)

or at Observation creation via

conn.experiments(experiment.id).observations().create(
  suggestion=suggestion.id,
  value=value,
  metadata=dict(instance_id=instance_id),
)

Information stored in metadata will be displayed on the SigOpt Experiment webpage. If instance_id is passed in, the Suggestion tab in each Suggestion entry will include the associated instance_id. Likewise, under the History tab, the user can see all metadata passed in with any Observation. In particular if the user clicks on entries with failed Observations, they can see which instance_id is associated with that result(see example below).

To find the instance_id of an EC2 instance, the UNIX command is

ec2metadata --instance-id

or in Python

import boto.utils
boto.utils.get_instance_metadata()["instance-id"]

If an instance crashes during a Suggestion computation, the user can reconnect to that instance and run the following commands to restart the open Suggestion(s) associated with that instance:

suggestions = conn.experiments(experiment_id).suggestion().fetch(state="open")
for suggestion in suggestions.iterate_pages():
  if suggestion.metadata["instance_id"] == current_instance_id:
    value_dict = evaluate_metric(suggestion.assignments)
    conn.experiments(experiment_id).observations().create(
      suggestion=suggestion.id,
      values=value_dict,
    )

When generating plots on the SigOpt webpage, the user can choose fields from metadata as coordinates for plots. In particular, Experiment History, Full Experiment History, and Parallel Coordinates plots all include fields from metadata as possible choices for coordinates. The user can select and combine them with other coordinates to generate custom visualizations.

Example where the metadata entry instance_id is included as one coordinate. Note that plotting metric value against instance_id is unlikely to show any correlation since Suggestions are served out independently of the instance_id.