Metric Thresholds

In Multimetric Experiments, SigOpt searches for the Pareto Frontier; these are the parameter assignments whose metric values cannot be improved uniformly. The final outcome of such an experiment is the subset of all observations which have efficient metric values. However, it may be the case that only a subset of the Pareto Frontier is actually relevant for a given application. Some points on the Pareto Frontier may not be viable because unique business requirements force one or both metrics to meet custom criteria.

The goal of Metric Thresholds is to allow you to provide a region of metric values which will define success for you. SigOpt’s Optimizer will use this information to focus on this region with the intent of better resolving the Pareto Frontier to satisfy these stated thresholds. If you want to place thresholds on non-optimized (guardrail) metrics, see Metric Constraints.

The SigOpt optimized efficient frontier is illustrated by the orange points (left), but not all solutions are relevant. By placing a threshold on the metric of inference_time (right), SigOpt can focus on sampling more promising regions.

Example: Balancing Accuracy with Speed

Neural networks have great capacity to perform well in complicated machine learning situations; however, this may come at the cost of creating a very complex model with a slow inference time. We discussed this situation in an earlier blog, and used the Metric Thresholds feature to state these practical limitations at experiment creation to direct the SigOpt search on the regions of interest.

Defining the Metric Thresholds

To assign a threshold to one or more of your metrics, specify the threshold field in the desired Metric object when creating your experiment. You can specify a threshold for any number of metrics in your multimetric experiment. To assign thresholds after an experiment has been created, update the metrics field, as shown below. SigOpt will interpret the specified threshold in the following ways.

• By default, SigOpt will assume you want to maximize each metric and try to find observations that are greater than or equal to the specified threshold.

• If you provide an objective of “minimize” for a metric, SigOpt will try to find observations that are less than or equal to the specified threshold.

Core Module

Python

from sigopt import Connection

conn = Connection(client_token="USER_TOKEN")
experiment = conn.experiments().create(
  name="Neural network",
  parameters=[
    dict(
      name="learning_rate",
      bounds=dict(
        min=0.0001,
        max=1
        ),
      transformation="log",
      type="double"
      ),
    dict(
      name="nodes_per_layer",
      bounds=dict(
        min=5,
        max=20
        ),
      type="int"
      )
    ],
  metrics=[
    dict(
      name="inference_time_milliseconds",
      objective="minimize",
      threshold=100
      ),
    dict(
      name="validation_accuracy",
      objective="maximize"
      )
    ],
  observation_budget=65,
  parallel_bandwidth=2,
  type="offline"
  )
print("Created experiment: https://app.sigopt.com/experiment/" + experiment.id)

Bash

EXPERIMENT=curl -s -X POST https://api.sigopt.com/v1/experiments -u "$SIGOPT_API_TOKEN": \
    -H "Content-Type: application/json" -d "`cat experiment_meta.json`"

JSON file defining the Experiment:

experiment_meta.json

{
  "name": "Neural network",
  "parameters": [
    {
      "name": "learning_rate",
      "bounds": {
        "min": 0.0001,
        "max": 1
      },
      "transformation": "log",
      "type": "double"
    },
    {
      "name": "nodes_per_layer",
      "bounds": {
        "min": 5,
        "max": 20
      },
      "type": "int"
    }
  ],
  "metrics": [
    {
      "name": "inference_time_milliseconds",
      "objective": "minimize",
      "threshold": 100
    },
    {
      "name": "validation_accuracy",
      "objective": "maximize"
    }
  ],
  "observation_budget": 65,
  "parallel_bandwidth": 2,
  "type": "offline"
}

Java

import com.sigopt.SigOpt;
import com.sigopt.exception.SigoptException;
import com.sigopt.model.*;

import java.util.Arrays;

public class YourSigoptExperiment {
  public static Experiment createExperiment() throws SigoptException {
    Experiment experiment = Experiment.create()
      .data(
        new Experiment.Builder()
          .name("Neural network")
          .parameters(java.util.Arrays.asList(
            new Parameter.Builder()
              .name("learning_rate")
              .bounds(new Bounds.Builder()
                .min(0.0001)
                .max(1)
                .build())
              .transformation("log")
              .type("double")
              .build(),
            new Parameter.Builder()
              .name("nodes_per_layer")
              .bounds(new Bounds.Builder()
                .min(5)
                .max(20)
                .build())
              .type("int")
              .build()
            ))
          .metrics(java.util.Arrays.asList(
            new Metric.Builder()
              .name("inference_time_milliseconds")
              .objective("minimize")
              .threshold(100)
              .build(),
            new Metric.Builder()
              .name("validation_accuracy")
              .objective("maximize")
              .build()
            ))
          .observationBudget(65)
          .parallelBandwidth(2)
          .type("offline")
          .build()
        )
      .call();
    return experiment;
  }

AI Module

Python

sigopt.create_experiment(
  name="Multimetric optimization with a threshold",
  type="offline",
  parameters=[
    dict(
      name="hidden_layer_size",
      type="int",
      bounds=dict(
        min=32,
        max=128
      )
    ),
    dict(
      name="activation_fn",
      type="categorical",
      categorical_values=[
        "relu",
        "tanh"
      ]
    )
  ],
  metrics=[
    dict(
      name="holdout_accuracy",
      strategy="optimize",
      objective="maximize"
    ),
    dict(
      name="inference_time",
      strategy="optimize",
      objective="minimize",
      threshold=0.1
    )
  ],
  parallel_bandwidth=1,
  budget=30
)

YAML

name: Multimetric optimization with a threshold
type: offline
parameters:
  - name: hidden_layer_size
    type: int
    bounds:
      min: 32
      max: 128
  - name: activation_fn
    type: categorical
    categorical_values:
      - relu
      - tanh
metrics:
  - name: holdout_accuracy
    strategy: optimize
    objective: maximize
  - name: inference_time
    strategy: optimize
    objective: minimize
    threshold: 0.1
parallel_bandwidth: 1
budget: 30

Updating Metric Thresholds

Metric thresholds can be updated on the properties page of an experiment at any time while the experiment is in progress.

Core Module

For core module users, the thresholds can also be updated directly via the API. An example is given below.

Python

experiment = conn.experiments(experiment.id).update(
  metrics=[
    dict(
      name="inference_time_milliseconds",
      threshold=85
      ),
    dict(
      name="validation_accuracy",
      threshold=0.79
      )
    ]
  )

Bash

EXPERIMENT=`curl -s -X PUT https://api.sigopt.com/v1/experiments/EXPERIMENT_ID -u "$SIGOPT_API_TOKEN": \
  -H 'Content-Type: application/json' \
  -d "{\"metrics\":[{\"name\":\"inference_time_milliseconds\",\"threshold\":85},{\"name\":\"validation_accuracy\",\"threshold\":0.79}]}"`

Java

import com.sigopt.SigOpt;
import com.sigopt.exception.SigoptException;
import com.sigopt.model.*;

import java.util.Arrays;

Experiment.update(experiment.getId())
  .data(
    new Experiment.Builder()
      .metrics(java.util.Arrays.asList(
        new Metric.Builder()
          .name("inference_time_milliseconds")
          .threshold(85)
          .build(),
        new Metric.Builder()
          .name("validation_accuracy")
          .threshold(0.79)
          .build()
        ))
      .build()
    )
  .call();

Notes on Feasibility

If the defined metric thresholds don’t feasibly exist anywhere in the domain, this feature will perform unpredictably. SigOpt does not know beforehand what is feasible, so setting unrealistic goals hinders our ability to exploit the information we gain to optimize towards the threshold you’ve specified. For example, if the accuracy threshold for a machine learning model is set to 984, when the actual accuracy exists between 0 and 1, this feature will assume the value 984 is actually achievable and explore the domain erratically trying to find solutions that satisfy this threshold.

When defining thresholds, it is best

• to state well-established thresholds at the start of an experiment if you have the data, or

• to update the experiment after some feasible observations have been observed.

Limitations

• Metric thresholds can be applied to optimized metrics in Multimetric Experiments.
• Thresholds can also be applied to constrained metrics.