Multimetric Optimization

In many applications, it may be necessary to consider multiple competing metrics which have optimal values for different parameters. SigOpt enables this through Multimetric Experiments.

How does Multimetric Optimization work?

The contour plots below depict two competing metrics, where parameter values of x1 and x2 cannot simultaneously meet the optima for both metrics f1 and f2. For example, it can be extremely challenging to maximize both model performance and minimize training time. SigOpt’s Multimetric Optimization tries to solve this problem by finding sets of feasible or optimal values. The result of a Multimetric Experiment is a Pareto Frontier.

Contour plots of f1 and f2

Defining a Multimetric Experiment in SigOpt

A SigOpt Multimetric Experiment can be conducted to explore the optimal values achievable for both metrics. Notice that, unlike for single metric functions, this multimetric function returns a set of optimal values which contain the name of the metric and the associated value for that metric.

Core Module

Python

from sigopt import Connection

conn = Connection(client_token="CLIENT_TOKEN")
experiment = conn.experiments().create(
  name="Multimetric optimization",
  type="offline",
  parameters=[
    dict(
      name="x1",
      type="double",
      bounds=dict(
        min=0,
        max=10
      )
    ),
    dict(
      name="x2",
      type="double",
      bounds=dict(
        min=0,
        max=10
      )
    )
  ],
  metrics=[
    dict(
      name="f1",
      strategy="optimize",
      objective="maximize"
    ),
    dict(
      name="f2",
      strategy="optimize",
      objective="maximize"
    )
  ],
  parallel_bandwidth=1,
  observation_budget=38
)

Bash

EXPERIMENT=curl -s -X POST https://api.sigopt.com/v1/experiments -u "$SIGOPT_API_TOKEN": \
    -H "Content-Type: application/json" -d "`cat experiment_meta.json`"

JSON file defining the Experiment:

experiment_meta.json

{
  "name": "Multimetric optimization",
  "type": "offline",
  "parameters": [
    {
      "name": "x1",
      "type": "double",
      "bounds": {
        "min": 0,
        "max": 10
      }
    },
    {
      "name": "x2",
      "type": "double",
      "bounds": {
        "min": 0,
        "max": 10
      }
    }
  ],
  "metrics": [
    {
      "name": "f1",
      "strategy": "optimize",
      "objective": "maximize"
    },
    {
      "name": "f2",
      "strategy": "optimize",
      "objective": "maximize"
    }
  ],
  "parallel_bandwidth": 1,
  "observation_budget": 38
}

Java

import com.sigopt.SigOpt;
import com.sigopt.exception.SigoptException;
import com.sigopt.model.*;

import java.util.Arrays;

public class YourSigoptExperiment {
  public static Experiment createExperiment() throws SigoptException {
    Experiment experiment = Experiment.create()
      .data(
        new Experiment.Builder()
          .name("Multimetric optimization")
          .parameters(java.util.Arrays.asList(
            new Parameter.Builder()
              .name("x1")
              .bounds(new Bounds.Builder()
                .min(0)
                .max(10)
                .build())
              .type("double")
              .build(),
            new Parameter.Builder()
              .name("x2")
              .bounds(new Bounds.Builder()
                .min(0)
                .max(10)
                .build())
              .type("double")
              .build()
            ))
          .metrics(java.util.Arrays.asList(
            new Metric.Builder()
              .name("f1")
              .objective("maximize")
              .strategy("optimize")
              .build(),
            new Metric.Builder()
              .name("f2")
              .objective("maximize")
              .strategy("optimize")
              .build()
            ))
          .observationBudget(38)
          .parallelBandwidth(1)
          .type("offline")
          .build()
        )
      .call();
    return experiment;
  }

AI Module

Python

sigopt.create_experiment(
  name="Multimetric optimization",
  type="offline",
  parameters=[
    dict(
      name="x1",
      type="double",
      bounds=dict(
        min=0,
        max=10
      )
    ),
    dict(
      name="x2",
      type="double",
      bounds=dict(
        min=0,
        max=10
      )
    )
  ],
  metrics=[
    dict(
      name="f1",
      strategy="optimize",
      objective="maximize"
    ),
    dict(
      name="f2",
      strategy="optimize",
      objective="maximize"
    )
  ],
  parallel_bandwidth=1,
  budget=38
)

YAML

name: Multimetric optimization
type: offline
parameters:
  - name: x1
    type: double
    bounds:
      min: 0
      max: 10
  - name: x2
    type: double
    bounds:
      min: 0
      max: 10
metrics:
  - name: f1
    strategy: optimize
    objective: maximize
  - name: f2
    strategy: optimize
    objective: maximize
parallel_bandwidth: 1
budget: 38

Interpreting the Solution

Below is an example of possible outcomes of a Multimetric Experiment involving the metrics from the contour plots above.

(Left) Pareto efficient frontier in metrics. (Right) Pareto optimal solutions in Parameter space.

The graph on the left shows the region of feasible points with the blue circles representing the efficient points that SigOpt found on the Pareto frontier, the red circles representing other suggested parameters from SigOpt, and the black dots representing randomly chosen parameters (to show the outline of the region). Each of the efficient blue circles in the left graph corresponds to an efficient parameter choice represented by a blue circle in the right graph.

A domain expert may be able to judge which of the results in the left graph is preferred, and then use the parameters from the right graph (also accessed through best assignments endpoint) to create the desired model in production. For further reading, check out our blog posts here and here.

Core Module

Python

pareto_efficient_results = conn.experiments(EXPERIMENT_ID).best_assignments().fetch()

Bash

PARETO_EFFICIENT_RESULTS=`curl -s -X GET https://api.sigopt.com/v1/clients/EXPERIMENT_ID/best_assignments -u "$SIGOPT_API_TOKEN":`

Java

Pagination<BestAssignment> paretoEfficientResults = new Experiment(EXPERIMENT_ID).bestAssignments().list().call();

AI Module

Python

pareto_efficient_results = sigopt.get_experiment(EXPERIMENT_ID).get_best_runs()

SigOpt will find trade-offs in each of your metrics in order to find the efficient frontier. If you want to quantify how much of a trade-off you’re willing to make, check out the Metric Thresholds feature.

Limitations

• Budget must be set when a Multimetric Experiment is created

• The maximum number of optimized metrics is 2

• Multisolution Experiments are not permitted