Multisolution

For some experiments you may be interested in exploring multiple performant solutions within the parameter space, despite only optimizing for a single objective. Multisolution enables you to find additional solutions that are sufficiently diverse, returning multiple sets of parameters and increasing the chances of finding models that meet unique business and research goals. You can further customize the process by combining Metric constraints with Multisolution to act as “guardrail metrics” and further target areas of interest.

In cases where multiple objectives must be considered, while still prioritizing the exploration of the parameter space, check out how to set up an all-constraints experiment instead.

Multisolution Intuition

Modeling in the real world frequently poses a mismatch problem between development and production environments. Either due to temporal differences from training and production (unseen data), or because there is an intrinsic difference between both settings, it is common to not entirely trust your developing results.

In these cases it is often useful to consider multiple distinct and diverse parameter configurations that yield high performing values. Sometimes in development, a lower-performing model with more “stable” parameters (configurations in which small changes in parameters do not drastically change the resulting metric value) can lead to more promising results in production.

Multisolution addresses this by searching for feasible regions of hyperparameter configurations that yield high-performing metric values and returning a set of single and diverse configurations. While we can't directly analyze or make a decision for you on which models work best for your particular application, SigOpt Multisolution offers you the ability to explore and narrow down options to make the most informed choice you can.

During the experiment progression SigOpt will seek to balance performance and diversity, finding multiple top-performing values that exist in distinct parameter spaces.

Figure 1: Multisolution SigOpt experiment with five distinct solutions (num_solution= 5). This approach searches for all optima values in the input space and returns the topmost diverse num_solutions, as represented by the opaque white dots within the feasibly performing orange regions.

Creating a SigOpt Experiment with Multiple Solutions

A SigOpt Experiment with multiple solutions can be conducted to return multiple sets of parameters that are sufficiently diverse. In the experiment create call, add the key num_solutions and value as the number of solutions desired. Budget is required for experiments with multiple solutions, and cannot be updated once the experiment has started.

Picking the Number of Solutions

While the max number of possible solutions that you can ask SigOpt for is bounded by the overall budget of the experiment, that is not to say that setting the num_solutions to a higher number will provide you with the best conclusions. As a start, we recommend setting the number of solutions to no more than 20% of your budget to extract actionable experiment insights.

Picking an Experiment Budget

As per our general recommendations around determining an appropriate budget, we would encourage you to first consider the resources and time you have available to dedicate to the experiment. In ideal circumstances, a budget of 30 times the number of parameter dimensions in the experiment should be allocated. Keep in mind that including categorical should also call for a budget increase.

Multiple Solutions

Calling the Best Assignments endpoint (Best Runs for AI module) will return a list of solutions ordered in ascending function value. Prior to completing the budget, SigOpt may not feel that it has sufficiently searched the space and may return fewer than num_solutions when calling Best Assignments; when the budget is exhausted, the full num_solutions will be returned.

Example Code

Core Module

Python

from sigopt import Connection

conn = Connection(client_token="CLIENT_TOKEN")
experiment = conn.experiments().create( 
  name="Multisolution example with Bimodal Function", 
  parameters=[ 
    dict( 
      name="x1", 
      bounds=dict( 
        min=-1, 
        max=1 
      ), 
      type="double" 
    ), 
    dict( 
      name="x2", 
      bounds=dict( 
        min=-1, 
        max=1 
      ), 
      type="double" 
    ) 
  ], 
  metrics=[
    dict( 
      name="function_value", 
      objective="maximize", 
      strategy="optimize" 
    ) 
  ], 
  observation_budget=50, 
  num_solutions=5, 
  parallel_bandwidth=1, 
) 

Bash

EXPERIMENT=curl -s -X POST https://api.sigopt.com/v1/experiments -u "$SIGOPT_API_TOKEN": \
    -H "Content-Type: application/json" -d "`cat experiment_meta.json`"

JSON file defining the Experiment:

experiment_meta.json

{
  "name": "Multisolution example with Bimodal Function",
  "parameters": [
    {
      "name": "x1",
      "bounds": {
        "min": -1,
        "max": 1
      },
      "type": "double"
    },
    {
      "name": "x2",
      "bounds": {
        "min": -1,
        "max": 1
      },
      "type": "double"
    }
  ],
  "metrics": [
    {
      "name": "function_value",
      "objective": "maximize",
      "strategy": "optimize"
    }
  ],
  "observation_budget": 50,
  "num_solutions": 5,
  "parallel_bandwidth": 1
}

Java

import com.sigopt.SigOpt;
import com.sigopt.exception.SigoptException;
import com.sigopt.model.*;

import java.util.Arrays;

public class YourSigoptExperiment {
  public static Experiment createExperiment() throws SigoptException {
    Experiment experiment = Experiment.create()
      .data(
        new Experiment.Builder()
          .name("Multisolution example with Bimodal Function")
          .parameters(java.util.Arrays.asList(
            new Parameter.Builder()
              .name("x1")
              .bounds(new Bounds.Builder()
                .min(-1)
                .max(1)
                .build())
              .type("double")
              .build(),
            new Parameter.Builder()
              .name("x2")
              .bounds(new Bounds.Builder()
                .min(-1)
                .max(1)
                .build())
              .type("double")
              .build()
            ))
          .metrics(java.util.Arrays.asList(
            new Metric.Builder()
              .name("function_value")
              .objective("maximize")
              .strategy("optimize")
              .build()
            ))
          .numSolutions(5)
          .observationBudget(50)
          .parallelBandwidth(1)
          .type("offline")
          .build()
        )
      .call();
    return experiment;
  }

AI Module

Python

experiment = sigopt.create_experiment( 
  name="Multisolution example with Bimodal Function",  
  parameters=[ 
    dict( 
      name="x1", 
      bounds=dict( 
        min=-1, 
        max=1 
      ), 
      type="double" 
    ), 
    dict( 
      name="x2", 
      bounds=dict( 
        min=-1, 
        max=1 
      ), 
      type="double" 
    ) 
  ], 
  metrics=[
    dict( 
      name="function_value", 
      objective="maximize", 
      strategy="optimize" 
    ) 
  ], 
  budget=50, 
  num_solutions=5, 
  parallel_bandwidth=1,
) 

YAML

name: Multisolution example with Bimodal Function
parameters:
  - name: x1
    bounds:
      min: -1
      max: 1
    type: double
  - name: x2
    bounds:
      min: -1
      max: 1
    type: double
metrics:
  - name: function_value
    objective: maximize
    strategy: optimize
budget: 50
num_solutions: 5
parallel_bandwidth: 1

Limitations

• Budget must be set when creating a Multisolution experiment

• A max number of 50 dimensions is allowed

• num_solutions cannot be modified

• Multimetric Experiments are not permitted

• Conditional parameters are not permitted