Multisolution
For some experiments you may be interested in exploring multiple performant solutions within the parameter space, despite only optimizing for a single objective. Multisolution enables you to find additional solutions that are sufficiently diverse, returning multiple sets of parameters and increasing the chances of finding models that meet unique business and research goals. You can further customize the process by combining Metric constraints with Multisolution to act as “guardrail metrics” and further target areas of interest.
In cases where multiple objectives must be considered, while still prioritizing the exploration of the parameter space, check out how to set up an all-constraints experiment instead.
Multisolution Intuition
Modeling in the real world frequently poses a mismatch problem between development and production environments. Either due to temporal differences from training and production (unseen data), or because there is an intrinsic difference between both settings, it is common to not entirely trust your developing results.
In these cases it is often useful to consider multiple distinct and diverse parameter configurations that yield high performing values. Sometimes in development, a lower-performing model with more “stable” parameters (configurations in which small changes in parameters do not drastically change the resulting metric value) can lead to more promising results in production.
Multisolution addresses this by searching for feasible regions of hyperparameter configurations that yield high-performing metric values and returning a set of single and diverse configurations. While we can't directly analyze or make a decision for you on which models work best for your particular application, SigOpt Multisolution offers you the ability to explore and narrow down options to make the most informed choice you can.
During the experiment progression SigOpt will seek to balance performance and diversity, finding multiple top-performing values that exist in distinct parameter spaces.
Figure 1: Multisolution SigOpt experiment with five distinct solutions (num_solution= 5). This approach searches for all optima values in the input space and returns the topmost diverse
num_solutions, as represented by the opaque white dots within the feasibly performing orange regions.Creating a SigOpt Experiment with Multiple Solutions
A SigOpt Experiment with multiple solutions can be conducted to return multiple sets of parameters that are sufficiently diverse. In the create_experiment call, add the key
num_solutions and value as the number of solutions desired. A budget is required for experiments with multiple solutions, and cannot be updated once the experiment has started.Picking the Number of Solutions
While the max number of possible solutions that you can ask SigOpt for is gated by the overall budget of the experiment, that is not to say that setting the
num_solutions to a higher number will provide you with the best conclusions. As a start, we recommend setting the number of solutions to no more than 20% of your budget to glean actionable experiment insights.Picking an Experiment Budget
As per our general recommendations around determining an appropriate budget, we would encourage you to first consider the resources and time you have available to dedicate to the experiment. In ideal circumstances, a budget of 30 times the number of parameter dimensions in the experiment should be allocated. Keep in mind that including categoricals should also call for a budget increase.
Best Runs for Multiple Solutions
Calling Best Runs will return a list of solutions ordered in ascending function value. Prior to completing the budget, SigOpt may not feel that it has sufficiently searched the space and may return fewer than
num_solutions when calling Best Runs; when the budget is exhausted, the full num_solutions will be returned.Example Code
1
experiment = sigopt.create_experiment(
2
name="Multisolution example with Bimodal Function",
3
parameters=[
4
dict(
5
name="x1",
6
bounds=dict(
7
min=-1,
8
max=1
9
),
10
type="double"
11
),
12
dict(
13
name="x2",
14
bounds=dict(
15
min=-1,
16
max=1
17
),
18
type="double"
19
)
20
],
21
metrics=[dict(
22
name="function_value",
23
objective="maximize",
24
strategy="optimize"
25
)
26
],
27
budget=50,
28
num_solutions=5,
29
parallel_bandwidth=1,
30
)
Copied!
Limitations
budgetmust be set when creating a Multisolution experiment- The max number of metric constraints available is 4
- A max number of 50 dimensions is allowed
- Conditional parameters are not permitted
Last modified 22d ago