Metric Thresholds
In Multimetric Experiments, SigOpt searches for the Pareto Frontier; these are the parameter assignments whose metric values cannot be improved uniformly. However, it may be the case that only a subset of the Pareto Frontier is actually relevant for a given application. Some points on the Pareto Frontier may not be viable because unique business requirements force one or both metrics to meet custom criteria.
The goal of Metric Thresholds is to allow you to provide a region of metric values which will define success for you. SigOpt’s Optimizer will use this information to focus on this region with the intent of better resolving the Pareto Frontier to satisfy these stated thresholds. If you want to place thresholds on non-optimized (guardrail) metrics, see Metric Constraints.
The SigOpt optimized efficient frontier is illustrated by the orange points (left), but not all solutions are relevant. By placing a threshold on the metric of inference_time (right), SigOpt can focus on sampling more promising regions.
Example: Balancing Accuracy with Speed
Neural networks have great capacity to perform well in complicated machine learning situations; however, this may come at the cost of creating a very complex model with a slow inference time. We discussed this situation in an earlier blog, and used the Metric Thresholds feature to state these practical limitations at experiment creation to direct the SigOpt search on the regions of interest.
Defining the Metric Thresholds
To assign a threshold to one or more of your metrics, specify the
threshold field in the desired Metric object when creating your experiment. You can specify a threshold for any number of metrics in your multimetric experiment.Python
YAML
1
sigopt.create_experiment(
2
name="Multimetric optimization with a threshold",
3
project="sigopt-examples",
4
type="offline",
5
parameters=[
6
dict(
7
name="hidden_layer_size",
8
type="int",
9
bounds=dict(
10
min=32,
11
max=128
12
)
13
),
14
dict(
15
name="activation_fn",
16
type="categorical",
17
categorical_values=[
18
"relu",
19
"tanh"
20
]
21
)
22
],
23
metrics=[
24
dict(
25
name="holdout_accuracy",
26
strategy="optimize",
27
objective="maximize"
28
),
29
dict(
30
name="inference_time",
31
strategy="optimize",
32
objective="minimize",
33
threshold=0.1
34
)
35
],
36
parallel_bandwidth=1,
37
budget=30
38
)
Copied!
1
name: Multimetric optimization with a threshold
2
project: sigopt-examples
3
type: offline
4
parameters:
5
- name: hidden_layer_size
6
type: int
7
bounds:
8
min: 32
9
max: 128
10
- name: activation_fn
11
type: categorical
12
categorical_values:
13
- relu
14
- tanh
15
metrics:
16
- name: holdout_accuracy
17
strategy: optimize
18
objective: maximize
19
- name: inference_time
20
strategy: optimize
21
objective: minimize
22
threshold: 0.1
23
parallel_bandwidth: 1
24
budget: 30
Copied!
Updating Metric Thresholds
Metric thresholds can be updated on the properties page of an experiment at any time while the experiment is in progress.
Notes on Feasibility
If the defined metric thresholds don’t feasibly exist anywhere in the domain, this feature will perform unpredictably. SigOpt does not know beforehand what is feasible, so setting unrealistic goals hinders our ability to exploit the information we gain to optimize towards the threshold you’ve specified. For example, if the accuracy threshold for a machine learning model is set to
984, when the actual accuracy exists between 0 and 1, this feature will assume the value 984 is actually achievable and explore the domain erratically trying to find solutions that satisfy this threshold.When defining thresholds, it is best
1. to state well-established thresholds at the start of an experiment if you have the data, or
2. to update the experiment after some Runs have been completed.
Limitations
Last modified 23d ago