Define and Set Up Metric Space
Similar to defining the hyperparameter space, you need to define the metric space to run a SigOpt Experiment.
The entire metric space is defined as a list of objects, where each object represents a metric.
Each metric is given a
name
, an objective
and a strategy
. Metrics with a strategy of constraint
are given a threshold
. Metrics with a strategy of optimize
can optionally be assigned a threshold
.The metric name is a string used to keep track of the different metrics in the experiment.
The objective is a flag that specifies if a metric is intended to be minimized or maximized by either setting
minimize
or maximize
.Python
YAML
dict(
name="name",
objective="minimize",
strategy="store"
)
metrics:
- name: name
objective: minimize
strategy: store
Python
YAML
dict(
name="name",
objective="maximize",
strategy="store"
)
metrics:
- name: name
objective: maximize
strategy: store
SigOpt allows three different types of metrics: stored, optimized and constrained. They are invoked by calling
store
, optimize
, and constraint
respectively.Store
metrics are for tracking purposes. SigOpt can handle up to 50 stored metrics per training run or experiment.Python
YAML
dict(
name="name",
objective="maximize",
strategy="store"
)
metrics:
- name: name
objective: maximize
strategy: store
Optimize
metrics are for finding a minimum or a maximum value. SigOpt handles up to two optimized metrics - for more information visit multimetric optimization.Each optimized metric has the ability to specify a threshold which is a floating point number representing a threshold for success - for more information visit metric threshold.
Python
YAML
dict(
name="name",
objective="minimize",
strategy="optimize"
)
metrics:
- name: name
objective: minimize
strategy: optimize
Python
YAML
dict(
name="name",
objective="minimize",
threshold=100,
strategy="optimize"
)
metrics:
- name: name
objective: minimize
threshold: 100
strategy: optimize
Constraint
metrics are for defining success thresholds without the need to find a minimum or maximum value. SigOpt handles up to four constraints - for more information visit metric constraints. The constraint is specified as a threshold
which is a floating point number representing a threshold for success.Python
YAML
dict(
name="name",
objective="minimize",
threshold=100,
strategy="constraint"
)
metrics:
- name: name
objective: minimize
threshold: 100
strategy: constraint
Observations or Runs report the value of your metric evaluated on a given set of parameters. However, sometimes a set of parameters is not feasible for evaluation. We call this a metric failure since the model failed to obtain a metric. SigOpt has the ability to handle and track such Failures to learn the feasible regions of your domain.
Cases we recommend marking as Failures:
- Evaluating a metric is not possible because the SigOpt provided parameter assignments are infeasible/ not of interest
- Certain parameter configurations lead to out-of-memory error or run-time errors
If model training abruptly stops because a machine randomly fails, we recommend deleting that Run or Observation instead of marking it as failed.
Python
Java
from sigopt import Connection
conn = Connection(client_token="USER_TOKEN")
observation = conn.experiments(EXPERIMENT_ID).observations().create(
failed=True,
suggestion="SUGGESTION_ID"
)
import com.sigopt.SigOpt;
import com.sigopt.exception.SigoptException;
import com.sigopt.model.*;
import java.util.Arrays;
public class YourSigoptExperiment {
public static Experiment createExperiment() throws SigoptException {
Observation observation = new Experiment(EXPERIMENT_ID).observations().create()
.data(
new Observation.Builder()
.failed(true)
.suggestion("SUGGESTION_ID")
.build()
)
.call();
return experiment;
}
sigopt.log_failure()
If an infeasible region of the parameter space is known beforehand, it may be possible to predefine with Parameter Constraints. In situations in which feasibility is defined through thresholding on auxiliary non-optimized metric values, it may be more beneficial to use Metric Constraints.
Last modified 6mo ago