Define and Set Up Metric Space

Search…
Similar to defining the hyperparameter space, you need to define the metric space to run a SigOpt Experiment.
The entire metric space is defined as a list of objects, where each object represents a metric.
Each metric is given a name, an objective and a strategy. Metrics with a strategy of constraint are given a threshold. Metrics with a strategy of optimize can optionally be assigned a threshold.
name
The metric name is a string used to keep track of the different metrics in the experiment.
objective
The objective is a flag that specifies if a metric is intended to be minimized or maximized by either setting minimize or maximize.
Python
YAML
1
dict(
2
  name="name",
3
  objective="minimize",
4
  strategy="store"
5
)
Copied!
1
metrics:
2
  - name: name
3
    objective: minimize
4
    strategy: store
Copied!
Python
YAML
1
dict(
2
  name="name",
3
  objective="maximize",
4
  strategy="store"
5
)
Copied!
1
metrics:
2
  - name: name
3
    objective: maximize
4
    strategy: store
Copied!
strategy
SigOpt allows three different types of metrics: stored, optimized and constrained. They are invoked by calling store, optimize, and constraint respectively.
Store metrics are for tracking purposes. SigOpt can handle up to 50 stored metrics per training run or experiment.
Python
YAML
1
dict(
2
  name="name",
3
  objective="maximize",
4
  strategy="store"
5
)
Copied!
1
metrics:
2
  - name: name
3
    objective: maximize
4
    strategy: store
Copied!
Optimize metrics are for finding a minimum or a maximum value. SigOpt handles up to two optimized metrics - for more information visit multimetric optimization.
Each optimized metric has the ability to specify a threshold which is a floating point number representing a threshold for success - for more information visit metric threshold.
Python
YAML
1
dict(
2
  name="name",
3
  objective="minimize",
4
  strategy="optimize"
5
)
Copied!
1
metrics:
2
  - name: name
3
    objective: minimize
4
    strategy: optimize
Copied!
Python
YAML
1
dict(
2
  name="name",
3
  objective="minimize",
4
  threshold=100,
5
  strategy="optimize"
6
)
Copied!
1
metrics:
2
  - name: name
3
    objective: minimize
4
    threshold: 100
5
    strategy: optimize
Copied!
Constraint metrics are for defining success thresholds without the need to find a minimum or maximum value. SigOpt handles up to four constraints - for more information visit metric constraints. The constraint is specified as a threshold which is a floating point number representing a threshold for success.
Python
YAML
1
dict(
2
  name="name",
3
  objective="minimize",
4
  threshold=100,
5
  strategy="constraint"
6
)
Copied!
1
metrics:
2
  - name: name
3
    objective: minimize
4
    threshold: 100
5
    strategy: constraint
Copied!
Metric Failures
SigOpt has the ability to handle and track Failures.
Cases we recommend marking as Failures:* Evaluating a metric was not possible because of SigOpt provided parameter assignments themselves
Certain parameter configurations led to out-of-memory error or run-time errors
The parameter assignments were infeasible/not of interest. If you know these regions prior to running the Experiment, we recommend adding these as constraints.
If model training abruptly stops because a machine randomly fails, we recommend deleting that Run instead of marking it as failed.
1
sigopt.log_failure()
Copied!
Define and Set Up Parameter Space
Wrap Up Your Experiment
Last modified 30d ago
Copy link
Contents
`name`

`objective`

`strategy`

Metric Failures
