Links
Comment on page

Manage Open Suggestions

If you run into an error, there may be some outstanding suggestions with no reported observations (called open suggestions in the API). Here are some strategies you can use to recover in this situation.

Iterate through all open suggestions and reevaluate

Python
Java
suggestions = conn.experiments(EXPERIMENT_ID).suggestions().fetch(state="open")
for suggestion in suggestions.iterate_pages():
value = evaluate_metric(suggestion.assignments) # implement this
conn.experiments(EXPERIMENT_ID).observations().create(
suggestion=suggestion.id,
values=values,
)
import com.sigopt.model.*;
import com.sigopt.*;
Sigopt.clientToken = "CLIENT_TOKEN";
Pagination<Suggestion> openSuggestions = new Experiment(EXPERIMENT_ID).suggestions().list()
.addParam("state", "open")
.call();
for (Suggestion os: openSuggestions.iteratePages()) {
double value = evaluateMetric(suggestion.getAssignments()); // implement this
Observation observation = new Experiment(EXPERIMENT_ID).observations().create()
.data(
new Observation.Builder()
.suggestion(suggestion.getID())
.values(java.util.Arrays.asList(
new Value.Builder()
.name("metric_1")
.value(value)
.build()
))
.build()
)
.call();
}

If you know the suggestion ID and wish to report it as a failure

Python
Bash
Java
conn.experiments(EXPERIMENT_ID).observations().create(
suggestion=SUGGESTION_ID,
failed=True,
)
OBSERVATION=`curl -s -X POST https://api.sigopt.com/v1/experiments/EXPERIMENT_ID/observations -u "$SIGOPT_API_TOKEN": \
-H 'Content-Type: application/json' \
-d "{\"suggestion\":\"SUGGESTION_ID\",\"failed\":true}"`
Observation observation = new Experiment(EXPERIMENT_ID).observations().create()
.data(
new Observation.Builder()
.failed(true)
.suggestion(SUGGESTION_ID)
.build()
)
.call();

If you know the suggestion ID and wish to retry evaluating and reporting

Python
Bash
Java
suggestion = conn.experiments(EXPERIMENT_ID).suggestions(SUGGESTION_ID).fetch()
values = evaluate_metric(suggestion.assignments) # Implement this
conn.experiments(EXPERIMENT_ID).observations().create(
suggestion=SUGGESTION_ID,
values=values,
)
The following Bash script requires the jq package for processing JSON.
SUGGESTION=`curl -s -X GET https://api.sigopt.com/v1/experiments/$EXPERIMENT_ID/suggestions/SUGGESTION_ID -u "$SIGOPT_API_TOKEN":`
# Parse the suggestion JSON with jq
SUGGESTION_ID=`echo $SUGGESTION | jq -r '.id'`
parameter_1=`echo $SUGGESTION | jq '.assignments.parameter_1'`
parameter_2=`echo $SUGGESTION | jq '.assignments.parameter_2'`
# implement the `evaluate_metric` script
metric_1 = ./evaluate_metric $parameter_1 $parameter_2
OBSERVATION=`curl -s -X POST https://api.sigopt.com/v1/experiments/$EXPERIMENT_ID/observations -u "$SIGOPT_API_TOKEN": \
-H 'Content-Type: application/json' \
-d "{\"suggestion\":$SUGGESTION_ID,\"values\":[{\"name\":\"metric_1\",\"value\":$metric_1}]}"`
Suggestion suggestion = new Experiment(EXPERIMENT_ID).suggestions(SUGGESTION_ID).fetch().call();
double value = evaluateMetric(suggestion.getAssignments()); // implement this
Observation observation = new Experiment(EXPERIMENT_ID).observations().create()
.data(
new Observation.Builder()
.suggestion(suggestion.id)
.values(java.util.Arrays.asList(
new MetricEvaluation.Builder()
.name("metric1")
.value(value)
.build()
))
.build()
)
.call();

If you know the suggestion ID and want to delete the suggestion

Python
Bash
Java
conn.experiments(EXPERIMENT_ID).suggestions(SUGGESTION_ID).delete()
curl -s -X DELETE https://api.sigopt.com/v1/experiments/EXPERIMENT_ID/suggetions/SUGGESTION_ID -u "$SIGOPT_API_TOKEN":
new Experiment(EXPERIMENT_ID).suggestions(SUGGESTION_ID).delete().call();

If you just want to delete all open suggestions

Python
Bash
Java
conn.experiments(EXPERIMENT_ID).suggestions().delete(state="open")
curl -s -X DELETE https://api.sigopt.com/v1/experiments/EXPERIMENT_ID/suggestions -u "$SIGOPT_API_TOKEN": \
-d state=open
new Experiment(EXPERIMENT_ID).suggestions().deleteList()
.addParam("state", "open")
.call();
Last modified 1yr ago