Manage Open Suggestions

If you run into an error, there may be some outstanding suggestions with no reported observations (called open suggestions in the API). Here are some strategies you can use to recover in this situation.
Iterate through all open suggestions and reevaluate
Python
Java
suggestions = conn.experiments(EXPERIMENT_ID).suggestions().fetch(state="open")
for suggestion in suggestions.iterate_pages():
  value = evaluate_metric(suggestion.assignments)  # implement this
  conn.experiments(EXPERIMENT_ID).observations().create(
    suggestion=suggestion.id,
    values=values,
  )
import com.sigopt.model.*;
import com.sigopt.*;
​
Sigopt.clientToken = "CLIENT_TOKEN";
​
Pagination<Suggestion> openSuggestions = new Experiment(EXPERIMENT_ID).suggestions().list()
  .addParam("state", "open")  
  .call();
​
for (Suggestion os: openSuggestions.iteratePages()) {
  double value = evaluateMetric(suggestion.getAssignments()); // implement this
  Observation observation = new Experiment(EXPERIMENT_ID).observations().create()
  .data(
    new Observation.Builder()
      .suggestion(suggestion.getID())
      .values(java.util.Arrays.asList(
        new Value.Builder()
          .name("metric_1")
          .value(value)
          .build()
        ))
      .build()
    )
  .call();
}
If you know the suggestion ID and wish to report it as a failure
Python
Bash
Java
conn.experiments(EXPERIMENT_ID).observations().create(
  suggestion=SUGGESTION_ID,
  failed=True,
)
OBSERVATION=`curl -s -X POST https://api.sigopt.com/v1/experiments/EXPERIMENT_ID/observations -u "$SIGOPT_API_TOKEN": \
  -H 'Content-Type: application/json' \
  -d "{\"suggestion\":\"SUGGESTION_ID\",\"failed\":true}"`
Observation observation = new Experiment(EXPERIMENT_ID).observations().create()
  .data(
    new Observation.Builder()
      .failed(true)
      .suggestion(SUGGESTION_ID)
      .build()
    )
  .call();
If you know the suggestion ID and wish to retry evaluating and reporting
Python
Bash
Java
suggestion = conn.experiments(EXPERIMENT_ID).suggestions(SUGGESTION_ID).fetch()
values = evaluate_metric(suggestion.assignments)  # Implement this
conn.experiments(EXPERIMENT_ID).observations().create(
  suggestion=SUGGESTION_ID,
  values=values,
)
The following Bash script requires the jq package for processing JSON.
SUGGESTION=`curl -s -X GET https://api.sigopt.com/v1/experiments/$EXPERIMENT_ID/suggestions/SUGGESTION_ID -u "$SIGOPT_API_TOKEN":`
​
# Parse the suggestion JSON with jq
SUGGESTION_ID=`echo $SUGGESTION | jq -r '.id'`
parameter_1=`echo $SUGGESTION | jq '.assignments.parameter_1'`
parameter_2=`echo $SUGGESTION | jq '.assignments.parameter_2'`
​
# implement the `evaluate_metric` script
metric_1 = ./evaluate_metric $parameter_1 $parameter_2
​
OBSERVATION=`curl -s -X POST https://api.sigopt.com/v1/experiments/$EXPERIMENT_ID/observations -u "$SIGOPT_API_TOKEN": \
  -H 'Content-Type: application/json' \
  -d "{\"suggestion\":$SUGGESTION_ID,\"values\":[{\"name\":\"metric_1\",\"value\":$metric_1}]}"`
​
Suggestion suggestion = new Experiment(EXPERIMENT_ID).suggestions(SUGGESTION_ID).fetch().call();
double value = evaluateMetric(suggestion.getAssignments()); // implement this
Observation observation = new Experiment(EXPERIMENT_ID).observations().create()
  .data(
    new Observation.Builder()
      .suggestion(suggestion.id)
      .values(java.util.Arrays.asList(
        new MetricEvaluation.Builder()
          .name("metric1")
          .value(value)
          .build()
        ))
      .build()
    )
  .call();
If you know the suggestion ID and want to delete the suggestion
Python
Bash
Java
conn.experiments(EXPERIMENT_ID).suggestions(SUGGESTION_ID).delete()
curl -s -X DELETE https://api.sigopt.com/v1/experiments/EXPERIMENT_ID/suggetions/SUGGESTION_ID -u "$SIGOPT_API_TOKEN":
new Experiment(EXPERIMENT_ID).suggestions(SUGGESTION_ID).delete().call();
If you just want to delete all open suggestions
Python
Bash
Java
conn.experiments(EXPERIMENT_ID).suggestions().delete(state="open")
curl -s -X DELETE https://api.sigopt.com/v1/experiments/EXPERIMENT_ID/suggestions -u "$SIGOPT_API_TOKEN": \
  -d state=open
new Experiment(EXPERIMENT_ID).suggestions().deleteList()
  .addParam("state", "open")  
  .call();

API Errors

Metadata

Last modified 6mo ago