Manage Open Suggestions

If you run into an error, there may be some outstanding suggestions with no reported observations (called open suggestions in the API). Here are some strategies you can use to recover in this situation.

Iterate through all open suggestions and reevaluate

Python

suggestions = conn.experiments(EXPERIMENT_ID).suggestions().fetch(state="open")
for suggestion in suggestions.iterate_pages():
  value = evaluate_metric(suggestion.assignments)  # implement this
  conn.experiments(EXPERIMENT_ID).observations().create(
    suggestion=suggestion.id,
    values=values,
  )

Java

import com.sigopt.model.*;
import com.sigopt.*;

Sigopt.clientToken = "CLIENT_TOKEN";

Pagination<Suggestion> openSuggestions = new Experiment(EXPERIMENT_ID).suggestions().list()
  .addParam("state", "open")  
  .call();

for (Suggestion os: openSuggestions.iteratePages()) {
  double value = evaluateMetric(suggestion.getAssignments()); // implement this
  Observation observation = new Experiment(EXPERIMENT_ID).observations().create()
  .data(
    new Observation.Builder()
      .suggestion(suggestion.getID())
      .values(java.util.Arrays.asList(
        new Value.Builder()
          .name("metric_1")
          .value(value)
          .build()
        ))
      .build()
    )
  .call();
}

If you know the suggestion ID and wish to report it as a failure

Python

conn.experiments(EXPERIMENT_ID).observations().create(
  suggestion=SUGGESTION_ID,
  failed=True,
)

Bash

OBSERVATION=`curl -s -X POST https://api.sigopt.com/v1/experiments/EXPERIMENT_ID/observations -u "$SIGOPT_API_TOKEN": \
  -H 'Content-Type: application/json' \
  -d "{\"suggestion\":\"SUGGESTION_ID\",\"failed\":true}"`

Java

Observation observation = new Experiment(EXPERIMENT_ID).observations().create()
  .data(
    new Observation.Builder()
      .failed(true)
      .suggestion(SUGGESTION_ID)
      .build()
    )
  .call();

If you know the suggestion ID and wish to retry evaluating and reporting

Python

suggestion = conn.experiments(EXPERIMENT_ID).suggestions(SUGGESTION_ID).fetch()
values = evaluate_metric(suggestion.assignments)  # Implement this
conn.experiments(EXPERIMENT_ID).observations().create(
  suggestion=SUGGESTION_ID,
  values=values,
)

Bash

The following Bash script requires the jq package for processing JSON.

SUGGESTION=`curl -s -X GET https://api.sigopt.com/v1/experiments/$EXPERIMENT_ID/suggestions/SUGGESTION_ID -u "$SIGOPT_API_TOKEN":`

# Parse the suggestion JSON with jq
SUGGESTION_ID=`echo $SUGGESTION | jq -r '.id'`
parameter_1=`echo $SUGGESTION | jq '.assignments.parameter_1'`
parameter_2=`echo $SUGGESTION | jq '.assignments.parameter_2'`

# implement the `evaluate_metric` script
metric_1 = ./evaluate_metric $parameter_1 $parameter_2

OBSERVATION=`curl -s -X POST https://api.sigopt.com/v1/experiments/$EXPERIMENT_ID/observations -u "$SIGOPT_API_TOKEN": \
  -H 'Content-Type: application/json' \
  -d "{\"suggestion\":$SUGGESTION_ID,\"values\":[{\"name\":\"metric_1\",\"value\":$metric_1}]}"`

Java

Suggestion suggestion = new Experiment(EXPERIMENT_ID).suggestions(SUGGESTION_ID).fetch().call();
double value = evaluateMetric(suggestion.getAssignments()); // implement this
Observation observation = new Experiment(EXPERIMENT_ID).observations().create()
  .data(
    new Observation.Builder()
      .suggestion(suggestion.id)
      .values(java.util.Arrays.asList(
        new MetricEvaluation.Builder()
          .name("metric1")
          .value(value)
          .build()
        ))
      .build()
    )
  .call();

If you know the suggestion ID and want to delete the suggestion

Python

conn.experiments(EXPERIMENT_ID).suggestions(SUGGESTION_ID).delete()

Bash

curl -s -X DELETE https://api.sigopt.com/v1/experiments/EXPERIMENT_ID/suggetions/SUGGESTION_ID -u "$SIGOPT_API_TOKEN":

Java

new Experiment(EXPERIMENT_ID).suggestions(SUGGESTION_ID).delete().call();

If you just want to delete all open suggestions

Python

conn.experiments(EXPERIMENT_ID).suggestions().delete(state="open")

Bash

curl -s -X DELETE https://api.sigopt.com/v1/experiments/EXPERIMENT_ID/suggestions -u "$SIGOPT_API_TOKEN": \
  -d state=open

Java

new Experiment(EXPERIMENT_ID).suggestions().deleteList()
  .addParam("state", "open")  
  .call();