Record SigOpt Runs with Python IDE and SigOpt CLI
With SigOpt installed and your Python environment set up, let's take a look at how to record a SigOpt Run in a Python IDE with the SigOpt CLI.

Instrument the Run

1
import tensorflow as tf
2
import sigopt
3
import os
4
5
os.environ["SIGOPT_API_TOKEN"] = # SIGOPT_API_TOKEN
6
os.environ["SIGOPT_PROJECT"] = "run-examples"
7
8
9
class KerasNNModel:
10
def __init__(self, hidden_layer_size, activation_fn):
11
model = tf.keras.Sequential(
12
[
13
tf.keras.layers.Flatten(input_shape=(28, 28)),
14
tf.keras.layers.Dense(hidden_layer_size, activation=activation_fn),
15
tf.keras.layers.Dense(10),
16
]
17
)
18
self.model = model
19
20
def get_keras_nn_model(self):
21
return self.model
22
23
def train_model(self, train_images, train_labels, optimizer_type, metrics_list, num_epochs):
24
self.model.compile(
25
optimizer=optimizer_type,
26
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
27
metrics=metrics_list,
28
)
29
self.model.fit(train_images, train_labels, epochs=num_epochs)
30
31
def evaluate_model(self, test_images, test_labels):
32
metrics_dict = self.model.evaluate(test_images, test_labels, verbose=2, return_dict=True)
33
return metrics_dict
34
35
36
def load_data_train_model():
37
sigopt.log_dataset(name="mnist")
38
(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()
39
40
# set model training, architecture parameters and hyperparameters
41
sigopt.params.num_epochs = 2
42
sigopt.params.hidden_layer_size = 200
43
sigopt.params.activation_fn = "relu"
44
45
# create the model
46
keras_nn_model = KerasNNModel(
47
hidden_layer_size=sigopt.params.hidden_layer_size, activation_fn=sigopt.params.activation_fn
48
)
49
sigopt.log_model("Keras NN Model with 1 Hidden layer")
50
51
# train the model
52
keras_nn_model.train_model(train_images, train_labels, "adam", ["accuracy"], sigopt.params.num_epochs)
53
sigopt.log_metadata("sgd optimizer", "adam")
54
metrics_dict = keras_nn_model.evaluate_model(test_images, test_labels)
55
56
# log performance metrics
57
sigopt.log_metric("accuracy", metrics_dict["accuracy"])
58
sigopt.log_metric("loss", metrics_dict["loss"])
59
60
61
if __name__ == "__main__":
62
load_data_train_model()
Copied!

Run the Code with the SigOpt CLI

1
$ sigopt run python keras_model.py
Copied!
1
Run started, view it on the SigOpt dashboard at https://app.sigopt.com/run/1234
2
Epoch 1/2
3
1875/1875 [==============================] - 5s 2ms/step - loss: 2.7513 - accuracy: 0.8826
4
Epoch 2/2
5
1875/1875 [==============================] - 4s 2ms/step - loss: 0.3313 - accuracy: 0.9265
6
313/313 - 0s - loss: 0.2941 - accuracy: 0.9478
7
Run finished, view it on the SigOpt dashboard at https://app.sigopt.com/run/1234
Copied!

Orchestrate a Run with the SigOpt CLI

SigOpt Orchestrate allows you to confidently scale your modeling experimentation on either your existing Kubernetes infrastructure or a Kubernetes cluster managed by SigOpt. To set up SigOpt Orchestrate, follow this installation guide. With SigOpt Orchestrate installed, all you have to do is set up yourrun.yml file and execute the following command locally to orchestrate your Run on your Kubernetes cluster:
1
#run.yml
2
name: My Run
3
run: python keras_model.py
4
resources:
5
limits:
6
cpu: 2
7
memory: 4Gi
8
gpus: 1
9
image: my-run bab
Copied!
1
$ sigopt cluster run --run-file run.yml
Copied!
When you execute the above, SigOpt Orchestrate will look into the specified run.yml file. It will allocate 2 CPUs, 4 GiB memory, and 1 GPU of your kubernetes cluster for your Run, and will execute the command provided for run.
1
Run started, view it on the SigOpt dashboard at https://app.sigopt.com/run/1234
2
Epoch 1/2
3
1875/1875 [==============================] - 5s 2ms/step - loss: 2.7513 - accuracy: 0.8826
4
Epoch 2/2
5
1875/1875 [==============================] - 4s 2ms/step - loss: 0.3313 - accuracy: 0.9265
6
313/313 - 0s - loss: 0.2941 - accuracy: 0.9478
7
Run finished, view it on the SigOpt dashboard at https://app.sigopt.com/run/1234
Copied!