We'll walk through an example of instrumenting and executing run code with SigOpt. In this tutorial, you will learn how to:
- Install the SigOpt Python client
- Set your SigOpt API token
- Set the project
- Instrument your model
- Create your first Run and log your model metric and parameters to SigOpt
- View your Run in the SigOpt web application
$ python --version
$ pip --version
For notebook instructions and tutorials, check out our GitHub notebook tutorials repo or open the SigOpt Run notebook tutorial in Google Colab.
Install the SigOpt Python package and the libraries required to run the model used for this tutorial.
# Install sigopt
$ pip install sigopt
# Confirm that sigopt >= 8.0.0 is installed
$ sigopt version
# Install XGBoost and scikit-learn. We have tested the sample model used in this tutorial with xgboost==1.5.2, and scikit-learn==1.0.2
$ pip install xgboost scikit-learn
Once you've installed SigOpt, you need to get your API token in order to use the SigOpt API and later explore your Runs and AI Experiments in the SigOpt app. To find your API token, go directly to the API Token page.
# Set sigopt basic configuration. You will be asked to fill in your API token,
# and whether you want SigOpt to collect your model logs and track your model code
$ sigopt config
Runs are created within Projects. The Project allows you to sort and filter through your Runs and AI Experiments and view useful charts to gain insights into everything you've tried. In order to complete this step, first create a Project on the Projects Page or follow the Project Tutorial.
# Set the environment variable to the SigOpt project where your Run will be saved.
$ export SIGOPT_PROJECT=my_first_project
The code below is a sample model instrumented with SigOpt where we highlight how to use SigOpt methods to log and track key model information.
Save the lines below in a script called model.py.
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
# Data preparation required to run and evaluate the sample model
X, y = sklearn.datasets.load_iris(return_X_y=True)
Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.33)
# Track the name of the dataset used for your Run
sigopt.log_dataset("iris 2/3 training, full test")
# Set n_estimators as the hyperparameter to explore for your AI Experiment
# Track the name of the model used for your Run
# Instantiate and train your sample model
model = XGBClassifier(
pred = model.predict(Xtest)
# Track the metric value and metric name for each Run
sigopt.log_metric("accuracy", sklearn.metrics.accuracy_score(pred, ytest))
Use the command
sigopt runto track and execute your model. In our example, we have a python script called model.py.
$ sigopt run python model.py
Once you've run the code above, SigOpt will conveniently output a link to the Run page on our web application.
Click on the Run link to view your completed Run in our web application. Here's a view of a Run page:
In this tutorial, we've covered the recommended way to instrument your Run with SigOpt. After your model has been instrumented, it is easy to take advantage of SigOpt's optimization features. Optimization helps find the parameters for your model that give you the best metric (e.g. maximizing an accuracy metric).
Check out the tutorial, AI Experiment and Optimization Tutorial, to see how you can create an Experiment.