Quickstart

Scale large Deep Learning models, deliver blazing fast inferences and optimize infraestructure costs with the Stochastic tools.

tip

It is recommended that you first familiarize yourself with the Stochastic platform before beginning this tutorial.

1. Set up the Stochastic

a) Sign up for a free account at https://app.stochastic.ai/signup.

b) Install the Stochastic library on your machine in a Python 3 environment using pip or conda. The CLI will be automatically installed in your computer.

Pip

pip install stochasticx

Conda

conda install stochasticx

c) Verify the installation.

Python

python -c "import stochasticx ; print(stochasticx.__version__)"

Output

0.0.1

CLI

stochasticx --help

CLI output

Usage: stochasticx [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  datasets
  deployments
  inference
  instances
  jobs
  login
  me
  models
  stable-diffusion

2. Sign in

Before you start using the CLI, you need to log in with your username and password.

Python

from stochasticx.auth.auth import Stochastic

username = "your_email@email.com"
password = "your_password"
Stochastic().login(username, password)

CLI

stochasticx login --username "my_email@email.com" --password "my_password"

CLI output

[+] Login successfully

Platform

You can log in on the Stochastic platform website: https://app.stochastic.ai/login

3. Available classes

Python

In the Stochastic platform documentation, you can upload and download models, the same for datasets, start optimization jobs and deploy these optimized models in a production-ready inference engine. So, all this functionalities have been encapsulated in Python classes and functions. Below you can see a brief summary of the available classes.

- `Models`: static class to list all the uploaded models in the Stochastic platform. You can also list the optimized models.
- `Model`: each model returned by the `Models` class is encapsulated in the `Model` class, which allows you to get specific information about that model, download it in your local machine, etc.
- `OptimizedModel`: it is similar to the Model class but for models that have already been optimized.
- `Datasets`: static class to list all the uploaded datasets in the Stochastic platform.
- `Dataset`: each dataset returned by the `Datasets` class is encapsulated in the `Dataset` class, which allows you to get specific information about that dataset, download it in your local machine, etc.
- `Jobs`: static class to list all the jobs launched in the Stochastic platform.
- `Job`: each job returned by the `Jobs` class is encapsulated in the `Job` class, which allows you to get specific information about that job.
- `Deployments`: static class to list all the deployments launched in the Stochastic platform.
- `Deployment`: each deployment returned by the `Deployments` class is encapsulated in the `Deployment` class, which allows you to get specific information about that deployment, like the endpoint, API key, ...

CLI

Usage: stochasticx [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  datasets
  deployments
  inference
  instances
  jobs
  login
  me
  models
  stable-diffusion

The CLI is organized into different commands. You have commands to work with datasets, models, deployments, inferences, ... These commands can in turn have other commands. For instance, you can list all available models that you have uploaded to the Stochastic platform by running this command stochasticx models ls

4. Models and datasets

Before starting an optimization job, we will upload a HuggingFace dataset and a HuggingFace model in the Stochastic Platform using the Python library.

First download the HuggingFace dataset and save it in the disk. We will use the SQuAD dataset that contains questions, answers and contexts.

from datasets import load_dataset
dataset = load_dataset("squad")
dataset.save_to_disk("./squad")

Once you have saved it in the disk, you can upload it to the Stochastic Platform.

Python

from stochasticx.datasets.datasets import Dataset, DatasetType

dataset_to_upload = Dataset(
    name="SQuAD",
    dir_path="./squad",
    type=DatasetType.HUGGINGFACE
)

print(dataset_to_upload)

dataset_to_upload.upload()
print(dataset_to_upload)

Output

ID: None ; Dataset name: SQuAD ; Directory path: squad/ ; Dataset type: hf ; Uploaded: False
ID: 62e3a4eab5beb300264408ee ; Dataset name: SQuAD ; Directory path: squad/ ; Dataset type: hf ; Uploaded: True

CLI

stochasticx datasets upload \
    --name "squad_dataset" \
    --dir_path "./squad" \
    --type "hf"

Platform

Go to the menu on the left and click on Datasets. Once there, you should see a Add dataset button in the top left corner.

Once you have clicked on Add dataset, a new window will be showed. You have to enter a dataset name, description, type (select HuggingFace), task (For Squad is question answering) and select the folder containing your SQuAD dataset that you downloaded from the HuggingFace Hub. See more details...

info

• name: will allow you to identify the dataset later.
• dir_path: directory path where your dataset is located.
• type: dataset type. Right now we are supporting Hugging Face, CSV, JSON, JSONL datasets.

Then, we will have to do the same for the model. In this tutorial, we will use the following one bert-base-uncased. Feel free to select another BERT model.

from transformers import AutoTokenizer, AutoModelForQuestionAnswering

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
tokenizer.save_pretrained("./bert")

model = AutoModelForQuestionAnswering.from_pretrained("bert-base-uncased")
model.save_pretrained("./bert")

Now you can upload this model to the Stochastic platform.

Python

from stochasticx.models.models import Model, ModelType

model_to_upload = Model(
    name="bert-squad",
    dir_path="bert-squad/",
    type=ModelType.HUGGINGFACE
)

print(model_to_upload)

model_to_upload.upload()
print(model_to_upload)

Output

Model ID: None ; Name: bert-squad ; Directory path: bert-squad/ ; Model type: hf ; Uploaded: False
Model ID: 62e3a5aab5beb3002644093e ; Name: bert-squad ; Directory path: bert-squad/ ; Model type: hf ; Uploaded: True

CLI

!stochasticx models upload \
    --name "BERT SQuAD" \
    --dir_path "./bert-squad" \
    --type "hf"

Platform

Go to the menu on the left and click on Models. Once there, you should see a Add model button in the top left corner.

Once you have clicked on Add model a new window will be showed. You have to enter a model name, the model type (select HuggingFace) and select the folder containing your BERT model that you downloaded from the HuggingFace Hub.

info

• name: will allow you to identify the model later.
• dir_path: directory path where your model is located.
• type: model type. In this case a Hugging Face model

You can list out your models:

Python

from stochasticx.models.models import Models

print("\n--- Models ---")
models = Models.get_models()

for model in models:
    print(model)

Output

--- Models ---
Model ID: 62e3a5aab5beb3002644093e ; Name: bert ; Directory path: None ; Model type: hf ; Uploaded: True

CLI

stochasticx models ls

CLI output

[+] Collecting uploaded models

┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━┓
┃ Id                       ┃ Name                    ┃ Directory path ┃ Type ┃ Uploaded ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━┩
│ 62e15b3db5beb3002644046e │ bert                    │                │ hf   │ True     │
│ 62d545be860f360027140b74 │ distill-bert            │                │ hf   │ True     │
└──────────────────────────┴─────────────────────────┴────────────────┴──────┴──────────┘

Platform

You should see your models listed on the models page. Here we can see the model is in status pending, it will become successful once we ensured all the files are correct. Only successfully verified models can be used for running jobs.

You can also list your datasets:

Python

from stochasticx.datasets.datasets import Datasets

print("--- Datasets ---\n")
datasets = Datasets.get_datasets()

for dataset in datasets:
    print(dataset)

Output

--- Datasets ---
ID: 62e3a4eab5beb300264408ee ; Dataset name: SQuAD ; Directory path: squad/ ; Dataset type: hf ; Uploaded: True
ID: 62d92ff2b3828f002719dba7 ; Dataset name: GLUE ; Directory path: None ; Dataset type: None ; Uploaded: True

CLI

stochasticx datasets ls

CLI output

[+] Collecting all datasets

┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┓
┃ Id                       ┃ Name                         ┃ Directory path ┃ Type  ┃ Uploaded ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━┩
│ 62e903a198855200266c03fe │ squad                        │                │ hf    │ True     │
│ 62d92ff2b3828f002719dba7 │ GLUE                         │                │       │ True     │
└──────────────────────────┴──────────────────────────────┴────────────────┴───────┴──────────┘

Platform

You should see your dataset listed on the datasets page.

5. Finetune or accelerate your favourite model for question answering

To start a finetuning or acceleration job, you will need to specify the following information:

• Job name: will allow you to identify the job later.
• Model ID: the model that you want to finetune and optimize.
• Dataset ID: dataset that will be used to finetune and evaluate your model.
• Optimization criteria: latency or lossless.
  • latency optimization will optimize the model latency and thoughput without taking into account the loss of accuracy. The accuracy is usually reduced around 3%, but the speedup can be x7 or more.
  • lossless optimization will optimize your model without loosing accuracy.
• Task type: right now we only support the following tasks: sequence classification, question answering, summarization, translation and token classification.
• Dataset columns: depending on the task you will have to map dataset columns to job metadata. For example, for question answering you will need to specify which column of the dataset contains the question, which one contains the context and which one contains the answer.

To get the columns of your dataset you can execute the following code or command, you also can try our web platform:

Python

dataset = Datasets.get_dataset("62e3a4eab5beb300264408ee")
print(dataset.get_column_names())

Output

['id', 'title', 'context', 'question', 'answers']

To run this example job, BERT model and squad dataset will be used. They had been listed in the previous section

from stochasticx.models.models import Models
from stochasticx.datasets.datasets import Datasets
from stochasticx.jobs.jobs import Job, OptimizationCriteria, QuestionAnsweringTask

model = Models.get_model("62e3a5aab5beb3002644093e")
dataset = Datasets.get_dataset("62e3a4eab5beb300264408ee")

q_a_task = QuestionAnsweringTask(
    question_column="question_column",
    answer_column="answer_column",
    context_column="context_column"
)

job = Job(
    name="quickstart_tutorial"
)

job.launch_auto(
    model=model,
    dataset=dataset,
    task_type=q_a_task,
    optimization_criteria=OptimizationCriteria.LATENCY
)

Now, list your jobs.

from stochasticx.jobs.jobs import Jobs

jobs = Jobs.get_jobs()

for job in jobs:
    print(job)

Output

Job ID: 62e9071c98855200266c0434 ; Name: quickstart_tutorial ; Status: New ; Created at: 2022-08-02T11:14:36.485Z ; Optimization type: auto ; Optimization criteria: latency

If you are seeing something similar, congratulations, you have started your first job. The job status you have started should be New, which means that the job is still running. When it finishes, its status should change to Successful.

CLI

stochasticx datasets columns --id 62e903a198855200266c03fe

CLI output

[+] Collecting columns from the dataset

['id', 'title', 'context', 'question', 'answers']

To run this example job, BERT model and squad dataset will be used. They had been listed in the previous section

stochasticx jobs launch question_answering \
    --job_name "quickstart_tutorial" \
    --model_id "62e15b3db5beb3002644046e" \
    --dataset_id "62e903a198855200266c03fe" \
    --optimization_criteria "latency" \
    --question_column "question" \
    --answer_column "answers" \
    --context_column "context"

Now, list your jobs.

stochasticx jobs ls

CLI Output

[+] Collecting all jobs

┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Id                       ┃ Name                ┃ Status       ┃ Created at          ┃ Optimization type ┃ Optimization criteria ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩
│ 62e9071c98855200266c0434 │ quickstart_tutorial │ New          │ 2022-08-02 11:14:36 │ auto              │ latency               │
└──────────────────────────┴─────────────────────┴──────────────┴─────────────────────┴───────────────────┴───────────────────────┘

If you are seeing something similar, congratulations, you have started your first job using the CLI. Now, wait until your job is finished.

Platform

Once you have uploaded some datasets and models, you can start an optimization job. For this, go to Jobs and click on Create job. Enter a job name, select Question answering as task type, Uploaded models as model source, select your uploaded model and dataset and finally select an optimization criteria. Before starting the job, you will also need to specify which column of the dataset contains the questions, which one contains the answers and which one contains the contexts.

info

• Lantecy optimization: reduce latency as much as possible, but the accuracy could be reduced (usually no more than a 3%).
• Lossless optimization: reduce latency without losing accuracy. Keep in mind that the latency reduction will be worse than the latency optimization.

After launching your jobs, you should see its New status, which means the job is running. Once the job has finished, you should be able to visualize the results. In this example, you can see that the optimized model is above 3.5 times faster than the original one in batch size 4 and sequence length 8. Go to the Accuracy tab to check the accuracy and other metrics of the models.

If you like the results of your optimized model, don't forget to save it. For this, go to the menu on the left and click on Jobs again. Now, you should see an arrow in the job, expand it and click on the save button. Give a name to this model and it will be permanently saved. If you want to download it go to the menu on the left and click on Models. There you should see all your saved models.