Optimization jobs

tip

Recommended tutorials before starting starting to work with optimization jobs.

• Stochastic platform introduction
• Installation
• Quickstart
• Upload and download datasets
• Upload and download models

1. Sign in

Before you start using the CLI, you need to log in with your username and password.

Python

from stochasticx.auth.auth import Stochastic

username = "your_email@email.com"
password = "your_password"
Stochastic().login(username, password)

CLI

stochasticx login --username "my_email@email.com" --password "my_password"

Platform

You can log in on the Stochastic platform website: https://app.stochastic.ai/login

2. Create a job

A job allows you to finetune and accelerate a model. In this tutorial a job will be launched to finetune and accelerate a BERT model using the SQuAD dataset downloaded from the Hugging Face Hub. This model will be able to retrieve the answer to a question, given the context. Perform the following steps to start the job:

1. Download the SQuAD dataset from the Hugging Face Hub:

from datasets import load_dataset
dataset = load_dataset("squad")
dataset.save_to_disk("./squad")

2. Upload the dataset to the Stochastic platform:

Python

from stochasticx.datasets.datasets import Dataset, DatasetType

dataset_to_upload = Dataset(
    name="squad_dataset",
    dir_path="./squad",
    type=DatasetType.HUGGINGFACE
)

dataset_to_upload.upload()

CLI

stochasticx datasets upload \
    --name "squad_dataset" \
    --dir_path "./squad" \
    --type "hf"

Platform

Go to the menu on the left and click on Datasets. Once there, you should see a Add dataset button in the top left corner.

Once you have clicked on Add dataset, a new window will be showed. You have to enter a dataset name, description, type (select HuggingFace), task (For Squad is question answering) and select the folder containing your SQuAD dataset that you downloaded from the HuggingFace Hub. See more details...

3. Download the BERT model from the Hugging Face Hub:

from transformers import AutoTokenizer, AutoModelForQuestionAnswering

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
tokenizer.save_pretrained("./bert")

model = AutoModelForQuestionAnswering.from_pretrained("bert-base-uncased")
model.save_pretrained("./bert")

4. Upload the model to the Stochastic platform:

Python

from stochasticx.models.models import Model, ModelType

model_to_upload = Model(
    name="BERT",
    dir_path="./bert",
    type=ModelType.HUGGINGFACE
)

model_to_upload.upload()

CLI

stochasticx models upload \
    --name "BERT" \
    --dir_path "./bert" \
    --type "hf"

Platform

Go to the menu on the left and click on Models. Once there, you should see a Add model button in the top left corner.

Once you have clicked on Add model a new window will be showed. You have to enter a model name, the model type (select HuggingFace) and select the folder containing your BERT model that you downloaded from the HuggingFace Hub.

5. Get the ID of the uploaded dataset.

Python

from stochasticx.datasets.datasets import Datasets

datasets = Datasets.get_datasets()

for dataset in datasets:
    print(dataset)

Output

ID: 62e912a598855200266c0478 ; Dataset name: squad_dataset ; Directory path: None ; Dataset type: hf ; Uploaded: True

CLI

stochasticx datasets ls

CLI output

[+] Collecting all datasets

┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┓
┃ Id                       ┃ Name                         ┃ Directory path ┃ Type  ┃ Uploaded ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━┩
│ 62e912a598855200266c0478 │ squad_dataset                │                │ json  │ True     │
└──────────────────────────┴──────────────────────────────┴────────────────┴───────┴──────────┘

Platform

Now you should see your dataset listed on the datasets page.

6. Get the ID of the uploaded model.

Python

from stochasticx.models.models import Models

models = Models.get_models()

for model in models:
    print(model)

Output

Model ID: 62e9195598855200266c0497 ; Name: BERT ; Directory path: None ; Model type: hf ; Uploaded: True

CLI

stochasticx models ls

CLI output

[+] Collecting uploaded models

┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┓
┃ Id                       ┃ Name                         ┃ Directory path ┃ Type  ┃ Uploaded ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━┩
│ 62e9195598855200266c0497 │ BERT                         │                │ hf  │ True     │
└──────────────────────────┴──────────────────────────────┴────────────────┴───────┴──────────┘

Platform

Now you should see your model listed on the models page.

7. Get the dataset columns

Python

dataset = Datasets.get_dataset("62e912a598855200266c0478")
print(dataset.get_column_names())

Output

['id', 'title', 'context', 'question', 'answers']

CLI

stochasticx datasets columns --id 62e912a598855200266c0478

CLI output

[+] Collecting columns from the dataset

['id', 'title', 'context', 'question', 'answers']

Platform

Click on a dataset card to get its description, number of training samples, columns and preview some of the data samples.

8. Start the optimization job

Python

from stochasticx.models.models import Models
from stochasticx.datasets.datasets import Datasets
from stochasticx.jobs.jobs import Job, OptimizationCriteria, QuestionAnsweringTask


model = Models.get_model("62e9195598855200266c0497")
dataset = Datasets.get_dataset("62e912a598855200266c0478")

q_a_task = QuestionAnsweringTask(
    question_column="question_column",
    answer_column="answer_column",
    context_column="context_column"
)

job = Job(
    name="job_tutorial"
)

job.launch_auto(
    model=model,
    dataset=dataset,
    task_type=q_a_task,
    optimization_criteria=OptimizationCriteria.LATENCY
)

CLI

stochasticx jobs launch question_answering \
    --job_name "job_tutorial" \
    --model_id "62e9195598855200266c0497" \
    --dataset_id "62e912a598855200266c0478" \
    --optimization_criteria "latency" \
    --question_column "question" \
    --answer_column "answers" \
    --context_column "context"

Platform

Once you have uploaded some datasets and models, you can start an optimization job. For this, go to Jobs and click on Create job. Enter a job name, select Question answering as task type, Uploaded models as model source, select your uploaded model and dataset and finally select an optimization criteria. Before starting the job, you will also need to specify which column of the dataset contains the questions, which one contains the answers and which one contains the contexts.

info

• Lantecy optimization: reduce latency as much as possible, but the accuracy could be reduced (usually no more than a 3%).
• Lossless optimization: reduce latency without losing accuracy. Keep in mind that the latency reduction will be worse than the latency optimization.

3. List your jobs

Use the following command to list the jobs. The job status you have started should be New, which means that the job is still running. When it finishes, its status should change to Successful.

Python

from stochasticx.jobs.jobs import Jobs

jobs = Jobs.get_jobs()

for job in jobs:
    print(job)

CLI output

Job ID: 62e92b8998855200266c0584 ; Name: job_tutorial ; Status: New ; Created at: 2022-08-02T13:50:01.240Z ; Optimization type: auto ; Optimization criteria: latency
Job ID: 62d93018b3828f002719dbc9 ; Name: Test ; Status: successful ; Created at: 2022-07-21T10:53:12.844Z ; Optimization type: auto ; Optimization criteria: latency

CLI

stochasticx jobs ls

CLI output

[+] Collecting all jobs

┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Id                       ┃ Name                      ┃ Status       ┃ Created at          ┃ Optimization type ┃ Optimization criteria ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩
│ 6357eace1da66b553c95f89d │ job_tutorial              │ New          │ 2022-08-02 13:55:26 │ auto              │ latency               │
│ 62d93018b3828f002719dbc9 │ Test                      │ successful   │ 2022-07-21 10:53:12 │ auto              │ latency               │
└──────────────────────────┴───────────────────────────┴──────────────┴─────────────────────┴───────────────────┴───────────────────────┘

Platform

In this example, you can see that the optimized model is above 3.5 times faster than the original one in batch size 4 and sequence length 8. Go to the Accuracy tab to check the accuracy and other metrics of the models.

If you like the results of your optimized model, don't forget to save it. For this, go to the menu on the left and click on Jobs again. Now, you should see an arrow in the job, expand it and click on the save button. Give a name to this model and it will be permanently saved. If you want to download it go to the menu on the left and click on Models. There you should see all your saved models.