Skip to main content

Inference

tip

Recommended tutorials before starting to work with local inference jobs.

1. Sign in​

Before you start using the CLI, you need to log in with your username and password.

stochasticx login

If you don't have an account, you can create one on the Stochastic platform website: https://app.stochastic.ai/signup

2. Initialize the local mode in the CLI​

By default, the stochasticx CLI will be communicating with the cloud infrastructure. If you want to work in the local mode, you can execute:

stochasticx local init

This will download and start some Docker containers needed to work locally. If you want to check in which mode you are working, you can execute the following command:

stochasticx config inspect
CLI output
[+] You are in the **local** mode...

To switch to the cloud mode, just execute:

stochasticx config cloud
CLI output
[+] Setting up **local** mode...

3. Requirements​

Before deploying a model you need to finetune or optimize it. You can follow the local finetuning job tutorial to get a model ready to be deployed.

4. Deploy your model​

In this guide we are going to deploy the model that was obtained after doing the finetuning. Execute the following command:

stochasticx deployments deploy sequence_classification --model_name_or_id finetuning_job_name_optimized --model_type hf
CLI output
[+] Starting deployment...
[+] Model deployed

List your deployments using the following command:

CLI output
[+] Collecting all deployments

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Model ID                             ┃ Model name                    ┃ Model type ┃ CUDA ┃ Task type               ┃ User params                                  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ d5063658-ac67-42cd-bbdb-7ec653b79c94 │ finetuning_job_name_optimized │ hf         │ True │ sequence_classification │ {'max_batch_size': 8, 'max_seq_length': 128} │
└──────────────────────────────────────┴───────────────────────────────┴────────────┴──────┴─────────────────────────┴──────────────────────────────────────────────┘

Now you can run inferences on this model.

stochasticx inference sequence_classification --deployment_id d5063658-ac67-42cd-bbdb-7ec653b79c94 --text "I love Stochastic"
CLI output
Labels: ['0']
Scores: [0.9722172021865845]

5. Delete your deployment​

When you have finished, you can delete your deployment executing the following command:

stochasticx deployments delete --model_name_or_id finetuning_job_name_optimized --model_type hf
CLI output
[+] Deleting deployment...

[+] Deleted

To be sure you have deleted the deployment, you can list your deployments:

stochasticx deployments ls
CLI output
[+] Collecting all deployments