Skip to main content

Model conversion

tip

Recommended tutorials before trying model conversion

1. Add model to registry​

Download a Hugging Face model and save on your disk. In this tutorial, deepset/bert-base-uncased-squad2 model will be downloaded from the Hugging Face Hub. It is a model finetuned on the SQuAD dataset. If the model is saved in ./bert-squad directory, we can upload it to our registry using the following command:

stochasticx models add \
    --name "bert-squad" \
    --dir_path "./bert-squad" \
    --type "hf"
info
  • name: will allow you to identify the model later.
  • dir_path: directory path where your model is located.
  • type: model type. In this case a Hugging Face model

You can list your models with the following command:

stochasticx models ls
CLI output
[+] Collecting all local models
┏━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━┓
┃ Id ┃ Name              ┃ Directory path                                            ┃ Type ┃ Uploaded ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━┩
│ 1  │ bert-squad        │ /vol/registry/models/a30418bc-16dd-4938-812f-3ff5c56d2a18 │ hf   │ True     │
└────┴───────────────────┴───────────────────────────────────────────────────────────┴──────┴──────────┘

2. Convert Pytorch model to ONNX​

You can convert the Pytorch model to ONNX using the following command:

stochasticx conversion convert --model_id 1 --model_type hf --task_type question_answering --convert_type onnx
CLI output
[+] Converting model ...


[+] Model is converted

The ONNX converted model (model_onnx.onnx) is stored in the same directory with your Pytorch model in the registry.

3. Convert ONNX model to TensorRT​

You can convert the ONNX model to TensorRT using the following command:

stochasticx conversion convert --model_id 1 --convert_type tensorrt
CLI output
[+] Converting model ...


[+] Model is converted

The TensorRT converted model (model_trt.engine) is stored in the same directory with your Pytorch model in the registry

4. Convert ONNX model to ONNX int8​

You can convert the ONNX model to ONNX int8 using the following command:

stochasticx conversion convert --model_id 1 --convert_type onnx_int8
CLI output
[+] Converting model ...


[+] Model is converted

The ONNX int8 converted model (model_onnx_quantized.onnx) is stored in the same directory with your Pytorch model in the registry