Benchmarking

tip

Recommended tutorials before starting to generate images with Stable Diffusion.

• Stochastic platform introduction
• Installation
• Local setup
• Manage models

Benchmarking is a proces of measuring model performance. In tutorial, we will benchmark optimized model in onnx format.

Running container locally

First of all we need to start local container. To do that we need to run following command:

stochasticx local init

We can check local container logs with following command:

stochasticx local logs

Model selection

To list all registered models we need to run following command:

stochasticx models ls

Here we should see list like that:

CLI output

[+] Collecting all local models

┏━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━┓
┃ Id ┃ Name       ┃ Directory path                                            ┃ Type ┃ Uploaded ┃
┡━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━┩
│ 1  │ onnx_model │ /vol/registry/models/115de1ed-7171-46ee-9cbe-3ecf66cb23f0 │ onnx │   True   │
└────┴────────────┴───────────────────────────────────────────────────────────┴──────┴──────────┘

If list is empty you should register some models before doing next steps.

When you have optimized model you want to benchmark in registry you can proceed to next step.

Benchmarking

To benchmark model we need to run following command:

stochasticx benchmarking start

With specified parameters.

info

• help: information about command and it's parameters

stochasticx benchmarking start --help

info

• job_name: name of benchmarking job, shold be unique
• params_file: path to json file with benchmarking parameters
• model_id: id of model to benchmark from model registry

For example for onnx_benchmark we should specify input_info and triton_info in params file.

CLI output

{
  "input_infos": [
    {
      "input_ids": {
        "shape": [
          16,
          64
        ],
        "dtype": "INT32"
      },
      "attention_mask": {
        "shape": [
          16,
          64
        ],
        "dtype": "INT32"
      }
    }
  ],
  "use_gpu": true,
  "triton_info": {
    "enable": true,
    "use_grpc": true,
    "use_trt": true,
    "fp16": true
  }
}

Results

After benchmarking is finished we should see results in a command line:

CLI output

{
  "instance_type": null,
  "benchmark_params": {
    "engine": "fastapi",
    "use_gpu": true,
    "use_cpu": false
  },
  "system_specs": {
    "system": "Linux",
    "architecture": "x86_64",
    "version": "#26~20.04.1-Ubuntu SMP Sat Oct 15 03:22:07 UTC 2022",
    "cpu": "Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz",
    "physical_cores": 2,
    "total_cores": 4,
    "max_cpu_freq": 0.0,
    "min_cpu_freq": 0.0,
    "ram_memory": 15705.52734375,
    "swap_memory": 0.0,
    "gpus": [
      {
        "id": 0,
        "name": "Tesla T4",
        "total_memory": 15360.0
      }
    ]
  },
  "time_benchmark": [
    {
      "info": {
        "input_ids": {
          "shape": [
            16,
            64
          ],
          "dtype": "INT32"
        },
        "attention_mask": {
          "shape": [
            16,
            64
          ],
          "dtype": "INT32"
        }
      },
      "median_time": 0.015364835009677336,
      "mean_time": 0.015390528332015188,
      "max_time": 0.015462222989299335,
      "min_time": 0.015344526997068897
    }
  ],
  "throughput_benchmark": {},
  "status": "Successful",
  "start_time": 1668637237.106553,
  "end_time": 1668637302.7877016
}

Errors

You can get error like that:

CLI output

{
  "Error": "job_name is not valid. It should be unique"
}

Other errors also will have description in response.