Inference Performance
| Model | Imported from | Batch Size | Latency (msec) | Throughput (samples per sec) | Mix Data Types - lowest Precision | Model source | |
|---|---|---|---|---|---|---|---|
| inception V1 | TF | 1 | 0.152 | 12091.4 | INT8 | Source | |
| 4 | 0.391 | 15604.4 | INT8 | ||||
| 8 | 0.692 | 16286.9 | INT8 | ||||
| 10 | 0.839 | 16470.3 | INT8 | ||||
| inception V3 | TF | 1 | 0.396 | 3374.2 | INT8 | Source | |
| 4 | 1.225 | 4324.6 | INT8 | ||||
| 8 | 2.338 | 4345.8 | INT8 | ||||
| 10 | 2.82 | 4354.7 | INT8 | ||||
| bninception | ONNX | 1 | 0.179 | 8438.5 | INT8 | open source model | |
| 4 | 0.432 | 12680.4 | INT8 | ||||
| 8 | 0.763 | 13420.1 | INT8 | ||||
| 10 | 0.887 | 13861 | INT8 | ||||
| 20 | 1.72 | 14416.4 | INT8 | ||||
| resnet18 V1 | ONNX | 1 | 0.135 | 14544.6 | INT8 | Source | |
| 224224 | 4 | 0.249 | 27628.1 | INT8 | |||
| 8 | 0.411 | 30676.6 | INT8 | ||||
| 10 | 0.493 | 31566.8 | INT8 | ||||
| resnet18 V2 | ONNX | 1 | 0.184 | 8836.1 | INT8 | Source | |
| 224224 | 4 | 0.426 | 13544.8 | INT8 | |||
| 8 | 0.736 | 14630.4 | INT8 | ||||
| 10 | 0.892 | 14845.9 | INT8 | ||||
| resnet34 V1 | ONNX | 1 | 0.175 | 8569.6 | INT8 | Source | |
| 224224 | 4 | 0.366 | 15778.3 | INT8 | |||
| 8 | 0.617 | 17406.8 | INT8 | ||||
| 10 | 0.725 | 18215.7 | INT8 | ||||
| resnet34 V2 | ONNX | 1 | 0.262 | 5563.5 | INT8 | Source | |
| 224224 | 4 | 0.602 | 9106.8 | INT8 | |||
| 8 | 0.993 | 9985.1 | INT8 | ||||
| 10 | 1.263 | 10217.1 | INT8 | ||||
| resnet50 V1 | TF | 1 | 0.202 | 7142.8 | INT8 | Source | |
| 224224 | 4 | 0.427 | 12791 | INT8 | |||
| 8 | 0.74 | 13964.7 | INT8 | ||||
| 10 | 0.872 | 14620.1 | INT8 | ||||
| ONNX | 1 | 0.199 | 7491.9 | INT8 | |||
| 4 | 0.406 | 13573.3 | INT8 | ||||
| 8 | 0.693 | 14766.6 | INT8 | ||||
| 10 | 0.819 | 15487.7 | INT8 | ||||
| resnet50 V1 | ONNX | 1 | 0.176 | 8759.9 | INT8 | Source | |
| 160x160 | 4 | 0.274 | 20836.9 | INT8 | |||
| 8 | 0.417 | 25095.5 | INT8 | ||||
| 10 | 0.48 | 27408.4 | INT8 | ||||
| resnet50 V1 slim | TF | 1 | 0.202 | 7433.6 | INT8 | Source | |
| 224224 | 4 | 0.391 | 14458.6 | INT8 | |||
| 8 | 0.655 | 15977 | INT8 | ||||
| 10 | 0.777 | 16798.2 | INT8 | ||||
| resnet50 V2 | TF | 1 | 0.293 | 4686.4 | INT8 | Source | |
| 224224 | 4 | 0.652 | 7472.4 | INT8 | |||
| 8 | 1.234 | 8154.3 | INT8 | ||||
| ONNX | 1 | 0.297 | 4672.3 | INT8 | Source | ||
| 4 | 0.662 | 7398.5 | INT8 | ||||
| 8 | 1.259 | 8058.1 | INT8 | ||||
| resnet101 V1 | ONNX | 1 | 0.301 | 4579 | INT8 | Source | |
| 224224 | 4 | 0.604 | 8213.1 | INT8 | |||
| 8 | 1.098 | 9169.6 | INT8 | ||||
| 10 | 1.286 | 9799.9 | INT8 | ||||
| resnet101 V2 | ONNX | 1 | 0.445 | 2766 | INT8 | Source | |
| 224224 | 4 | 1.119 | 4154.5 | INT8 | |||
| 8 | 2.059 | 4513.1 | INT8 | ||||
| 10 | 2.475 | 4596.7 | INT8 | ||||
| resnet152 v1 | TF | 1 | 0.659 | 1759.4 | INT8 | open source model | |
| 224224 | 4 | 0.901 | 4949.3 | INT8 | |||
| 8 | 1.495 | 6614.2 | INT8 | ||||
| 10 | 1.677 | 7095.3 | INT8 | ||||
| ONNX | 1 | 0.458 | 2644.6 | INT8 | Source | ||
| 4 | 0.823 | 5742.3 | INT8 | ||||
| 8 | 1.519 | 6419.7 | INT8 | ||||
| 10 | 1.717 | 6856.3 | INT8 | ||||
| resnet152 V1 slim | TF | 1 | 0.663 | 1757.4 | INT8 | Source | |
| 224224 | 4 | 0.903 | 4935.4 | INT8 | |||
| 8 | 1.502 | 6625.5 | INT8 | ||||
| 10 | 1.712 | 7096 | INT8 | ||||
| resnet152 V2 | ONNX | 1 | 0.63 | 1939.2 | INT8 | Source | |
| 224224 | 4 | 1.634 | 2948.8 | INT8 | |||
| 8 | 2.796 | 3208.2 | INT8 | ||||
| 10 | 3.346 | 3258.4 | INT8 | ||||
| ResNext50-32_4d | ONNX | 1 | 0.349 | 3827.4 | INT8 | open source model | |
| 224224 | 4 | 0.783 | 6004.2 | INT8 | |||
| 8 | 1.475 | 6522.6 | INT8 | ||||
| 10 | 1.766 | 6655.6 | INT8 | ||||
| resnext101_32_4d | ONNX | 1 | 0.409 | 3065.6 | INT8 | open source model | |
| 224224 | 4 | 1.001 | 4614.8 | INT8 | |||
| 8 | 1.874 | 5092.8 | INT8 | ||||
| 10 | 2.116 | 5433.4 | INT8 | ||||
| tiny yolo v2 | 10881920 | Pytorch | 1 | 2.073 | 533.6 | INT8 | Source |
| 1 | 0.169 | 10971.3 | INT8 | Source | |||
| 320320 | |||||||
| 1 | 0.227 | 8117.5 | INT8 | Source | |||
| 416416 | |||||||
| 1 | 0.414 | 3968.6 | INT8 | Source | |||
| 608608 | |||||||
| 960960 | 1 | 0.849 | 1796.7 | INT8 | Source | ||
| Yolo V2 | 1088_1920 | Pytorch | 1 | 6.359 | 162.9 | INT8 | Source |
| 320320 | 1 | 0.561 | 2226.6 | INT8 | Source | ||
| 416416 | 1 | 0.713 | 1688.5 | INT8 | Source | ||
| 544736 | 1 | 1.411 | 816.7 | INT8 | Source | ||
| 608608 | 1 | 1.391 | 880.2 | INT8 | Source | ||
| 960960 | 1 | 3.733 | 291.7 | INT8 | Source | ||
| yolo v3 | 416416 | Pytorch | 1 | 1.111 | 1102.2 | INT8 | Source |
| 960960 | Pytorch | 4 | 11.375 | 360.9 | INT8 | Source | |
| BERT squad | MX | 1 | 2.809 | 404 | INT8 | Source | |
| BASE | 4 | 2.798 | 1611.4 | INT8 | |||
| max sequnce length = 128 | 8 | 2.816 | 2226.2 | INT8 | |||
| 10 | 3.191 | 2477.7 | INT8 | ||||
| 10 | 5.762 | 1726.4 | INT16 | ||||
| 12 | 3.59 | 2773.8 | INT8 | ||||
| bert mrpc | MX | 1 | 2.822 | 402.8 | INT8 | ||
| BASE | 4 | 2.762 | 1605.9 | INT8 | |||
| max sequnce length = 128 | 8 | 2.83 | 2211.6 | INT8 | |||
| 10 | 3.196 | 2455.4 | INT8 | ||||
| 10 | 5.777 | 1722.6 | INT16 | ||||
| 12 | 6.877 | 1762.8 | INT16 | ||||
| bvlc_googlenet | ONNX | 1 | 0.278 | 5113.7 | INT8 | Source | |
| 224224 | |||||||
| googlenet_bn_no_lrn | ONNX | 1 | 0.141 | 12787.4 | INT8 | Developed inhouse based on Googlenet with batch norm and w/o LRN | |
| 224224 | 4 | 0.345 | 17558 | INT8 | |||
| 8 | 0.592 | 18424.7 | INT8 | ||||
| 10 | 0.715 | 18530.5 | INT8 | ||||
| squeezenet1.1 | ONNX | 1 | 0.107 | 23281 | INT8 | Source | |
| 224224 | 4 | 0.209 | 36321.4 | INT8 | |||
| 8 | 0.373 | 37740.1 | INT8 | ||||
| 10 | 0.46 | 37843.9 | INT8 | ||||
| ssd-vgg16 300300 | MX | 1 | 0.833 | 1466.1 | INT8 | Source |
Software Configuration: Ubuntu v-18.04, SynapseAI v-0.11.0-447
Hardware Configuration: Goya HL-100 PCIe card, Host: Xeon Gold [email protected]0GhzHabana