Inference Performance

Model Imported from Batch SizeLatency (msec)Throughput (samples per sec)Mix Data Types - lowest PrecisionModel source
inception V1TF10.15212091.4INT8Source
40.39115604.4INT8
80.69216286.9INT8
100.83916470.3INT8
inception V3TF10.3963374.2INT8Source
41.2254324.6INT8
82.3384345.8INT8
102.824354.7INT8
bninceptionONNX10.1798438.5INT8open source model
40.43212680.4INT8
80.76313420.1INT8
100.88713861INT8
201.7214416.4INT8
resnet18 V1ONNX10.13514544.6INT8Source
22422440.24927628.1INT8
80.41130676.6INT8
100.49331566.8INT8
resnet18 V2ONNX10.1848836.1INT8Source
22422440.42613544.8INT8
80.73614630.4INT8
100.89214845.9INT8
resnet34 V1ONNX10.1758569.6INT8Source
22422440.36615778.3INT8
80.61717406.8INT8
100.72518215.7INT8
resnet34 V2ONNX10.2625563.5INT8Source
22422440.6029106.8INT8
80.9939985.1INT8
101.26310217.1INT8
resnet50 V1TF10.2027142.8INT8Source
22422440.42712791INT8
80.7413964.7INT8
100.87214620.1INT8
ONNX10.1997491.9INT8
40.40613573.3INT8
80.69314766.6INT8
100.81915487.7INT8
resnet50 V1ONNX10.1768759.9INT8Source
160x16040.27420836.9INT8
80.41725095.5INT8
100.4827408.4INT8
resnet50 V1 slimTF10.2027433.6INT8Source
22422440.39114458.6INT8
80.65515977INT8
100.77716798.2INT8
resnet50 V2TF10.2934686.4INT8Source
22422440.6527472.4INT8
81.2348154.3INT8
ONNX10.2974672.3INT8Source
40.6627398.5INT8
81.2598058.1INT8
resnet101 V1ONNX10.3014579INT8Source
22422440.6048213.1INT8
81.0989169.6INT8
101.2869799.9INT8
resnet101 V2ONNX10.4452766INT8Source
22422441.1194154.5INT8
82.0594513.1INT8
102.4754596.7INT8
resnet152 v1TF10.6591759.4INT8open source model
22422440.9014949.3INT8
81.4956614.2INT8
101.6777095.3INT8
ONNX10.4582644.6INT8Source
40.8235742.3INT8
81.5196419.7INT8
101.7176856.3INT8
resnet152 V1 slimTF10.6631757.4INT8Source
22422440.9034935.4INT8
81.5026625.5INT8
101.7127096INT8
resnet152 V2ONNX10.631939.2INT8Source
22422441.6342948.8INT8
82.7963208.2INT8
103.3463258.4INT8
ResNext50-32_4dONNX10.3493827.4INT8open source model
22422440.7836004.2INT8
81.4756522.6INT8
101.7666655.6INT8
resnext101_32_4dONNX10.4093065.6INT8open source model
22422441.0014614.8INT8
81.8745092.8INT8
102.1165433.4INT8
tiny yolo v210881920Pytorch12.073533.6INT8Source
10.16910971.3INT8Source
320320
10.2278117.5INT8Source
416416
10.4143968.6INT8Source
608608
96096010.8491796.7INT8Source
Yolo V21088_1920Pytorch16.359162.9INT8Source
32032010.5612226.6INT8Source
41641610.7131688.5INT8Source
54473611.411816.7INT8Source
60860811.391880.2INT8Source
96096013.733291.7INT8Source
yolo v3416416Pytorch11.1111102.2INT8Source
960960Pytorch411.375360.9INT8Source
BERT squadMX12.809404INT8Source
BASE42.7981611.4INT8
max sequnce length = 12882.8162226.2INT8
103.1912477.7INT8
105.7621726.4INT16
123.592773.8INT8
bert mrpcMX12.822402.8INT8
BASE42.7621605.9INT8
max sequnce length = 12882.832211.6INT8
103.1962455.4INT8
105.7771722.6INT16
126.8771762.8INT16
bvlc_googlenetONNX10.2785113.7INT8Source
224224
googlenet_bn_no_lrnONNX10.14112787.4INT8Developed inhouse based on Googlenet with batch norm and w/o LRN
22422440.34517558INT8
80.59218424.7INT8
100.71518530.5INT8
squeezenet1.1ONNX10.10723281INT8Source
22422440.20936321.4INT8
80.37337740.1INT8
100.4637843.9INT8
ssd-vgg16
300300
MX10.8331466.1INT8Source

Software Configuration: Ubuntu v-18.04, SynapseAI v-0.11.0-447
Hardware Configuration: Goya HL-100 PCIe card, Host: Xeon Gold [email protected]