Benchmark

Here we have presented benchmark results for some selected neural networks.

Our benchmark is an open simple benchmark that measures inference time of ONNX models on ENOT Lite backend versus PyTorch native inference time and transforms it to FPS (frame-per-second, the bigger the better) metric.

All values in tables below are given in FPS. For natural language processing neural networks FPS = QPS.

Benchmarks:
CPU Benchmarks:

ResNet-50

input: (batch_size, 3, 224, 224)
batch_size = 1

Device / Backend

ENOT Lite

ONNX CUDA

Torch CUDA

RTX 3060 Ti

1849.6

340.6

209.1

RTX 2080 Ti

1463.7

317.4

215.7

GTX 1080 Ti

882.7

282.5

231.2
batch_size = 16

Device / Backend

ENOT Lite

ONNX CUDA

Torch CUDA

RTX 3060 Ti

6381.4

973.8

624.3

RTX 2080 Ti

4713.9

842.3

770.4

GTX 1080 Ti

2620.5

935.1

595.6

MobileNetV2

input: (batch_size, 3, 224, 224)
batch_size = 1

Device / Backend

ENOT Lite

ONNX CUDA

Torch CUDA

RTX 3060 Ti

2934.6

779.1

254.0

RTX 2080 Ti

2287.7

658.3

203.0

GTX 1080 Ti

1647.9

649.3

343.3
batch_size = 16

Device / Backend

ENOT Lite

ONNX CUDA

Torch CUDA

RTX 3060 Ti

11275.7

1855.5

1746.2

RTX 2080 Ti

6434.3

2038.3

1855.3

GTX 1080 Ti

6305.3

1411.6

1344.6

MobileNetV2-SSD

input: (batch_size, 3, 224, 224)
batch_size = 1

Device / Backend

ENOT Lite

ONNX CUDA

Torch CUDA

RTX 3060 Ti

622.7

105.7

119.3

RTX 2080 Ti

451.4

107.3

79.1

GTX 1080 Ti

483.5

159.6

126.2
batch_size = 16

Device / Backend

ENOT Lite

ONNX CUDA

Torch CUDA

RTX 3060 Ti

2419.5

211.9

230.6

RTX 2080 Ti

1411.0

238.0

222.9

GTX 1080 Ti

2128.6

275.3

256.7

YOLOv5s

input: (batch_size, 3, 640, 640)
batch_size = 1

Device / Backend

ENOT Lite

ONNX CUDA

Torch CUDA

RTX 3060 Ti

601.3

158.8

148.5

RTX 2080 Ti

441.4

172.0

84.5

GTX 1080 Ti

281.9

127.3

111.6
batch_size = 16

Device / Backend

ENOT Lite

ONNX CUDA

Torch CUDA

RTX 3060 Ti

777.9

196.0

120.7

RTX 2080 Ti

649.4

243.6

126.5

GTX 1080 Ti

440.4

170.0

138.4

ViT

Vision Transformer (ViT), patch = 16, resolution = 224.

input: (batch_size, 3, 224, 224)
batch_size = 1

Device / Backend

ENOT Lite

ONNX CUDA

Torch CUDA

RTX 3060 Ti

318.1

172.9

175.7

RTX 2080 Ti

374.9

175.3

132.6

GTX 1080 Ti

123.1

135.5

108.3
batch_size = 16

Device / Backend

ENOT Lite

ONNX CUDA

Torch CUDA

RTX 3060 Ti

595.3

291.3

279.7

RTX 2080 Ti

435.8

153.7

123.7

GTX 1080 Ti

169.6

182.7

166.0

BERT

input length: 1941 characters

Device / Backend

ENOT Lite

ONNX CUDA

Torch CUDA

RTX 3060 Ti

220.3

99.2

91.8

RTX 2080 Ti

257.0

94.7

73.4

GTX 1080 Ti

43.9

21.8

25.1

ResNet-50 CPU

input: (batch_size, 3, 224, 224)
batch_size = 1

Device / Backend

ENOT Lite

ONNX CPU

Torch CPU

11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz

268.4

101.5

46.2
batch_size = 8

Device / Backend

ENOT Lite

ONNX CPU

Torch CPU

11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz

254.2

100.4

50.0

MobileNetV2 CPU

input: (batch_size, 3, 224, 224)
batch_size = 1

Device / Backend

ENOT Lite

ONNX CPU

Torch CPU

11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz

1535.7

842.2

135.5
batch_size = 8

Device / Backend

ENOT Lite

ONNX CPU

Torch CPU

11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz

2176.9

453.0

139.8

YOLOv5s CPU

input: (batch_size, 3, 224, 224)
batch_size = 1

Device / Backend

ENOT Lite

ONNX CPU

Torch CPU

11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz

82.8

33.2

22.6
batch_size = 8

Device / Backend

ENOT Lite

ONNX CPU

Torch CPU

11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz

45.1

22.1

18.8

ViT CPU

Vision Transformer (ViT), patch = 16, resolution = 224.

input: (batch_size, 3, 224, 224)
batch_size = 1

Device / Backend

ENOT Lite

ONNX CPU

Torch CPU

11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz

32.8

15.5

14.9
batch_size = 8

Device / Backend

ENOT Lite

ONNX CPU

Torch CPU

11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz

29.0

17.4

16.6

BERT CPU

input length: 1941 characters

Device / Backend

ENOT Lite

ONNX CPU

Torch CPU

11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz

10.6

10.8

7.8