Triton Inference Server Feature Requests

Triton Inference Server Feature Requests

Add a new idea

Filter by status

1 Vote
- Add my vote +1
- +2
- +3
Timeout threshold in dynamic and sequence batcher

Add a timeout threshold to the dynamic and/or sequence batcher per model, which would hold the request in queue until it meets the threshold, then reject it

Created 04 Feb 19:54

0 Shipped
1 Vote
- Add my vote +1
- +2
- +3
Model warming

Warming part of the model loading process (ie caching values so that the first few inferences are not slower than the rest)

Created 24 Sep 00:27

0 Shipped
1 Vote
- Add my vote +1
- +2
- +3
Windows support with TRT and ONNX RT

NBX Windows Requirements: Platform: Windows 10 64-bit Version: 1.0.0.30 GPU: GeForce RTX 2060, Quadro RTX 3000, TITAN RTX or higher CPU: recommended: Intel Core i5 8600, AMD Ryzen r5 2600 or higher Driver: NVIDIA Display Driver version 456.38 or h...

Created 08 Feb 22:27

0 Shipped
0 Vote
- Add my vote +1
- +2
- +3
Example DALI custom backend

Have an example custom backend using DALI for image pre-processing to be used in an ensemble inside TRTIS. NVIDIA DALI: https://github.com/NVIDIA/DALI

Created 21 Nov 20:10

0 Shipped
0 Vote
- Add my vote +1
- +2
- +3
TF 2.0 Support

Add support for new Tensorflow 2.0 as a backend to TRTIS.

Created 20 Nov 00:40

0 Shipped