Triton Inference Server Feature Requests

Unix sockets for communication between containers, sending model inputs

TRTIS client for Python includes a C library which is ubuntu specific. Some customers have to build their own gRPC library to TRTIS in Python so that they didn't have a dependency on OS or Linux distribution

Want to be able to load more models into TRTIS GPU memory. Concerned about the inability to anticipate GPU memory consumption/fragmentation. Having UVM would solve this

In addition to the Python and C++ client libraries provided in TRTIS, also have a C# client to support customers on Windows.

Telegraph is easier to push metrics to AWS for their SW stack

Apache Thrift: https://thrift.apache.org/

Want to be able to run PyTorch models without having to convert to Torchscript. May be useful for certain customers who have a mix of production models and models not in production and in PyTorch. Would be useful to have both run in the same TRTIS...

Have TRTIS keep state of which models were deployed where and when they were called last by adding a metric for when a particular model was last called (time stamp of last inference).

Have TRTIS be able to pull models from a model repository hosted on Azure in addition to S3, Google Cloud Storage, and a local persistent volume.

Support for deploying and serving models using TRTIS on embedded GPUs (Jetson, Xavier, etc)