Torch Compile 進階用法

此互動式腳本旨在概述 torch_tensorrt.compile(…, ir=”torch_compile”, …) 的運作方式，以及它如何與 torch.compile API 整合。

匯入和模型定義

import torch
import torch_tensorrt

# We begin by defining a model
class Model(torch.nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.relu = torch.nn.ReLU()

    def forward(self, x: torch.Tensor, y: torch.Tensor):
        x_out = self.relu(x)
        y_out = self.relu(y)
        x_y_out = x_out + y_out
        return torch.mean(x_y_out)

使用預設設定透過 torch.compile 編譯

# Define sample float inputs and initialize model
sample_inputs = [torch.rand((5, 7)).cuda(), torch.rand((5, 7)).cuda()]
model = Model().eval().cuda()

# Next, we compile the model using torch.compile
# For the default settings, we can simply call torch.compile
# with the backend "torch_tensorrt", and run the model on an
# input to cause compilation, as so:
optimized_model = torch.compile(model, backend="torch_tensorrt", dynamic=False)
optimized_model(*sample_inputs)

使用自訂設定透過 torch.compile 編譯

# First, we use Torch utilities to clean up the workspace
# after the previous compile invocation
torch._dynamo.reset()

# Define sample half inputs and initialize model
sample_inputs_half = [
    torch.rand((5, 7)).half().cuda(),
    torch.rand((5, 7)).half().cuda(),
]
model_half = Model().eval().cuda()

# If we want to customize certain options in the backend,
# but still use the torch.compile call directly, we can provide
# custom options to the backend via the "options" keyword
# which takes in a dictionary mapping options to values.
#
# For accepted backend options, see the CompilationSettings dataclass:
# py/torch_tensorrt/dynamo/_settings.py
backend_kwargs = {
    "enabled_precisions": {torch.half},
    "debug": True,
    "min_block_size": 2,
    "torch_executed_ops": {"torch.ops.aten.sub.Tensor"},
    "optimization_level": 4,
    "use_python_runtime": False,
}

# Run the model on an input to cause compilation, as so:
optimized_model_custom = torch.compile(
    model_half,
    backend="torch_tensorrt",
    options=backend_kwargs,
    dynamic=False,
)
optimized_model_custom(*sample_inputs_half)

清除

# Finally, we use Torch utilities to clean up the workspace
torch._dynamo.reset()

Cuda 驅動程式錯誤注意事項

偶爾，在透過 torch_tensorrt 使用 Dynamo 編譯後退出 Python 執行階段時，可能會遇到 Cuda 驅動程式錯誤。此問題與 https://github.com/NVIDIA/TensorRT/issues/2052 相關，並且可以透過將編譯/推論包裝在函式中並使用範圍呼叫來解決，如下所示

if __name__ == '__main__':
    compile_engine_and_infer()