TRT wrappers for Vista3D by borisfom · Pull Request #28 · Project-MONAI/VISTA

borisfom · 2024-07-24T04:12:07Z

Description

Adds TRT acceleration.
Depends on Project-MONAI/MONAI#7990

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
In-line docstrings updated.

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

mingxueg-nv · 2024-07-29T02:22:32Z

@heyufan1995 @Nic-Ma
Please help review this draft PR.
Vista3D TRT end-to-end is about 10% faster than PyTorch.

mingxueg-nv · 2024-07-29T09:59:20Z

@yiheng-wang-nv please also help review it. Thanks.

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

…3d-export

for more information, see https://pre-commit.ci

borisfom · 2024-08-01T00:56:04Z

The perf gain from TRT end-to-end is about 20% on infer_everything sample I ran (18s vs 22s):
python3 -m scripts.infer --config_file 'configs/infer.yaml' - infer_everything --image_file 'example-1.nii.gz'

Nic-Ma · 2024-08-01T07:03:11Z

Hi @heyufan1995 ,

Could you please help review this PR?

Thanks in advance.

mingxueg-nv · 2024-08-01T10:42:57Z

Hi, @bfomitchev, I notice that you have added TRT wrapper in latest version.
Could you please help adopt previous method for both supports of TensorRT and PyTorch.
For instance,
PyTorch uses
python3 -m scripts.infer --config_file 'configs/infer.yaml' - infer_everything --image_file 'example-1.nii.gz'
TensorRT uses
python3 -m scripts.export --config_file 'configs/infer.yaml' - infer_everything --image_file 'example-1.nii.gz'
It's fine for users to infer even without the TensorRT environment.
Thanks.

borisfom · 2024-08-01T19:41:06Z

@mingxueg-nv : I have removed export.py in the latest version to avoid duplication. TRT use is being controlled by "trt" config flag in infer.yaml. I will add graceful fallback for the case when TRT is not installed.

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

…3d-export

for more information, see https://pre-commit.ci

mingxueg-nv · 2024-08-02T02:32:13Z

@mingxueg-nv : I have removed export.py in the latest version to avoid duplication. TRT use is being controlled by "trt" config flag in infer.yaml. I will add graceful fallback for the case when TRT is not installed.

Thanks.

I will measure accuracy today to supplement more benchmark details.

…to vista3d-export

for more information, see https://pre-commit.ci

yiheng-wang-nv · 2024-08-05T02:51:15Z

we can enable trt version in model zoo after this PR merge cc @heyufan1995

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

…3d-export Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

for more information, see https://pre-commit.ci

borisfom · 2024-08-05T06:42:59Z

Now depends on Project-MONAI/MONAI#7990

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

…3d-export Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

…3d-export

for more information, see https://pre-commit.ci

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

…3d-export

for more information, see https://pre-commit.ci

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

for more information, see https://pre-commit.ci

borisfom added 5 commits July 15, 2024 19:34

Initial profiling/export

6e3201f

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

stash

a1da5e2

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

Working TRT wrappers for encoder and class head

d854bc5

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

Merge remote-tracking branch 'origin/vista3d' into vista3d-export

a82ce56

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

Cleaned up, working TRT wrapping

818a548

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

borisfom changed the title ~~TRT wrappers for Vista3D~~ Draft: TRT wrappers for Vista3D Jul 24, 2024

Nic-Ma requested review from KumoLiu and heyufan1995 July 31, 2024 08:31

Cleanup

3e4a84b

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

borisfom force-pushed the vista3d-export branch from 01ed8ab to d30f656 Compare July 31, 2024 20:02

Merge remote-tracking branch 'origin/vista3d' into vista3d-export

a45deed

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

borisfom force-pushed the vista3d-export branch from d30f656 to a45deed Compare July 31, 2024 22:15

[pre-commit.ci] auto fixes from pre-commit.com hooks

91606c3

for more information, see https://pre-commit.ci

borisfom changed the title ~~Draft: TRT wrappers for Vista3D~~ TRT wrappers for Vista3D Aug 1, 2024

borisfom and others added 3 commits July 31, 2024 17:48

Fixing CI issues

1004dad

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

Merge branch 'vista3d-export' of github.com:borisfom/VISTA into vista…

e141954

…3d-export

[pre-commit.ci] auto fixes from pre-commit.com hooks

1942aa5

for more information, see https://pre-commit.ci

borisfom and others added 3 commits August 1, 2024 19:25

Improved TRT engine handling, fallback added

20dce0e

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

Merge branch 'vista3d-export' of github.com:borisfom/VISTA into vista…

6844725

…3d-export

[pre-commit.ci] auto fixes from pre-commit.com hooks

47006eb

for more information, see https://pre-commit.ci

mingxueg-nv and others added 3 commits August 2, 2024 12:04

add accuracy benchmark results

8b1408a

Merge branch 'vista3d-export' of https://github.com/borisfom/VISTA in…

4650e1b

…to vista3d-export

[pre-commit.ci] auto fixes from pre-commit.com hooks

e0c0e7a

for more information, see https://pre-commit.ci

borisfom and others added 3 commits August 4, 2024 23:28

Refactored to use TRTWrapper from MONAI

2d4dfc0

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

Merge branch 'vista3d-export' of github.com:borisfom/VISTA into vista…

20f1b19

…3d-export Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

92cda1b

for more information, see https://pre-commit.ci

borisfom and others added 20 commits August 5, 2024 12:44

Using optional import

15e37e3

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

Merge branch 'vista3d-export' of github.com:borisfom/VISTA into vista…

20cf733

…3d-export Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

8a01bb5

for more information, see https://pre-commit.ci

Using optional import, take 2

9cadc97

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

6a3d2ac

for more information, see https://pre-commit.ci

precision_constraints=obey

939abc2

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

Merge branch 'vista3d-export' of github.com:borisfom/VISTA into vista…

6b711f9

…3d-export

[pre-commit.ci] auto fixes from pre-commit.com hooks

4f1d21c

for more information, see https://pre-commit.ci

update accuracy benchmark

b14474d

[pre-commit.ci] auto fixes from pre-commit.com hooks

1b7d13a

for more information, see https://pre-commit.ci

Fixed ruff

cd3ee1e

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

9b6cd9e

for more information, see https://pre-commit.ci

Adjusted for TRWrapper move

fd7f6af

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

90b5b47

for more information, see https://pre-commit.ci

Adjusted for TRTWrapper API change

c0641e5

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

Merge branch 'vista3d-export' of github.com:borisfom/VISTA into vista…

b668233

…3d-export

[pre-commit.ci] auto fixes from pre-commit.com hooks

b24abb3

for more information, see https://pre-commit.ci

Adjusted TRTWrapper args

6053338

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

Adjusted for TRT wrapper refactoring

2ec59eb

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

2047da0

for more information, see https://pre-commit.ci

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TRT wrappers for Vista3D#28

TRT wrappers for Vista3D#28
borisfom wants to merge 40 commits intoProject-MONAI:vista3dfrom
borisfom:vista3d-export

borisfom commented Jul 24, 2024 •

edited

Loading

Uh oh!

mingxueg-nv commented Jul 29, 2024 •

edited

Loading

Uh oh!

mingxueg-nv commented Jul 29, 2024

Uh oh!

borisfom commented Aug 1, 2024 •

edited

Loading

Uh oh!

Nic-Ma commented Aug 1, 2024

Uh oh!

mingxueg-nv commented Aug 1, 2024

Uh oh!

borisfom commented Aug 1, 2024

Uh oh!

mingxueg-nv commented Aug 2, 2024 •

edited

Loading

Uh oh!

yiheng-wang-nv commented Aug 5, 2024

Uh oh!

borisfom commented Aug 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

borisfom commented Jul 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Uh oh!

mingxueg-nv commented Jul 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mingxueg-nv commented Jul 29, 2024

Uh oh!

borisfom commented Aug 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Nic-Ma commented Aug 1, 2024

Uh oh!

mingxueg-nv commented Aug 1, 2024

Uh oh!

borisfom commented Aug 1, 2024

Uh oh!

mingxueg-nv commented Aug 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yiheng-wang-nv commented Aug 5, 2024

Uh oh!

borisfom commented Aug 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

borisfom commented Jul 24, 2024 •

edited

Loading

mingxueg-nv commented Jul 29, 2024 •

edited

Loading

borisfom commented Aug 1, 2024 •

edited

Loading

mingxueg-nv commented Aug 2, 2024 •

edited

Loading