Assuming Avidemux has been compiled with support for it and your PC supports it, you should enable both the HW decoding and matching display driver
Both happen in preference
For example, for windows we'll want to use dxva2
So Enable Dxva2 HW decoding
and use the matching display driver
You can check it works by watching the two strings on the upper left
The left one is the decoder used The right one is the display used
If the hardware accelerated fails, it should revert automatically to software
In the above example we are using software for both
If you see VDPAU/VDPAU or LIBVA/LIBVA or DXVA2/DXVA2 it means it's fully operationnal
The rule is :
Using wrapper is a bad idea. For example VDPAU on top of Intel video driver.
The current DXVA2 is not fully optimised, there is back and forth copy. It will be fixed in the D3D11 version.