That certainly works but it’s a pretty big limitation. When catering with dynamic viewports on the web you’re really going to struggle to line everything up if you absolutely anything in the background, providing an alpha channel really is a much easier alternative.
The canonical way to handle responsive videos is to embed multiple source elements within the video element, and them use the media attribute to respond to the user's viewport size, and preferred color scheme: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/so...