Extend EncodedVideoChunkMetadata for Spatial SVC

[aboba]

Fixes #619

Rebase of PR #636

Work-in-progress do not merge

---

Preview | Diff

[aboba]

Discussed at the April 14 MEDIA WG meeting. Questions:

• Is the RTCEncodedVideoChunk metadata (e.g. spatialIndex, Dependencies, etc.) shipping? Answer: spatialIndex and dependencies are shipping.
• Are DTIs, chainkLinks implemented? Answer: They are implemented in VP9, but not yet AV1.

[kalradivyanshu]

@aboba what is blocking this PR? Spatial SVC is really needed if devs want to use webcodec for conferencing etc use-cases.

[aboba]

@kalradivyanshu One question is whether WebCodecs (as a low-level API) should just provide spatialIndex and dependencies similarly to RTCEncodedFrameMetadata, or whether it should also include chains and DTIs. Do you have an opinion?

[kalradivyanshu]

@aboba thankyou for taking the time to reply!

So I went through the new API, and I am fairly new to SVC, but I think the new API is what should be done. Libraries can be used to get simple spatialIndex if someone wants, but I think one of the design goals of webcodecs + webtransport was that it will be low-level with maximum control to app developer. I didn't even know SVC could be this flexible, I just always thought of it from L-xT-y standpoint. I recently tried insertableStreams in webrtc, to get access to L3T3 AV1, atleast I tried, WebRTC is so opinionated in so many ways, that the quality was all over the place, and even something as simple as keyframe distance, I couldn't set. So while spatialIndex might sound good enough right now, with the pace AV1 is evolving, I am sure in a year or so, it will feel very restrictive. Ofc the main question is can the new API be done, do chrome folks agree? Are there rough timelines?

Also, can switch frames #747 be done using the new API?

[aboba]

@kalradivyanshu Agree that spatial support in Encoded Transform API is not a proper substitute for WebCodecs. Adding spatialIndex and dependencies to WebCodecs metadata is a much smaller task than implementing the new video encoder API, which could take quite a while to fully design, implement and test. Without the new API it will be hard for WebCodecs to support LTR, layer refresh or arbitrary modes. But hopefully developers can get by with keyframes for recovery in the interim. Am looking forward to ironing out the issues with this PR and hopefully moving forward on implementation.

[kalradivyanshu]

,@aboba got it. Is there no way that the spatialIndex is added first then the new API is added? Because from the sound of the new API is many months away, and no spatial SVC is a big blocker for people like me who are using webcodecs in production.
Keyframes don't remove the fact that the sender needs to run multiple encoders in parallel for 480p, 720p and 1080p. And for advanced codecs like AV1 this is very resource intensive and low and maybe even mid level devices will struggle.

[Djuffin]

most likely we will indeed add spatialLayerId first to support scalability modes with spatial layers.

[aboba]

Obsoleted by PR #756