How should be populated codecs capabilities fmtp lines?

[youennf]

UAs might expose several entries for codecs capabilities with fmtp variants.
Even though different UAs may have the same underlying support, the capabilities exposed might be different.

We should evaluate how different the exposed capabilities are and see how we can converge.

[youennf]

FWIW, restricting to H264, VP8 and VP9 which are in all browsers for now:

For VP8

• Chrome has 1 entry (no parameters)
• Firefox has 1 entry for VP8 (max-fs and max-fr parameters)
• Safari has 1 entry (no parameters)

For H264

• Chrome has 10 entries (5 profiles, with level-asymnetry-allowed and packetization-mode=0 or 1)
• Firefox has 2 entries for H264 (1 profile with level-asymnetry-allowed and packetization-mode=1 or omitted)
• Safari has 4 entries for H264 (2 profiles with level-asymnetry-allowed and packetization-mode=0 or 1)

For VP9

• Chrome has 4 entries (different profiles)
• Firefox has 1 entry (no profile but max-fs and max-fr parameters)
• Safari has 2 entries (two profiles)

I wonder how we could converge here.

[aboba]

If we are clear about what constitutes the "media format" for each codec, and what the defaults are in the absence of fmtp parameters, we can avoid unnecessary negotiation failures. For example:

For VP8

• max-fs and max-fr parameters are not part of the "media format", only mimeType
• Browser advertising no parameters should be able to succesfully negotiate with browser including max-fs/max-fr

For H264

• "media format" includes mimeType, profile-level-id (because profile and level-id are combined), packetization-mode, level-asymmetry-allowed
• packetization-mode = 0 assumed if omitted
• With level-asymmetry-allowed in Offer & Answer, Offer of level-id A and Answer of level-id B will fail negotiation if B != A even if profile and packetization-mode matches. This is a design error in RFC 6184, so not clear we can fix it.

For VP9

• "media format" includes mimeType, profile-id
• profile-id = 0 assumed if omitted
• Browser advertising no parameters should be able to succesfully negotiate with browser including profile-id=0, max-fs/max-fr

For AV1

• "media format" includes mimeType, profile, tier
• profile=0, tier=0, level-idx=5 assumed if omitted
• level-idx represents max level that can be sent (sendonly), received (recvonly)
• level-idx represents max level that can be sent/received (sendrecv)
• Offer of level-idx 4 is observed to cause a negotiation failure if Answerer supports level-idx 5. Bug?

[alvestrand]

max-fr and max-fs are examples of what I'm thinking of as "scalar" parameters, where matching two codecs needs to consider them, but there are too many possible values to imagine listing them all.

The issue here is of course that understanding how these need to match is strongly codec specific.

H.264 also has max-recv-level, max-mbps, max-smbps, max-fs, max-cbp, max-dbp and max-br (which, AFAIK, nobody generates, checks or cares about) , which are also "scalars".