[RFC,v5,00/11] Add audio support in v4l2 framework

  Audio signal processing also has the requirement for memory to
memory similar as Video.

This asrc memory to memory (memory ->asrc->memory) case is a non
real time use case.

User fills the input buffer to the asrc module, after conversion, then asrc
sends back the output buffer to user. So it is not a traditional ALSA playback
and capture case.

It is a specific use case,  there is no reference in current kernel.
v4l2 memory to memory is the closed implementation,  v4l2 current
support video, image, radio, tuner, touch devices, so it is not
complicated to add support for this specific audio case.

Because we had implemented the "memory -> asrc ->i2s device-> codec"
use case in ALSA.  Now the "memory->asrc->memory" needs
to reuse the code in asrc driver, so the first 3 patches is for refining
the code to make it can be shared by the "memory->asrc->memory"
driver.

The main change is in the v4l2 side, A /dev/vl4-audioX will be created,
user applications only use the ioctl of v4l2 framework.

Other change is to add memory to memory support for two kinds of i.MX ASRC
module.

changes in v5:
- remove V4L2_AUDIO_FMT_LPCM
- define audio pixel format like V4L2_AUDIO_FMT_S8...
- remove rate and format in struct v4l2_audio_format.
- Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls
- updata document accordingly.

changes in v4:
- update document style
- separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit

changes in v3:
- Modify documents for adding audio m2m support
- Add audio virtual m2m driver
- Defined V4L2_AUDIO_FMT_LPCM format type for audio.
- Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case.
- with modification in v4l-utils, pass v4l2-compliance test.

changes in v2:
- decouple the implementation in v4l2 and ALSA
- implement the memory to memory driver as a platfrom driver
  and move it to driver/media
- move fsl_asrc_common.h to include/sound folder

Shengjiu Wang (11):
  ASoC: fsl_asrc: define functions for memory to memory usage
  ASoC: fsl_easrc: define functions for memory to memory usage
  ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound
  ASoC: fsl_asrc: register m2m platform device
  ASoC: fsl_easrc: register m2m platform device
  media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag
  media: v4l2: Add audio capture and output support
  media: uapi: Add audio rate controls support
  media: uapi: define audio sample format fourcc type
  media: imx-asrc: Add memory to memory driver
  media: audm2m: add virtual driver for audio memory to memory

 .../userspace-api/media/v4l/buffer.rst        |    6 +
 .../userspace-api/media/v4l/common.rst        |    1 +
 .../media/v4l/dev-audio-mem2mem.rst           |   71 +
 .../userspace-api/media/v4l/devices.rst       |    1 +
 .../media/v4l/ext-ctrls-asrc-rate.rst         |   36 +
 .../userspace-api/media/v4l/pixfmt-audio.rst  |  277 ++++
 .../userspace-api/media/v4l/pixfmt.rst        |    1 +
 .../media/v4l/vidioc-enum-fmt.rst             |    2 +
 .../media/v4l/vidioc-g-ext-ctrls.rst          |    4 +
 .../userspace-api/media/v4l/vidioc-g-fmt.rst  |    4 +
 .../media/v4l/vidioc-querycap.rst             |    3 +
 .../media/v4l/vidioc-queryctrl.rst            |    7 +
 .../media/videodev2.h.rst.exceptions          |    4 +
 .../media/common/videobuf2/videobuf2-v4l2.c   |    4 +
 drivers/media/platform/nxp/Kconfig            |   12 +
 drivers/media/platform/nxp/Makefile           |    1 +
 drivers/media/platform/nxp/imx-asrc.c         | 1251 +++++++++++++++++
 drivers/media/test-drivers/Kconfig            |    9 +
 drivers/media/test-drivers/Makefile           |    1 +
 drivers/media/test-drivers/audm2m.c           |  808 +++++++++++
 drivers/media/v4l2-core/v4l2-ctrls-core.c     |    5 +
 drivers/media/v4l2-core/v4l2-ctrls-defs.c     |    4 +
 drivers/media/v4l2-core/v4l2-dev.c            |   17 +
 drivers/media/v4l2-core/v4l2-ioctl.c          |  104 ++
 include/media/v4l2-ctrls.h                    |    1 +
 include/media/v4l2-dev.h                      |    2 +
 include/media/v4l2-ioctl.h                    |   34 +
 .../fsl => include/sound}/fsl_asrc_common.h   |   58 +
 include/uapi/linux/v4l2-controls.h            |   13 +
 include/uapi/linux/videodev2.h                |   77 +
 sound/soc/fsl/fsl_asrc.c                      |  136 ++
 sound/soc/fsl/fsl_asrc.h                      |    4 +-
 sound/soc/fsl/fsl_asrc_dma.c                  |    2 +-
 sound/soc/fsl/fsl_easrc.c                     |  214 +++
 sound/soc/fsl/fsl_easrc.h                     |    8 +-
 35 files changed, 3179 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst
 create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-asrc-rate.rst
 create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst
 create mode 100644 drivers/media/platform/nxp/imx-asrc.c
 create mode 100644 drivers/media/test-drivers/audm2m.c
 rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%)
  

On Thu, Sep 28, 2023 at 05:00:08PM +0800, Shengjiu Wang wrote:
> Audio signal processing also has the requirement for memory to
> memory similar as Video.

> This asrc memory to memory (memory ->asrc->memory) case is a non
> real time use case.

Other than the naming thing I sent in reply to one of the patches the
ASoC bits look fine.
  

Hi Shengjiu,

On 28/09/2023 11:00, Shengjiu Wang wrote:
> The audio sample format definition is from alsa,
> the header file is include/uapi/sound/asound.h, but
> don't include this header file directly, because in
> user space, there is another copy in alsa-lib.
> There will be conflict in userspace for include
> videodev2.h & asound.h and asoundlib.h
> 
> Here still use the fourcc format.
> 
> Signed-off-by: Shengjiu Wang <shengjiu.wang@nxp.com>
> ---
>  .../userspace-api/media/v4l/pixfmt-audio.rst  | 277 ++++++++++++++++++
>  .../userspace-api/media/v4l/pixfmt.rst        |   1 +
>  drivers/media/v4l2-core/v4l2-ioctl.c          |  51 ++++
>  include/uapi/linux/videodev2.h                |  56 ++++
>  4 files changed, 385 insertions(+)
>  create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst

I think it would make more sense if this patch came after 07/11, so swap this
and the previous patch around.

> 
> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-audio.rst b/Documentation/userspace-api/media/v4l/pixfmt-audio.rst
> new file mode 100644
> index 000000000000..6ff114dfc2d1
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/pixfmt-audio.rst
> @@ -0,0 +1,277 @@
> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
> +
> +.. _pixfmt-audio:
> +
> +*************
> +Audio Formats
> +*************
> +
> +These formats are used for :ref:`audiomem2mem` interface only.
> +
> +.. tabularcolumns:: |p{5.8cm}|p{1.2cm}|p{10.3cm}|
> +
> +.. cssclass:: longtable
> +
> +.. flat-table:: Audio Format
> +    :header-rows:  1
> +    :stub-columns: 0
> +    :widths:       3 1 4
> +
> +    * - Identifier
> +      - Code
> +      - Details
> +    * .. _V4L2-AUDIO-FMT-S8:
> +
> +      - ``V4L2_AUDIO_FMT_S8``
> +      - 'S8'
> +      - Correspond to SNDRV_PCM_FORMAT_S8 in ALSA

Correspond -> Corresponds

(fix everywhere below)

> +    * .. _V4L2-AUDIO-FMT-U8:
> +
> +      - ``V4L2_AUDIO_FMT_U8``
> +      - 'U8'
> +      - Correspond to SNDRV_PCM_FORMAT_U8 in ALSA
> +    * .. _V4L2-AUDIO-FMT-S16-LE:
> +
> +      - ``V4L2_AUDIO_FMT_S16_LE``
> +      - 'S16_LE'
> +      - Correspond to SNDRV_PCM_FORMAT_S16_LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-S16-BE:
> +
> +      - ``V4L2_AUDIO_FMT_S16_BE``
> +      - 'S16_BE'
> +      - Correspond to SNDRV_PCM_FORMAT_S16_BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-U16-LE:
> +
> +      - ``V4L2_AUDIO_FMT_U16_LE``
> +      - 'U16_LE'
> +      - Correspond to SNDRV_PCM_FORMAT_U16_LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-U16-BE:
> +
> +      - ``V4L2_AUDIO_FMT_U16_BE``
> +      - 'U16_BE'
> +      - Correspond to SNDRV_PCM_FORMAT_U16_BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-S24-LE:
> +
> +      - ``V4L2_AUDIO_FMT_S24_LE``
> +      - 'S24_LE'
> +      - Correspond to SNDRV_PCM_FORMAT_S24_LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-S24-BE:
> +
> +      - ``V4L2_AUDIO_FMT_S24_BE``
> +      - 'S24_BE'
> +      - Correspond to SNDRV_PCM_FORMAT_S24_BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-U24-LE:
> +
> +      - ``V4L2_AUDIO_FMT_U24_LE``
> +      - 'U24_LE'
> +      - Correspond to SNDRV_PCM_FORMAT_U24_LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-U24-BE:
> +
> +      - ``V4L2_AUDIO_FMT_U24_BE``
> +      - 'U24_BE'
> +      - Correspond to SNDRV_PCM_FORMAT_U24_BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-S32-LE:
> +
> +      - ``V4L2_AUDIO_FMT_S32_LE``
> +      - 'S32_LE'
> +      - Correspond to SNDRV_PCM_FORMAT_S32_LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-S32-BE:
> +
> +      - ``V4L2_AUDIO_FMT_S32_BE``
> +      - 'S32_BE'
> +      - Correspond to SNDRV_PCM_FORMAT_S32_BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-U32-LE:
> +
> +      - ``V4L2_AUDIO_FMT_U32_LE``
> +      - 'U32_LE'
> +      - Correspond to SNDRV_PCM_FORMAT_U32_LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-U32-BE:
> +
> +      - ``V4L2_AUDIO_FMT_U32_BE``
> +      - 'U32_BE'
> +      - Correspond to SNDRV_PCM_FORMAT_U32_BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-FLOAT-LE:
> +
> +      - ``V4L2_AUDIO_FMT_FLOAT_LE``
> +      - 'FLOAT_LE'
> +      - Correspond to SNDRV_PCM_FORMAT_FLOAT_LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-FLOAT-BE:
> +
> +      - ``V4L2_AUDIO_FMT_FLOAT_BE``
> +      - 'FLOAT_BE'
> +      - Correspond to SNDRV_PCM_FORMAT_FLOAT_BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-FLOAT64-LE:
> +
> +      - ``V4L2_AUDIO_FMT_FLOAT64_LE``
> +      - 'FLOAT64_LE'
> +      - Correspond to SNDRV_PCM_FORMAT_FLOAT64_LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-FLOAT64-BE:
> +
> +      - ``V4L2_AUDIO_FMT_FLOAT64_BE``
> +      - 'FLOAT64_BE'
> +      - Correspond to SNDRV_PCM_FORMAT_FLOAT64_BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-IEC958-SUBFRAME-LE:
> +
> +      - ``V4L2_AUDIO_FMT_IEC958_SUBFRAME_LE``
> +      - 'IEC958_SUBFRAME_LE'
> +      - Correspond to SNDRV_PCM_FORMAT_IEC958_SUBFRAME_LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-IEC958-SUBFRAME-BE:
> +
> +      - ``V4L2_AUDIO_FMT_IEC958_SUBFRAME_BE``
> +      - 'IEC958_SUBFRAME_BE'
> +      - Correspond to SNDRV_PCM_FORMAT_IEC958_SUBFRAME_BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-MU-LAW:
> +
> +      - ``V4L2_AUDIO_FMT_MU_LAW``
> +      - 'MU_LAW'
> +      - Correspond to SNDRV_PCM_FORMAT_MU_LAW in ALSA
> +    * .. _V4L2-AUDIO-FMT-A-LAW:
> +
> +      - ``V4L2_AUDIO_FMT_A_LAW``
> +      - 'A_LAW'
> +      - Correspond to SNDRV_PCM_FORMAT_A_LAW in ALSA
> +    * .. _V4L2-AUDIO-FMT-IMA-ADPCM:
> +
> +      - ``V4L2_AUDIO_FMT_IMA_ADPCM``
> +      - 'IMA_ADPCM'
> +      - Correspond to SNDRV_PCM_FORMAT_IMA_ADPCM in ALSA
> +    * .. _V4L2-AUDIO-FMT-MPEG:
> +
> +      - ``V4L2_AUDIO_FMT_MPEG``
> +      - 'MPEG'
> +      - Correspond to SNDRV_PCM_FORMAT_MPEG in ALSA
> +    * .. _V4L2-AUDIO-FMT-GSM:
> +
> +      - ``V4L2_AUDIO_FMT_GSM``
> +      - 'GSM'
> +      - Correspond to SNDRV_PCM_FORMAT_GSM in ALSA
> +    * .. _V4L2-AUDIO-FMT-S20-LE:
> +
> +      - ``V4L2_AUDIO_FMT_S20_LE``
> +      - 'S20_LE'
> +      - Correspond to SNDRV_PCM_FORMAT_S20_LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-S20-BE:
> +
> +      - ``V4L2_AUDIO_FMT_S20_BE``
> +      - 'S20_BE'
> +      - Correspond to SNDRV_PCM_FORMAT_S20_BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-U20-LE:
> +
> +      - ``V4L2_AUDIO_FMT_U20_LE``
> +      - 'U20_LE'
> +      - Correspond to SNDRV_PCM_FORMAT_U20_LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-U20-BE:
> +
> +      - ``V4L2_AUDIO_FMT_U20_BE``
> +      - 'U20_BE'
> +      - Correspond to SNDRV_PCM_FORMAT_U20_BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-SPECIAL:
> +
> +      - ``V4L2_AUDIO_FMT_SPECIAL``
> +      - 'SPECIAL'
> +      - Correspond to SNDRV_PCM_FORMAT_SPECIAL in ALSA

In alsa it says:

	/* FIXME: the following format is not defined properly yet */
        [SNDRV_PCM_FORMAT_SPECIAL] = {
                .le = -1, .signd = -1,
        },

Which suggests to me that we should just skip it.

> +    * .. _V4L2-AUDIO-FMT-S24-3LE:
> +
> +      - ``V4L2_AUDIO_FMT_S24_3LE``
> +      - 'S24_3LE'
> +      - Correspond to SNDRV_PCM_FORMAT_S24_3LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-S24-3BE:
> +
> +      - ``V4L2_AUDIO_FMT_S24_3BE``
> +      - 'S24_3BE'
> +      - Correspond to SNDRV_PCM_FORMAT_S24_3BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-U24-3LE:
> +
> +      - ``V4L2_AUDIO_FMT_U24_3LE``
> +      - 'U24_3LE'
> +      - Correspond to SNDRV_PCM_FORMAT_U24_3LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-U24-3BE:
> +
> +      - ``V4L2_AUDIO_FMT_U24_3BE``
> +      - 'U24_3BE'
> +      - Correspond to SNDRV_PCM_FORMAT_U24_3BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-S20-3LE:
> +
> +      - ``V4L2_AUDIO_FMT_S20_3LE``
> +      - 'S20_3LE'
> +      - Correspond to SNDRV_PCM_FORMAT_S24_3LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-S20-3BE:
> +
> +      - ``V4L2_AUDIO_FMT_S20_3BE``
> +      - 'S20_3BE'
> +      - Correspond to SNDRV_PCM_FORMAT_S20_3BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-U20-3LE:
> +
> +      - ``V4L2_AUDIO_FMT_U20_3LE``
> +      - 'U20_3LE'
> +      - Correspond to SNDRV_PCM_FORMAT_U20_3LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-U20-3BE:
> +
> +      - ``V4L2_AUDIO_FMT_U20_3BE``
> +      - 'U20_3BE'
> +      - Correspond to SNDRV_PCM_FORMAT_U20_3BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-S18-3LE:
> +
> +      - ``V4L2_AUDIO_FMT_S18_3LE``
> +      - 'S18_3LE'
> +      - Correspond to SNDRV_PCM_FORMAT_S18_3LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-S18-3BE:
> +
> +      - ``V4L2_AUDIO_FMT_S18_3BE``
> +      - 'S18_3BE'
> +      - Correspond to SNDRV_PCM_FORMAT_S18_3BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-U18-3LE:
> +
> +      - ``V4L2_AUDIO_FMT_U18_3LE``
> +      - 'U18_3LE'
> +      - Correspond to SNDRV_PCM_FORMAT_U18_3LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-U18-3BE:
> +
> +      - ``V4L2_AUDIO_FMT_U18_3BE``
> +      - 'U18_3BE'
> +      - Correspond to SNDRV_PCM_FORMAT_U18_3BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-G723-24:
> +
> +      - ``V4L2_AUDIO_FMT_G723_24``
> +      - 'G723_24'
> +      - Correspond to SNDRV_PCM_FORMAT_G723_24 in ALSA
> +    * .. _V4L2-AUDIO-FMT-G723-24-1B:
> +
> +      - ``V4L2_AUDIO_FMT_G723_24_1B``
> +      - 'G723_24_1B'
> +      - Correspond to SNDRV_PCM_FORMAT_G723_24_1B in ALSA
> +    * .. _V4L2-AUDIO-FMT-G723-40:
> +
> +      - ``V4L2_AUDIO_FMT_G723_40``
> +      - 'G723_40'
> +      - Correspond to SNDRV_PCM_FORMAT_G723_40 in ALSA
> +    * .. _V4L2-AUDIO-FMT-G723-40-1B:
> +
> +      - ``V4L2_AUDIO_FMT_G723_40_1B``
> +      - 'G723_40_1B'
> +      - Correspond to SNDRV_PCM_FORMAT_G723_40_1B in ALSA
> +    * .. _V4L2-AUDIO-FMT-DSD-U8:
> +
> +      - ``V4L2_AUDIO_FMT_DSD-U8``
> +      - 'DSD_U8'
> +      - Correspond to SNDRV_PCM_FORMAT_DSD_U8 in ALSA
> +    * .. _V4L2-AUDIO-FMT-DSD-U16-LE:
> +
> +      - ``V4L2_AUDIO_FMT_DSD-U16-LE``
> +      - 'DSD_U16_LE'
> +      - Correspond to SNDRV_PCM_FORMAT_DSD_U16_LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-DSD-U32-LE:
> +
> +      - ``V4L2_AUDIO_FMT_DSD-U32-LE``
> +      - 'DSD_U32_LE'
> +      - Correspond to SNDRV_PCM_FORMAT_DSD_U32_LE in ALSA
> +    * .. _V4L2-AUDIO-FMT-DSD-U16-BE:
> +
> +      - ``V4L2_AUDIO_FMT_DSD-U16-BE``
> +      - 'DSD_U16_BE'
> +      - Correspond to SNDRV_PCM_FORMAT_DSD_U16_BE in ALSA
> +    * .. _V4L2-AUDIO-FMT-DSD-U32-BE:
> +
> +      - ``V4L2_AUDIO_FMT_DSD-U32-BE``
> +      - 'DSD_U32_BE'
> +      - Correspond to SNDRV_PCM_FORMAT_DSD_U32_BE in ALSA
> diff --git a/Documentation/userspace-api/media/v4l/pixfmt.rst b/Documentation/userspace-api/media/v4l/pixfmt.rst
> index 11dab4a90630..2eb6fdd3b43d 100644
> --- a/Documentation/userspace-api/media/v4l/pixfmt.rst
> +++ b/Documentation/userspace-api/media/v4l/pixfmt.rst
> @@ -36,3 +36,4 @@ see also :ref:`VIDIOC_G_FBUF <VIDIOC_G_FBUF>`.)
>      colorspaces
>      colorspaces-defs
>      colorspaces-details
> +    pixfmt-audio
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 1a40090d8287..044611d5d3f8 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1471,6 +1471,57 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>  	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
>  	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
>  	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
> +	case V4L2_AUDIO_FMT_S8:		descr = "8-bit Signed"; break;
> +	case V4L2_AUDIO_FMT_U8:		descr = "8-bit Unsigned"; break;
> +	case V4L2_AUDIO_FMT_S16_LE:	descr = "16-bit Signed LE"; break;
> +	case V4L2_AUDIO_FMT_S16_BE:		descr = "16-bit Signed BE"; break;
> +	case V4L2_AUDIO_FMT_U16_LE:		descr = "16-bit Unsigned LE"; break;
> +	case V4L2_AUDIO_FMT_U16_BE:		descr = "16-bit Unsigned BE"; break;
> +	case V4L2_AUDIO_FMT_S24_LE:		descr = "24(32)-bit Signed LE"; break;
> +	case V4L2_AUDIO_FMT_S24_BE:		descr = "24(32)-bit Signed BE"; break;
> +	case V4L2_AUDIO_FMT_U24_LE:		descr = "24(32)-bit Unsigned LE"; break;
> +	case V4L2_AUDIO_FMT_U24_BE:		descr = "24(32)-bit Unsigned BE"; break;
> +	case V4L2_AUDIO_FMT_S32_LE:		descr = "32-bit Signed LE"; break;
> +	case V4L2_AUDIO_FMT_S32_BE:		descr = "32-bit Signed BE"; break;
> +	case V4L2_AUDIO_FMT_U32_LE:		descr = "32-bit Unsigned LE"; break;
> +	case V4L2_AUDIO_FMT_U32_BE:		descr = "32-bit Unsigned BE"; break;
> +	case V4L2_AUDIO_FMT_FLOAT_LE:		descr = "32-bit Float LE"; break;
> +	case V4L2_AUDIO_FMT_FLOAT_BE:		descr = "32-bit Float BE"; break;
> +	case V4L2_AUDIO_FMT_FLOAT64_LE:		descr = "64-bit Float LE"; break;
> +	case V4L2_AUDIO_FMT_FLOAT64_BE:		descr = "64-bit Float BE"; break;
> +	case V4L2_AUDIO_FMT_IEC958_SUBFRAME_LE:	descr = "32-bit IEC958 LE"; break;
> +	case V4L2_AUDIO_FMT_IEC958_SUBFRAME_BE:	descr = "32-bit IEC958 BE"; break;
> +	case V4L2_AUDIO_FMT_MU_LAW:		descr = "Mu Law"; break;
> +	case V4L2_AUDIO_FMT_A_LAW:		descr = "A Law"; break;
> +	case V4L2_AUDIO_FMT_IMA_ADPCM:		descr = "IMA ADPCM"; break;
> +	case V4L2_AUDIO_FMT_MPEG:		descr = "MPEG Audio"; break;

Compressed formats are handled in the default case, and those set theV4L2_FMT_FLAG_COMPRESSED
flag. That's true for MPEG and perhaps also for some of the other audio formats?

I'm no audio expert, so I don't know which are compressed or not.

> +	case V4L2_AUDIO_FMT_GSM:		descr = "GSM Audio"; break;
> +	case V4L2_AUDIO_FMT_S20_LE:		descr = "20-bit Signed LE"; break;
> +	case V4L2_AUDIO_FMT_S20_BE:		descr = "20-bit Signed BE"; break;
> +	case V4L2_AUDIO_FMT_U20_LE:		descr = "20-bit Unsigned LE"; break;
> +	case V4L2_AUDIO_FMT_U20_BE:		descr = "20-bit Unsigned BE"; break;
> +	case V4L2_AUDIO_FMT_SPECIAL:		descr = "Special Audio"; break;
> +	case V4L2_AUDIO_FMT_S24_3LE:		descr = "24(24)-bit Signed LE"; break;
> +	case V4L2_AUDIO_FMT_S24_3BE:		descr = "24(24)-bit Signed BE"; break;
> +	case V4L2_AUDIO_FMT_U24_3LE:		descr = "24(24)-bit Unsigned LE"; break;
> +	case V4L2_AUDIO_FMT_U24_3BE:		descr = "24(24)-bit Unsigned BE"; break;
> +	case V4L2_AUDIO_FMT_S20_3LE:		descr = "20(24)-bit Signed LE"; break;
> +	case V4L2_AUDIO_FMT_S20_3BE:		descr = "20(24)-bit Signed BE"; break;
> +	case V4L2_AUDIO_FMT_U20_3LE:		descr = "20(24)-bit Unsigned LE"; break;
> +	case V4L2_AUDIO_FMT_U20_3BE:		descr = "20(24)-bit Unsigned BE"; break;
> +	case V4L2_AUDIO_FMT_S18_3LE:		descr = "18(24)-bit Signed LE"; break;
> +	case V4L2_AUDIO_FMT_S18_3BE:		descr = "18(24)-bit Signed BE"; break;
> +	case V4L2_AUDIO_FMT_U18_3LE:		descr = "18(24)-bit Unsigned LE"; break;
> +	case V4L2_AUDIO_FMT_U18_3BE:		descr = "18(24)-bit Unsigned BE"; break;
> +	case V4L2_AUDIO_FMT_G723_24:		descr = "G723 24"; break;
> +	case V4L2_AUDIO_FMT_G723_24_1B:		descr = "G723 24 1b"; break;
> +	case V4L2_AUDIO_FMT_G723_40:		descr = "G723 40"; break;
> +	case V4L2_AUDIO_FMT_G723_40_1B:		descr = "G723 40 1b"; break;
> +	case V4L2_AUDIO_FMT_DSD_U8:		descr = "8-bit DSD"; break;
> +	case V4L2_AUDIO_FMT_DSD_U16_LE:		descr = "16-bit DSD LE"; break;
> +	case V4L2_AUDIO_FMT_DSD_U32_LE:		descr = "32-bit DSD LE"; break;
> +	case V4L2_AUDIO_FMT_DSD_U16_BE:		descr = "16-bit DSD BE"; break;
> +	case V4L2_AUDIO_FMT_DSD_U32_BE:		descr = "32-bit DSD BE"; break;
>  
>  	default:
>  		/* Compressed formats */
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 166c51f537cc..72d7d71050ee 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -842,6 +842,62 @@ struct v4l2_pix_format {
>  #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
>  #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
>  
> +/* Audio-data formats */

I think that for now you should only include the formats that your hardware
actually supports.

But it is important to mention in the comment that all these audio formats use
a fourcc starting with 'AU' followed by the SNDRV_PCM_FORMAT_ value from asound.h.

> +#define V4L2_AUDIO_FMT_S8			v4l2_fourcc('A', 'U', '0', '0')
> +#define V4L2_AUDIO_FMT_U8			v4l2_fourcc('A', 'U', '0', '1')
> +#define V4L2_AUDIO_FMT_S16_LE			v4l2_fourcc('A', 'U', '0', '2')
> +#define V4L2_AUDIO_FMT_S16_BE			v4l2_fourcc('A', 'U', '0', '3')
> +#define V4L2_AUDIO_FMT_U16_LE			v4l2_fourcc('A', 'U', '0', '4')
> +#define V4L2_AUDIO_FMT_U16_BE			v4l2_fourcc('A', 'U', '0', '5')
> +#define V4L2_AUDIO_FMT_S24_LE			v4l2_fourcc('A', 'U', '0', '6')
> +#define V4L2_AUDIO_FMT_S24_BE			v4l2_fourcc('A', 'U', '0', '7')
> +#define V4L2_AUDIO_FMT_U24_LE			v4l2_fourcc('A', 'U', '0', '8')
> +#define V4L2_AUDIO_FMT_U24_BE			v4l2_fourcc('A', 'U', '0', '9')
> +
> +#define V4L2_AUDIO_FMT_S32_LE			v4l2_fourcc('A', 'U', '1', '0')
> +#define V4L2_AUDIO_FMT_S32_BE			v4l2_fourcc('A', 'U', '1', '1')
> +#define V4L2_AUDIO_FMT_U32_LE			v4l2_fourcc('A', 'U', '1', '2')
> +#define V4L2_AUDIO_FMT_U32_BE			v4l2_fourcc('A', 'U', '1', '3')
> +#define V4L2_AUDIO_FMT_FLOAT_LE			v4l2_fourcc('A', 'U', '1', '4')
> +#define V4L2_AUDIO_FMT_FLOAT_BE			v4l2_fourcc('A', 'U', '1', '5')
> +#define V4L2_AUDIO_FMT_FLOAT64_LE		v4l2_fourcc('A', 'U', '1', '6')
> +#define V4L2_AUDIO_FMT_FLOAT64_BE		v4l2_fourcc('A', 'U', '1', '7')
> +#define V4L2_AUDIO_FMT_IEC958_SUBFRAME_LE	v4l2_fourcc('A', 'U', '1', '8')
> +#define V4L2_AUDIO_FMT_IEC958_SUBFRAME_BE	v4l2_fourcc('A', 'U', '1', '9')
> +
> +#define V4L2_AUDIO_FMT_MU_LAW			v4l2_fourcc('A', 'U', '2', '0')
> +#define V4L2_AUDIO_FMT_A_LAW			v4l2_fourcc('A', 'U', '2', '1')
> +#define V4L2_AUDIO_FMT_IMA_ADPCM		v4l2_fourcc('A', 'U', '2', '2')
> +#define V4L2_AUDIO_FMT_MPEG			v4l2_fourcc('A', 'U', '2', '3')
> +#define V4L2_AUDIO_FMT_GSM			v4l2_fourcc('A', 'U', '2', '4')
> +#define V4L2_AUDIO_FMT_S20_LE			v4l2_fourcc('A', 'U', '2', '5')
> +#define V4L2_AUDIO_FMT_S20_BE			v4l2_fourcc('A', 'U', '2', '6')
> +#define V4L2_AUDIO_FMT_U20_LE			v4l2_fourcc('A', 'U', '2', '7')
> +#define V4L2_AUDIO_FMT_U20_BE			v4l2_fourcc('A', 'U', '2', '8')
> +
> +#define V4L2_AUDIO_FMT_SPECIAL			v4l2_fourcc('A', 'U', '3', '1')
> +#define V4L2_AUDIO_FMT_S24_3LE			v4l2_fourcc('A', 'U', '3', '2')
> +#define V4L2_AUDIO_FMT_S24_3BE			v4l2_fourcc('A', 'U', '3', '3')
> +#define V4L2_AUDIO_FMT_U24_3LE			v4l2_fourcc('A', 'U', '3', '4')
> +#define V4L2_AUDIO_FMT_U24_3BE			v4l2_fourcc('A', 'U', '3', '5')
> +#define V4L2_AUDIO_FMT_S20_3LE			v4l2_fourcc('A', 'U', '3', '6')
> +#define V4L2_AUDIO_FMT_S20_3BE			v4l2_fourcc('A', 'U', '3', '7')
> +#define V4L2_AUDIO_FMT_U20_3LE			v4l2_fourcc('A', 'U', '3', '8')
> +#define V4L2_AUDIO_FMT_U20_3BE			v4l2_fourcc('A', 'U', '3', '9')
> +#define V4L2_AUDIO_FMT_S18_3LE			v4l2_fourcc('A', 'U', '4', '0')
> +#define V4L2_AUDIO_FMT_S18_3BE			v4l2_fourcc('A', 'U', '4', '1')
> +#define V4L2_AUDIO_FMT_U18_3LE			v4l2_fourcc('A', 'U', '4', '2')
> +#define V4L2_AUDIO_FMT_U18_3BE			v4l2_fourcc('A', 'U', '4', '3')
> +#define V4L2_AUDIO_FMT_G723_24			v4l2_fourcc('A', 'U', '4', '4')
> +#define V4L2_AUDIO_FMT_G723_24_1B		v4l2_fourcc('A', 'U', '4', '5')
> +#define V4L2_AUDIO_FMT_G723_40			v4l2_fourcc('A', 'U', '4', '6')
> +#define V4L2_AUDIO_FMT_G723_40_1B		v4l2_fourcc('A', 'U', '4', '7')
> +#define V4L2_AUDIO_FMT_DSD_U8			v4l2_fourcc('A', 'U', '4', '8')
> +#define V4L2_AUDIO_FMT_DSD_U16_LE		v4l2_fourcc('A', 'U', '4', '9')
> +#define V4L2_AUDIO_FMT_DSD_U32_LE		v4l2_fourcc('A', 'U', '5', '0')
> +#define V4L2_AUDIO_FMT_DSD_U16_BE		v4l2_fourcc('A', 'U', '5', '1')
> +#define V4L2_AUDIO_FMT_DSD_U32_BE		v4l2_fourcc('A', 'U', '5', '2')
> +
>  /* priv field value to indicates that subsequent fields are valid. */
>  #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
>  

Regards,

	Hans