Message ID | 1694670845-17070-1-git-send-email-shengjiu.wang@nxp.com |
---|---|
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp151864vqi; Wed, 13 Sep 2023 23:39:13 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFtuz/8CuIxJlPtb2mS04mHx1d3EfEnKWUEd7E2leSoIy1Yyxg5xoLRHhax5g/KcavpfrEA X-Received: by 2002:a05:6a20:3254:b0:14b:a3f8:1c17 with SMTP id hm20-20020a056a20325400b0014ba3f81c17mr4086050pzc.38.1694673553402; Wed, 13 Sep 2023 23:39:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694673553; cv=none; d=google.com; s=arc-20160816; b=T/Av1WMSxwPpYpxCTuGC8J2AdkVJgoXtBvgTCNhNKrgRSdKAHn5yHlu6G9Msypnbby 5VTe2oKhnDuWoJo4wof80RihwajOPOIw7wTxndG7VGfoxecuoc2GJNIIp/jfweCD+/hV rfsmZXwTr9+orlPLfpt3lhryiL141gZvypx9Fs70fF3DnwIcUiECjb+FZyYUQeqGnPCw FGC8BMA/GJcprt/seMx0Rkc9ZNyKy/DCZ90hE373M6ghDP8+uKscDx1UwktgLJh6nOCt toU2Aj53s+h1KB9qBjVddVse6zbDAa3UAx7MLi6hAHE2YqIonXa7HI00coKavEbdUwrP Bebg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:message-id:date:subject:to:from; bh=LPzMUpF3eAUjQmp08IXQ5fE62dTx7trR4FmN7F1UDzk=; fh=ufiESpW7Hvn5oNmN0l6/qCUiQrKfLLrOwb5uR1Dftds=; b=eLefqd+QvOk6QrzQcwFyuaTkLucB9Bx73nLaacNBN2KAWVhbjZuu/Mw2L84M3Hh+mi kJmdL+dZ+rMV3/HMeCLqPFYOxfFiRa7WJO1rDFIXkAU4+ariZVSFBq+icWtWL5YeFlAn GYkLGn26Bi2XwLppL//7Hc0N/eZCKadBZz+d5Rb8xBnMvHJC766QHQZmHPBnaUt7I9WX mzONiEDSEyBmSPya5dNuO2LAK5nRI9xCCSCMR65/3hY1ZgHr4mNY3JSQH6lX8mXTlt4k hKtMkw7UpnWGLH1+sm1ywNU578sFVpab154hmoBc9oe9mVMWSS3nsWCoNG5rf0Vm88Sk 1xEg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id o5-20020a17090323c500b001bbc6846a21si994470plh.429.2023.09.13.23.39.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 23:39:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id C0E7E8021385; Wed, 13 Sep 2023 23:31:43 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235537AbjINGaq (ORCPT <rfc822;chrisfriedt@gmail.com> + 35 others); Thu, 14 Sep 2023 02:30:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49170 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235498AbjINGap (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Thu, 14 Sep 2023 02:30:45 -0400 Received: from inva020.nxp.com (inva020.nxp.com [92.121.34.13]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D2B8CC7 for <linux-kernel@vger.kernel.org>; Wed, 13 Sep 2023 23:30:41 -0700 (PDT) Received: from inva020.nxp.com (localhost [127.0.0.1]) by inva020.eu-rdc02.nxp.com (Postfix) with ESMTP id A8EA11A0638; Thu, 14 Sep 2023 08:30:39 +0200 (CEST) Received: from aprdc01srsp001v.ap-rdc01.nxp.com (aprdc01srsp001v.ap-rdc01.nxp.com [165.114.16.16]) by inva020.eu-rdc02.nxp.com (Postfix) with ESMTP id 45DC31A062C; Thu, 14 Sep 2023 08:30:39 +0200 (CEST) Received: from localhost.localdomain (shlinux2.ap.freescale.net [10.192.224.44]) by aprdc01srsp001v.ap-rdc01.nxp.com (Postfix) with ESMTP id 5F6141834869; Thu, 14 Sep 2023 14:30:37 +0800 (+08) From: Shengjiu Wang <shengjiu.wang@nxp.com> To: hverkuil@xs4all.nl, sakari.ailus@iki.fi, tfiga@chromium.org, m.szyprowski@samsung.com, mchehab@kernel.org, linux-media@vger.kernel.org, linux-kernel@vger.kernel.org, shengjiu.wang@gmail.com, Xiubo.Lee@gmail.com, festevam@gmail.com, nicoleotsuka@gmail.com, lgirdwood@gmail.com, broonie@kernel.org, perex@perex.cz, tiwai@suse.com, alsa-devel@alsa-project.org, linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH v3 0/9] Add audio support in v4l2 framework Date: Thu, 14 Sep 2023 13:53:56 +0800 Message-Id: <1694670845-17070-1-git-send-email-shengjiu.wang@nxp.com> X-Mailer: git-send-email 2.7.4 X-Virus-Scanned: ClamAV using ClamSMTP Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Wed, 13 Sep 2023 23:31:43 -0700 (PDT) X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776994015841481279 X-GMAIL-MSGID: 1776994015841481279 |
Series |
Add audio support in v4l2 framework
|
|
Message
Shengjiu Wang
Sept. 14, 2023, 5:53 a.m. UTC
Audio signal processing also has the requirement for memory to memory similar as Video. This asrc memory to memory (memory ->asrc->memory) case is a non real time use case. User fills the input buffer to the asrc module, after conversion, then asrc sends back the output buffer to user. So it is not a traditional ALSA playback and capture case. It is a specific use case, there is no reference in current kernel. v4l2 memory to memory is the closed implementation, v4l2 current support video, image, radio, tuner, touch devices, so it is not complicated to add support for this specific audio case. Because we had implemented the "memory -> asrc ->i2s device-> codec" use case in ALSA. Now the "memory->asrc->memory" needs to reuse the code in asrc driver, so the first 3 patches is for refining the code to make it can be shared by the "memory->asrc->memory" driver. The main change is in the v4l2 side, A /dev/vl4-audioX will be created, user applications only use the ioctl of v4l2 framework. Other change is to add memory to memory support for two kinds of i.MX ASRC module. changes in v3: - Modify documents for adding audio m2m support - Add audio virtual m2m driver - Defined V4L2_AUDIO_FMT_LPCM format type for audio. - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case. - with modification in v4l-utils, pass v4l2-compliance test. changes in v2: - decouple the implementation in v4l2 and ALSA - implement the memory to memory driver as a platfrom driver and move it to driver/media - move fsl_asrc_common.h to include/sound folder Shengjiu Wang (9): ASoC: fsl_asrc: define functions for memory to memory usage ASoC: fsl_easrc: define functions for memory to memory usage ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound ASoC: fsl_asrc: register m2m platform device ASoC: fsl_easrc: register m2m platform device media: v4l2: Add audio capture and output support media: uapi: Add V4L2_CID_USER_IMX_ASRC_RATIO_MOD control media: audm2m: add virtual driver for audio memory to memory media: imx-asrc: Add memory to memory driver .../userspace-api/media/v4l/audio-formats.rst | 15 + .../userspace-api/media/v4l/buffer.rst | 6 + .../userspace-api/media/v4l/control.rst | 5 + .../userspace-api/media/v4l/dev-audio.rst | 63 + .../userspace-api/media/v4l/devices.rst | 1 + .../media/v4l/pixfmt-aud-lpcm.rst | 31 + .../userspace-api/media/v4l/pixfmt.rst | 1 + .../media/v4l/vidioc-enum-fmt.rst | 2 + .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 + .../media/v4l/vidioc-querycap.rst | 3 + .../media/videodev2.h.rst.exceptions | 2 + .../media/common/videobuf2/videobuf2-v4l2.c | 4 + drivers/media/platform/nxp/Kconfig | 12 + drivers/media/platform/nxp/Makefile | 1 + drivers/media/platform/nxp/imx-asrc.c | 1058 +++++++++++++++++ drivers/media/test-drivers/Kconfig | 9 + drivers/media/test-drivers/Makefile | 1 + drivers/media/test-drivers/audm2m.c | 767 ++++++++++++ drivers/media/v4l2-core/v4l2-ctrls-defs.c | 1 + drivers/media/v4l2-core/v4l2-dev.c | 17 + drivers/media/v4l2-core/v4l2-ioctl.c | 53 + include/media/v4l2-dev.h | 2 + include/media/v4l2-ioctl.h | 34 + .../fsl => include/sound}/fsl_asrc_common.h | 54 + include/uapi/linux/v4l2-controls.h | 1 + include/uapi/linux/videodev2.h | 25 + sound/soc/fsl/fsl_asrc.c | 162 +++ sound/soc/fsl/fsl_asrc.h | 4 +- sound/soc/fsl/fsl_asrc_dma.c | 2 +- sound/soc/fsl/fsl_easrc.c | 239 ++++ sound/soc/fsl/fsl_easrc.h | 8 +- 31 files changed, 2584 insertions(+), 3 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/audio-formats.rst create mode 100644 Documentation/userspace-api/media/v4l/dev-audio.rst create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-aud-lpcm.rst create mode 100644 drivers/media/platform/nxp/imx-asrc.c create mode 100644 drivers/media/test-drivers/audm2m.c rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%)
Comments
Hi Shenjiu, Thanks for the update. On Thu, Sep 14, 2023 at 01:54:02PM +0800, Shengjiu Wang wrote: > Audio signal processing has the requirement for memory to > memory similar as Video. > > This patch is to add this support in v4l2 framework, defined > new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and > V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format > for audio case usage. > > Defined V4L2_AUDIO_FMT_LPCM format type for audio. This would be nicer as a separate patch. Also see the related comments below. > > Defined V4L2_CAP_AUDIO_M2M capability type for audio memory > to memory case. > > The created audio device is named "/dev/v4l-audioX". > > Signed-off-by: Shengjiu Wang <shengjiu.wang@nxp.com> > --- > .../userspace-api/media/v4l/audio-formats.rst | 15 +++++ > .../userspace-api/media/v4l/buffer.rst | 6 ++ > .../userspace-api/media/v4l/dev-audio.rst | 63 +++++++++++++++++++ > .../userspace-api/media/v4l/devices.rst | 1 + > .../media/v4l/pixfmt-aud-lpcm.rst | 31 +++++++++ > .../userspace-api/media/v4l/pixfmt.rst | 1 + > .../media/v4l/vidioc-enum-fmt.rst | 2 + > .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 ++ > .../media/v4l/vidioc-querycap.rst | 3 + > .../media/videodev2.h.rst.exceptions | 2 + > .../media/common/videobuf2/videobuf2-v4l2.c | 4 ++ > drivers/media/v4l2-core/v4l2-dev.c | 17 +++++ > drivers/media/v4l2-core/v4l2-ioctl.c | 53 ++++++++++++++++ > include/media/v4l2-dev.h | 2 + > include/media/v4l2-ioctl.h | 34 ++++++++++ > include/uapi/linux/videodev2.h | 25 ++++++++ > 16 files changed, 263 insertions(+) > create mode 100644 Documentation/userspace-api/media/v4l/audio-formats.rst > create mode 100644 Documentation/userspace-api/media/v4l/dev-audio.rst > create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-aud-lpcm.rst > > diff --git a/Documentation/userspace-api/media/v4l/audio-formats.rst b/Documentation/userspace-api/media/v4l/audio-formats.rst > new file mode 100644 > index 000000000000..bc52712d20d3 > --- /dev/null > +++ b/Documentation/userspace-api/media/v4l/audio-formats.rst > @@ -0,0 +1,15 @@ > +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later > + > +.. _audio-formats: > + > +************* > +Audio Formats > +************* > + > +These formats are used for :ref:`audio` interface only. > + > + > +.. toctree:: > + :maxdepth: 1 > + > + pixfmt-aud-lpcm > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst > index 04dec3e570ed..80cf2cb20dfe 100644 > --- a/Documentation/userspace-api/media/v4l/buffer.rst > +++ b/Documentation/userspace-api/media/v4l/buffer.rst > @@ -438,6 +438,12 @@ enum v4l2_buf_type > * - ``V4L2_BUF_TYPE_META_OUTPUT`` > - 14 > - Buffer for metadata output, see :ref:`metadata`. > + * - ``V4L2_BUF_TYPE_AUDIO_CAPTURE`` > + - 15 > + - Buffer for audio capture, see :ref:`audio`. > + * - ``V4L2_BUF_TYPE_AUDIO_OUTPUT`` > + - 16 > + - Buffer for audio output, see :ref:`audio`. > > > .. _buffer-flags: > diff --git a/Documentation/userspace-api/media/v4l/dev-audio.rst b/Documentation/userspace-api/media/v4l/dev-audio.rst > new file mode 100644 > index 000000000000..f9bcf0c7b056 > --- /dev/null > +++ b/Documentation/userspace-api/media/v4l/dev-audio.rst > @@ -0,0 +1,63 @@ > +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later > + > +.. _audiodev: > + > +****************** > +audio Interface Capital "A"? > +****************** Too many asterisks (same a few lines above, too). > + > +The audio interface is implemented on audio device nodes. The audio device > +which uses application software for modulation or demodulation. This > +interface is intended for controlling and data streaming of such devices > + > +Audio devices are accessed through character device special files named > +``/dev/v4l-audio`` > + > +Querying Capabilities > +===================== > + > +Device nodes supporting the audio capture and output interface set the > +``V4L2_CAP_AUDIO_M2M`` flag in the ``device_caps`` field of the > +:c:type:`v4l2_capability` structure returned by the :c:func:`VIDIOC_QUERYCAP` > +ioctl. > + > +At least one of the read/write or streaming I/O methods must be supported. > + > + > +Data Format Negotiation > +======================= > + > +The audio device uses the :ref:`format` ioctls to select the capture format. > +The audio buffer content format is bound to that selected format. In addition > +to the basic :ref:`format` ioctls, the :c:func:`VIDIOC_ENUM_FMT` ioctl must be > +supported as well. > + > +To use the :ref:`format` ioctls applications set the ``type`` field of the > +:c:type:`v4l2_format` structure to ``V4L2_BUF_TYPE_AUDIO_CAPTURE`` or to > +``V4L2_BUF_TYPE_AUDIO_OUTPUT``. Both drivers and applications must set the > +remainder of the :c:type:`v4l2_format` structure to 0. > + > +.. c:type:: v4l2_audio_format > + > +.. tabularcolumns:: |p{1.4cm}|p{2.4cm}|p{13.5cm}| > + > +.. flat-table:: struct v4l2_audio_format > + :header-rows: 0 > + :stub-columns: 0 > + :widths: 1 1 2 > + > + * - __u32 > + - ``rate`` > + - The sample rate, set by the application. The range is [5512, 768000]. > + * - __u32 > + - ``format`` > + - The sample format, set by the application. format is defined as > + SNDRV_PCM_FORMAT_S8, SNDRV_PCM_FORMAT_U8, ..., > + * - __u32 > + - ``channels`` > + - The channel number, set by the application. channel number range is > + [1, 32]. > + * - __u32 > + - ``buffersize`` > + - Maximum buffer size in bytes required for data. The value is set by the > + driver. > diff --git a/Documentation/userspace-api/media/v4l/devices.rst b/Documentation/userspace-api/media/v4l/devices.rst > index 8bfbad65a9d4..8261f3468489 100644 > --- a/Documentation/userspace-api/media/v4l/devices.rst > +++ b/Documentation/userspace-api/media/v4l/devices.rst > @@ -24,3 +24,4 @@ Interfaces > dev-event > dev-subdev > dev-meta > + dev-audio > diff --git a/Documentation/userspace-api/media/v4l/pixfmt-aud-lpcm.rst b/Documentation/userspace-api/media/v4l/pixfmt-aud-lpcm.rst > new file mode 100644 > index 000000000000..f9ebe2a05f69 > --- /dev/null > +++ b/Documentation/userspace-api/media/v4l/pixfmt-aud-lpcm.rst > @@ -0,0 +1,31 @@ > +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later > + > +.. _v4l2-aud-fmt-lpcm: > + > +************************* > +V4L2_AUDIO_FMT_LPCM ('LPCM') > +************************* > + > +Linear Pulse-Code Modulation (LPCM) > + > + > +Description > +=========== > + > +This describes audio format used by the audio memory to memory driver. > + > +It contains the following fields: > + > +.. flat-table:: > + :widths: 1 4 > + :header-rows: 1 > + :stub-columns: 0 > + > + * - Field > + - Description > + * - u32 samplerate; > + - which is the number of times per second that samples are taken. > + * - u32 sampleformat; > + - which determines the number of possible digital values that can be used to represent each sample 80 characters (or less) per line, please. Which values could this field have and what do they signify? > + * - u32 channels; > + - channel number for each sample. I suppose the rest of the buffer would be samples? This should be documented. I think there are also different ways the data could be arrangeed and this needs to be documented, too.