Message ID | 1700552255-5364-1-git-send-email-shengjiu.wang@nxp.com |
---|---|
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2b07:b0:403:3b70:6f57 with SMTP id io7csp471131vqb; Tue, 21 Nov 2023 00:17:55 -0800 (PST) X-Google-Smtp-Source: AGHT+IEOFIiLjRXo7E4iUEzhgiBiEvbebTwiiaYi5rSI0eJcWvwk4cJl/II9BKWF5HOLY3wgVLix X-Received: by 2002:a05:6a20:914a:b0:18a:df45:fad4 with SMTP id x10-20020a056a20914a00b0018adf45fad4mr1545056pzc.37.1700554674814; Tue, 21 Nov 2023 00:17:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700554674; cv=none; d=google.com; s=arc-20160816; b=z3tf1usTyMKvVLG3eRoGNSlNras6+JCYQeX1h9eSkGG2kBsk5y6ZEvNh8qRnB77quH MnclU08RZw8DPTEIg5o84vkZ3ymldXgzhcK/Wb3aJ/weHvjEBNwP22VkKlXLOzLMB7w1 z4uMOH2daNvZBNcL6frBrbPS6vXdJkfslYlGCxJYsus+4MbxshgC5NBeQq8AbKzEuBgz ULrav37LkY2eCwru5MwYuS98iX3a0yCy825pe0uWCoqhDwkOYYS/KbkJAK0SqDcxbL65 cjANS+pEdrSocNhKeuThR2hveUq631WfVwJbhikENuwKZJlpzoCshxGbu9/9lJHtAPwp +5FA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:message-id:date:subject:to:from; bh=7qFVPzHZ8e/BpJxPbdfAtyNhOjsNtioecNNO5110khw=; fh=ufiESpW7Hvn5oNmN0l6/qCUiQrKfLLrOwb5uR1Dftds=; b=TaFigc46g0K/p586CiTcJ+F2/RJi5tOFuzDWSktfF7uGa+KnVyJrpRSkDg/ffOR3EL fVTjiGZkB/pwprcnjGSHOJnqMowvNvlspo5X5KrqqiM5yLPIr7+7CjcyRB7RU6JfE07Z x0aMd+FebMhhaK8z9BCVwkf19aC6rW5uw2L4veAyepLT+QTVFNaRJkmblPdhOettuCNV 6NgVscCvE9UfHaTVHqY1UAkU1vUNIhwNc55qVof5TLEw+1kto0W1y/lfn1wyj+LfIn9U QSmMMgFYCnXxHHOeq0hwBuGQqiqY0fwZsQZhtDSi7sDKnBa6x7qK6mWsQI28X/oUJjXB oU+g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id y8-20020a056a00180800b006cb6cc056f8si6091336pfa.334.2023.11.21.00.17.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Nov 2023 00:17:54 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id A077A80BE600; Tue, 21 Nov 2023 00:17:41 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230028AbjKUIRi (ORCPT <rfc822;ouuuleilei@gmail.com> + 99 others); Tue, 21 Nov 2023 03:17:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229457AbjKUIRg (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Tue, 21 Nov 2023 03:17:36 -0500 Received: from inva021.nxp.com (inva021.nxp.com [92.121.34.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B829CB; Tue, 21 Nov 2023 00:17:32 -0800 (PST) Received: from inva021.nxp.com (localhost [127.0.0.1]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id 2F456200EEA; Tue, 21 Nov 2023 09:17:31 +0100 (CET) Received: from aprdc01srsp001v.ap-rdc01.nxp.com (aprdc01srsp001v.ap-rdc01.nxp.com [165.114.16.16]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id BDA8620005B; Tue, 21 Nov 2023 09:17:30 +0100 (CET) Received: from localhost.localdomain (shlinux2.ap.freescale.net [10.192.224.44]) by aprdc01srsp001v.ap-rdc01.nxp.com (Postfix) with ESMTP id C626D1802201; Tue, 21 Nov 2023 16:17:28 +0800 (+08) From: Shengjiu Wang <shengjiu.wang@nxp.com> To: hverkuil@xs4all.nl, sakari.ailus@iki.fi, tfiga@chromium.org, m.szyprowski@samsung.com, mchehab@kernel.org, linux-media@vger.kernel.org, linux-kernel@vger.kernel.org, shengjiu.wang@gmail.com, Xiubo.Lee@gmail.com, festevam@gmail.com, nicoleotsuka@gmail.com, lgirdwood@gmail.com, broonie@kernel.org, perex@perex.cz, tiwai@suse.com, alsa-devel@alsa-project.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v10 00/14] Add audio support in v4l2 framework Date: Tue, 21 Nov 2023 15:37:21 +0800 Message-Id: <1700552255-5364-1-git-send-email-shengjiu.wang@nxp.com> X-Mailer: git-send-email 2.7.4 X-Virus-Scanned: ClamAV using ClamSMTP X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Tue, 21 Nov 2023 00:17:41 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783160818855490962 X-GMAIL-MSGID: 1783160818855490962 |
Series |
Add audio support in v4l2 framework
|
|
Message
Shengjiu Wang
Nov. 21, 2023, 7:37 a.m. UTC
Audio signal processing also has the requirement for memory to memory similar as Video. This asrc memory to memory (memory ->asrc->memory) case is a non real time use case. User fills the input buffer to the asrc module, after conversion, then asrc sends back the output buffer to user. So it is not a traditional ALSA playback and capture case. It is a specific use case, there is no reference in current kernel. v4l2 memory to memory is the closed implementation, v4l2 current support video, image, radio, tuner, touch devices, so it is not complicated to add support for this specific audio case. Because we had implemented the "memory -> asrc ->i2s device-> codec" use case in ALSA. Now the "memory->asrc->memory" needs to reuse the code in asrc driver, so the first 3 patches is for refining the code to make it can be shared by the "memory->asrc->memory" driver. The main change is in the v4l2 side, A /dev/vl4-audioX will be created, user applications only use the ioctl of v4l2 framework. Other change is to add memory to memory support for two kinds of i.MX ASRC module. changes in v10 - remove FIXED_POINT type - change code base on media: v4l2-ctrls: add support for fraction_bits - fix issue reported by kernel test robot - remove module_alias changes in v9: - add MEDIA_ENT_F_PROC_AUDIO_RESAMPLER. - add MEDIA_INTF_T_V4L_AUDIO - add media controller support - refine the vim2m-audio to support 8k<->16k conversion. changes in v8: - refine V4L2_CAP_AUDIO_M2M to be 0x00000008 - update doc for FIXED_POINT - address comments for imx-asrc changes in v7: - add acked-by from Mark - separate commit for fixed point, m2m audio class, audio rate controls - use INTEGER_MENU for rate, FIXED_POINT for rate offset - remove used fmts - address other comments for Hans changes in v6: - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two. - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT - fix warning by kernel test rebot - remove some unused format V4L2_AUDIO_FMT_XX - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver. - rename audm2m to viaudm2m. changes in v5: - remove V4L2_AUDIO_FMT_LPCM - define audio pixel format like V4L2_AUDIO_FMT_S8... - remove rate and format in struct v4l2_audio_format. - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls - updata document accordingly. changes in v4: - update document style - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit changes in v3: - Modify documents for adding audio m2m support - Add audio virtual m2m driver - Defined V4L2_AUDIO_FMT_LPCM format type for audio. - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case. - with modification in v4l-utils, pass v4l2-compliance test. changes in v2: - decouple the implementation in v4l2 and ALSA - implement the memory to memory driver as a platfrom driver and move it to driver/media - move fsl_asrc_common.h to include/sound folder Shengjiu Wang (14): ASoC: fsl_asrc: define functions for memory to memory usage ASoC: fsl_easrc: define functions for memory to memory usage ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound ASoC: fsl_asrc: register m2m platform device ASoC: fsl_easrc: register m2m platform device media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag media: v4l2: Add audio capture and output support media: uapi: Define audio sample format fourcc type media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO media: uapi: Add audio rate controls support media: uapi: Declare interface types for Audio media: uapi: Add an entity type for audio resampler media: imx-asrc: Add memory to memory driver media: vim2m-audio: add virtual driver for audio memory to memory .../media/mediactl/media-types.rst | 11 + .../userspace-api/media/v4l/buffer.rst | 6 + .../userspace-api/media/v4l/common.rst | 1 + .../media/v4l/dev-audio-mem2mem.rst | 71 + .../userspace-api/media/v4l/devices.rst | 1 + .../media/v4l/ext-ctrls-audio-m2m.rst | 41 + .../userspace-api/media/v4l/pixfmt-audio.rst | 87 ++ .../userspace-api/media/v4l/pixfmt.rst | 1 + .../media/v4l/vidioc-enum-fmt.rst | 2 + .../media/v4l/vidioc-g-ext-ctrls.rst | 4 + .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 + .../media/v4l/vidioc-querycap.rst | 3 + .../media/videodev2.h.rst.exceptions | 3 + .../media/common/videobuf2/videobuf2-v4l2.c | 4 + drivers/media/platform/nxp/Kconfig | 13 + drivers/media/platform/nxp/Makefile | 1 + drivers/media/platform/nxp/imx-asrc.c | 1264 +++++++++++++++++ drivers/media/test-drivers/Kconfig | 11 + drivers/media/test-drivers/Makefile | 1 + drivers/media/test-drivers/vim2m-audio.c | 799 +++++++++++ drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 9 + drivers/media/v4l2-core/v4l2-ctrls-defs.c | 10 + drivers/media/v4l2-core/v4l2-dev.c | 21 + drivers/media/v4l2-core/v4l2-ioctl.c | 66 + drivers/media/v4l2-core/v4l2-mem2mem.c | 13 +- include/media/v4l2-dev.h | 2 + include/media/v4l2-ioctl.h | 34 + .../fsl => include/sound}/fsl_asrc_common.h | 60 + include/uapi/linux/media.h | 2 + include/uapi/linux/v4l2-controls.h | 9 + include/uapi/linux/videodev2.h | 41 + sound/soc/fsl/fsl_asrc.c | 144 ++ sound/soc/fsl/fsl_asrc.h | 4 +- sound/soc/fsl/fsl_asrc_dma.c | 2 +- sound/soc/fsl/fsl_easrc.c | 233 +++ sound/soc/fsl/fsl_easrc.h | 6 +- 36 files changed, 2977 insertions(+), 7 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst create mode 100644 drivers/media/platform/nxp/imx-asrc.c create mode 100644 drivers/media/test-drivers/vim2m-audio.c rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%)
Comments
On 21/11/2023 08:37, Shengjiu Wang wrote: > Audio signal processing also has the requirement for memory to > memory similar as Video. > > This asrc memory to memory (memory ->asrc->memory) case is a non > real time use case. > > User fills the input buffer to the asrc module, after conversion, then asrc > sends back the output buffer to user. So it is not a traditional ALSA playback > and capture case. > > It is a specific use case, there is no reference in current kernel. > v4l2 memory to memory is the closed implementation, v4l2 current > support video, image, radio, tuner, touch devices, so it is not > complicated to add support for this specific audio case. > > Because we had implemented the "memory -> asrc ->i2s device-> codec" > use case in ALSA. Now the "memory->asrc->memory" needs > to reuse the code in asrc driver, so the first 3 patches is for refining > the code to make it can be shared by the "memory->asrc->memory" > driver. > > The main change is in the v4l2 side, A /dev/vl4-audioX will be created, > user applications only use the ioctl of v4l2 framework. > > Other change is to add memory to memory support for two kinds of i.MX ASRC > module. > > changes in v10 > - remove FIXED_POINT type > - change code base on media: v4l2-ctrls: add support for fraction_bits > - fix issue reported by kernel test robot > - remove module_alias Note that I still need a patch for vivid adding a fixed point test control. I think I want two controls: one INTEGER Q16 and one INTEGER64 Q63 (a nice corner case). Regards, Hans > > changes in v9: > - add MEDIA_ENT_F_PROC_AUDIO_RESAMPLER. > - add MEDIA_INTF_T_V4L_AUDIO > - add media controller support > - refine the vim2m-audio to support 8k<->16k conversion. > > changes in v8: > - refine V4L2_CAP_AUDIO_M2M to be 0x00000008 > - update doc for FIXED_POINT > - address comments for imx-asrc > > changes in v7: > - add acked-by from Mark > - separate commit for fixed point, m2m audio class, audio rate controls > - use INTEGER_MENU for rate, FIXED_POINT for rate offset > - remove used fmts > - address other comments for Hans > > changes in v6: > - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace > m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two. > - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT > - fix warning by kernel test rebot > - remove some unused format V4L2_AUDIO_FMT_XX > - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver. > - rename audm2m to viaudm2m. > > changes in v5: > - remove V4L2_AUDIO_FMT_LPCM > - define audio pixel format like V4L2_AUDIO_FMT_S8... > - remove rate and format in struct v4l2_audio_format. > - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls > - updata document accordingly. > > changes in v4: > - update document style > - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit > > changes in v3: > - Modify documents for adding audio m2m support > - Add audio virtual m2m driver > - Defined V4L2_AUDIO_FMT_LPCM format type for audio. > - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case. > - with modification in v4l-utils, pass v4l2-compliance test. > > changes in v2: > - decouple the implementation in v4l2 and ALSA > - implement the memory to memory driver as a platfrom driver > and move it to driver/media > - move fsl_asrc_common.h to include/sound folder > > Shengjiu Wang (14): > ASoC: fsl_asrc: define functions for memory to memory usage > ASoC: fsl_easrc: define functions for memory to memory usage > ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound > ASoC: fsl_asrc: register m2m platform device > ASoC: fsl_easrc: register m2m platform device > media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag > media: v4l2: Add audio capture and output support > media: uapi: Define audio sample format fourcc type > media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO > media: uapi: Add audio rate controls support > media: uapi: Declare interface types for Audio > media: uapi: Add an entity type for audio resampler > media: imx-asrc: Add memory to memory driver > media: vim2m-audio: add virtual driver for audio memory to memory > > .../media/mediactl/media-types.rst | 11 + > .../userspace-api/media/v4l/buffer.rst | 6 + > .../userspace-api/media/v4l/common.rst | 1 + > .../media/v4l/dev-audio-mem2mem.rst | 71 + > .../userspace-api/media/v4l/devices.rst | 1 + > .../media/v4l/ext-ctrls-audio-m2m.rst | 41 + > .../userspace-api/media/v4l/pixfmt-audio.rst | 87 ++ > .../userspace-api/media/v4l/pixfmt.rst | 1 + > .../media/v4l/vidioc-enum-fmt.rst | 2 + > .../media/v4l/vidioc-g-ext-ctrls.rst | 4 + > .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 + > .../media/v4l/vidioc-querycap.rst | 3 + > .../media/videodev2.h.rst.exceptions | 3 + > .../media/common/videobuf2/videobuf2-v4l2.c | 4 + > drivers/media/platform/nxp/Kconfig | 13 + > drivers/media/platform/nxp/Makefile | 1 + > drivers/media/platform/nxp/imx-asrc.c | 1264 +++++++++++++++++ > drivers/media/test-drivers/Kconfig | 11 + > drivers/media/test-drivers/Makefile | 1 + > drivers/media/test-drivers/vim2m-audio.c | 799 +++++++++++ > drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 9 + > drivers/media/v4l2-core/v4l2-ctrls-defs.c | 10 + > drivers/media/v4l2-core/v4l2-dev.c | 21 + > drivers/media/v4l2-core/v4l2-ioctl.c | 66 + > drivers/media/v4l2-core/v4l2-mem2mem.c | 13 +- > include/media/v4l2-dev.h | 2 + > include/media/v4l2-ioctl.h | 34 + > .../fsl => include/sound}/fsl_asrc_common.h | 60 + > include/uapi/linux/media.h | 2 + > include/uapi/linux/v4l2-controls.h | 9 + > include/uapi/linux/videodev2.h | 41 + > sound/soc/fsl/fsl_asrc.c | 144 ++ > sound/soc/fsl/fsl_asrc.h | 4 +- > sound/soc/fsl/fsl_asrc_dma.c | 2 +- > sound/soc/fsl/fsl_easrc.c | 233 +++ > sound/soc/fsl/fsl_easrc.h | 6 +- > 36 files changed, 2977 insertions(+), 7 deletions(-) > create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst > create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst > create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst > create mode 100644 drivers/media/platform/nxp/imx-asrc.c > create mode 100644 drivers/media/test-drivers/vim2m-audio.c > rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%) >
On Tue, Nov 21, 2023 at 5:22 PM Hans Verkuil <hverkuil@xs4all.nl> wrote: > > On 21/11/2023 08:37, Shengjiu Wang wrote: > > Audio signal processing also has the requirement for memory to > > memory similar as Video. > > > > This asrc memory to memory (memory ->asrc->memory) case is a non > > real time use case. > > > > User fills the input buffer to the asrc module, after conversion, then asrc > > sends back the output buffer to user. So it is not a traditional ALSA playback > > and capture case. > > > > It is a specific use case, there is no reference in current kernel. > > v4l2 memory to memory is the closed implementation, v4l2 current > > support video, image, radio, tuner, touch devices, so it is not > > complicated to add support for this specific audio case. > > > > Because we had implemented the "memory -> asrc ->i2s device-> codec" > > use case in ALSA. Now the "memory->asrc->memory" needs > > to reuse the code in asrc driver, so the first 3 patches is for refining > > the code to make it can be shared by the "memory->asrc->memory" > > driver. > > > > The main change is in the v4l2 side, A /dev/vl4-audioX will be created, > > user applications only use the ioctl of v4l2 framework. > > > > Other change is to add memory to memory support for two kinds of i.MX ASRC > > module. > > > > changes in v10 > > - remove FIXED_POINT type > > - change code base on media: v4l2-ctrls: add support for fraction_bits > > - fix issue reported by kernel test robot > > - remove module_alias > > Note that I still need a patch for vivid adding a fixed point test control. > > I think I want two controls: one INTEGER Q16 and one INTEGER64 Q63 (a nice > corner case). > Not sure if we can do like this: diff --git a/drivers/media/test-drivers/vivid/vivid-core.h b/drivers/media/test-drivers/vivid/vivid-core.h index cfb8e66083f6..def8cf6c30c1 100644 --- a/drivers/media/test-drivers/vivid/vivid-core.h +++ b/drivers/media/test-drivers/vivid/vivid-core.h @@ -222,6 +222,8 @@ struct vivid_dev { struct v4l2_ctrl *boolean; struct v4l2_ctrl *int32; struct v4l2_ctrl *int64; + struct v4l2_ctrl *int32Q16; + struct v4l2_ctrl *int64Q63; struct v4l2_ctrl *menu; struct v4l2_ctrl *string; struct v4l2_ctrl *bitmask; diff --git a/drivers/media/test-drivers/vivid/vivid-ctrls.c b/drivers/media/test-drivers/vivid/vivid-ctrls.c index f2b20e25a7a4..c912b6776775 100644 --- a/drivers/media/test-drivers/vivid/vivid-ctrls.c +++ b/drivers/media/test-drivers/vivid/vivid-ctrls.c @@ -182,6 +182,28 @@ static const struct v4l2_ctrl_config vivid_ctrl_int64 = { .step = 1, }; +static const struct v4l2_ctrl_config vivid_ctrl_int32Q16 = { + .ops = &vivid_user_gen_ctrl_ops, + .id = VIVID_CID_INTEGER, + .name = "Integer 32 Bits Q16", + .type = V4L2_CTRL_TYPE_INTEGER, + .min = 0xffffffff80000000ULL, + .max = 0x7fffffff, + .step = 1, + .fraction_bits = 16, +}; + +static const struct v4l2_ctrl_config vivid_ctrl_int64Q63 = { + .ops = &vivid_user_gen_ctrl_ops, + .id = VIVID_CID_INTEGER64, + .name = "Integer 64 Bits Q63", + .type = V4L2_CTRL_TYPE_INTEGER64, + .min = 0x8000000000000000ULL, + .max = 0x7fffffffffffffffLL, + .step = 1, + .fraction_bits = 63, +}; + static const struct v4l2_ctrl_config vivid_ctrl_u32_array = { .ops = &vivid_user_gen_ctrl_ops, .id = VIVID_CID_U32_ARRAY, @@ -1670,6 +1692,8 @@ int vivid_create_controls(struct vivid_dev *dev, bool show_ccs_cap, dev->button = v4l2_ctrl_new_custom(hdl_user_gen, &vivid_ctrl_button, NULL); dev->int32 = v4l2_ctrl_new_custom(hdl_user_gen, &vivid_ctrl_int32, NULL); dev->int64 = v4l2_ctrl_new_custom(hdl_user_gen, &vivid_ctrl_int64, NULL); + dev->int32Q16 = v4l2_ctrl_new_custom(hdl_user_gen, &vivid_ctrl_int32Q16, NULL); + dev->int64Q63 = v4l2_ctrl_new_custom(hdl_user_gen, &vivid_ctrl_int64Q63, NULL); dev->boolean = v4l2_ctrl_new_custom(hdl_user_gen, &vivid_ctrl_boolean, NULL); dev->menu = v4l2_ctrl_new_custom(hdl_user_gen, &vivid_ctrl_menu, NULL); dev->string = v4l2_ctrl_new_custom(hdl_user_gen, &vivid_ctrl_string, NULL); (END) > Regards, > > Hans > > > > > changes in v9: > > - add MEDIA_ENT_F_PROC_AUDIO_RESAMPLER. > > - add MEDIA_INTF_T_V4L_AUDIO > > - add media controller support > > - refine the vim2m-audio to support 8k<->16k conversion. > > > > changes in v8: > > - refine V4L2_CAP_AUDIO_M2M to be 0x00000008 > > - update doc for FIXED_POINT > > - address comments for imx-asrc > > > > changes in v7: > > - add acked-by from Mark > > - separate commit for fixed point, m2m audio class, audio rate controls > > - use INTEGER_MENU for rate, FIXED_POINT for rate offset > > - remove used fmts > > - address other comments for Hans > > > > changes in v6: > > - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace > > m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two. > > - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT > > - fix warning by kernel test rebot > > - remove some unused format V4L2_AUDIO_FMT_XX > > - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver. > > - rename audm2m to viaudm2m. > > > > changes in v5: > > - remove V4L2_AUDIO_FMT_LPCM > > - define audio pixel format like V4L2_AUDIO_FMT_S8... > > - remove rate and format in struct v4l2_audio_format. > > - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls > > - updata document accordingly. > > > > changes in v4: > > - update document style > > - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit > > > > changes in v3: > > - Modify documents for adding audio m2m support > > - Add audio virtual m2m driver > > - Defined V4L2_AUDIO_FMT_LPCM format type for audio. > > - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case. > > - with modification in v4l-utils, pass v4l2-compliance test. > > > > changes in v2: > > - decouple the implementation in v4l2 and ALSA > > - implement the memory to memory driver as a platfrom driver > > and move it to driver/media > > - move fsl_asrc_common.h to include/sound folder > > > > Shengjiu Wang (14): > > ASoC: fsl_asrc: define functions for memory to memory usage > > ASoC: fsl_easrc: define functions for memory to memory usage > > ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound > > ASoC: fsl_asrc: register m2m platform device > > ASoC: fsl_easrc: register m2m platform device > > media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag > > media: v4l2: Add audio capture and output support > > media: uapi: Define audio sample format fourcc type > > media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO > > media: uapi: Add audio rate controls support > > media: uapi: Declare interface types for Audio > > media: uapi: Add an entity type for audio resampler > > media: imx-asrc: Add memory to memory driver > > media: vim2m-audio: add virtual driver for audio memory to memory > > > > .../media/mediactl/media-types.rst | 11 + > > .../userspace-api/media/v4l/buffer.rst | 6 + > > .../userspace-api/media/v4l/common.rst | 1 + > > .../media/v4l/dev-audio-mem2mem.rst | 71 + > > .../userspace-api/media/v4l/devices.rst | 1 + > > .../media/v4l/ext-ctrls-audio-m2m.rst | 41 + > > .../userspace-api/media/v4l/pixfmt-audio.rst | 87 ++ > > .../userspace-api/media/v4l/pixfmt.rst | 1 + > > .../media/v4l/vidioc-enum-fmt.rst | 2 + > > .../media/v4l/vidioc-g-ext-ctrls.rst | 4 + > > .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 + > > .../media/v4l/vidioc-querycap.rst | 3 + > > .../media/videodev2.h.rst.exceptions | 3 + > > .../media/common/videobuf2/videobuf2-v4l2.c | 4 + > > drivers/media/platform/nxp/Kconfig | 13 + > > drivers/media/platform/nxp/Makefile | 1 + > > drivers/media/platform/nxp/imx-asrc.c | 1264 +++++++++++++++++ > > drivers/media/test-drivers/Kconfig | 11 + > > drivers/media/test-drivers/Makefile | 1 + > > drivers/media/test-drivers/vim2m-audio.c | 799 +++++++++++ > > drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 9 + > > drivers/media/v4l2-core/v4l2-ctrls-defs.c | 10 + > > drivers/media/v4l2-core/v4l2-dev.c | 21 + > > drivers/media/v4l2-core/v4l2-ioctl.c | 66 + > > drivers/media/v4l2-core/v4l2-mem2mem.c | 13 +- > > include/media/v4l2-dev.h | 2 + > > include/media/v4l2-ioctl.h | 34 + > > .../fsl => include/sound}/fsl_asrc_common.h | 60 + > > include/uapi/linux/media.h | 2 + > > include/uapi/linux/v4l2-controls.h | 9 + > > include/uapi/linux/videodev2.h | 41 + > > sound/soc/fsl/fsl_asrc.c | 144 ++ > > sound/soc/fsl/fsl_asrc.h | 4 +- > > sound/soc/fsl/fsl_asrc_dma.c | 2 +- > > sound/soc/fsl/fsl_easrc.c | 233 +++ > > sound/soc/fsl/fsl_easrc.h | 6 +- > > 36 files changed, 2977 insertions(+), 7 deletions(-) > > create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst > > create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst > > create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst > > create mode 100644 drivers/media/platform/nxp/imx-asrc.c > > create mode 100644 drivers/media/test-drivers/vim2m-audio.c > > rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%) > > >
On 11/21/23 11:03, Shengjiu Wang wrote: > On Tue, Nov 21, 2023 at 5:22 PM Hans Verkuil <hverkuil@xs4all.nl> wrote: >> >> On 21/11/2023 08:37, Shengjiu Wang wrote: >>> Audio signal processing also has the requirement for memory to >>> memory similar as Video. >>> >>> This asrc memory to memory (memory ->asrc->memory) case is a non >>> real time use case. >>> >>> User fills the input buffer to the asrc module, after conversion, then asrc >>> sends back the output buffer to user. So it is not a traditional ALSA playback >>> and capture case. >>> >>> It is a specific use case, there is no reference in current kernel. >>> v4l2 memory to memory is the closed implementation, v4l2 current >>> support video, image, radio, tuner, touch devices, so it is not >>> complicated to add support for this specific audio case. >>> >>> Because we had implemented the "memory -> asrc ->i2s device-> codec" >>> use case in ALSA. Now the "memory->asrc->memory" needs >>> to reuse the code in asrc driver, so the first 3 patches is for refining >>> the code to make it can be shared by the "memory->asrc->memory" >>> driver. >>> >>> The main change is in the v4l2 side, A /dev/vl4-audioX will be created, >>> user applications only use the ioctl of v4l2 framework. >>> >>> Other change is to add memory to memory support for two kinds of i.MX ASRC >>> module. >>> >>> changes in v10 >>> - remove FIXED_POINT type >>> - change code base on media: v4l2-ctrls: add support for fraction_bits >>> - fix issue reported by kernel test robot >>> - remove module_alias >> >> Note that I still need a patch for vivid adding a fixed point test control. >> >> I think I want two controls: one INTEGER Q16 and one INTEGER64 Q63 (a nice >> corner case). >> > > Not sure if we can do like this: > > diff --git a/drivers/media/test-drivers/vivid/vivid-core.h > b/drivers/media/test-drivers/vivid/vivid-core.h > index cfb8e66083f6..def8cf6c30c1 100644 > --- a/drivers/media/test-drivers/vivid/vivid-core.h > +++ b/drivers/media/test-drivers/vivid/vivid-core.h > @@ -222,6 +222,8 @@ struct vivid_dev { > struct v4l2_ctrl *boolean; > struct v4l2_ctrl *int32; > struct v4l2_ctrl *int64; > + struct v4l2_ctrl *int32Q16; I would call this int32_q16. This to avoid checkpatch CamelCase warnings. > + struct v4l2_ctrl *int64Q63; ditto > struct v4l2_ctrl *menu; > struct v4l2_ctrl *string; > struct v4l2_ctrl *bitmask; > diff --git a/drivers/media/test-drivers/vivid/vivid-ctrls.c > b/drivers/media/test-drivers/vivid/vivid-ctrls.c > index f2b20e25a7a4..c912b6776775 100644 > --- a/drivers/media/test-drivers/vivid/vivid-ctrls.c > +++ b/drivers/media/test-drivers/vivid/vivid-ctrls.c > @@ -182,6 +182,28 @@ static const struct v4l2_ctrl_config vivid_ctrl_int64 = { > .step = 1, > }; > > +static const struct v4l2_ctrl_config vivid_ctrl_int32Q16 = { > + .ops = &vivid_user_gen_ctrl_ops, > + .id = VIVID_CID_INTEGER, You need to add new CIDs. E.g. VIVID_CID_INT_Q4_16. > + .name = "Integer 32 Bits Q16", Q4.16 > + .type = V4L2_CTRL_TYPE_INTEGER, > + .min = 0xffffffff80000000ULL, Let's make this a Q4.16 integer. I think it is worth adding this to the v4l2-ctrls.h header: #define v4l2_ctrl_fp_compose(i, f, fraction_bits) \ (((i) << fraction_bits) + (f)) Then you can do: .min = v4l2_ctrl_fp_compose(-16, 0), .max = v4l2_ctrl_fp_compose(15, 0xffff), > + .max = 0x7fffffff, > + .step = 1, > + .fraction_bits = 16, > +}; > + > +static const struct v4l2_ctrl_config vivid_ctrl_int64Q63 = { > + .ops = &vivid_user_gen_ctrl_ops, > + .id = VIVID_CID_INTEGER64, > + .name = "Integer 64 Bits Q63", > + .type = V4L2_CTRL_TYPE_INTEGER64, > + .min = 0x8000000000000000ULL, > + .max = 0x7fffffffffffffffLL, .min = v4l2_ctrl_fp_compose(-1, 0), .max = v4l2_ctrl_fp_compose(0, LLONG_MAX), > + .step = 1, > + .fraction_bits = 63, > +}; Looks good otherwise. The purpose is that v4l2-compliance can test such control types using a hw emulation driver like vivid. Regards, Hans > + > static const struct v4l2_ctrl_config vivid_ctrl_u32_array = { > .ops = &vivid_user_gen_ctrl_ops, > .id = VIVID_CID_U32_ARRAY, > @@ -1670,6 +1692,8 @@ int vivid_create_controls(struct vivid_dev *dev, > bool show_ccs_cap, > dev->button = v4l2_ctrl_new_custom(hdl_user_gen, > &vivid_ctrl_button, NULL); > dev->int32 = v4l2_ctrl_new_custom(hdl_user_gen, > &vivid_ctrl_int32, NULL); > dev->int64 = v4l2_ctrl_new_custom(hdl_user_gen, > &vivid_ctrl_int64, NULL); > + dev->int32Q16 = v4l2_ctrl_new_custom(hdl_user_gen, > &vivid_ctrl_int32Q16, NULL); > + dev->int64Q63 = v4l2_ctrl_new_custom(hdl_user_gen, > &vivid_ctrl_int64Q63, NULL); > dev->boolean = v4l2_ctrl_new_custom(hdl_user_gen, > &vivid_ctrl_boolean, NULL); > dev->menu = v4l2_ctrl_new_custom(hdl_user_gen, &vivid_ctrl_menu, NULL); > dev->string = v4l2_ctrl_new_custom(hdl_user_gen, > &vivid_ctrl_string, NULL); > (END) > > >> Regards, >> >> Hans >> >>> >>> changes in v9: >>> - add MEDIA_ENT_F_PROC_AUDIO_RESAMPLER. >>> - add MEDIA_INTF_T_V4L_AUDIO >>> - add media controller support >>> - refine the vim2m-audio to support 8k<->16k conversion. >>> >>> changes in v8: >>> - refine V4L2_CAP_AUDIO_M2M to be 0x00000008 >>> - update doc for FIXED_POINT >>> - address comments for imx-asrc >>> >>> changes in v7: >>> - add acked-by from Mark >>> - separate commit for fixed point, m2m audio class, audio rate controls >>> - use INTEGER_MENU for rate, FIXED_POINT for rate offset >>> - remove used fmts >>> - address other comments for Hans >>> >>> changes in v6: >>> - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace >>> m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two. >>> - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT >>> - fix warning by kernel test rebot >>> - remove some unused format V4L2_AUDIO_FMT_XX >>> - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver. >>> - rename audm2m to viaudm2m. >>> >>> changes in v5: >>> - remove V4L2_AUDIO_FMT_LPCM >>> - define audio pixel format like V4L2_AUDIO_FMT_S8... >>> - remove rate and format in struct v4l2_audio_format. >>> - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls >>> - updata document accordingly. >>> >>> changes in v4: >>> - update document style >>> - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit >>> >>> changes in v3: >>> - Modify documents for adding audio m2m support >>> - Add audio virtual m2m driver >>> - Defined V4L2_AUDIO_FMT_LPCM format type for audio. >>> - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case. >>> - with modification in v4l-utils, pass v4l2-compliance test. >>> >>> changes in v2: >>> - decouple the implementation in v4l2 and ALSA >>> - implement the memory to memory driver as a platfrom driver >>> and move it to driver/media >>> - move fsl_asrc_common.h to include/sound folder >>> >>> Shengjiu Wang (14): >>> ASoC: fsl_asrc: define functions for memory to memory usage >>> ASoC: fsl_easrc: define functions for memory to memory usage >>> ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound >>> ASoC: fsl_asrc: register m2m platform device >>> ASoC: fsl_easrc: register m2m platform device >>> media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag >>> media: v4l2: Add audio capture and output support >>> media: uapi: Define audio sample format fourcc type >>> media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO >>> media: uapi: Add audio rate controls support >>> media: uapi: Declare interface types for Audio >>> media: uapi: Add an entity type for audio resampler >>> media: imx-asrc: Add memory to memory driver >>> media: vim2m-audio: add virtual driver for audio memory to memory >>> >>> .../media/mediactl/media-types.rst | 11 + >>> .../userspace-api/media/v4l/buffer.rst | 6 + >>> .../userspace-api/media/v4l/common.rst | 1 + >>> .../media/v4l/dev-audio-mem2mem.rst | 71 + >>> .../userspace-api/media/v4l/devices.rst | 1 + >>> .../media/v4l/ext-ctrls-audio-m2m.rst | 41 + >>> .../userspace-api/media/v4l/pixfmt-audio.rst | 87 ++ >>> .../userspace-api/media/v4l/pixfmt.rst | 1 + >>> .../media/v4l/vidioc-enum-fmt.rst | 2 + >>> .../media/v4l/vidioc-g-ext-ctrls.rst | 4 + >>> .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 + >>> .../media/v4l/vidioc-querycap.rst | 3 + >>> .../media/videodev2.h.rst.exceptions | 3 + >>> .../media/common/videobuf2/videobuf2-v4l2.c | 4 + >>> drivers/media/platform/nxp/Kconfig | 13 + >>> drivers/media/platform/nxp/Makefile | 1 + >>> drivers/media/platform/nxp/imx-asrc.c | 1264 +++++++++++++++++ >>> drivers/media/test-drivers/Kconfig | 11 + >>> drivers/media/test-drivers/Makefile | 1 + >>> drivers/media/test-drivers/vim2m-audio.c | 799 +++++++++++ >>> drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 9 + >>> drivers/media/v4l2-core/v4l2-ctrls-defs.c | 10 + >>> drivers/media/v4l2-core/v4l2-dev.c | 21 + >>> drivers/media/v4l2-core/v4l2-ioctl.c | 66 + >>> drivers/media/v4l2-core/v4l2-mem2mem.c | 13 +- >>> include/media/v4l2-dev.h | 2 + >>> include/media/v4l2-ioctl.h | 34 + >>> .../fsl => include/sound}/fsl_asrc_common.h | 60 + >>> include/uapi/linux/media.h | 2 + >>> include/uapi/linux/v4l2-controls.h | 9 + >>> include/uapi/linux/videodev2.h | 41 + >>> sound/soc/fsl/fsl_asrc.c | 144 ++ >>> sound/soc/fsl/fsl_asrc.h | 4 +- >>> sound/soc/fsl/fsl_asrc_dma.c | 2 +- >>> sound/soc/fsl/fsl_easrc.c | 233 +++ >>> sound/soc/fsl/fsl_easrc.h | 6 +- >>> 36 files changed, 2977 insertions(+), 7 deletions(-) >>> create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst >>> create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst >>> create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst >>> create mode 100644 drivers/media/platform/nxp/imx-asrc.c >>> create mode 100644 drivers/media/test-drivers/vim2m-audio.c >>> rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%) >>> >>