From patchwork Thu Sep 28 09:00:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shengjiu Wang X-Patchwork-Id: 14642 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp3191018vqu; Thu, 28 Sep 2023 02:53:23 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE1WBhp1Ju0kujdcP9kfS7oZE/NK7Aql0TdcvvSzaNP1N2r1Z0YsJS2bEMhKk+DcUquUtm3 X-Received: by 2002:a81:728a:0:b0:570:89bf:6f10 with SMTP id n132-20020a81728a000000b0057089bf6f10mr592785ywc.25.1695894803004; Thu, 28 Sep 2023 02:53:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695894802; cv=none; d=google.com; s=arc-20160816; b=obolTS5QpeWsTaCF+ldkv33NnDdkkc+l9LjbI+xmBWqFbVNM9GkViSlCQAg1sNKwKk 8hhtQ7N9uOcMuCPD/3evB1CeZsrJ6idxXhqUTS1cQK6ZDNp+wWEh75pkOTfhZ3iaO37J Zt1J/fWmD7t+fzu3e6gJJz+wk/LEOehV+VWSyikv+J/6NtJrY/hX8hK7DXyU4Jwvr3Ij K6cXwENeFn462mWPfJoPREF54FynI9QOeGIT/2/JWRs4NGd7g7HFPkMSYX8AtKgJbrte Y5V2JlDr+m2kEwUoyLfDFSF1EhxUTYp4hOLQgk3GSWP6+VIgcBKG9SwaBFwzzRFUnlaf MwXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:message-id:date:subject:to:from; bh=ELZ56O6p8Tv6vkngoP4O0TXcKlHITTc9YGke8YArQX8=; fh=ufiESpW7Hvn5oNmN0l6/qCUiQrKfLLrOwb5uR1Dftds=; b=GldP2M/hWcvVMA/xgYnQKRNpYndGlv3si3LEcWfhKsP0GtlK92nNr5TQ4v+q36z0Ox GvJTGNlDfqeyX9dt7eWFsnEErLng70WdAk7ktxP/BQnNgKLzoc2tMNyLPZ7dm8jkLfFY eFPd5zScLPUuzPbAOqR6kWGQ6e5gGbYHrElFy5w01erF39eabFCxO7y/Fy5StyaXOVNA toj+1T44GvSrCkeVhbkCV8b1cHcW1vDtlZEdKdXyEnpAda39+NUHDSX2gDCzPOg9xmPK rz1CoR6tDSWne7QS5Nk7m3w4yu7K6yXwCdtLA6ohq5VrQG4j2/QeRgyTUBIQQTnEweZ5 2dTw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id j2-20020a63e742000000b00578b8bd30b0si17526527pgk.205.2023.09.28.02.53.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Sep 2023 02:53:22 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id E6EF78313015; Thu, 28 Sep 2023 02:37:57 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231691AbjI1Jhn (ORCPT + 21 others); Thu, 28 Sep 2023 05:37:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51170 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231571AbjI1Jhm (ORCPT ); Thu, 28 Sep 2023 05:37:42 -0400 Received: from inva021.nxp.com (inva021.nxp.com [92.121.34.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 78FD795; Thu, 28 Sep 2023 02:37:38 -0700 (PDT) Received: from inva021.nxp.com (localhost [127.0.0.1]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id 9179B20160F; Thu, 28 Sep 2023 11:37:36 +0200 (CEST) Received: from aprdc01srsp001v.ap-rdc01.nxp.com (aprdc01srsp001v.ap-rdc01.nxp.com [165.114.16.16]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id 283D8200221; Thu, 28 Sep 2023 11:37:36 +0200 (CEST) Received: from localhost.localdomain (shlinux2.ap.freescale.net [10.192.224.44]) by aprdc01srsp001v.ap-rdc01.nxp.com (Postfix) with ESMTP id 3D4A7181D0C0; Thu, 28 Sep 2023 17:37:34 +0800 (+08) From: Shengjiu Wang To: hverkuil@xs4all.nl, sakari.ailus@iki.fi, tfiga@chromium.org, m.szyprowski@samsung.com, mchehab@kernel.org, linux-media@vger.kernel.org, linux-kernel@vger.kernel.org, shengjiu.wang@gmail.com, Xiubo.Lee@gmail.com, festevam@gmail.com, nicoleotsuka@gmail.com, lgirdwood@gmail.com, broonie@kernel.org, perex@perex.cz, tiwai@suse.com, alsa-devel@alsa-project.org, linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH v5 00/11] Add audio support in v4l2 framework Date: Thu, 28 Sep 2023 17:00:08 +0800 Message-Id: <1695891619-32393-1-git-send-email-shengjiu.wang@nxp.com> X-Mailer: git-send-email 2.7.4 X-Virus-Scanned: ClamAV using ClamSMTP X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Thu, 28 Sep 2023 02:37:58 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778274589189284459 X-GMAIL-MSGID: 1778274589189284459 Audio signal processing also has the requirement for memory to memory similar as Video. This asrc memory to memory (memory ->asrc->memory) case is a non real time use case. User fills the input buffer to the asrc module, after conversion, then asrc sends back the output buffer to user. So it is not a traditional ALSA playback and capture case. It is a specific use case, there is no reference in current kernel. v4l2 memory to memory is the closed implementation, v4l2 current support video, image, radio, tuner, touch devices, so it is not complicated to add support for this specific audio case. Because we had implemented the "memory -> asrc ->i2s device-> codec" use case in ALSA. Now the "memory->asrc->memory" needs to reuse the code in asrc driver, so the first 3 patches is for refining the code to make it can be shared by the "memory->asrc->memory" driver. The main change is in the v4l2 side, A /dev/vl4-audioX will be created, user applications only use the ioctl of v4l2 framework. Other change is to add memory to memory support for two kinds of i.MX ASRC module. changes in v5: - remove V4L2_AUDIO_FMT_LPCM - define audio pixel format like V4L2_AUDIO_FMT_S8... - remove rate and format in struct v4l2_audio_format. - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls - updata document accordingly. changes in v4: - update document style - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit changes in v3: - Modify documents for adding audio m2m support - Add audio virtual m2m driver - Defined V4L2_AUDIO_FMT_LPCM format type for audio. - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case. - with modification in v4l-utils, pass v4l2-compliance test. changes in v2: - decouple the implementation in v4l2 and ALSA - implement the memory to memory driver as a platfrom driver and move it to driver/media - move fsl_asrc_common.h to include/sound folder Shengjiu Wang (11): ASoC: fsl_asrc: define functions for memory to memory usage ASoC: fsl_easrc: define functions for memory to memory usage ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound ASoC: fsl_asrc: register m2m platform device ASoC: fsl_easrc: register m2m platform device media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag media: v4l2: Add audio capture and output support media: uapi: Add audio rate controls support media: uapi: define audio sample format fourcc type media: imx-asrc: Add memory to memory driver media: audm2m: add virtual driver for audio memory to memory .../userspace-api/media/v4l/buffer.rst | 6 + .../userspace-api/media/v4l/common.rst | 1 + .../media/v4l/dev-audio-mem2mem.rst | 71 + .../userspace-api/media/v4l/devices.rst | 1 + .../media/v4l/ext-ctrls-asrc-rate.rst | 36 + .../userspace-api/media/v4l/pixfmt-audio.rst | 277 ++++ .../userspace-api/media/v4l/pixfmt.rst | 1 + .../media/v4l/vidioc-enum-fmt.rst | 2 + .../media/v4l/vidioc-g-ext-ctrls.rst | 4 + .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 + .../media/v4l/vidioc-querycap.rst | 3 + .../media/v4l/vidioc-queryctrl.rst | 7 + .../media/videodev2.h.rst.exceptions | 4 + .../media/common/videobuf2/videobuf2-v4l2.c | 4 + drivers/media/platform/nxp/Kconfig | 12 + drivers/media/platform/nxp/Makefile | 1 + drivers/media/platform/nxp/imx-asrc.c | 1251 +++++++++++++++++ drivers/media/test-drivers/Kconfig | 9 + drivers/media/test-drivers/Makefile | 1 + drivers/media/test-drivers/audm2m.c | 808 +++++++++++ drivers/media/v4l2-core/v4l2-ctrls-core.c | 5 + drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 + drivers/media/v4l2-core/v4l2-dev.c | 17 + drivers/media/v4l2-core/v4l2-ioctl.c | 104 ++ include/media/v4l2-ctrls.h | 1 + include/media/v4l2-dev.h | 2 + include/media/v4l2-ioctl.h | 34 + .../fsl => include/sound}/fsl_asrc_common.h | 58 + include/uapi/linux/v4l2-controls.h | 13 + include/uapi/linux/videodev2.h | 77 + sound/soc/fsl/fsl_asrc.c | 136 ++ sound/soc/fsl/fsl_asrc.h | 4 +- sound/soc/fsl/fsl_asrc_dma.c | 2 +- sound/soc/fsl/fsl_easrc.c | 214 +++ sound/soc/fsl/fsl_easrc.h | 8 +- 35 files changed, 3179 insertions(+), 3 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-asrc-rate.rst create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst create mode 100644 drivers/media/platform/nxp/imx-asrc.c create mode 100644 drivers/media/test-drivers/audm2m.c rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%)