From patchwork Mon Feb 26 08:28:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shengjiu Wang X-Patchwork-Id: 20953 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp1956554dyb; Mon, 26 Feb 2024 01:21:28 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVAT/RXDO0fqz9C+M4FUQGufJ3JRVfqMPLfOXXJnqfqMLw48mBiuLUj0nXmqwr5yia6QG2TYiUjS2yVxCARuz2DNeR6ew== X-Google-Smtp-Source: AGHT+IHOq3pDRpD2D0cKZOpaV1mTAFjAr47f73XJ2v+XHmbsxGsn7yPi9v8TvZt5bsKnj/T+yo8g X-Received: by 2002:a05:6a00:4fc5:b0:6e5:c31:88d8 with SMTP id le5-20020a056a004fc500b006e50c3188d8mr4506502pfb.15.1708939288436; Mon, 26 Feb 2024 01:21:28 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708939288; cv=pass; d=google.com; s=arc-20160816; b=l0dPvoCOpd2GPW4h5pfanZJEkVZZ9+0vXb8ZhDOyA+boIXqyHXhKauI4flkpEgLfpV nvmH+7LcM/Ldl8xa/bO/y++syQyeX58pa9GDPNZ/1RtOTkfsKtT7tsoi3pn/Ds7U1C/1 gTnKEKsP6sQGXvt9P/luSaRy8upNNvB5iELF2JONoGCKuz2gWMCOB9NOdQNLcQ/uOsik 2+crETf4ns2PP9YRi4ckmOuf83QGowonVtzyJJr17nu9UU3dc72lEH87Njs/HIvtEAJF Lv6kNQtZ72zaH8vOIXutkbKTC1l+9j7zhIAlllmgryXSAlpUJILRl9SKMIb47h9h58g8 nSfA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-id:precedence:message-id:date :subject:to:from; bh=2IIqjv9cUoJxWLa4escxG04ptylI8fyEwe+MBKxPg9Y=; fh=ufiESpW7Hvn5oNmN0l6/qCUiQrKfLLrOwb5uR1Dftds=; b=yWLT6G0egQhc/345sVBNksSvzquGAF6VEmM2m+qbjplVauDrtfmoKpV0L7NdiyR5Yx +0S7aeK7Sc3vK/W6Vw/xaieeRlhLArASEGV7Twxf7m8UVwpfOUdwj6ECNDHhTjJYsNlZ Tc1Ng469HRRIh3MKtb5EnPt1lTAuBTpCa0Ne3gypy3OcnjK87i88tJS2Lzwi6bXaIIfa hZpIlQOPx6raQ/iAQ4i3U8q2WOFj8XpB6BBX7kPkq0aVdJM+nUffPAd5ZegRy7gFSm/l 3xSKLXsXFXG4jUnGbu1Jmy2Fw+d6qm9g4rCFmEH1weMKj0G2Ed7ViAyGJPHjApY/oyY4 BA8w==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=nxp.com dmarc=pass fromdomain=nxp.com); spf=pass (google.com: domain of linux-kernel+bounces-80926-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80926-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id le6-20020a056a004fc600b006e531ba990bsi1064998pfb.326.2024.02.26.01.21.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 01:21:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-80926-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=nxp.com dmarc=pass fromdomain=nxp.com); spf=pass (google.com: domain of linux-kernel+bounces-80926-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80926-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 6B37128622B for ; Mon, 26 Feb 2024 09:20:45 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id F0E1854900; Mon, 26 Feb 2024 08:44:19 +0000 (UTC) Received: from inva021.nxp.com (inva021.nxp.com [92.121.34.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8DA001D69E; Mon, 26 Feb 2024 08:44:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=92.121.34.21 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708937057; cv=none; b=TYvsApdvgw35fkxakmebXVDCWYXNOg6MLRE4bWz44FVIgtwHZ2BW6vGkixAbD0hmbHSjYTdGU5gbvA6e90FnW63+upeSyQ8oQn9gHYErU4phCavwIni1xdpwNNZ5nmrQoIy/XRcJPt+zi4TUEYa98xDUXjNb7IJZuVHTa1wq46c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708937057; c=relaxed/simple; bh=uscKEYignr3+K4U+SNizooeP+KV5pNE3Uw2IIlWJNN8=; h=From:To:Subject:Date:Message-Id; b=fqP9fcJXkTh+jp3D1Dck3rbP80kMl9MZW8Js4gqHN/JhB6iB5mj7TiQc6aze0zfDT6nJOQEDHHyU/jqUje6UXVMPHjgckojBSevoZqg8vLDWPuu8LFmpdC0qeiBFbmqAN9UUDoZ3atMphAffTrm0gSfEYOOoEj7X2g15O6yafyw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nxp.com; spf=pass smtp.mailfrom=nxp.com; arc=none smtp.client-ip=92.121.34.21 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nxp.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=nxp.com Received: from inva021.nxp.com (localhost [127.0.0.1]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id 957EE20061B; Mon, 26 Feb 2024 09:44:08 +0100 (CET) Received: from aprdc01srsp001v.ap-rdc01.nxp.com (aprdc01srsp001v.ap-rdc01.nxp.com [165.114.16.16]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id 2C1B02010FC; Mon, 26 Feb 2024 09:44:08 +0100 (CET) Received: from localhost.localdomain (shlinux2.ap.freescale.net [10.192.224.44]) by aprdc01srsp001v.ap-rdc01.nxp.com (Postfix) with ESMTP id 784F7180031D; Mon, 26 Feb 2024 16:44:06 +0800 (+08) From: Shengjiu Wang To: hverkuil@xs4all.nl, sakari.ailus@iki.fi, tfiga@chromium.org, m.szyprowski@samsung.com, mchehab@kernel.org, linux-media@vger.kernel.org, linux-kernel@vger.kernel.org, shengjiu.wang@gmail.com, Xiubo.Lee@gmail.com, festevam@gmail.com, nicoleotsuka@gmail.com, lgirdwood@gmail.com, broonie@kernel.org, perex@perex.cz, tiwai@suse.com, alsa-devel@alsa-project.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v13 00/16] Add audio support in v4l2 framework Date: Mon, 26 Feb 2024 16:28:13 +0800 Message-Id: <1708936109-11587-1-git-send-email-shengjiu.wang@nxp.com> X-Mailer: git-send-email 2.7.4 X-Virus-Scanned: ClamAV using ClamSMTP Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791952723177426140 X-GMAIL-MSGID: 1791952723177426140 Audio signal processing also has the requirement for memory to memory similar as Video. This asrc memory to memory (memory ->asrc->memory) case is a non real time use case. User fills the input buffer to the asrc module, after conversion, then asrc sends back the output buffer to user. So it is not a traditional ALSA playback and capture case. It is a specific use case, there is no reference in current kernel. v4l2 memory to memory is the closed implementation, v4l2 current support video, image, radio, tuner, touch devices, so it is not complicated to add support for this specific audio case. Because we had implemented the "memory -> asrc ->i2s device-> codec" use case in ALSA. Now the "memory->asrc->memory" needs to reuse the code in asrc driver, so the first 3 patches is for refining the code to make it can be shared by the "memory->asrc->memory" driver. The main change is in the v4l2 side, A /dev/vl4-audioX will be created, user applications only use the ioctl of v4l2 framework. Other change is to add memory to memory support for two kinds of i.MX ASRC module. changes in v13 - change 'pixelformat' to 'audioformat' in dev-audio-mem2mem.rst - add more description for clock drift in ext-ctrls-audio-m2m.rst - Add "media: v4l2-ctrls: add support for fraction_bits" from Hans to avoid build issue for kernel test robot changes in v12 - minor changes according to comments - drop min_buffers_needed = 1 and V4L2_CTRL_FLAG_UPDATE flag - drop bus_info changes in v11 - add add-fixed-point-test-controls in vivid. - add v4l2_ctrl_fp_compose() helper function for min and max changes in v10 - remove FIXED_POINT type - change code base on media: v4l2-ctrls: add support for fraction_bits - fix issue reported by kernel test robot - remove module_alias changes in v9: - add MEDIA_ENT_F_PROC_AUDIO_RESAMPLER. - add MEDIA_INTF_T_V4L_AUDIO - add media controller support - refine the vim2m-audio to support 8k<->16k conversion. changes in v8: - refine V4L2_CAP_AUDIO_M2M to be 0x00000008 - update doc for FIXED_POINT - address comments for imx-asrc changes in v7: - add acked-by from Mark - separate commit for fixed point, m2m audio class, audio rate controls - use INTEGER_MENU for rate, FIXED_POINT for rate offset - remove used fmts - address other comments for Hans changes in v6: - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two. - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT - fix warning by kernel test rebot - remove some unused format V4L2_AUDIO_FMT_XX - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver. - rename audm2m to viaudm2m. changes in v5: - remove V4L2_AUDIO_FMT_LPCM - define audio pixel format like V4L2_AUDIO_FMT_S8... - remove rate and format in struct v4l2_audio_format. - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls - updata document accordingly. changes in v4: - update document style - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit changes in v3: - Modify documents for adding audio m2m support - Add audio virtual m2m driver - Defined V4L2_AUDIO_FMT_LPCM format type for audio. - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case. - with modification in v4l-utils, pass v4l2-compliance test. changes in v2: - decouple the implementation in v4l2 and ALSA - implement the memory to memory driver as a platfrom driver and move it to driver/media - move fsl_asrc_common.h to include/sound folder Hans Verkuil (1): media: v4l2-ctrls: add support for fraction_bits Shengjiu Wang (15): ASoC: fsl_asrc: define functions for memory to memory usage ASoC: fsl_easrc: define functions for memory to memory usage ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound ASoC: fsl_asrc: register m2m platform device ASoC: fsl_easrc: register m2m platform device media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag media: v4l2: Add audio capture and output support media: uapi: Define audio sample format fourcc type media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO media: uapi: Add audio rate controls support media: uapi: Declare interface types for Audio media: uapi: Add an entity type for audio resampler media: vivid: add fixed point test controls media: imx-asrc: Add memory to memory driver media: vim2m-audio: add virtual driver for audio memory to memory .../media/mediactl/media-types.rst | 11 + .../userspace-api/media/v4l/buffer.rst | 6 + .../userspace-api/media/v4l/common.rst | 1 + .../media/v4l/dev-audio-mem2mem.rst | 71 + .../userspace-api/media/v4l/devices.rst | 1 + .../media/v4l/ext-ctrls-audio-m2m.rst | 59 + .../userspace-api/media/v4l/pixfmt-audio.rst | 87 ++ .../userspace-api/media/v4l/pixfmt.rst | 1 + .../media/v4l/vidioc-enum-fmt.rst | 2 + .../media/v4l/vidioc-g-ext-ctrls.rst | 4 + .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 + .../media/v4l/vidioc-querycap.rst | 3 + .../media/v4l/vidioc-queryctrl.rst | 11 +- .../media/videodev2.h.rst.exceptions | 3 + .../media/common/videobuf2/videobuf2-v4l2.c | 4 + drivers/media/platform/nxp/Kconfig | 13 + drivers/media/platform/nxp/Makefile | 1 + drivers/media/platform/nxp/imx-asrc.c | 1256 +++++++++++++++++ drivers/media/test-drivers/Kconfig | 10 + drivers/media/test-drivers/Makefile | 1 + drivers/media/test-drivers/vim2m-audio.c | 793 +++++++++++ drivers/media/test-drivers/vivid/vivid-core.h | 2 + .../media/test-drivers/vivid/vivid-ctrls.c | 26 + drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 9 + drivers/media/v4l2-core/v4l2-ctrls-api.c | 1 + drivers/media/v4l2-core/v4l2-ctrls-core.c | 93 +- drivers/media/v4l2-core/v4l2-ctrls-defs.c | 10 + drivers/media/v4l2-core/v4l2-dev.c | 21 + drivers/media/v4l2-core/v4l2-ioctl.c | 66 + drivers/media/v4l2-core/v4l2-mem2mem.c | 13 +- include/media/v4l2-ctrls.h | 13 +- include/media/v4l2-dev.h | 2 + include/media/v4l2-ioctl.h | 34 + .../fsl => include/sound}/fsl_asrc_common.h | 60 + include/uapi/linux/media.h | 2 + include/uapi/linux/v4l2-controls.h | 9 + include/uapi/linux/videodev2.h | 44 +- sound/soc/fsl/fsl_asrc.c | 144 ++ sound/soc/fsl/fsl_asrc.h | 4 +- sound/soc/fsl/fsl_asrc_dma.c | 2 +- sound/soc/fsl/fsl_easrc.c | 233 +++ sound/soc/fsl/fsl_easrc.h | 6 +- 42 files changed, 3109 insertions(+), 27 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst create mode 100644 drivers/media/platform/nxp/imx-asrc.c create mode 100644 drivers/media/test-drivers/vim2m-audio.c rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%)