[v9,2/3] fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about PTEs
Commit Message
This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear
the info about page table entries. The following operations are supported
in this ioctl:
- Get the information if the pages have been written-to (PAGE_IS_WRITTEN),
file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT) or swapped
(PAGE_IS_SWAPPED).
- Write-protect the pages (PAGEMAP_WP_ENGAGE) to start finding which
pages have been written-to.
- Find pages which have been written-to and write protect the pages
(atomic PAGE_IS_WRITTEN + PAGEMAP_WP_ENGAGE)
The uffd should have been registered on the memory range before performing
any WP/WT (Write Protect/Writtern-To) related operations with the IOCTL.
struct pagemap_scan_args is used as the argument of the IOCTL. In this
struct:
- The range is specified through start and len.
- The output buffer of struct page_region array and size is specified as
vec and vec_len.
- The optional maximum requested pages are specified in the max_pages.
- The flags can be specified in the flags field. The PAGEMAP_WP_ENGAGE
is the only added flag at this time.
- The masks are specified in required_mask, anyof_mask, excluded_ mask
and return_mask.
This IOCTL can be extended to get information about more PTE bits. This
IOCTL doesn't support hugetlbs at the moment. No information about
hugetlb can be obtained. This patch has evolved from a basic patch from
Gabriel Krisman Bertazi.
Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
Change in v8:
- Correct is_pte_uffd_wp()
- Improve readability and error checks
- Remove some un-needed code
Changes in v7:
- Rebase on top of latest next
- Fix some corner cases
- Base soft-dirty on the uffd wp async
- Update the terminologies
- Optimize the memory usage inside the ioctl
Changes in v6:
- Rename variables and update comments
- Make IOCTL independent of soft_dirty config
- Change masks and bitmap type to _u64
- Improve code quality
Changes in v5:
- Remove tlb flushing even for clear operation
Changes in v4:
- Update the interface and implementation
Changes in v3:
- Tighten the user-kernel interface by using explicit types and add more
error checking
Changes in v2:
- Convert the interface from syscall to ioctl
- Remove pidfd support as it doesn't make sense in ioctl
---
fs/proc/task_mmu.c | 290 ++++++++++++++++++++++++++++++++++
include/uapi/linux/fs.h | 50 ++++++
tools/include/uapi/linux/fs.h | 50 ++++++
3 files changed, 390 insertions(+)
Comments
Hi Muhammad,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on shuah-kselftest/fixes]
[also build test ERROR on linus/master v6.2-rc6 next-20230131]
[cannot apply to shuah-kselftest/next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230131-163537
base: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git fixes
patch link: https://lore.kernel.org/r/20230131083257.3302830-3-usama.anjum%40collabora.com
patch subject: [PATCH v9 2/3] fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about PTEs
config: arc-defconfig (https://download.01.org/0day-ci/archive/20230131/202301312359.8WtBkSkQ-lkp@intel.com/config)
compiler: arc-elf-gcc (GCC) 12.1.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/11677b6b7fda958031115ea40aa219fc32c7dea4
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230131-163537
git checkout 11677b6b7fda958031115ea40aa219fc32c7dea4
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=arc olddefconfig
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=arc SHELL=/bin/bash
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
fs/proc/task_mmu.c: In function 'pagemap_scan_pmd_entry':
>> fs/proc/task_mmu.c:1927:17: error: implicit declaration of function 'uffd_wp_range' [-Werror=implicit-function-declaration]
1927 | uffd_wp_range(walk->mm, vma, start, addr - start, true);
| ^~~~~~~~~~~~~
cc1: some warnings being treated as errors
vim +/uffd_wp_range +1927 fs/proc/task_mmu.c
1915
1916 pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
1917 if (IS_GET_OP(p)) {
1918 for (addr = start; addr < end; pte++, addr += PAGE_SIZE) {
1919 ret = pagemap_scan_output(!is_pte_uffd_wp(*pte), vma->vm_file,
1920 pte_present(*pte), is_swap_pte(*pte), p, addr, 1);
1921 if (ret)
1922 break;
1923 }
1924 }
1925 pte_unmap_unlock(pte - 1, ptl);
1926 if ((!ret || ret == -ENOSPC) && IS_WP_ENGAGE_OP(p) && (addr - start))
> 1927 uffd_wp_range(walk->mm, vma, start, addr - start, true);
1928
1929 cond_resched();
1930 return ret;
1931 }
1932
On 1/31/23 8:52 PM, kernel test robot wrote:
> Hi Muhammad,
>
> Thank you for the patch! Yet something to improve:
>
> [auto build test ERROR on shuah-kselftest/fixes]
> [also build test ERROR on linus/master v6.2-rc6 next-20230131]
> [cannot apply to shuah-kselftest/next]
> [If your patch is applied to the wrong git tree, kindly drop us a note.
> And when submitting patch, we suggest to use '--base' as documented in
> https://git-scm.com/docs/git-format-patch#_base_tree_information]
>
> url: https://github.com/intel-lab-lkp/linux/commits/Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230131-163537
> base: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git fixes
> patch link: https://lore.kernel.org/r/20230131083257.3302830-3-usama.anjum%40collabora.com
> patch subject: [PATCH v9 2/3] fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about PTEs
> config: arc-defconfig (https://download.01.org/0day-ci/archive/20230131/202301312359.8WtBkSkQ-lkp@intel.com/config)
> compiler: arc-elf-gcc (GCC) 12.1.0
> reproduce (this is a W=1 build):
> wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
> chmod +x ~/bin/make.cross
> # https://github.com/intel-lab-lkp/linux/commit/11677b6b7fda958031115ea40aa219fc32c7dea4
> git remote add linux-review https://github.com/intel-lab-lkp/linux
> git fetch --no-tags linux-review Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230131-163537
> git checkout 11677b6b7fda958031115ea40aa219fc32c7dea4
> # save the config file
> mkdir build_dir && cp config build_dir/.config
> COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=arc olddefconfig
> COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=arc SHELL=/bin/bash
>
> If you fix the issue, kindly add following tag where applicable
> | Reported-by: kernel test robot <lkp@intel.com>
>
> All errors (new ones prefixed by >>):
>
> fs/proc/task_mmu.c: In function 'pagemap_scan_pmd_entry':
>>> fs/proc/task_mmu.c:1927:17: error: implicit declaration of function 'uffd_wp_range' [-Werror=implicit-function-declaration]
> 1927 | uffd_wp_range(walk->mm, vma, start, addr - start, true);
> | ^~~~~~~~~~~~~
> cc1: some warnings being treated as errors
I'll fix this by adding the following patch in next version:
--- a/include/linux/userfaultfd_k.h
+++ b/include/linux/userfaultfd_k.h
@@ -190,6 +190,11 @@ static inline vm_fault_t handle_userfault(struct
vm_fault *vmf,
return VM_FAULT_SIGBUS;
}
+static inline void uffd_wp_range(struct mm_struct *dst_mm, struct
vm_area_struct *vma,
+ unsigned long start, unsigned long len,
bool enable_wp)
+{
+}
+
>
>
> vim +/uffd_wp_range +1927 fs/proc/task_mmu.c
>
> 1915
> 1916 pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
> 1917 if (IS_GET_OP(p)) {
> 1918 for (addr = start; addr < end; pte++, addr += PAGE_SIZE) {
> 1919 ret = pagemap_scan_output(!is_pte_uffd_wp(*pte), vma->vm_file,
> 1920 pte_present(*pte), is_swap_pte(*pte), p, addr, 1);
> 1921 if (ret)
> 1922 break;
> 1923 }
> 1924 }
> 1925 pte_unmap_unlock(pte - 1, ptl);
> 1926 if ((!ret || ret == -ENOSPC) && IS_WP_ENGAGE_OP(p) && (addr - start))
>> 1927 uffd_wp_range(walk->mm, vma, start, addr - start, true);
> 1928
> 1929 cond_resched();
> 1930 return ret;
> 1931 }
> 1932
>
Hi Muhammad,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on shuah-kselftest/fixes]
[also build test ERROR on linus/master v6.2-rc6]
[cannot apply to shuah-kselftest/next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230131-163537
base: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git fixes
patch link: https://lore.kernel.org/r/20230131083257.3302830-3-usama.anjum%40collabora.com
patch subject: [PATCH v9 2/3] fs/proc/task_mmu: Implement IOCTL to get and/or the clear info about PTEs
config: x86_64-rhel-8.3-rust (https://download.01.org/0day-ci/archive/20230201/202302011447.7lSFpVO5-lkp@intel.com/config)
compiler: clang version 14.0.6 (https://github.com/llvm/llvm-project f28c006a5895fc0e329fe15fead81e37457cb1d1)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/11677b6b7fda958031115ea40aa219fc32c7dea4
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Muhammad-Usama-Anjum/userfaultfd-Add-UFFD-WP-Async-support/20230131-163537
git checkout 11677b6b7fda958031115ea40aa219fc32c7dea4
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=x86_64 olddefconfig
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=x86_64 SHELL=/bin/bash fs/
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
>> fs/proc/task_mmu.c:1907:5: error: implicit declaration of function 'uffd_wp_range' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
uffd_wp_range(walk->mm, vma, start, HPAGE_SIZE, true);
^
fs/proc/task_mmu.c:1927:3: error: implicit declaration of function 'uffd_wp_range' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
uffd_wp_range(walk->mm, vma, start, addr - start, true);
^
2 errors generated.
vim +/uffd_wp_range +1907 fs/proc/task_mmu.c
1874
1875 static inline int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
1876 unsigned long end, struct mm_walk *walk)
1877 {
1878 struct pagemap_scan_private *p = walk->private;
1879 struct vm_area_struct *vma = walk->vma;
1880 unsigned long addr = end;
1881 spinlock_t *ptl;
1882 int ret = 0;
1883 pte_t *pte;
1884
1885 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
1886 ptl = pmd_trans_huge_lock(pmd, vma);
1887 if (ptl) {
1888 bool pmd_wt;
1889
1890 pmd_wt = !is_pmd_uffd_wp(*pmd);
1891 /*
1892 * Break huge page into small pages if operation needs to be performed is
1893 * on a portion of the huge page.
1894 */
1895 if (pmd_wt && IS_WP_ENGAGE_OP(p) && (end - start < HPAGE_SIZE)) {
1896 spin_unlock(ptl);
1897 split_huge_pmd(vma, pmd, start);
1898 goto process_smaller_pages;
1899 }
1900 if (IS_GET_OP(p))
1901 ret = pagemap_scan_output(pmd_wt, vma->vm_file, pmd_present(*pmd),
1902 is_swap_pmd(*pmd), p, start,
1903 (end - start)/PAGE_SIZE);
1904 spin_unlock(ptl);
1905 if (!ret) {
1906 if (pmd_wt && IS_WP_ENGAGE_OP(p))
> 1907 uffd_wp_range(walk->mm, vma, start, HPAGE_SIZE, true);
1908 }
1909 return ret;
1910 }
1911 process_smaller_pages:
1912 if (pmd_trans_unstable(pmd))
1913 return 0;
1914 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
1915
1916 pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
1917 if (IS_GET_OP(p)) {
1918 for (addr = start; addr < end; pte++, addr += PAGE_SIZE) {
1919 ret = pagemap_scan_output(!is_pte_uffd_wp(*pte), vma->vm_file,
1920 pte_present(*pte), is_swap_pte(*pte), p, addr, 1);
1921 if (ret)
1922 break;
1923 }
1924 }
1925 pte_unmap_unlock(pte - 1, ptl);
1926 if ((!ret || ret == -ENOSPC) && IS_WP_ENGAGE_OP(p) && (addr - start))
1927 uffd_wp_range(walk->mm, vma, start, addr - start, true);
1928
1929 cond_resched();
1930 return ret;
1931 }
1932
@@ -19,6 +19,7 @@
#include <linux/shmem_fs.h>
#include <linux/uaccess.h>
#include <linux/pkeys.h>
+#include <linux/minmax.h>
#include <asm/elf.h>
#include <asm/tlb.h>
@@ -1135,6 +1136,22 @@ static inline void clear_soft_dirty(struct vm_area_struct *vma,
}
#endif
+static inline bool is_pte_uffd_wp(pte_t pte)
+{
+ if ((pte_present(pte) && pte_uffd_wp(pte)) ||
+ (pte_swp_uffd_wp_any(pte)))
+ return true;
+ return false;
+}
+
+static inline bool is_pmd_uffd_wp(pmd_t pmd)
+{
+ if ((pmd_present(pmd) && pmd_uffd_wp(pmd)) ||
+ (is_swap_pmd(pmd) && pmd_swp_uffd_wp(pmd)))
+ return true;
+ return false;
+}
+
#if defined(CONFIG_MEM_SOFT_DIRTY) && defined(CONFIG_TRANSPARENT_HUGEPAGE)
static inline void clear_soft_dirty_pmd(struct vm_area_struct *vma,
unsigned long addr, pmd_t *pmdp)
@@ -1763,11 +1780,284 @@ static int pagemap_release(struct inode *inode, struct file *file)
return 0;
}
+#define PAGEMAP_BITS_ALL (PAGE_IS_WRITTEN | PAGE_IS_FILE | \
+ PAGE_IS_PRESENT | PAGE_IS_SWAPPED)
+#define PAGEMAP_NON_WRITTEN_BITS (PAGE_IS_FILE | PAGE_IS_PRESENT | PAGE_IS_SWAPPED)
+#define IS_WP_ENGAGE_OP(a) (a->flags & PAGEMAP_WP_ENGAGE)
+#define IS_GET_OP(a) (a->vec)
+#define HAS_NO_SPACE(p) (p->max_pages && (p->found_pages == p->max_pages))
+
+#define PAGEMAP_SCAN_BITMAP(wt, file, present, swap) \
+ (wt | file << 1 | present << 2 | swap << 3)
+#define IS_WT_REQUIRED(a) \
+ ((a->required_mask & PAGE_IS_WRITTEN) || \
+ (a->anyof_mask & PAGE_IS_WRITTEN))
+
+struct pagemap_scan_private {
+ struct page_region *vec;
+ struct page_region prev;
+ unsigned long vec_len, vec_index;
+ unsigned int max_pages, found_pages, flags;
+ unsigned long required_mask, anyof_mask, excluded_mask, return_mask;
+};
+
+static int pagemap_scan_test_walk(unsigned long start, unsigned long end, struct mm_walk *walk)
+{
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+
+ if (IS_WT_REQUIRED(p) && !userfaultfd_wp(vma) && !userfaultfd_wp_async(vma))
+ return -EPERM;
+ if (vma->vm_flags & VM_PFNMAP)
+ return 1;
+ return 0;
+}
+
+static inline int pagemap_scan_output(bool wt, bool file, bool pres, bool swap,
+ struct pagemap_scan_private *p, unsigned long addr,
+ unsigned int len)
+{
+ unsigned long bitmap, cur = PAGEMAP_SCAN_BITMAP(wt, file, pres, swap);
+ bool cpy = true;
+ struct page_region *prev = &p->prev;
+
+ if (HAS_NO_SPACE(p))
+ return -ENOSPC;
+
+ if (p->max_pages && p->found_pages + len >= p->max_pages)
+ len = p->max_pages - p->found_pages;
+ if (!len)
+ return -EINVAL;
+
+ if (p->required_mask)
+ cpy = ((p->required_mask & cur) == p->required_mask);
+ if (cpy && p->anyof_mask)
+ cpy = (p->anyof_mask & cur);
+ if (cpy && p->excluded_mask)
+ cpy = !(p->excluded_mask & cur);
+ bitmap = cur & p->return_mask;
+ if (cpy && bitmap) {
+ if ((prev->len) && (prev->bitmap == bitmap) &&
+ (prev->start + prev->len * PAGE_SIZE == addr)) {
+ prev->len += len;
+ p->found_pages += len;
+ } else if (p->vec_index < p->vec_len) {
+ if (prev->len) {
+ memcpy(&p->vec[p->vec_index], prev, sizeof(struct page_region));
+ p->vec_index++;
+ }
+ prev->start = addr;
+ prev->len = len;
+ prev->bitmap = bitmap;
+ p->found_pages += len;
+ } else {
+ return -ENOSPC;
+ }
+ }
+ return 0;
+}
+
+static inline int export_prev_to_out(struct pagemap_scan_private *p, struct page_region __user *vec,
+ unsigned long *vec_index)
+{
+ struct page_region *prev = &p->prev;
+
+ if (prev->len) {
+ if (copy_to_user(&vec[*vec_index], prev, sizeof(struct page_region)))
+ return -EFAULT;
+ p->vec_index++;
+ (*vec_index)++;
+ prev->len = 0;
+ }
+ return 0;
+}
+
+static inline int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
+ unsigned long end, struct mm_walk *walk)
+{
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+ unsigned long addr = end;
+ spinlock_t *ptl;
+ int ret = 0;
+ pte_t *pte;
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+ ptl = pmd_trans_huge_lock(pmd, vma);
+ if (ptl) {
+ bool pmd_wt;
+
+ pmd_wt = !is_pmd_uffd_wp(*pmd);
+ /*
+ * Break huge page into small pages if operation needs to be performed is
+ * on a portion of the huge page.
+ */
+ if (pmd_wt && IS_WP_ENGAGE_OP(p) && (end - start < HPAGE_SIZE)) {
+ spin_unlock(ptl);
+ split_huge_pmd(vma, pmd, start);
+ goto process_smaller_pages;
+ }
+ if (IS_GET_OP(p))
+ ret = pagemap_scan_output(pmd_wt, vma->vm_file, pmd_present(*pmd),
+ is_swap_pmd(*pmd), p, start,
+ (end - start)/PAGE_SIZE);
+ spin_unlock(ptl);
+ if (!ret) {
+ if (pmd_wt && IS_WP_ENGAGE_OP(p))
+ uffd_wp_range(walk->mm, vma, start, HPAGE_SIZE, true);
+ }
+ return ret;
+ }
+process_smaller_pages:
+ if (pmd_trans_unstable(pmd))
+ return 0;
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+
+ pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
+ if (IS_GET_OP(p)) {
+ for (addr = start; addr < end; pte++, addr += PAGE_SIZE) {
+ ret = pagemap_scan_output(!is_pte_uffd_wp(*pte), vma->vm_file,
+ pte_present(*pte), is_swap_pte(*pte), p, addr, 1);
+ if (ret)
+ break;
+ }
+ }
+ pte_unmap_unlock(pte - 1, ptl);
+ if ((!ret || ret == -ENOSPC) && IS_WP_ENGAGE_OP(p) && (addr - start))
+ uffd_wp_range(walk->mm, vma, start, addr - start, true);
+
+ cond_resched();
+ return ret;
+}
+
+static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, int depth,
+ struct mm_walk *walk)
+{
+ struct pagemap_scan_private *p = walk->private;
+ struct vm_area_struct *vma = walk->vma;
+ int ret = 0;
+
+ if (vma)
+ ret = pagemap_scan_output(false, vma->vm_file, false, false, p, addr,
+ (end - addr)/PAGE_SIZE);
+ return ret;
+}
+
+/* No hugetlb support is present. */
+static const struct mm_walk_ops pagemap_scan_ops = {
+ .test_walk = pagemap_scan_test_walk,
+ .pmd_entry = pagemap_scan_pmd_entry,
+ .pte_hole = pagemap_scan_pte_hole,
+};
+
+static long do_pagemap_cmd(struct mm_struct *mm, struct pagemap_scan_arg *arg)
+{
+ unsigned long empty_slots, vec_index = 0;
+ unsigned long __user start, end;
+ unsigned long __start, __end;
+ struct page_region __user *vec;
+ struct pagemap_scan_private p;
+ int ret = 0;
+
+ start = (unsigned long)untagged_addr(arg->start);
+ vec = (struct page_region *)(unsigned long)untagged_addr(arg->vec);
+
+ /* Validate memory ranges */
+ if ((!IS_ALIGNED(start, PAGE_SIZE)) || (!access_ok((void __user *)start, arg->len)))
+ return -EINVAL;
+ if (IS_GET_OP(arg) && ((arg->vec_len == 0) ||
+ (!access_ok((void __user *)vec, arg->vec_len * sizeof(struct page_region)))))
+ return -EINVAL;
+
+ /* Detect illegal flags and masks */
+ if ((arg->flags & ~PAGEMAP_WP_ENGAGE) || (arg->required_mask & ~PAGEMAP_BITS_ALL) ||
+ (arg->anyof_mask & ~PAGEMAP_BITS_ALL) || (arg->excluded_mask & ~PAGEMAP_BITS_ALL) ||
+ (arg->return_mask & ~PAGEMAP_BITS_ALL))
+ return -EINVAL;
+ if (IS_GET_OP(arg) && ((!arg->required_mask && !arg->anyof_mask && !arg->excluded_mask) ||
+ !arg->return_mask))
+ return -EINVAL;
+ /* The non-WT flags cannot be obtained if PAGEMAP_WP_ENGAGE is also specified. */
+ if (IS_WP_ENGAGE_OP(arg) && ((arg->required_mask & PAGEMAP_NON_WRITTEN_BITS) ||
+ (arg->anyof_mask & PAGEMAP_NON_WRITTEN_BITS)))
+ return -EINVAL;
+
+ end = start + arg->len;
+ p.max_pages = arg->max_pages;
+ p.found_pages = 0;
+ p.flags = arg->flags;
+ p.required_mask = arg->required_mask;
+ p.anyof_mask = arg->anyof_mask;
+ p.excluded_mask = arg->excluded_mask;
+ p.return_mask = arg->return_mask;
+ p.prev.len = 0;
+ p.vec_len = (PAGEMAP_WALK_SIZE >> PAGE_SHIFT);
+
+ if (IS_GET_OP(arg)) {
+ p.vec = kmalloc_array(p.vec_len, sizeof(struct page_region), GFP_KERNEL);
+ if (!p.vec)
+ return -ENOMEM;
+ } else {
+ p.vec = NULL;
+ }
+ __start = __end = start;
+ while (!ret && __end < end) {
+ p.vec_index = 0;
+ empty_slots = arg->vec_len - vec_index;
+ if (p.vec_len > empty_slots)
+ p.vec_len = empty_slots;
+
+ __end = (__start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK;
+ if (__end > end)
+ __end = end;
+
+ mmap_read_lock(mm);
+ ret = walk_page_range(mm, __start, __end, &pagemap_scan_ops, &p);
+ mmap_read_unlock(mm);
+ if (!(!ret || ret == -ENOSPC))
+ goto free_data;
+
+ __start = __end;
+ if (IS_GET_OP(arg) && p.vec_index) {
+ if (copy_to_user(&vec[vec_index], p.vec,
+ p.vec_index * sizeof(struct page_region))) {
+ ret = -EFAULT;
+ goto free_data;
+ }
+ vec_index += p.vec_index;
+ }
+ }
+ ret = export_prev_to_out(&p, vec, &vec_index);
+ if (!ret)
+ ret = vec_index;
+free_data:
+ if (IS_GET_OP(arg))
+ kfree(p.vec);
+
+ return ret;
+}
+
+static long pagemap_scan_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
+{
+ struct pagemap_scan_arg __user *uarg = (struct pagemap_scan_arg __user *)arg;
+ struct mm_struct *mm = file->private_data;
+ struct pagemap_scan_arg argument;
+
+ if (cmd == PAGEMAP_SCAN) {
+ if (copy_from_user(&argument, uarg, sizeof(struct pagemap_scan_arg)))
+ return -EFAULT;
+ return do_pagemap_cmd(mm, &argument);
+ }
+ return -EINVAL;
+}
+
const struct file_operations proc_pagemap_operations = {
.llseek = mem_lseek, /* borrow this */
.read = pagemap_read,
.open = pagemap_open,
.release = pagemap_release,
+ .unlocked_ioctl = pagemap_scan_ioctl,
+ .compat_ioctl = pagemap_scan_ioctl,
};
#endif /* CONFIG_PROC_PAGE_MONITOR */
@@ -305,4 +305,54 @@ typedef int __bitwise __kernel_rwf_t;
#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
RWF_APPEND)
+/* Pagemap ioctl */
+#define PAGEMAP_SCAN _IOWR('f', 16, struct pagemap_scan_arg)
+
+/* Bits are set in the bitmap of the page_region and masks in pagemap_scan_args */
+#define PAGE_IS_WRITTEN (1 << 0)
+#define PAGE_IS_FILE (1 << 1)
+#define PAGE_IS_PRESENT (1 << 2)
+#define PAGE_IS_SWAPPED (1 << 3)
+
+/*
+ * struct page_region - Page region with bitmap flags
+ * @start: Start of the region
+ * @len: Length of the region
+ * bitmap: Bits sets for the region
+ */
+struct page_region {
+ __u64 start;
+ __u64 len;
+ __u64 bitmap;
+};
+
+/*
+ * struct pagemap_scan_arg - Pagemap ioctl argument
+ * @start: Starting address of the region
+ * @len: Length of the region (All the pages in this length are included)
+ * @vec: Address of page_region struct array for output
+ * @vec_len: Length of the page_region struct array
+ * @max_pages: Optional max return pages
+ * @flags: Flags for the IOCTL
+ * @required_mask: Required mask - All of these bits have to be set in the PTE
+ * @anyof_mask: Any mask - Any of these bits are set in the PTE
+ * @excluded_mask: Exclude mask - None of these bits are set in the PTE
+ * @return_mask: Bits that are to be reported in page_region
+ */
+struct pagemap_scan_arg {
+ __u64 start;
+ __u64 len;
+ __u64 vec;
+ __u64 vec_len;
+ __u32 max_pages;
+ __u32 flags;
+ __u64 required_mask;
+ __u64 anyof_mask;
+ __u64 excluded_mask;
+ __u64 return_mask;
+};
+
+/* Special flags */
+#define PAGEMAP_WP_ENGAGE (1 << 0)
+
#endif /* _UAPI_LINUX_FS_H */
@@ -305,4 +305,54 @@ typedef int __bitwise __kernel_rwf_t;
#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
RWF_APPEND)
+/* Pagemap ioctl */
+#define PAGEMAP_SCAN _IOWR('f', 16, struct pagemap_scan_arg)
+
+/* Bits are set in the bitmap of the page_region and masks in pagemap_scan_args */
+#define PAGE_IS_WRITTEN (1 << 0)
+#define PAGE_IS_FILE (1 << 1)
+#define PAGE_IS_PRESENT (1 << 2)
+#define PAGE_IS_SWAPPED (1 << 3)
+
+/*
+ * struct page_region - Page region with bitmap flags
+ * @start: Start of the region
+ * @len: Length of the region
+ * bitmap: Bits sets for the region
+ */
+struct page_region {
+ __u64 start;
+ __u64 len;
+ __u64 bitmap;
+};
+
+/*
+ * struct pagemap_scan_arg - Pagemap ioctl argument
+ * @start: Starting address of the region
+ * @len: Length of the region (All the pages in this length are included)
+ * @vec: Address of page_region struct array for output
+ * @vec_len: Length of the page_region struct array
+ * @max_pages: Optional max return pages
+ * @flags: Flags for the IOCTL
+ * @required_mask: Required mask - All of these bits have to be set in the PTE
+ * @anyof_mask: Any mask - Any of these bits are set in the PTE
+ * @excluded_mask: Exclude mask - None of these bits are set in the PTE
+ * @return_mask: Bits that are to be reported in page_region
+ */
+struct pagemap_scan_arg {
+ __u64 start;
+ __u64 len;
+ __u64 vec;
+ __u64 vec_len;
+ __u32 max_pages;
+ __u32 flags;
+ __u64 required_mask;
+ __u64 anyof_mask;
+ __u64 excluded_mask;
+ __u64 return_mask;
+};
+
+/* Special flags */
+#define PAGEMAP_WP_ENGAGE (1 << 0)
+
#endif /* _UAPI_LINUX_FS_H */