mirror of
https://github.com/raspberrypi/linux.git
synced 2025-12-06 01:49:46 +00:00
00c010e130
Pull MM updates from Andrew Morton: - "Add folio_mk_pte()" from Matthew Wilcox simplifies the act of creating a pte which addresses the first page in a folio and reduces the amount of plumbing which architecture must implement to provide this. - "Misc folio patches for 6.16" from Matthew Wilcox is a shower of largely unrelated folio infrastructure changes which clean things up and better prepare us for future work. - "memory,x86,acpi: hotplug memory alignment advisement" from Gregory Price adds early-init code to prevent x86 from leaving physical memory unused when physical address regions are not aligned to memory block size. - "mm/compaction: allow more aggressive proactive compaction" from Michal Clapinski provides some tuning of the (sadly, hard-coded (more sadly, not auto-tuned)) thresholds for our invokation of proactive compaction. In a simple test case, the reduction of a guest VM's memory consumption was dramatic. - "Minor cleanups and improvements to swap freeing code" from Kemeng Shi provides some code cleaups and a small efficiency improvement to this part of our swap handling code. - "ptrace: introduce PTRACE_SET_SYSCALL_INFO API" from Dmitry Levin adds the ability for a ptracer to modify syscalls arguments. At this time we can alter only "system call information that are used by strace system call tampering, namely, syscall number, syscall arguments, and syscall return value. This series should have been incorporated into mm.git's "non-MM" branch, but I goofed. - "fs/proc: extend the PAGEMAP_SCAN ioctl to report guard regions" from Andrei Vagin extends the info returned by the PAGEMAP_SCAN ioctl against /proc/pid/pagemap. This permits CRIU to more efficiently get at the info about guard regions. - "Fix parameter passed to page_mapcount_is_type()" from Gavin Shan implements that fix. No runtime effect is expected because validate_page_before_insert() happens to fix up this error. - "kernel/events/uprobes: uprobe_write_opcode() rewrite" from David Hildenbrand basically brings uprobe text poking into the current decade. Remove a bunch of hand-rolled implementation in favor of using more current facilities. - "mm/ptdump: Drop assumption that pxd_val() is u64" from Anshuman Khandual provides enhancements and generalizations to the pte dumping code. This might be needed when 128-bit Page Table Descriptors are enabled for ARM. - "Always call constructor for kernel page tables" from Kevin Brodsky ensures that the ctor/dtor is always called for kernel pgtables, as it already is for user pgtables. This permits the addition of more functionality such as "insert hooks to protect page tables". This change does result in various architectures performing unnecesary work, but this is fixed up where it is anticipated to occur. - "Rust support for mm_struct, vm_area_struct, and mmap" from Alice Ryhl adds plumbing to permit Rust access to core MM structures. - "fix incorrectly disallowed anonymous VMA merges" from Lorenzo Stoakes takes advantage of some VMA merging opportunities which we've been missing for 15 years. - "mm/madvise: batch tlb flushes for MADV_DONTNEED and MADV_FREE" from SeongJae Park optimizes process_madvise()'s TLB flushing. Instead of flushing each address range in the provided iovec, we batch the flushing across all the iovec entries. The syscall's cost was approximately halved with a microbenchmark which was designed to load this particular operation. - "Track node vacancy to reduce worst case allocation counts" from Sidhartha Kumar makes the maple tree smarter about its node preallocation. stress-ng mmap performance increased by single-digit percentages and the amount of unnecessarily preallocated memory was dramaticelly reduced. - "mm/gup: Minor fix, cleanup and improvements" from Baoquan He removes a few unnecessary things which Baoquan noted when reading the code. - ""Enhance sysfs handling for memory hotplug in weighted interleave" from Rakie Kim "enhances the weighted interleave policy in the memory management subsystem by improving sysfs handling, fixing memory leaks, and introducing dynamic sysfs updates for memory hotplug support". Fixes things on error paths which we are unlikely to hit. - "mm/damon: auto-tune DAMOS for NUMA setups including tiered memory" from SeongJae Park introduces new DAMOS quota goal metrics which eliminate the manual tuning which is required when utilizing DAMON for memory tiering. - "mm/vmalloc.c: code cleanup and improvements" from Baoquan He provides cleanups and small efficiency improvements which Baoquan found via code inspection. - "vmscan: enforce mems_effective during demotion" from Gregory Price changes reclaim to respect cpuset.mems_effective during demotion when possible. because presently, reclaim explicitly ignores cpuset.mems_effective when demoting, which may cause the cpuset settings to violated. This is useful for isolating workloads on a multi-tenant system from certain classes of memory more consistently. - "Clean up split_huge_pmd_locked() and remove unnecessary folio pointers" from Gavin Guo provides minor cleanups and efficiency gains in in the huge page splitting and migrating code. - "Use kmem_cache for memcg alloc" from Huan Yang creates a slab cache for `struct mem_cgroup', yielding improved memory utilization. - "add max arg to swappiness in memory.reclaim and lru_gen" from Zhongkun He adds a new "max" argument to the "swappiness=" argument for memory.reclaim MGLRU's lru_gen. This directs proactive reclaim to reclaim from only anon folios rather than file-backed folios. - "kexec: introduce Kexec HandOver (KHO)" from Mike Rapoport is the first step on the path to permitting the kernel to maintain existing VMs while replacing the host kernel via file-based kexec. At this time only memblock's reserve_mem is preserved. - "mm: Introduce for_each_valid_pfn()" from David Woodhouse provides and uses a smarter way of looping over a pfn range. By skipping ranges of invalid pfns. - "sched/numa: Skip VMA scanning on memory pinned to one NUMA node via cpuset.mems" from Libo Chen removes a lot of pointless VMA scanning when a task is pinned a single NUMA mode. Dramatic performance benefits were seen in some real world cases. - "JFS: Implement migrate_folio for jfs_metapage_aops" from Shivank Garg addresses a warning which occurs during memory compaction when using JFS. - "move all VMA allocation, freeing and duplication logic to mm" from Lorenzo Stoakes moves some VMA code from kernel/fork.c into the more appropriate mm/vma.c. - "mm, swap: clean up swap cache mapping helper" from Kairui Song provides code consolidation and cleanups related to the folio_index() function. - "mm/gup: Cleanup memfd_pin_folios()" from Vishal Moola does that. - "memcg: Fix test_memcg_min/low test failures" from Waiman Long addresses some bogus failures which are being reported by the test_memcontrol selftest. - "eliminate mmap() retry merge, add .mmap_prepare hook" from Lorenzo Stoakes commences the deprecation of file_operations.mmap() in favor of the new file_operations.mmap_prepare(). The latter is more restrictive and prevents drivers from messing with things in ways which, amongst other problems, may defeat VMA merging. - "memcg: decouple memcg and objcg stocks"" from Shakeel Butt decouples the per-cpu memcg charge cache from the objcg's one. This is a step along the way to making memcg and objcg charging NMI-safe, which is a BPF requirement. - "mm/damon: minor fixups and improvements for code, tests, and documents" from SeongJae Park is yet another batch of miscellaneous DAMON changes. Fix and improve minor problems in code, tests and documents. - "memcg: make memcg stats irq safe" from Shakeel Butt converts memcg stats to be irq safe. Another step along the way to making memcg charging and stats updates NMI-safe, a BPF requirement. - "Let unmap_hugepage_range() and several related functions take folio instead of page" from Fan Ni provides folio conversions in the hugetlb code. * tag 'mm-stable-2025-05-31-14-50' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (285 commits) mm: pcp: increase pcp->free_count threshold to trigger free_high mm/hugetlb: convert use of struct page to folio in __unmap_hugepage_range() mm/hugetlb: refactor __unmap_hugepage_range() to take folio instead of page mm/hugetlb: refactor unmap_hugepage_range() to take folio instead of page mm/hugetlb: pass folio instead of page to unmap_ref_private() memcg: objcg stock trylock without irq disabling memcg: no stock lock for cpu hot-unplug memcg: make __mod_memcg_lruvec_state re-entrant safe against irqs memcg: make count_memcg_events re-entrant safe against irqs memcg: make mod_memcg_state re-entrant safe against irqs memcg: move preempt disable to callers of memcg_rstat_updated memcg: memcg_rstat_updated re-entrant safe against irqs mm: khugepaged: decouple SHMEM and file folios' collapse selftests/eventfd: correct test name and improve messages alloc_tag: check mem_profiling_support in alloc_tag_init Docs/damon: update titles and brief introductions to explain DAMOS selftests/damon/_damon_sysfs: read tried regions directories in order mm/damon/tests/core-kunit: add a test for damos_set_filters_default_reject() mm/damon/paddr: remove unused variable, folio_list, in damon_pa_stat() mm/damon/sysfs-schemes: fix wrong comment on damons_sysfs_quota_goal_metric_strs ...
305 lines
7.6 KiB
C
305 lines
7.6 KiB
C
/*
|
|
* Copyright (C) 2011 Tobias Klauser <tklauser@distanz.ch>
|
|
* Copyright (C) 2009 Wind River Systems Inc
|
|
*
|
|
* Based on asm/pgtable-32.h from mips which is:
|
|
*
|
|
* Copyright (C) 1994, 95, 96, 97, 98, 99, 2000, 2003 Ralf Baechle
|
|
* Copyright (C) 1999, 2000, 2001 Silicon Graphics, Inc.
|
|
*
|
|
* This file is subject to the terms and conditions of the GNU General Public
|
|
* License. See the file "COPYING" in the main directory of this archive
|
|
* for more details.
|
|
*/
|
|
|
|
#ifndef _ASM_NIOS2_PGTABLE_H
|
|
#define _ASM_NIOS2_PGTABLE_H
|
|
|
|
#include <linux/io.h>
|
|
#include <linux/bug.h>
|
|
#include <asm/page.h>
|
|
#include <asm/cacheflush.h>
|
|
#include <asm/tlbflush.h>
|
|
|
|
#include <asm/pgtable-bits.h>
|
|
#include <asm-generic/pgtable-nopmd.h>
|
|
|
|
#define VMALLOC_START CONFIG_NIOS2_KERNEL_MMU_REGION_BASE
|
|
#define VMALLOC_END (CONFIG_NIOS2_KERNEL_REGION_BASE - SZ_32M - 1)
|
|
|
|
#define MODULES_VADDR (CONFIG_NIOS2_KERNEL_REGION_BASE - SZ_32M)
|
|
#define MODULES_END (CONFIG_NIOS2_KERNEL_REGION_BASE - 1)
|
|
|
|
struct mm_struct;
|
|
|
|
/* Helper macro */
|
|
#define MKP(x, w, r) __pgprot(_PAGE_PRESENT | _PAGE_CACHED | \
|
|
((x) ? _PAGE_EXEC : 0) | \
|
|
((r) ? _PAGE_READ : 0) | \
|
|
((w) ? _PAGE_WRITE : 0))
|
|
/*
|
|
* These are the macros that generic kernel code needs
|
|
* (to populate protection_map[])
|
|
*/
|
|
|
|
/* Remove W bit on private pages for COW support */
|
|
|
|
/* Shared pages can have exact HW mapping */
|
|
|
|
/* Used all over the kernel */
|
|
#define PAGE_KERNEL __pgprot(_PAGE_PRESENT | _PAGE_CACHED | _PAGE_READ | \
|
|
_PAGE_WRITE | _PAGE_EXEC | _PAGE_GLOBAL)
|
|
|
|
#define PAGE_SHARED __pgprot(_PAGE_PRESENT | _PAGE_CACHED | _PAGE_READ | \
|
|
_PAGE_WRITE | _PAGE_ACCESSED)
|
|
|
|
#define PAGE_COPY MKP(0, 0, 1)
|
|
|
|
#define PTRS_PER_PGD (PAGE_SIZE / sizeof(pgd_t))
|
|
#define PTRS_PER_PTE (PAGE_SIZE / sizeof(pte_t))
|
|
|
|
#define USER_PTRS_PER_PGD \
|
|
(CONFIG_NIOS2_KERNEL_MMU_REGION_BASE / PGDIR_SIZE)
|
|
|
|
#define PGDIR_SHIFT 22
|
|
#define PGDIR_SIZE (1UL << PGDIR_SHIFT)
|
|
#define PGDIR_MASK (~(PGDIR_SIZE-1))
|
|
|
|
/*
|
|
* ZERO_PAGE is a global shared page that is always zero: used
|
|
* for zero-mapped memory areas etc..
|
|
*/
|
|
extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)];
|
|
#define ZERO_PAGE(vaddr) (virt_to_page(empty_zero_page))
|
|
|
|
extern pgd_t swapper_pg_dir[PTRS_PER_PGD];
|
|
extern pte_t invalid_pte_table[PAGE_SIZE/sizeof(pte_t)];
|
|
|
|
/*
|
|
* (pmds are folded into puds so this doesn't get actually called,
|
|
* but the define is needed for a generic inline function.)
|
|
*/
|
|
static inline void set_pmd(pmd_t *pmdptr, pmd_t pmdval)
|
|
{
|
|
*pmdptr = pmdval;
|
|
}
|
|
|
|
static inline int pte_write(pte_t pte) \
|
|
{ return pte_val(pte) & _PAGE_WRITE; }
|
|
static inline int pte_dirty(pte_t pte) \
|
|
{ return pte_val(pte) & _PAGE_DIRTY; }
|
|
static inline int pte_young(pte_t pte) \
|
|
{ return pte_val(pte) & _PAGE_ACCESSED; }
|
|
|
|
#define pgprot_noncached pgprot_noncached
|
|
|
|
static inline pgprot_t pgprot_noncached(pgprot_t _prot)
|
|
{
|
|
unsigned long prot = pgprot_val(_prot);
|
|
|
|
prot &= ~_PAGE_CACHED;
|
|
|
|
return __pgprot(prot);
|
|
}
|
|
|
|
static inline int pte_none(pte_t pte)
|
|
{
|
|
return !(pte_val(pte) & ~(_PAGE_GLOBAL|0xf));
|
|
}
|
|
|
|
static inline int pte_present(pte_t pte) \
|
|
{ return pte_val(pte) & _PAGE_PRESENT; }
|
|
|
|
/*
|
|
* The following only work if pte_present() is true.
|
|
* Undefined behaviour if not..
|
|
*/
|
|
static inline pte_t pte_wrprotect(pte_t pte)
|
|
{
|
|
pte_val(pte) &= ~_PAGE_WRITE;
|
|
return pte;
|
|
}
|
|
|
|
static inline pte_t pte_mkclean(pte_t pte)
|
|
{
|
|
pte_val(pte) &= ~_PAGE_DIRTY;
|
|
return pte;
|
|
}
|
|
|
|
static inline pte_t pte_mkold(pte_t pte)
|
|
{
|
|
pte_val(pte) &= ~_PAGE_ACCESSED;
|
|
return pte;
|
|
}
|
|
|
|
static inline pte_t pte_mkwrite_novma(pte_t pte)
|
|
{
|
|
pte_val(pte) |= _PAGE_WRITE;
|
|
return pte;
|
|
}
|
|
|
|
static inline pte_t pte_mkdirty(pte_t pte)
|
|
{
|
|
pte_val(pte) |= _PAGE_DIRTY;
|
|
return pte;
|
|
}
|
|
|
|
static inline pte_t pte_mkyoung(pte_t pte)
|
|
{
|
|
pte_val(pte) |= _PAGE_ACCESSED;
|
|
return pte;
|
|
}
|
|
|
|
static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
|
|
{
|
|
const unsigned long mask = _PAGE_READ | _PAGE_WRITE | _PAGE_EXEC;
|
|
|
|
pte_val(pte) = (pte_val(pte) & ~mask) | (pgprot_val(newprot) & mask);
|
|
return pte;
|
|
}
|
|
|
|
static inline int pmd_present(pmd_t pmd)
|
|
{
|
|
return (pmd_val(pmd) != (unsigned long) invalid_pte_table)
|
|
&& (pmd_val(pmd) != 0UL);
|
|
}
|
|
|
|
static inline void pmd_clear(pmd_t *pmdp)
|
|
{
|
|
pmd_val(*pmdp) = (unsigned long) invalid_pte_table;
|
|
}
|
|
|
|
#define pte_pfn(pte) (pte_val(pte) & 0xfffff)
|
|
#define pfn_pte(pfn, prot) (__pte(pfn | pgprot_val(prot)))
|
|
#define pte_page(pte) (pfn_to_page(pte_pfn(pte)))
|
|
|
|
/*
|
|
* Store a linux PTE into the linux page table.
|
|
*/
|
|
static inline void set_pte(pte_t *ptep, pte_t pteval)
|
|
{
|
|
*ptep = pteval;
|
|
}
|
|
|
|
#define PFN_PTE_SHIFT 0
|
|
|
|
static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
|
|
pte_t *ptep, pte_t pte, unsigned int nr)
|
|
{
|
|
unsigned long paddr = (unsigned long)page_to_virt(pte_page(pte));
|
|
|
|
flush_dcache_range(paddr, paddr + nr * PAGE_SIZE);
|
|
for (;;) {
|
|
set_pte(ptep, pte);
|
|
if (--nr == 0)
|
|
break;
|
|
ptep++;
|
|
pte_val(pte) += 1;
|
|
}
|
|
}
|
|
#define set_ptes set_ptes
|
|
|
|
static inline int pmd_none(pmd_t pmd)
|
|
{
|
|
return (pmd_val(pmd) ==
|
|
(unsigned long) invalid_pte_table) || (pmd_val(pmd) == 0UL);
|
|
}
|
|
|
|
#define pmd_bad(pmd) (pmd_val(pmd) & ~PAGE_MASK)
|
|
|
|
static inline void pte_clear(struct mm_struct *mm,
|
|
unsigned long addr, pte_t *ptep)
|
|
{
|
|
pte_t null;
|
|
|
|
pte_val(null) = (addr >> PAGE_SHIFT) & 0xf;
|
|
|
|
set_pte(ptep, null);
|
|
}
|
|
|
|
/*
|
|
* Conversion functions: convert a page and protection to a page entry,
|
|
* and a page entry and page directory to the page they refer to.
|
|
*/
|
|
#define pmd_phys(pmd) virt_to_phys((void *)pmd_val(pmd))
|
|
#define pmd_pfn(pmd) (pmd_phys(pmd) >> PAGE_SHIFT)
|
|
#define pmd_page(pmd) (pfn_to_page(pmd_phys(pmd) >> PAGE_SHIFT))
|
|
|
|
static inline unsigned long pmd_page_vaddr(pmd_t pmd)
|
|
{
|
|
return pmd_val(pmd);
|
|
}
|
|
|
|
#define pte_ERROR(e) \
|
|
pr_err("%s:%d: bad pte %08lx.\n", \
|
|
__FILE__, __LINE__, pte_val(e))
|
|
#define pgd_ERROR(e) \
|
|
pr_err("%s:%d: bad pgd %08lx.\n", \
|
|
__FILE__, __LINE__, pgd_val(e))
|
|
|
|
/*
|
|
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
|
* are !pte_none() && !pte_present().
|
|
*
|
|
* Format of swap PTEs:
|
|
*
|
|
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
|
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
|
* E < type -> 0 0 0 0 0 0 <-------------- offset --------------->
|
|
*
|
|
* E is the exclusive marker that is not stored in swap entries.
|
|
*
|
|
* Note that the offset field is always non-zero if the swap type is 0, thus
|
|
* !pte_none() is always true.
|
|
*/
|
|
#define __swp_type(swp) (((swp).val >> 26) & 0x1f)
|
|
#define __swp_offset(swp) ((swp).val & 0xfffff)
|
|
#define __swp_entry(type, off) ((swp_entry_t) { (((type) & 0x1f) << 26) \
|
|
| ((off) & 0xfffff) })
|
|
#define __swp_entry_to_pte(swp) ((pte_t) { (swp).val })
|
|
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
|
|
|
static inline int pte_swp_exclusive(pte_t pte)
|
|
{
|
|
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
|
}
|
|
|
|
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
|
{
|
|
pte_val(pte) |= _PAGE_SWP_EXCLUSIVE;
|
|
return pte;
|
|
}
|
|
|
|
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
|
{
|
|
pte_val(pte) &= ~_PAGE_SWP_EXCLUSIVE;
|
|
return pte;
|
|
}
|
|
|
|
extern void __init paging_init(void);
|
|
extern void __init mmu_init(void);
|
|
|
|
void update_mmu_cache_range(struct vm_fault *vmf, struct vm_area_struct *vma,
|
|
unsigned long address, pte_t *ptep, unsigned int nr);
|
|
|
|
#define update_mmu_cache(vma, addr, ptep) \
|
|
update_mmu_cache_range(NULL, vma, addr, ptep, 1)
|
|
|
|
static inline int pte_same(pte_t pte_a, pte_t pte_b);
|
|
|
|
#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS
|
|
static inline int ptep_set_access_flags(struct vm_area_struct *vma,
|
|
unsigned long address, pte_t *ptep,
|
|
pte_t entry, int dirty)
|
|
{
|
|
if (!pte_same(*ptep, entry))
|
|
set_ptes(vma->vm_mm, address, ptep, entry, 1);
|
|
/*
|
|
* update_mmu_cache will unconditionally execute, handling both
|
|
* the case that the PTE changed and the spurious fault case.
|
|
*/
|
|
return true;
|
|
}
|
|
|
|
#endif /* _ASM_NIOS2_PGTABLE_H */
|