Jens Axboe 7fcbbaf183 mm/filemap.c: avoid always dirtying mapping->flags on O_DIRECT ...

In some testing I ran today (some fio jobs that spread over two nodes),
we end up spending 40% of the time in filemap_check_errors().  That
smells fishy.  Looking further, this is basically what happens:

blkdev_aio_read()
    generic_file_aio_read()
        filemap_write_and_wait_range()
            if (!mapping->nr_pages)
                filemap_check_errors()

and filemap_check_errors() always attempts two test_and_clear_bit() on
the mapping flags, thus dirtying it for every single invocation.  The
patch below tests each of these bits before clearing them, avoiding this
issue.  In my test case (4-socket box), performance went from 1.7M IOPS
to 4.0M IOPS.

Signed-off-by: Jens Axboe <axboe@fb.com>
Acked-by: Jeff Moyer <jmoyer@redhat.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

2014-05-23 09:37:29 -07:00

..

backing-dev.c

bdi: avoid oops on device removal

2014-04-03 16:20:49 -07:00

balloon_compaction.c

mm: print more details for bad_page()

2014-01-23 16:36:50 -08:00

bootmem.c

mm/bootmem.c: remove unused local `map'

2013-11-13 12:09:09 +09:00

bounce.c

block: Convert bio_for_each_segment() to bvec_iter

2013-11-23 22:33:49 -08:00

cleancache.c

mm: dump page when hitting a VM_BUG_ON using VM_BUG_ON_PAGE

2014-01-23 16:36:50 -08:00

compaction.c

mm/compaction: make isolate_freepages start at pageblock boundary

2014-05-06 13:04:59 -07:00

debug-pagealloc.c

mm, x86: Remove debug_pagealloc_enabled

2011-12-06 09:24:07 +01:00

dmapool.c

dmapool: make DMAPOOL_DEBUG detect corruption of free marker

2012-12-11 17:22:24 -08:00

early_ioremap.c

mm: create generic early_ioremap() support

2014-04-07 16:36:15 -07:00

fadvise.c

teach SYSCALL_DEFINE<n> how to deal with long long/unsigned long long

2013-03-03 22:46:22 -05:00

failslab.c

switch debugfs to umode_t

2012-01-03 22:54:56 -05:00

filemap_xip.c

seqcount: Add lockdep functionality to seqcount/seqlock structures

2013-11-06 12:40:26 +01:00

filemap.c

mm/filemap.c: avoid always dirtying mapping->flags on O_DIRECT

2014-05-23 09:37:29 -07:00

fremap.c

mm: fix bad rss-counter if remap_file_pages raced migration

2014-03-19 16:21:49 -07:00

frontswap.c

frontswap: fix incorrect zeroing and allocation size for frontswap_map

2013-06-12 16:29:46 -07:00

highmem.c

Merge tag 'virtio-next-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux

2012-12-20 08:37:05 -08:00

huge_memory.c

thp: close race between split and zap huge pages

2014-04-18 16:40:09 -07:00

hugetlb_cgroup.c

cgroup: drop const from @buffer of cftype->write_string()

2014-03-19 10:23:54 -04:00

hugetlb.c

hugetlb: ensure hugepage access is denied if hugepages are not supported

2014-05-06 13:04:58 -07:00

hwpoison-inject.c

mm/hwpoison: add '#' to hwpoison_inject

2014-01-21 16:19:48 -08:00

init-mm.c

atomic: use <linux/atomic.h>

2011-07-26 16:49:47 -07:00

internal.h

mm/readahead.c: inline ra_submit

2014-04-07 16:35:58 -07:00

interval_tree.c

mm: add CONFIG_DEBUG_VM_RB build option

2012-10-09 16:22:42 +09:00

iov_iter.c

take iov_iter stuff to mm/iov_iter.c

2014-04-01 23:19:30 -04:00

Kconfig

parisc,metag: Do not hardcode maximum userspace stack size

2014-05-15 00:01:41 +01:00

Kconfig.debug

mm: more intensive memory corruption debugging

2012-01-10 16:30:42 -08:00

kmemcheck.c

kmemcheck: Fix build errors due to missing slab.h

2010-03-30 22:02:32 +09:00

kmemleak-test.c

kmemleak: remove memset by using kzalloc

2011-01-27 18:31:51 +00:00

kmemleak.c

mm: postpone the disabling of kmemleak early logging

2014-05-11 17:55:48 +09:00

ksm.c

mm: close PageTail race

2014-03-04 07:55:47 -08:00

list_lru.c

mm: keep page cache radix tree nodes in check

2014-04-03 16:21:01 -07:00

maccess.c

mm: Map most files to use export.h instead of module.h

2011-10-31 09:20:12 -04:00

madvise.c

mm/hwpoison: fix traversal of hugetlbfs pages to avoid printk flood

2013-09-30 14:31:02 -07:00

Makefile

Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs

2014-04-12 14:49:50 -07:00

memblock.c

mm/memblock.c: use PFN_PHYS()

2014-04-07 16:35:58 -07:00

memcontrol.c

mm: filemap: update find_get_pages_tag() to deal with shadow entries

2014-05-06 13:04:59 -07:00

memory_hotplug.c

mm/memory_hotplug.c: move register_memory_resource out of the lock_memory_hotplug

2014-01-23 16:36:52 -08:00

memory-failure.c

hwpoison, hugetlb: lock_page/unlock_page does not match for handling a free hugepage

2014-05-23 09:37:29 -07:00

memory.c

mm: split 'tlb_flush_mmu()' into tlb flushing and memory freeing parts

2014-04-25 16:05:40 -07:00

mempolicy.c

mm, mempolicy: remove per-process flag

2014-04-07 16:35:54 -07:00

mempool.c

mempool: add unlikely and likely hints

2014-04-07 16:35:55 -07:00

migrate.c

mm: fix swapops.h:131 bug if remap_file_pages raced migration

2014-03-20 22:09:09 -07:00

mincore.c

mm + fs: prepare for non-page entries in page cache radix trees

2014-04-03 16:21:00 -07:00

mlock.c

mm: try_to_unmap_cluster() should lock_page() before mlocking

2014-04-07 16:35:57 -07:00

mm_init.c

mm: bring back /sys/kernel/mm

2014-01-27 21:02:39 -08:00

mmap.c

mm: per-thread vma caching

2014-04-07 16:35:53 -07:00

mmu_context.c

sched/mm: call finish_arch_post_lock_switch in idle_task_exit and use_mm

2014-02-21 08:50:17 +01:00

mmu_notifier.c

mm: audit/fix non-modular users of module_init in core code

2014-01-23 16:36:52 -08:00

mmzone.c

mm: numa: Change page last {nid,pid} into {cpu,pid}

2013-10-09 14:47:45 +02:00

mprotect.c

mm: move mmu notifier call from change_protection to change_pmd_range

2014-04-07 16:35:50 -07:00

mremap.c

mm, thp: close race between mremap() and split_huge_page()

2014-05-11 17:55:48 +09:00

msync.c

sanitize vfs_fsync calling conventions

2010-05-21 18:31:21 -04:00

nobootmem.c

mm/nobootmem.c: mark function as static

2014-04-03 16:21:02 -07:00

nommu.c

mm: fix 'ERROR: do not initialise globals to 0 or NULL' and coding style

2014-04-07 16:35:55 -07:00

oom_kill.c

mm, oom: base root bonus on current usage

2014-01-30 16:56:56 -08:00

page_alloc.c

mm/page_alloc.c: change mm debug routines back to EXPORT_SYMBOL

2014-04-07 16:35:59 -07:00

page_cgroup.c

mm/page_cgroup.c: mark functions as static

2014-04-03 16:21:02 -07:00

page_io.c

Merge branch 'for-3.14/core' of git://git.kernel.dk/linux-block

2014-01-30 11:19:05 -08:00

page_isolation.c

mm: memory-hotplug: enable memory hotplug to handle hugepage

2013-09-11 15:57:48 -07:00

page-writeback.c

mm/page-writeback.c: fix divide by zero in pos_ratio_polynom

2014-05-06 13:04:58 -07:00

pagewalk.c

mm/pagewalk.c: fix walk_page_range() access of wrong PTEs

2013-10-30 14:27:03 -07:00

percpu-km.c

percpu: clear memory allocated with the km allocator

2010-10-02 10:28:42 +03:00

percpu-vm.c

mm: fix kernel-doc warnings

2012-06-20 14:39:36 -07:00

percpu.c

percpu: make pcpu_alloc_chunk() use pcpu_mem_free() instead of kfree()

2014-04-14 16:18:06 -04:00

pgtable-generic.c

mm: fix TLB flush race between migration, and change_protection_range

2013-12-18 19:04:51 -08:00

process_vm_access.c

Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs

2014-04-12 14:49:50 -07:00

quicklist.c

mm: delete various needless include <linux/module.h>

2011-10-31 09:20:11 -04:00

readahead.c

mm/readahead.c: inline ra_submit

2014-04-07 16:35:58 -07:00

rmap.c

mm: try_to_unmap_cluster() should lock_page() before mlocking

2014-04-07 16:35:57 -07:00

shmem.c

mm: Initialize error in shmem_file_aio_read()

2014-04-13 14:10:26 -07:00

slab_common.c

slub: use sysfs'es release mechanism for kmem_cache

2014-05-06 13:04:59 -07:00

slab.c

slab: Fix off by one in object max number tests.

2014-05-05 20:38:49 -07:00

slab.h

slub: use sysfs'es release mechanism for kmem_cache

2014-05-06 13:04:59 -07:00

slob.c

mm: slab/slub: use page->list consistently instead of page->lru

2014-04-11 10:06:06 +03:00

slub.c

slub: use sysfs'es release mechanism for kmem_cache

2014-05-06 13:04:59 -07:00

sparse-vmemmap.c

mm/sparse: use memblock apis for early memory allocations

2014-01-21 16:19:47 -08:00

sparse.c

mm: use macros from compiler.h instead of __attribute__((...))

2014-04-07 16:35:54 -07:00

swap_state.c

swap: add a simple detector for inappropriate swapin readahead

2014-02-06 13:48:51 -08:00

swap.c

mm: thrash detection-based file cache sizing

2014-04-03 16:21:01 -07:00

swapfile.c

mm/swap: fix race on swap_info reuse between swapoff and swapon

2014-02-06 13:48:51 -08:00

truncate.c

mm: filemap: update find_get_pages_tag() to deal with shadow entries

2014-05-06 13:04:59 -07:00

util.c

nick kvfree() from apparmor

2014-05-06 14:02:53 -04:00

vmacache.c

mm: don't pointlessly use BUG_ON() for sanity check

2014-04-28 14:24:09 -07:00

vmalloc.c

mm/vmalloc.c: enhance vm_map_ram() comment

2014-04-07 16:35:55 -07:00

vmpressure.c

arm, pm, vmpressure: add missing slab.h includes

2014-02-03 13:24:01 -05:00

vmscan.c

revert "mm: vmscan: do not swap anon pages just because free+file is low"

2014-05-06 13:04:59 -07:00

vmstat.c

Merge tag 'cpu-hotplug-3.15-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm

2014-04-07 14:55:46 -07:00

workingset.c

mm: keep page cache radix tree nodes in check

2014-04-03 16:21:01 -07:00

zbud.c

mm/zbud: fix some trivial typos in comments

2013-09-11 15:57:35 -07:00

zsmalloc.c

zsmalloc: Fix CPU hotplug callback registration

2014-03-20 13:43:45 +01:00

zswap.c

Merge branch 'akpm' (incoming from Andrew)

2014-04-07 16:38:06 -07:00