Pavel Begunkov 873ebaf3c1 io_uring: fix overflow resched cqe reordering ...

[ Upstream commit a7d755ed9c ]

Leaving the CQ critical section in the middle of a overflow flushing
can cause cqe reordering since the cache cq pointers are reset and any
new cqe emitters that might get called in between are not going to be
forced into io_cqe_cache_refill().

Fixes: eac2ca2d68 ("io_uring: check if we need to reschedule during overflow flush")
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/90ba817f1a458f091f355f407de1c911d2b93bbf.1747483784.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Sasha Levin <sashal@kernel.org>

2025-05-29 11:03:17 +02:00

..

advise.c

io_uring/advise: support 64-bit lengths

2024-06-16 14:54:55 -06:00

advise.h

…

alloc_cache.h

io_uring/alloc_cache: switch to array based caching

2024-04-15 08:10:25 -06:00

cancel.c

io_uring: fix warnings on shadow variables

2024-04-15 08:10:26 -06:00

cancel.h

io_uring: fix cancellation overwriting req->flags

2024-06-13 19:25:28 -06:00

epoll.c

io_uring: undeprecate epoll_ctl support

2023-05-26 20:22:41 -06:00

epoll.h

…

eventfd.c

io_uring/eventfd: ensure io_eventfd_signal() defers another RCU period

2025-01-17 13:40:58 +01:00

eventfd.h

io_uring/eventfd: move eventfd handling to separate file

2024-06-16 14:54:55 -06:00

fdinfo.c

io_uring/fdinfo: annotate racy sq/cq head/tail reads

2025-05-29 11:03:11 +02:00

fdinfo.h

…

filetable.c

io_uring/filetable: don't unnecessarily clear/reset bitmap

2024-05-08 08:27:45 -06:00

filetable.h

io_uring: expand main struct io_kiocb flags to 64-bits

2024-02-08 13:27:03 -07:00

fs.c

io_uring/fs: consider link->flags when getting path for LINKAT

2023-11-20 09:01:42 -07:00

fs.h

…

futex.c

futex: Pass in task to futex_queue()

2025-03-22 12:54:14 -07:00

futex.h

io_uring/alloc_cache: switch to array based caching

2024-04-15 08:10:25 -06:00

io_uring.c

io_uring: fix overflow resched cqe reordering

2025-05-29 11:03:17 +02:00

io_uring.h

io_uring/sqpoll: ensure task state is TASK_RUNNING when running task_work

2024-10-17 08:38:04 -06:00

io-wq.c

io-wq: backoff when retrying worker creation

2025-03-22 12:54:19 -07:00

io-wq.h

io_uring/io-wq: make io_wq_work flags atomic

2024-06-16 14:54:55 -06:00

kbuf.c

io_uring/kbuf: reject zero sized provided buffers

2025-04-20 10:15:40 +02:00

kbuf.h

io_uring/kbuf: add support for incremental buffer consumption

2024-08-29 08:44:58 -06:00

Makefile

io_uring: add GCOV_PROFILE_URING Kconfig option

2024-08-30 10:52:02 -06:00

memmap.c

io_uring: check for overflows in io_pin_pages

2024-12-05 14:02:50 +01:00

memmap.h

io_uring: move mapping/allocation helpers to a separate file

2024-04-15 08:10:26 -06:00

msg_ring.c

io_uring/msg: initialise msg request opcode

2025-05-29 11:02:03 +02:00

msg_ring.h

io_uring/msg_ring: add an alloc cache for io_kiocb entries

2024-06-24 08:39:55 -06:00

napi.c

io_uring: user registered clockid for wait timeouts

2024-08-25 08:27:01 -06:00

napi.h

io_uring/napi: postpone napi timeout adjustment

2024-08-25 08:27:01 -06:00

net.c

io_uring/net: fix io_req_post_cqe abuse by send bundle

2025-04-20 10:15:40 +02:00

net.h

io_uring: Introduce IORING_OP_LISTEN

2024-06-19 07:57:21 -06:00

nop.c

io_uring: support to inject result for NOP

2024-05-10 06:09:45 -06:00

nop.h

…

notif.c

io_uring/notif: disable LAZY_WAKE for linked notifs

2024-04-30 13:06:27 -06:00

notif.h

io_uring/notif: implement notification stacking

2024-04-22 19:31:18 -06:00

opdef.c

io_uring: Fix probe of disabled operations

2024-06-19 08:58:00 -06:00

opdef.h

io_uring: Fix probe of disabled operations

2024-06-19 08:58:00 -06:00

openclose.c

io_uring: enable audit and restrict cred override for IORING_OP_FIXED_FD_INSTALL

2024-01-23 15:25:14 -07:00

openclose.h

io_uring/openclose: add support for IORING_OP_FIXED_FD_INSTALL

2023-12-12 07:42:57 -07:00

poll.c

io_uring/net: don't retry connect operation on EPOLLERR

2025-02-17 10:05:38 +01:00

poll.h

io_uring/poll: shrink alloc cache size to 32

2024-04-15 08:10:25 -06:00

refs.h

io_uring: always do atomic put from iowq

2025-05-02 07:59:21 +02:00

register.c

io_uring: clean up a type in io_uring_register_get_file()

2024-09-16 12:04:10 -06:00

register.h

io_uring: clean up a type in io_uring_register_get_file()

2024-09-16 12:04:10 -06:00

rsrc.c

io_uring/rsrc: require cloned buffers to share accounting contexts

2025-02-01 18:39:40 +01:00

rsrc.h

io_uring/rsrc: change ubuf->ubuf_end to length tracking

2024-09-15 09:15:22 -06:00

rw.c

block: add a rq_list type

2025-04-25 10:48:06 +02:00

rw.h

io_uring/alloc_cache: switch to array based caching

2024-04-15 08:10:25 -06:00

slist.h

io_uring: silence variable ‘prev’ set but not used warning

2023-03-09 10:10:58 -07:00

splice.c

splice: return type ssize_t from all helpers

2023-12-12 16:19:59 +01:00

splice.h

…

sqpoll.c

io_uring/sqpoll: Increase task_work submission batch size

2025-05-18 08:24:57 +02:00

sqpoll.h

io_uring/sqpoll: statistics of the true utilization of sq threads

2024-03-01 06:28:19 -07:00

statx.c

vfs: retire user_path_at_empty and drop empty arg from getname_flags

2024-06-05 17:03:57 +02:00

statx.h

…

sync.c

io_uring: for requests that require async, force it

2023-01-29 15:18:26 -07:00

sync.h

…

tctx.c

io_uring/tctx: work around xa_store() allocation error issue

2024-12-14 20:04:10 +01:00

tctx.h

io_uring: simplify __io_uring_add_tctx_node

2022-10-07 12:25:30 -06:00

timeout.c

io_uring/timeout: fix multishot updates

2025-01-17 13:40:51 +01:00

timeout.h

io_uring: remove unused return from io_disarm_next

2022-09-21 13:15:01 -06:00

truncate.c

io_uring: add support for ftruncate

2024-02-09 09:04:39 -07:00

truncate.h

io_uring: add support for ftruncate

2024-02-09 09:04:39 -07:00

uring_cmd.c

io_uring/uring_cmd: remove dead req_has_async_data() check

2025-02-21 14:01:20 +01:00

uring_cmd.h

io_uring/alloc_cache: switch to array based caching

2024-04-15 08:10:25 -06:00

waitid.c

io_uring/waitid: don't abuse io_tw_state

2025-02-21 14:01:20 +01:00

waitid.h

io_uring: add IORING_OP_WAITID support

2023-09-21 12:04:45 -06:00

xattr.c

vfs: retire user_path_at_empty and drop empty arg from getname_flags

2024-06-05 17:03:57 +02:00

xattr.h

…