UPSTREAM: io_uring: ensure that cached task references are always put on exit
commit e775f93f2ab976a2cdb4a7b53063cbe890904f73 upstream.
io_uring caches task references to avoid doing atomics for each of them
per request. If a request is put from the same task that allocated it,
then we can maintain a per-ctx cache of them. This obviously relies
on io_uring always pruning caches in a reliable way, and there's
currently a case off io_uring fd release where we can miss that.
One example is a ring setup with IOPOLL, which relies on the task
polling for completions, which will free them. However, if such a task
submits a request and then exits or closes the ring without reaping
the completion, then ring release will reap and put. If release happens
from that very same task, the completed request task refs will get
put back into the cache pool. This is problematic, as we're now beyond
the point of pruning caches.
Manually drop these caches after doing an IOPOLL reap. This releases
references from the current task, which is enough. If another task
happens to be doing the release, then the caching will not be
triggered and there's no issue.
Cc: [email protected]
Fixes: e98e49b2bbf7 ("io_uring: extend task put optimisations")
Reported-by: Homin Rhee <[email protected]>
Change-Id: I9495121af065424141fa9c39840ab9aa91f45c72
Signed-off-by: Jens Axboe <[email protected]>
Signed-off-by: Sasha Levin <[email protected]>
Bug: 268174392
(cherry picked from commit e9c6556708d3b1f77f1a9e08881f8bb01e98e919)
Signed-off-by: Greg Kroah-Hartman <[email protected]>
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 07ad834..713a243 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -9523,6 +9523,10 @@ static void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx)
/* if we failed setting up the ctx, we might not have any rings */
io_iopoll_try_reap_events(ctx);
+ /* drop cached put refs after potentially doing completions */
+ if (current->io_uring)
+ io_uring_drop_tctx_refs(current);
+
INIT_WORK(&ctx->exit_work, io_ring_exit_work);
/*
* Use system_unbound_wq to avoid spawning tons of event kworkers
