Skip to content

Commit 81cb947

Browse files
committed
smp_load_acquire(ring->cq.khead) in __io_uring_peek_cqe() for
consistent dereferencing __io_uring_peek_cqe() is on the critical path for the user to consume completion-queue entries. It load_acquires the tail, loads the head, uses the head to index into the cq ring. The C memory consistency model does not enforce ordering for address dependencies, like the one seen in: ``` LD1: unsigned head = *ring->cq.khead; ... LD2: cqe = &ring->cq.cqes[(head & mask) << shift]; ``` where LD2 is address-dependent on LD1. This means that LD2 could load a stale cqe due to LD1 being ordered after LD2. If instead, LD1 was a smp_load_acquire, the ordering would be enforced and correct cqes would be loaded. Signed-off by: Soham Bagchi <soham.bagchi@utah.edu> Co-authored by: Marco Elver <elver@google.com> Co-authored by: Vijay Nagarajan <vijay@cs.utah.edu> Co-authored by: Ryan Stutsman <stutsman@cs.utah.edu>
1 parent 2fd7904 commit 81cb947

1 file changed

Lines changed: 6 additions & 1 deletion

File tree

src/include/liburing.h

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1576,7 +1576,12 @@ IOURINGINLINE int __io_uring_peek_cqe(struct io_uring *ring,
15761576

15771577
do {
15781578
unsigned tail = io_uring_smp_load_acquire(ring->cq.ktail);
1579-
unsigned head = *ring->cq.khead;
1579+
1580+
/**
1581+
* A load_acquire on the head prevents reordering with the
1582+
* cqe load below, ensuring that we see the correct cq entry.
1583+
*/
1584+
unsigned head = io_uring_smp_load_acquire(ring->cq.khead);
15801585

15811586
cqe = NULL;
15821587
available = tail - head;

0 commit comments

Comments
 (0)