Remove User Button Clickable Area Very Large

[streatim]

While working on the Workflow/UserGroup Administration form, I noticed that the button to remove assigned users takes up the entire width of the box. This caught at least me off guard when I accidentally clicked while in the blank area of the row, removing the user in that row. See the attached screenshot below - the yellow background encompasses the entire area of the screen where a user can click to remove a user:

It looks like this is from the .form-grid class in style/css (:971 for my screen width, but I know it's defined elsewhere), where the grid-template-columns is flexing the second value. I'm not wholly sure what the best solution would be (or if this is just a "me issue"); while I feel like while there should be some allowance clicking outside the X to remove the user, having the entire row be the clickable button increases the likelihood (in my head) that users will be accidentally clicking and removing a user.

I'm not sure what the recommended solution for this would be, and thought I'd at least pass it along for discussion or to see if folks with more front-end experience would have a better sense for how to approach it (or could tell me I'm off base on being concerned about it). Thanks!