Clicking on the third (bright red) link [“Access Requests & Invitations”] in the left column of this post’s featured image (above) takes you to this handy dashboard below.  To change settings for a specific individual, use the three dots (general advice in SharePoint: when in doubt, hover on the three dots) to view th

 

So you can un-invite people impolite enough not to respond to the collaboration invitation you so thoughtfully extended — or remind them of something that can only be categorized as socially suicidal tendency negligence <smirk>.