Drivers are never penalized for rejecting trips.
As an Independent Contractor, drivers decide which trips suit their needs and availability. This gives the requestors an immediate answer so they can continue to search for the next driver.
Not responding tells the requestors you either didn’t get the request or you are no longer interested in driving.
Several “unanswered” requests may preclude you from being sent future requests since they aren’t getting an answer. So always hit either “accept” or “reject”.