5 mm change is generally a doable offset change with minimal to moderate effect on the handling and suspension. 10 mm is a significant change and will have noticeable effects on handling, even to a normal person. This includes loss of steering feel (and this is already pretty bad on the impreza), tread squirm, decreased stability under braking and accelerating, worsened bump steer/road imperfection following character (where the steering wheel jerks to one side going over a bump), and the usual increased wheel bearing and suspension wear. These are all things that will affect daily driving, not just trying to trim tenths of a second off your lap time.
Stock offset is 55 mm on the Impreza and I wouldn't venture lower than 48 mm if you want to retain similar driving characteristics to what you have now.
As an FYI, most tire shops show rims that physically fit, not rims that are proper for the handling and suspension of your vehicle. Subarus have much higher offset than almost all other vehicles and as such there are very few rims that properly fit without switching over to aftermarket suspension and adjusting the suspension geometry.