the smaller the pipe the larger the pressure drop from point a to point b. The higher velocity creates more friction on the pipe walls, and is overly sensitive to directional changes (pipe bends)
There is no hard rule on what diameter the pipe should be for a given setup, but I would at least go over the compressor housing exit ID by 25%. IE, if the ID is 1.5" go with 2"
larger pipe, at an extreme would slow down the boost hit, because the vollume of air takes longer to pressurize, but it takes a lot of excess to be noticable. Where as, if you are too small and have a large pressure drop, you are moving the compressor further up the compressor map, causing more heat in the intake and possibly pushing the turbo out of its efficiency range (that is the extreme of course). The heat in the intake is very bad. More heat = detonation, which is how you blow up motors.