A 4mm hex is what takes the side panels off and he said that it's hitting the chain links. 3 to 3½mm of clearance aint much.
A 4mm fits in my bike with plenty of room to spare. In fact the next size up (5mm you use to remove the seat) fits between the tire ridge and chain on my bike.
Something has got to be out of whack.
Maybe I'm missing something here, but how can worn cush rubbers cause lateral play? There's a bearing in the sprocket mounting drum so the sprocket can NOT go side-to-side. Of course if the bearing is shot, the force of the chain pulling on it can throw things out of alignment.
I don't think this is one to play around with...time to take it to a good bike mechanic.