Well, don't take it as an offence, but in my opninion it's quite pointless to complain.
-The gray building is not relevant in any way. It contains no effects or wallpaper events. There is a trigger for the mad clown event, but that could be triggered in many other ways as well.
-It meant to be a secret area, therefore, you can't really blame the designer for hiding it quite well. The fact that many players are unable to find it at all, or without wandering for hours, is the proof that the designer did a great job hiding it.
-I said this many times, on this site as well on Uboachan: this is an exploration game. If you don't like wandering, don't play it.