I'll talk to practically anyone who comes by my office as long as I'm not currently working on something that needs immediate attention. Then I don't even bother to look up, I just quickly say "No time right now, sorry." No one ever seems offended by it.