CHUCK: The easy answer to your question is: You can't know for sure. At least until you get to know him better. And even then...
The clues that you're taking right now could be right, but could just as likely be wrong. Let's face it: Work gossip is hardly 100% reliable. Think of anything you may have heard about yourself on the work grapevine and let that be your judge of its veracity. All men are not the same, contrary to what you may have been told. So if he interacts with his female co-workers in a way that seems strange to you, there might not be anything to it. That may just be his way.
Your friend may be concealing his true sexuality for personal and business reasons. The current climate isn't always welcoming to gays, especially men. But if you really are interested in getting to know him better, spend more time hanging out with him away from work. Maybe arrange to hang out with him at his place. The only thing that going to prove things to you one way or the other will be more time and more information.
GARLAND: You say you're "hanging out" with this guy. Now to me there are two different kinds of 'hanging out' - one kind where you go downtown after work and screen a movie, then the other kind where he goes downtown after work and you scream do me!!! So, I'm gonna try to touch on both.
If you are intimate with this guy, you're REALLY going to have to slow things down to a halt and seriously talk to this guy. Men on the DL play by some dangerous rules, and more often than not, it is the women in their lives that truly end up paying the price. There are plenty of little games you could play to check his preference - I've heard of women that tease questionable guys with gay porn and with other gay men and stuff like that. I don't know whether it works or not, but I think you should approach the subject gently with him, but make sure you get your concerns across to him.
I wouldn't give in too much to office gossip, I've seen women call guys gay because the guy didn't want to get with them, and well... I've also seen guys called gay because THEY were. You just never can tell.
Try talking to him and see what he has to say. But now if this guy is a buddy of yours that you share a decent plutonic relationship with, unless you have a problem with gay people, I'd just leave it alone. If you enjoy his company and he enjoys yours, I'd just leave it alone. Like Chuck said - basically, being gay today comes with so many stigmas that most guys just want to keep their sexuality to themselves.
Take your time with this guy, feel him out, ask him questions and share your concerns and thoughts. Communications is the key.