I certainly think it's possible. I would like to believe it is so, but I haven't seen any clear evidence of it. As for the universe being way too big for us to be the only game, I'd like to believe that is true too. But we haven't fully explored our own planet, much less our own solar system, our own galaxy, or the neighboring galaxies, much less the billions of galaxies that we've so far detected with our telescopes. We have so little first-hand observation of the universe, that we really don't have enough data to assume that life is the rule or the exception. If we can visit or send a probe to another oxygen atmosphere planet in the Goldilocks orbit of its sun and find it teaming with life, I'd be more willing to bet that there is lots of life out there and probably some of it is intelligent.
Another possibility is that there was intelligent life in some of the older solar systems in the galaxy, but their stars went super nova before we were born, or they reached the nuclear age and wiped themselves out, or they had a civilization that lasted millions of years and then died out for reasons we can't imagine.
I'll tell you one thing: If Harold White at NASA manages to perfect the Alcubierre "warp drive" before I'm 90, I think I'll volunteer for the first trip. :-)