You can make your blazer less “country club” by changing the buttons from gold to a dark navy blue (if they aren’t already).  Changing buttons is a fairly simple task for a tailor and shouldn’t be too pricey. 
   
As for what to wear with your blue blazer, you can’t go wrong with jeans, as long as they aren’t too similar in color.  You can wear blue, of course, but you can experiment with other colors, such as a darker gray denim.  For more formal occasions, you can still look young in gray pants, such as chinos or flannel trousers. Earthy tones such as brown or olive also work here.  Avoid khakis, which will give your blue blazer an older feel.  As for fit, look for pants that are well fitted and slimmer rather than baggier.

For shirts, a crisp white oxford works well here (when doesn’t it work well?).  You can make the blue blazer more casual by wearing a graphic tee.  For something different, though, try a windowpane plaid or smaller gingham shirt, but don’t get too crazy here.  For those blazers that are dark and inky (the ones I like), a skinny-but-not-too-skinny tie looks great here especially in a dark purpleish/gray. 

Complete the look with brown shoes, a brown belt and even a crisp white pocket square.  You could even get away with wearing white sneakers, but be careful – if they’re too athletic-looking you might give off the Jerry Seinfeld vibe. What to wear?  You can’t go wrong with lowtop Jack Purcells.