I think part of the joke falling flat for you might be your excellent familiarity with Japanese tea culture, which doesn't have strong gender associations in the West. (In fact, I'd say any association would lean toward the masculine.) This is not something a great majority of Americans share. In fact, I hate to say it...but I'm not really sure a good percentage of my countrymen could even find Japan on a map. I recently had a woman approach me in a store holding a tetsubin that was prominently labeled "MADE IN JAPAN," then ask me if I thought there was any Chinese lead in it. Honestly, I got the impression that she thought that Japan and China were the same place.
Anyway, I think it is safe to say that when you say 'tea' the associations most people will list are England, doilies, flowered china, crumpets, scones, cakes, and an inexhaustible set of pretentious manners. Unfortunately, these artifacts tend to 'read' as feminine...and the idea of a burly man not only sitting down to a dainty cup and a spread of wee little treats, but doing it of his own volition is rather comical.