I have had oolong tea mixed with dried Osmanthus. The taste and fragrance was wonderful. I usually buy this tea each year when I visit Taiwan.
"The flowers of the osmanthus, or sweet olive tree, are simple, but their fragrance is very fine, something like apricot and jasmine combined. You can often find this small tree planted in city parks, sometimes in clusters of three or four to intensify the power of their perfume. The variety known as kinmokusei (golden olive) has warm apricot-colored flowers,"
Quote taken from The Japan Times online.