I think what you are describing is not silver fish
- they are pretty big, not like the little white dot bugs. Although silverfish can show up from time to time, but mostly to eat the paper wrappers.
Like Marshaln said, they are indicative of a wetter storage environment. I have also seen them on some fresh (one year or less) ripe teas, left over from the humidity of the pile. Only in one case can I remember their presence being so severe that I abandoned a cake.
If you keep the cake for long enough, they usually just go away. Another method I have heard, is to break off a chunk of cake and toss it in the freezer for a little bit to kill them, and then brew. Not sure if that effects the flavor and in what way - I imagine a brief snap of cold would not destroy a tea, as many green teas are refrigerated for extended periods of time without folly. Then you will be brewing with some small bugs, but if you are into puer, such things probably don't bother you anyhow.