It's an educated guess.
You have to understand how these things show up in populations. A carcinogen that in normal exposure increases the chance of a rare cancer by twentyfold, say from one in 200,000 to one in ten thousand, would be pretty powerful, but it would still only have a direct affect on less than 0.01% of the population. If you or a loved one is in that 0.01%, the effect is devastating, and depending on the cost involved in banning that chemical, a ban may be excellent public policy, but the average effect will still be quite small.
Of course these things are additive to some extent, and there are always cases of people who are subject to increased exposure (agricultural workers, for example) and people with unusual sensitivity, such as women in preganancy and small children.
But I was comparing these effects to lifestyle choices that can have enormous impact on individual health in almost every person in society, like smoking, diet and exercise, and to societal changes like the development of modern agriculture that have had literally incalculable effects on public health. Almost anything is going to be dwarfed by these sort of issues, just like your cholesterol level isn't too important when your house is on fire.
Finally, keep in mind that most developed nations have governmental standards regulating toxins in food and the environment. These are extremely difficult to craft and enforce effectively, but they do their best where there are easily detectable health effects at a population level - meaning a noticeable average impact. A substance with an unidentified toxic effect is one that almost by definition has a small average effect at the level of the whole population. If it were a large effect at that scale, it would be hard not to detect it (if someone is looking for it).
The differences in health between people who eat foods high in antioxidants and those who eat healthy diets with low antioxidant levels is small enough that it is still in dispute. We don't know that drinking tea is better for you than not drinking tea. We also don't know that eating organics is better for you than not eating organics. Both are probably true, but the difference is small enough to be hard to find, even when you're looking hard for it. Compare that to something like eating a healthy diet or even having enough food at all or access to clean drinking water. It's not just comparing apples and oranges, it's comparing apples and atom bombs!