Trouble is brewing for coffee lovers in California, where a judge ruled that sellers must post scary warnings about cancer risks. But how frightened should we be of a daily cup of joe? Not very, some scientists and available evidence seem to suggest.

Scientific concerns about coffee have eased in recent years, and many studies even suggest it can help health.

"At the minimum, coffee is neutral. If anything, there is fairly good evidence of the benefit of coffee on cancer," said Dr. Edward Giovannucci, a nutrition expert at the Harvard School of Public Health.

CBS News medical contributor Dr. David Agus, director of the Westside Cancer Center at USC, says he believes it is too early to put this kind of blanket warning on coffee.

"When you put a bold declaration that 'X may cause cancer' when there isn't data to that effect in humans, to me it causes panic rather than informed knowledge," he told "CBS This Morning."

The World Health Organization's cancer agency moved coffee off the "possible carcinogen" list two years ago, though it says evidence is insufficient to rule out any possible role.

The current flap isn't about coffee itself, but a chemical called acrylamide that's made when the beans are roasted. Government agencies call it a probable or likely carcinogen, based on animal research, and a group sued to require coffee sellers to warn of that under a California law passed by voters in 1986.

Coffee and your health

The problem: No one knows what levels are safe or risky for people. The U.S. Environmental Protection Agency sets acrylamide limits for drinking water, but there aren't any for food.

"A cup of coffee a day, exposure probably is not that high," and probably should not change your habit, said Dr. Bruce Y. Lee of Johns Hopkins Bloomberg School of Public Health. "If you drink a lot of cups a day, this is one of the reasons you might consider cutting that down."

Here's what's known about the risks.

The chemical

Start with the biggest known risk factor for cancer - smoking - which generates acrylamide . In the diet, French fries, potato chips, crackers, cookies, cereal and other high-carbohydrate foods contain it as a byproduct of roasting, baking, toasting or frying.

Food and Drug Administration tests of acrylamide levels found they ranged from 175 to 351 parts per billion (a measure of concentration for a contaminant) for six brands of coffee tested; the highest was for one type of decaf coffee crystals. By comparison, French fries at one fast food chain ranged from 117 to 313 parts per billion, depending on the location tested. Some commercial fries had more than 1,000.

Even some baby foods contain acrylamide, such as teething biscuits and crackers. One brand of organic sweet potatoes tested as having 121 parts per billion.

What's the risk?

The "probable" or "likely" carcinogen label is based on studies of animals given high levels of acrylamide in drinking water. But people and rodents absorb the chemical at different rates and metabolize it differently, so its relevance to human health is unknown.

A group of 23 scientists convened by the WHO's cancer agency in 2016 looked at coffee - not acrylamide directly - and decided coffee was unlikely to cause breast, prostate or pancreatic cancer,  and that it seemed to lower the risks for liver and uterine cancers. Evidence was inadequate to determine its effect on dozens of other cancer types.