This has been addresses in another thread, see here =>
https://www.watchrepairtalk.com/topic/10086-radon-hazards-of-luminous-timepieces/?tab=comments#comment-91769
The short answer, is no, you should not be worried. However, you should take a little extra care in future.
There was a very small risk when handling that single watch, but that risk is not worth loosing any sleep over.
You almost certainly got more exposure to radiation that day, from simply going about your daily business.
The trouble comes from cumulative exposure without protection. Read the thread I linked above, and you will get a better idea of what I mean.
It is true, that any exposure is not a good idea, but nor is exposure to lead, asbestos, mercury, benzine and other similar hydrocarbons, chlorine.. I could go on, but in all of these cases, the biggest risk when dealing with low levels of exposure is from the cumulative effect of repeated exposure. The more you come in to contact with hazardous materials (especially things like heavy metals, which your body finds difficult to remove, or radioactive materials which can be absorbed into some body tissues), the greater the risk.
Radiation exposure is generally poorly understood by most of us. This chart may give you a better idea of doses. https://xkcd.com/radiation/
In summary, the more you handle without protection, or the more you ingest small quantities of these materials, the more cumulative dose you receive, and the more likely that is to cause health problems. Your body is designed to deal with the constant damage from small levels of background radiation. One brief, small exposure is therefore of little consequence. Multiple exposures over the long term, or large exposure to highly radioactive material (like the core of a nuclear power plant) over a very small period on the other hand are a bad thing.
So use gloves, keep exposure risk to a minimum, and clean up any loose material you find. Dispose of it carefully and wash your hands with soap and water afterwards, and you should be fine. For the same reason you shouldn't cross the road without looking, you shouldn't handle potentially dangerous materials without understanding and mitigating the risks either.