Simple explanation:
Both gold and silver are toxic because they get stuck in tissue, both Colloidal Gold and silver are too small to get stuck.
If you are really worried start with Manna or Monatomic Gold, the particles are so small they cannot be recognized by any testing method other than some Alchemy Magic.