Just the weight loss process itself has probably shifted the estrogen balance in your body away from testosterone as loss of fatty deposits releases estrogen. The excess stress will only compound that effect.
Here's the deal (in short): the steroid pathway works something like this... cholesterol is used as a building block to form progestogens. Those are used as building blocks to form either corticosteroids OR androgens and estrogens. Cortisol is a corticosteroid, and it is produced in high supply when a body is under stress. If we're producing high levels of cortisol, we are taking away from testosterone production.
Saw Palmetto, even in small doses, begins to block the action of 5-alpha-reductase which is necessary to convert testosterone to dihydrotestosterone. DHT is needed to produce an erection. I've experimented with Saw Palmetto on myself just to see what it does, and it definitely stops a person from getting it up! I've talked to a couple of other guys (online) who experienced the same results. Basically, we're all looking to build muscle and were trying things to improve testosterone levels for that purpose, but we ended up noticing that nasty side-effect. :) Discontinuing Saw Palmetto will begin erasing its effects within a few days to weeks.
Of course, there is a reverse issue whereby excess DHT can screw up the prostate (hence the popularity of Saw Palmetto these days). However, if your body, and subsequently its endocrine system, are in good balance, then the prostate probably won't suffer from "normal" production of DHT.
The mixed nuts are a good idea -- nothing wrong with them; however, between the nuts and the supplements, you're likely getting too much zinc. You might just try taking half of that zinc supplement. Actually, it might be worth taking a week off of zinc to let your body clean out a little and then returning to the zinc by just taking half of what you're taking now.
I'm guessing your liver's out of balance given your past condition. A good liver cleanse will help that. You'll need to carefully watch your blood Sugar while cleansing so you don't go hypoglycemic. Milk thistle is a potent herb which helps protect the liver and restore liver function; however, a good cleanse is essential for getting the crap out of the bile ducts.
Fish and fish oil are generally good choices. Of course, there's always the potential for mercury contamination there. Supposedly selenium and compounds high in sulfurs (onions, garlic) are good to help eliminate mercury contamination from what I've read in this area of "purification."
Time of day is the last thing I can think of at the moment. If you're a morning person, maybe morning is a better time for sex. Perhaps night is better. I, personally, do better at night because I'm a night person though technically testosterone levels peak in the morning and drop off throughout the day.
There are some products (though I can't remember them at the moment) which inhibit creation of aromatase -- a compound necessary in the production of sex hormone binding globulin. You might look into one of them as reduction of SHBG will help free some testosterone (though this might be a limited effect).