I agree with Archus, do not beat up on yourself this way. Concentrate on your accomplishment, and definitely, definitely do be gentle with your body right now...even though you already ate whole fruits, try to spend a day with fresh fruit juice cut with water, just to feel a day of how the fast should have been broken, or make a pot of simple vegetable soup with very little or no salt added, and enjoy the broth one day, the vegetables the next. If you think you may try a fast soon again, eat very gently and even mostly raw. That, too, may be a difficult task, but after having weathered fasting, will be much, much easier and possibly even enjoyable for you.
I cant speak for all fasters, but I certainly have had my share of false starts. So, you aren't alone in that. And, you answered many of your own questions regarding associating with anyone who is negative toward fasting, or leaving the house before you were ready. My very first fast, I recall spending the entire time at home before venturing out to purchase the foods I would use to break the fast. It was the only way for me, and subsequent fasts involved more activity, and more tolerance to people, places and "usual" stuff.