I agree with you to a point. However, when individuals undergo severe emotional trauma, this can also be responsible for the overwhelming surges in adrenaline and cortisol. It can also serve to zap important nutrients out of the body. Prolonged unresolved trauma can affect the chemicals in the brain and cause that chemical imbalance that the medical profession uses as an excuse to prescribe antidepressants. Diet plays an important part and a poor diet can make the symptoms of panic and anxiety much worse. Unresolved issues are also the cause of deep seated depression. Life's circumstances are complicated and anxiety and panic result when you are caught between a "rock and a hard spot." In some cases, yes, it is purely physical.
That is why when dealing with anxiety and panic it is imperative to address both the physical and emotional causes. It's all about Body, Mind, and Spirit.