Now that it looks like your allergy has nothing to do with grains or with milk, I would try the other part of the spectrum, part you never eliminated before.
Fruits and veggies and oils.
You never tried eliminating all vegetables and all fruits for at least 5 - 7 days.
I would also eliminate all plant oils and would focus on animal fat as a main source of fat.
Try that for 7 -14 days. Don't worry, you will not die of malnutrition.
If that diet works for you, you will see your lips healing within the first 7 days, giving you clue to contine diet for at least one more week.
Try eating just meat, liver, white fish, and if you want, some yogurt, rice, oats, ...
It should not be very difficult diet for 5-7 days, considering you managed to live off grains which are everywhere.
Ragweed Allergy: "Ragweed, in theory, cross-reacts with bananas and melons, so people with ragweed allergies may react to honeydew, cantaloupe, and watermelons, or tomatoes," says Warren V. Filley, MD, from the Oklahoma Allergy & Asthma Clinic in Oklahoma City.
Zucchini, sunflower seeds, dandelions, chamomile tea, and echinacea also go on that list.
Birch Pollen Allergy: People with birch pollen allergies may react to kiwi, apples, pears, peaches, kiwi, plums, coriander, fennel, parsley, celery, cherries, carrots, hazelnuts, and almonds.
Grass Allergy: People with grass allergy may react to peaches, celery, tomatoes, melons, and oranges, according to the AAAAI.
Latex Rubber Allergy: Like pollen allergy, people allergic to latex rubber may react to bananas, avocados, kiwi, chestnut, and papaya.
Good luck with your detective work. Very few people would have tried a specific diet as long as you tried. I believe only one out of 50 people are able to follow strict diet the way you did it.