Updated: Jan 3, 2021

Photo by form PxHere
I buy "specialty" foods both online and at stores in person. If I shop in person, I go to Stop & Shop most often. They carry a lot of organic and gluten free items. They also offer a delivery service, so you can order online. I also shop at Whole Foods. They are known to carry specialty items, though I find them more expensive. Sam's Club also carries some items. Of course, you can shop in the club and online.
Why I don't eat meat!

Online, I use Vitacost. Their delivery time is quick and their prices are decent.

Shoprite has some items that I can only get there. I buy my organic, gluten free Ricotta from them. It freezes well, so I stock up.
Unfortunately, I can't get everything in one place. This means that I shop online, and sometimes go to two or three different stores, never in one day.
Shopping for gluten free, organic food that does not have any corn, nuts, eggs or soy as ingredients can cause anxiety. It's a bit daunting, but I've been doing it so many years, I've gotten used to it.

My biggest tip is to ALWAYS read labels from top to bottom. Just because it says gluten free on the front doesn't mean you can eat it. READ, READ, READ. It may be manufactured in a factory that also manufactures products containing gluten or any one of the other things I'm allergic to. Cross contamination is a big concern.
If necessary, make a phone call and ask questions. Almost every brand has a customer service department that is very knowledgeable and happy to answer any question you may have.

Photo by form PxHere
If you are new to eating gluten free, take a look at this list of naturally gluten free foods. Some I can't eat or won't eat, but for those just eating gluten free, this list could be helpful. Some tips for beginner shoppers.
Happy and safe shopping and cooking.