There's a bit of a stigma with Paleo in that it is expensive to eat this way. My retort is: “What is more costly, medical bills or a better diet?”. Once you give up on boxed foods and start buying whole foods you should begin to see a difference in meal costs for the better. You can easily put together a meal for four for about 15 dollars which is a lot cheaper than eating out. Speaking of eating out, you'll likely reduce that considerably since it is somewhat difficult to source a restaurant that is conducive to your eating habits.
The initial costs can be a bit off-putting and a lot of times I recommend starting slow in replacing items. Eat down what you may have stocked up as you build up new supplies, or if you have long term storage, give it away or keep it stashed for emergencies. I went the cold turkey route and tossed out everything in favor of my new lifestyle. The choice is yours if you can afford it or if your health can hold out for a little while longer as you make the transition. Some people are in dire straights with obesity or metabolic issues and need to shock the system as far as I am concerned. If the temptation is there then they'll never achieve their goal.