Lunch times are great opportunities to network and make connections that could improve your career growth.
So unless there is a common eating area for brown baggers, you may choose to limit brown bagging lunch to three days each week.
Find a balance between saving some money and making the connection.
In my case, I take my lunch with me 2-3 times a week and eat out the rest of the time.