Cracker Barrel used to be an incredibly popular spot to grab a meal, but many former fans now claim there's one big reason the chain is going downhill.