A common atheist response is that we should reject any belief for which we don't have evidence. Graham Oppy calls this "Clifford's principle." Oppy thinks it's untenable.
Oppy starts off his book with a discusion of beliefs and their formation and revision. He approves of Harman's Principle of Conservatism:
One is justified in continuing to fully accept something in the absence of special reason not to.
Specifically, Harman says I shouldn't change my mind simply because my belief is not adequately justified (denial of the Principle of Negative Undermining). But I should change my mind if I have positive reasons for thinking my belief is no good (the Principle of Positive Undermining).
Oppy doesn't go into great detail on these principles, but I think what he is getting at is something like the following. We each have a complex network of interlocking beliefs. Many of these beliefs have never been critically examined (by me), and hence aren't adequately justified. For example, I believe Minsk is a city in Russia. I have never been to Russia, let alone Minsk, nor have I ever met anyone from there. I probably learned about Minsk from a book or a casual conversation, or possibly in school. According to Oppy, I should go on believing this, even though I do not have adequate justification. If someone comes along and says, "Minsk isn't in Russia, you idiot, it's the capital of Belarus!" or if I bother to look it up in an atlas, then I have reason to change my belief.
Most of our beliefs are like this. I believe a squirrel is a mammal, even though I have never seen one suckle its young. I believe water is composed of hydrogen and oxygen, even though I have never performed an experiment to prove this. I believe my father was born in Philadelphia, even though I have never seen his birth certificate. And so forth.
In this view, the atheist who says, "I never believe anything unless I have evidence for it" is saying something profoundly stupid. Such a principle would require a retreat to near total skepticism.
Oppy's position here is a sophisticated version of the religionist who says, "But you have faith, too!" And I think he's right. If "faith" means "belief without adequate supporting reasons," then we all have to have faith. Life is simply too short to completely, or even adequately, examine all of our beliefs.
This helps us understand why so many people have irrational beliefs, and why they are so hard to change. It is actually better (in an evolutionary sense) to continue to believe unsupported things than to jettison too many beliefs and be paralyzed into inaction. It may be that something like the Principle of Conservatism is built into our psychological makeup to prevent this kind of paralysis.