Exactly my thoughts. Most of the people I've spoken with honestly in my CS classes have expressed these feelings of inadequacy. Often it's accompanied by things like "oh, I didn't take CS in high school and it seems like everyone else did." If they're a certain gender or race they may feel this has hindered them, even though many others in the class secretly have the same feelings of inadequacy.

I'm not sure whether it'd be comforting to feel (correctly or not) that your background is partially to blame for your lack of confidence or success. On the one hand you may discourage yourself and set yourself up for failure because you believe you have a good reason to fail. On the other hand it may prevent you from beating yourself up as much since there are some external factors at play that are out of your control.