Why are African-Americans called 'Black' and Caucasians "White"?

I never understood this. If people are going to call us an actual color at least get it right! Majority of "black" people aren't actually black. We're brown, mocha-late, caramel, whatever you wanna call it. And Caucasians aren't literally white either. Some are peach, baige, all kinds of colors. So Why generalize one race as black or white? I don't get it..