It's Not White "Privilege"; It's White "Insulation"
...however well-intended its original purpose (and let's assume here that it was well-intended), it has devolved into a tool for delegitimizing people on the basis of their race and/or sex. It doesn't allow for looking people at individuals, but defines them based solely on group characteristics. In that, it perfectly mimics the racism it supposedly targets.
The concept embodied by that use of "privilege" is real, but the actual word "Privilege" is probably the wrong one to use, in part for the reasons he mentions. I prefer the suggestion of another friend of mine, who once said that the word should instead be replaced by "insulation." If your house is insulated, you're not completely protected from temperature extremes, but the more insulation you have, the less likely those extremes are to bother you.
My being white and speaking with what Americans call an educated middle- or upper-middle-class accent doesn't guarantee me immunity from bad cops, for example, but it gives me a lot of insulation compared to any American black person, even with the same or better educational and financial status.
Imagine if you will an upper-class modern black family -- no doubt their wealth in many ways gives them an easier life than I have had, and opportunities I lack. On the other hand: I've never had cops arrest me for trying to enter my own house (or a friend's house in a ritzy neighborhood), whereas Henry Louis Gates did -- even though he was surely dressed better at the time than I usually am. Gates is a Harvard professor who is friends with an ex-president, much richer than me, and has a far better career too -- I am not "privileged" compared to him, but I have a hell of a lot more insulation than he does, against such indignities as "Cops in a rich neighborhood look at my complexion and automatically, wrongly assume I must be up to no good."
The idea "innocent person minding his or her own business is not hassled by the cops" should not be considered a "privilege" in an ostensibly free country. In the country we actually have, unfortunately, even innocent people who mind their own business often find themselves harassed by cops, arrested by cops, even murdered by cops (who rarely face any legal consequences, provided the cop remembers to say "Whoops, I was really totally scared for my life" afterwards). I cannot guarantee nothing like this will ever happen to me -- but so far, my skin tone has provided me excellent insulation against this extreme example of modern American injustice.