Scrubs are great for an easy fix, but I think the turning point in really sorting out my KP began when I started relying less on physical exfoliants and more on chemical ones. Alpha hydroxy and beta hydroxy acids (AHAs and BHAs) break down and slough away dead skin cells without any of the harsh scrubbing, and they work wonders to reduce those little plugs of hard keratin. Look for glycolic, lactic, or salicylic acid in the ingredients list, and you're on the right track. As a general rule, a BHA will exfoliate deeper than an AHA, but I find the latter works better for my KP. The concentration makes a difference, too. Start out gentle, especially if you have sensitive skin.
I originally bought Pixi Glow Tonic ($46) to use on my face, but it was when I started swiping a cotton pad soaked in the liquid over my arms that I really started to recognise the wonders of chemical exfoliants. Glycolic acid (an AHA) reduces the appearance of my KP like nothing else. However, it does increase skin sensitivity, so as I am fair skinned and prone to sunburn and sun damage, I only use it two times a week and never on the same day as the physical scrub. Anyone using an AHA or BHA should also wear SPF protection on exposed skin afterwards. Yes, even in Winter.
If you're ready to use a slightly stronger product, I'd recommend Nip + Fab's famous Glycolic Fix Cleaning Pads ($35), or even the brand's Extreme Night Pads ($35). Again, start off gently and don't use them more than you need, a couple of times a week should deliver results.