It occurred to me after posting yesterday that perhaps I made it sound as if the villain's transformation to hero is simply a matter of the hero waking up and smelling the coffee, as in, "Hey, I don't have to be bad and if I'm good, I get the girl!"
It shouldn't be nearly as simple as that.
For the villain-to-hero transformation to really work, the villain has to already be a three-dimensional, interesting character, and that means having an understandable reason for his bad deeds, and one with which the reader can empathize. The reader should be able to think, "Yeah, if that had happened to me, I might want to do those things, too." Not that the reader necessarily would, but he or she can understand and appreciate why a character might.
And the more powerful that motive, the stronger the character, and the stronger the character, the more he's going to resist altering his path. His goal is so important and so fixed, it takes a major upheaval to make him want to change.
That major upheaval is provided by the heroine, and his love for her. Note I don't say her love for him. That certainly helps, but I truly think it's the hero's love for the heroine that is the catalyst for his change. That's why the villain-to-hero can do good before he will even admit to himself that he's in love, or the heroine has revealed her true feelings. He can act on that change even if he doesn't yet know if his feelings are reciprocated -- and oh, what a gut-wrenching moment that can be, if he thinks she'll never know how she's affected him and that he loves her.
The villain's metamorphosis from bad to good should be a long, tough, intense struggle for him. He should resist, fight it, try to ignore his feelings, refuse to admit he cares, act worse before he acts better, or do any combination of these things and more until he realizes that his feelings for the heroine have changed him and there's no going back -- but he doesn't want to. Because of his love for her, he wants to change. He wants to earn her respect and be worthy of her love. He obeys the promptings of his troublesome conscience and gives in to better impulses he's been trying to ignore. And what he wanted so much before no longer seems so important.
It should take nearly the whole book for this transformation to occur and for the hero to realize that he's not the man he was and his initial goal isn't number one any more. He should struggle against his changing feelings the whole way.
Because the harder the struggle, the more satisfying the victory. And that, I think, explains why this type of hero can be so effective.