I wrote my BA thesis on why the Joker is a great villain
I kind of got it down to two reasons, both stemming from storytelling: first, he was created by actions of the hero, and second, the hero, by definition, cannot defeat this particular villain (without getting himself out of the equation).
I really see both Batman and Joker as concepts, so I'm not really down on the psychological aspects of the Joker - most people would start analysing even Ledger's Joker from a psychology point of view, but I think that's wrong.
The same way Bruce Wayne became Batman, some random guy became the Joker. Real persons becoming "concepts" or "ideas", thus becoming indestructible by real people. That's the idea behind Batman, and Joker realises that. So, in order to stand up to Batman, he became the Joker, to match him on his own level.
Gordon talks about Batman's effect on society, about escalation, and that leads, of course, to the arrival of the Joker. And I think Batman only ever thought of defeating real people, and never even devised a plan to defeat a Joker-(or Batman-)like entity. That's why he fails, because he's stubborn in his methods or weapons against this new threat. This is apparent in the interrogation scene ("You have nothing to threaten me with").
So in a way you have the hero who is indirectly responsible for this new villain, and who is essentially stubborn and short-sighted and is incapable of defeating this threat by his usual methods. Then, at the end, he realises that he has to vanish in order for the other threat to vanish as well. Both on a conceptual and physical level.
Because physically, of course, we know that he disappears from Gotham, but on a conceptual level: because of his murder of Dent, he is not Batman anymore. And, as the Joker is right to assume, the only way Batman can defeat him is by murdering him, and he won't do that. But the Joker makes Batman murder someone anyway, so at that point Batman, the original idea or concept, ceases to be. And that's why it's a powerful situation, from a storytelling point of view. The hero's fate is entangled and inseparable from the villain's fate. And it's also a bit like a paradox: Batman can only win if he loses.
So, not talking about any other aspect of Ledger's Joker (portrayal, etc.), I think these are at least some of the reasons why this villain works so much more than most recent blockbuster villains.

I really see both Batman and Joker as concepts, so I'm not really down on the psychological aspects of the Joker - most people would start analysing even Ledger's Joker from a psychology point of view, but I think that's wrong.
The same way Bruce Wayne became Batman, some random guy became the Joker. Real persons becoming "concepts" or "ideas", thus becoming indestructible by real people. That's the idea behind Batman, and Joker realises that. So, in order to stand up to Batman, he became the Joker, to match him on his own level.
Gordon talks about Batman's effect on society, about escalation, and that leads, of course, to the arrival of the Joker. And I think Batman only ever thought of defeating real people, and never even devised a plan to defeat a Joker-(or Batman-)like entity. That's why he fails, because he's stubborn in his methods or weapons against this new threat. This is apparent in the interrogation scene ("You have nothing to threaten me with").
So in a way you have the hero who is indirectly responsible for this new villain, and who is essentially stubborn and short-sighted and is incapable of defeating this threat by his usual methods. Then, at the end, he realises that he has to vanish in order for the other threat to vanish as well. Both on a conceptual and physical level.
Because physically, of course, we know that he disappears from Gotham, but on a conceptual level: because of his murder of Dent, he is not Batman anymore. And, as the Joker is right to assume, the only way Batman can defeat him is by murdering him, and he won't do that. But the Joker makes Batman murder someone anyway, so at that point Batman, the original idea or concept, ceases to be. And that's why it's a powerful situation, from a storytelling point of view. The hero's fate is entangled and inseparable from the villain's fate. And it's also a bit like a paradox: Batman can only win if he loses.
So, not talking about any other aspect of Ledger's Joker (portrayal, etc.), I think these are at least some of the reasons why this villain works so much more than most recent blockbuster villains.