Maybe he hasn't come outright and turned down a fight but he certainly bitches about his potential challengers a lot in the media. When Dana White has to spend 3 hours talking him into a Chael Sonnen match there's some problems there. Especially since Chael represented the greatest PPV rates for Anderson.
Hating on Anderson Silva before it was the cool thing to do.
The idea of a "true number one contender" is the stupidiest MMA idea to be perpetuated by fans and media, EVER.