The comic says he understands why people feel they see a lot of him, but jokes: "I do say no to stuff."