I agree that its never in your best interest to honestly spill the beans about what you think is wrong with a company or your manager during an exit interview. You have nothing to gain, and everything to lose. It seems the official position here is that one should try and get out of doing an exit interview if at all possible.
What about the cases where not doing an exit interview also creates a big red flag? A colleague has endured a very difficult manager and finally quit. He followed the advice of respectfully declining the interview request in the most polite and professional way possible. HR reported that back to his manager before he left, who then spent the last 20min of his last day tearing a strip off him. Basically said that refusing such a meeting is tantamount to giving your manager a terrible review, and was unprofessional. He also suggested that it would affect any future reference requests.
He's not the only one with that sort of difficult manager with a volatile temper. In such cases, is it maybe "safer" to accept the interview, and simply give benign, vague platitudes about how it's all great, but you have a better opportunity, than it is to decline the exit interview altogether?