It's marginally better to invite a (former) politician for a formal talk. But those events aren't about "free exchange of ideas" either. It's embarrassingly disingenuous for colleges to try to defend the visit of a controversial figure on those grounds. The speakers who get big fees to show up and defend their record (or, theoretically, speak about some other area in their expertise) aren't picked to represent a broad diversity of views. No struggling activist for an unpopular cause gets a $150,000 speaking fee, and are likely not even invited to the same forum. Lecture series are a tool of the establishment if ever there was one.
So while I roll my eyes at the recent controversy over Condoleezza Rice's speaking engagements (U Minnesota and Rutgers), the language of free debate being used in service of funneling money to ex-politicians leaves a bad taste in my mouth.