There is a reason it is exceptionally rare and that's because Speakers don't normally go against the wishes of 90% of their parties members (the claim made here). It is a great way to no longer be speaker. Likewise, a small vocal minority who may want to join with Dems won't do it because it pisses off the party leadership, so it has to be for something worth it.