Both times, Judge Richard Mark Gergel declined to step down from a three-judge panel in a suit filed in the District of South Carolina that challenges the constitutionality of 28 state House districts, alleging they are drawn to dilute the votes of African American voters.