Keep in mind a few things:
1. If you ask your CB to change auditors, according to the IAQG Rules, you will have to provide justification for the request.
2. Some CB’s have very few auditors contracted, and if you decide to stick with the current registrar it is possible that a new auditor could come from 3 time zones away, making the travel expenses potentially higher than the audit fees. Your management likely would not appreciate that.
3. If you decide to change CB’s, do realize that most auditors contract with a few of them, so, it is possible that you could have this same auditor assigned to you with another CB unless you tell them this individual is persona non grata in your organization.
Good luck.