The chairman of the Sheku Bayoh Inquiry has refused a request to step down following concerns over his impartiality.