My mother always annoys me how would you address this?

Next time she starts speaking to you in Swahili, tell her you'll only talk to her if she says it in English. If she starts talking in Swahili again, just don't answer her.