Actually, anti-semitism not only refers to Jewish people, but also to Arabic people as they are both semitic ethnicities.
This is actually not quite true. While Arab people are semites (which refers not to ethnicity, but to language), the term "anti-semitism" was popularized specifically to reference hatred against Jews. This was done primarily by Wilhelm Marr in the late 1800s as an attempt to find a more scientific-sounding word for Judenhass. He wanted something that carried not just a religious connotation, but a racial connotation, as well.
Until another word for discrimination against Jewish people becomes more popularized, anti-semitism is okay to use when discussing hatred or prejudice against Jews, as that it's how it's been used for over 100 years now.
And that's a fair point, but it's a microcosm of the broader problem that the only descendants of Shem who are linguistically included are those who are Jewish.