Is there such a thing as "going over the line" in comedy?

Most definitely. What's considered "over the line" changes with societal values. Racist and sexist jokes are widely considered to be in poor taste currently, but it was not always so. For the most part, comedians can get away with making light of an ethnic group or sexual identity that they belong to themselves. Personally, I think the most successful comedians are the ones who can get us to laugh at ourselves instead of laughing at some perceived "other."