Trying to invalidate veganism by saying vegans still cause harm by buying from sweatshops is a form of the “al tu quoque” fallacy. The truth of a statement or philosophy not determined by the individuals who believe in it. If a murderer says it’s wrong to commit murder, that doesn’t make murdering people right. If a vegan says it’s wrong to kill animals for food, but causes some harm elsewhere, that doesn’t make killing animals right. It is impossible to cause zero harm, and no vegan claims perfection, but we’re trying to reduce our impact as far as practicable and possible.
Plus, buying animal products is not helping workers in unfair conditions. Being vegan and buying from sweatshops is better than not being vegan and still buying from sweatshops. But what if we saw the argument the other way round? In other words, that the logical conclusion, if you’re against slave labour, is that you’re against all oppression, including animal oppression.