They're one of several brands who are really closely associated with far-right skinheads, mods and football hooligans from decades ago. Even today they're still . Even without being associated with the far-right (which the brand themselves nor the overwhelming majority of their customers aren't) they're just really common in the uk and europe and can be looked down upon as a "chav" brand.
Nothing wrong with the clothes, they're just a brand burdened with pre-conceptions, judgement, and a difficult and distinctive legacy.
That association with tribes, gangs and youthful rage is obviously right up Raf's street hence their recurring collaborative collections.