Jean fuzz on the back of my white shorts?

I went dancing a while ago and I was wearing white shorts and there were little traces of jean material from me dancing with people wearing jeans. I washed them but its still there and the lint roller doesnt work! Anyone know how to get rid of it?