Never say anything negative about your old boss. If he is called for a referral, then let him say something bad about you, this way you can sew him for that harassment against you. Just say it was time for you to look for a change in jobs.