Clifford was married to Elizabeth Porter from 1967 until her death in 2003. He has a daughter by her. He married Jo Westwood in 2010. She began divorce proceedings against him in August 2013 on the grounds of unreasonable behaviour. A decree nisi was granted in May 2014. He lived in Hersham before being sent to prison.
He was arrested in December 2012 by Metropolitan Police officers as part of Operation Yewtree an investigation which was set up to investigate child sexual abuse committed by DJ Jimmy Savile who died in 2011, and by other people. In 2014, he was tried at Southwark Crown Court for 11 counts of indecent assault against teenage girls. On 28 April 2014 he was convicted of 8 counts. On 2 May 2014, he was sentenced to 8 years imprisonment and is now incarcerated at HM Prison Littlehey in Cambridgeshire. He collapsed in his cell and was taken to hospital, where he died of a heart attack.