The architect - who triggered an outcry when he admitted his affiliations at the end of last week - has promised to stay in the race.
Speaking to Radio 4, he defended the far-right party, claiming its aims and policies were misconstrued by the mainstream press.
'People's political affiliations are irrelevant,' he told the programme. 'Nobody's asked the other candidates what their political views are, or affiliations, and there's no need to; it's irrelevant.'
Mr Phillips added that there was much 'hysteria' about the BNP because of the way the far-right group was depicted by the media.
He went on: 'I'm not a misogynist and I've never actually talked about race in the RIBA.'
He added that he had spent the first two years of his career in Kenya, which was 'not something that a racist does'.