After culture secretary Tessa Jowell informed the Commonwealth Secretariat on Wednesday of her decision not to delist the building in London's Holland Park, Commonwealth officials were rumoured to be considering a judicial review to contest the issue.
Twentieth Century Society caseworker Cordula Zeidler said she felt the directors were 'over-reacting'.
Details of a statement by Commonwealth Secretary General Don McKinnon were released on Wednesday.
In it he said: 'The UK's decision is selfish imperialism. This scandalous act robs millions of children in the developing world of educational opportunities.
'By having this white elephant delisted, the Commonwealth Institute could have realised funds for education programmes for 75 million children in the Commonwealth who have never seen the walls of a classroom.'
But Zeidler responded: 'The cultural value of a building is not always reflected in its value in monetary terms.
'The funny thing is, they had a lot of interest when they first marketed the building. Lots of things could help starving kids, but trying to demolish a listed building is not the way forward. They must take responsibility for their buildings.'
McKinnon claimed that the 1960s institute's own architect, Robert Matthew, Johnson-Marshall & Partners, said it 'is without great architectural merit' and has 'serious design flaws which cannot be rectified, including a permanently leaking roof.'
In December, the Twentieth Century Society wrote a letter to English Heritage demanding the rejection of an application to delist the west London 1960s edifice (AJ 02.12.04).