Campaigners, led by the Twentieth Century Society (C20), have greeted the decision with relief.
They feared that if the ODPM inspector upheld Runnymede Council's decision to grant permission for the demolition, it would have caused havoc in the listing and planning systems.
In giving its support to owner David Beadle's demolition of the 1937 house, the council's planning committee had set a precedent - they had supported the owner's argument that to stop him knocking the building down would be to infringe his human rights.
If the inspector had agreed with the decision, conservationists feared that the same argument would be used by other owners of listed properties who wanted to demolish.
However the inspector's decision will not bring the important Modernist building back. Beadle illegally demolished the building, despite the fact that the controversial planning permission was subject to a Holding Directive from the ODPM.
He has faced separate legal action for ignoring this Holding Order and also faced a fine from the High Court.
But C20 is still pleased to have proved that Beadle's original argument was flawed.
Speaking immediately after the decision, C20 director Catherine Croft said: 'This means that Mr Beadle, the owner who demolished Greenside, now has neither his distinguished original house, nor any prospect of erecting another one on the land.
'We were very disappointed by the tiny fine he received when convicted of the criminal offence of unauthorized demolition of a listed building in the Crown Court.
'£15,000 plus £10,000 costs was a derisory penalty in relation to the amount of money he would have made from replacing Greenside with a new building.
'However, this verdict means that his own actions have effectively reduced the value of his property to a fraction of its previous worth. In this way the case is likely to act as a severe deterrent to any other owners who might consider doing likewise.
'We anticipate that Mr Beadle is likely to appeal the Secretary of State's verdict, but we are confident that the ODPM has taken extensive legal advice and that the decision is both morally right and legally sound,' Croft added.