The Twentieth Century Society (C20) wants to know why the landmark is falling apart, even though there is a mountain of funding.
The conservation group has claimed that urgent repairs have not been undertaken, despite the stadium receiving £2.5 million a year funding from the public purse.
In a letter to the freeholder the London Borough of Bromley, last week, C20 caseworker Cordula Zeidler said: 'We understand that the building is currently being run down with no maintenance work being carried out.
'This includes a broken window in the pool area, missing locks in changing rooms and rusted tiles on the floor in the pool area caused by metal posts (now removed). You are, of course, aware that the council has a legal responsibility to maintain its listed buildings,' she added.
However, the council's planning department pointed the finger at quango Sport England, which rents the facility from it.
Peter Martin, Bromley's head of heritage and urban design, said: 'The state of the centre does concern me but its future is a little uncertain and it does cost a lot of money to maintain.'
But Sport England has refuted the accusations, claiming that much of the funding it has received has to be used to subsidise the centre's revenue, which is running at a heavy loss.
Greg Smith, manager of national projects at Sport England, said: 'It needs to be recognised that this building is 40 years old. It receives a large number of visits every year and will show signs of wear and tear.
'We are aware that work needs to be done. Over the next 12 months a lot will be going on,' he added.
Sport England's lease will expire in April 2006, when the London Development Agency (LDA) will take over the lease and responsibility for the facility.
Discussions on the long-term future of the building are ongoing between the LDA, Bromley council and Sport England, but any decision will depend on the success of London's bid to host the 2012 Olympics.