Although CABE's design review panel said it 'admired' the £200 million scheme, it felt the 144m-tall mixed-use tower - originally intended to be 172m high - would impact on 'photo opportunity' views of Grade I-listed Somerset House directly across the River Thames.
These amendments included hacking down the tower from 172m to its current height and ditching stainless-steel cladding in favour of white stone and bronze.
Commenting on the revised design, CABE said: 'The significance of Somerset House is due, in part, to its symmetrical courtyard form, the planned skyline and southerly entry sequence from the Strand into the courtyard and these will be highly compromised by this proposal.
'We say this with some considerable regret, as this is clearly an intelligent, high-quality building, which could bring a largely derelict space into active use and provide much needed facilities to the local community,' the watchdog added.
But Lifschutz Davidson Sandilands director Alex Lifschutz has vowed to plough on with the scheme despite CABE's criticism.
He said: 'This is a step too far. Because we have reduced the tower it is invisible from the Classical view of Somerset House courtyard.
'The part [CABE] is talking about is a transitional view, and very few take pictures or takes notice of this. There is signage and lamps which already detract from this view anyway,' said Lifschutz.