CABE has slammed the scheme's 'banality' and EH has expressed 'substantial reservations' over the project.
However, architect Piers Gough, who has acted as an adviser to Gehry, said he would be surprised if the recently resubmitted project failed to reach planning committee.
In a letter to Brighton & Hove Council last week, CABE director of design review Selina Mason said: 'The earlier material we saw was promising, but the planning application has not realised its potential.
'In looking at the new images, we are struck by how banal we would find this scheme if it were not for the towers.'
EH's concern continues to be the project's 'perimeter blocks and public-realm aspects', which it reaffirmed in a letter dated 21 April.
Gough also revealed that Gehry is exasperated with the British planning system: 'He might think twice about taking up another offer of building in Britain.'
Josh Arghiros, Karis Managing Director, said: 'We remain absolutely confident that our design philosophy for King Alfred is robust and that we have fulfilled the stringent development and design brief set by Brighton & Hove Council.'
A Council spokesperson said: 'This is all part of the process of arriving at a good scheme and such an exchange of views is expected and healthy.
'Consultation with CABE and EH is designed to get their input and utilise their expertise. This is what's happening. The council will continue to work with Karis to improve the scheme as it would with any developer.'