The appointment, which emerged this week, comes after British Land sold the site to McAleer & Rushe for £47 million last October.
Current plans being considered include redeveloping the site - which sits at the north-west corner of the square and was built to promote Swiss trade and tourism - as a hotel.
Avery Associates chief Brian Avery said: 'What happened was that British Land commissioned us to draw up our scheme and that's what we did. But after a while it all went very quiet and it transpired that the site had been put on the market.
'We're not really annoyed, just a little disappointed. We don't do a lot of buildings and every one that we do takes a lot of care, so it's sad that we aren't going to build it.
'It was always clear that the site could be used as a hotel, so we cannot blame the new landowner for not taking on our scheme. It's not exactly my area of expertise, so it's not like I can expect them to take me on,' he added.
Jestico + Whiles refused to comment.