Caruso St John, Jamie Fobert Architects and Tony Fretton Architects are among six practices in the running to land the 'prestige' project in the heart of the historic city.
Unicorn theatre designer Keith Williams, Panter Hudspith and van Heyningen and Haward were also named as finalists in the RIBA-organised contest - a competition which attracted more than 60 entries.
The practices were asked to draw up proposals for a £3.34 million mixed-use museum, office and housing scheme for the Tower Street site, close to the city's cathedral.
Myles Cullen, leader of Chichester District Council, said: 'We were very impressed by the quality of the architects who applied to this competition.
'It was a very difficult decision for the interview panel to make, but it does mean that we have a very strong shortlist. This is a very exciting and important opportunity to add a building of quality to Chichester's cityscape and to improve the services to our community.'
The new museum will house 'significant geological, archaeological and historical collections' and will tell the story of the town's Roman origins, its medieval history and its Georgian townscape.
An honorarium of £1,500 has been awarded to all six shortlisted firms and these practices will face competitive interviews (on 9 November) before a winner is announced.