The practice, which has developed an impressive international reputation in the last few years, saw off competition from another Hibernian big name, Heneghan Peng.
Also on the shortlist, which was picked from a host of international entries, were the Consarc Design Group, Niall McLaughlin and Panter Hudspith.
O'Donnell + Tuomey Architects was Stirling-shortlisted last year for the Glucksman Gallery in Cork, which received international acclaim.
The practice will now draw up detailed proposals for the Coleraine museum, which may or may not include a library as well.
London-based Claire Wright, who was the RIBA advisor for the competition, said she thought the winning scheme was outstanding.
'It was a pleasure to be involved in selecting a team from such a high calibre field,' she said. 'O'Donnell + Tuomey's outstanding talent and sensitivity were once again evident in their ideas for this project.
'I look forward to visiting a wonderful new museum in Coleraine in the not-too-distant future,' she added.