Nearly half of the Grade-I listed house in North Yorkshire was destroyed in the blaze which broke out on Saturday morning.
No one was injured in the devastating fire, which took almost 100 firefighters to bring under control, and fire investigation officers are looking into its causes.
Designed by London architect George Martin, the gothic mansion was built between 1848-1851 on the site of the former home of Prince Frederick, the Duke of York and brother to King George IV.
The surrounding park, in particular the hill on which The Temple of Victory stands, is also believed to have been the inspiration for the nursery rhyme The Grand Old Duke of York.
The owner of Allerton Castle, American businessman Gerald Rolph, has now pledged to restore the home to its former glory and English Heritage has already sent its own structural engineers to advise him on the full extent of the damage.
Meanwhile on Tuesday, another Victorian gothic mansion was devastated by fire in Lincolnshire.
Stoke Rochford Hall, designed by the Scottish architect William Burn during the 1840s, was also 'seriously damaged' in a similar blaze.
Neither fire is being treated as suspicious.
Photograph courtesy of HACS Construction Group