Geographically Fort York was in St George's Ward. But in the 1861 census it was called an "institution" and the returns are on the microfilm which also includes the final 40 or so houses in St Patrick's Ward. At last check this microfilm still has not appeared on Ancestry with the rest of the 1861 Canada West census. But I have an indexed transcription! (email@example.com)You say you found 1851 details. Considering that the Toronto 1851 (52) census was lost, I wonder where you found it?
Got a better answer or further advice? Post a comment!