Mills are all over the place... there really isn't a District. Believe it or not there are textile mills to be found in the yellow pages. Places like the Toronto Fashion Incubator (google that) help designers find resources.
I don't know much about Montreal, but it would certainly be a place I would look.
In Toronto: Queen West from Spadina to Bathurst for jobbers and stuff... some shops do wholesale... try King Textiles, I think they have a website now too. MacFab for costumiers supplies, upholstery and notions.
Up at Dufferin and Wingold is the other fashion District in Toronto... Many designers wholesale their clothes there, and also there are more jobbers, fabric wholesalers and leather dealers, etc. up there.