On August 7, 1864, Dr. Kerfoot and his assistant, Henry Coit (who later became the founding headmaster of Saint Paul's School in New Hampshire) were arrested by Confederate officers. They were paroled a day later on the condition that they secure the release of the Rev. Hunter Boyd, a Presbyterian clergyman from Virginia, who was being held by the Union.
During the same month, Dr. Kerfoot received an offer from Trinity College in Hartford, Connecticut, to become its president. His concern about Saint James School's financial condition and the drastic decline in enrollment led him to accept the offer. He left in 1864 with a heavy heart, expressing his remorse in a letter to Bishop Whittingham, "Here I have loved, as I think I never can again love, any work. It seems impossible that I should leave it..." (Kerfoot, 280)
Bishop Whittingham agreed that the fight to keep the school open must be abandoned, at least for the duration of the war. In his address to the 1865 Diocesan Convention, he spoke of his reaction to the closing of the school: "What your bishop lost in all this process, brethren, I shall not attempt to tell. For him it makes a large part of the work of a quarter of a century a blank."