This phrase originated in the history of American slavery: the river was the Mississippi and down implied the transfer of slaves from north to south.