In the summer of 1863 prisoner exchanges between North and South were stopped, for the most part, because the South would not exchange captured black soldiers.
From the Richmond Daily Dispatch August 27, 1864:
Captured negroes.
–Among the captures from the Yankee army on the occasion of the explosion of one of their mines near Petersburg, about six weeks since, were eighty-odd runaway negroes, who were enlisted troops in Burnside’s corps. Soon after their capture they were sent to Danville, Virginia, for confinement, but on Wednesday last they were transferred to Castle Thunder. For the information of persons whose servants have run away the following list is appended:
William, slave of Alexander Gray, Charles co., Md.; Churchill, slave of Miss M. A. Wilson, Annapolis, Md.; Joseph, slave of Alexander Butcher, Baltimore; Luke, slave of Major Pri, Charles co., Md.; George, slave of John G. Perry, St. Mary’s co., Md.; Washington, slave of Robert Young, Charles co., Md.; Lewis, slave of George B. Wilson, Norfolk; Henry, Geo. W. Owen, Dorchester, Md.; John, slave of John Wayland, Howard co., Md.; Charles, slave of Captain Arnold, Colburn, Md.; Frank, slave of Alfred Oden, Pike co., Mo.; John, slave of Jim Leighter, Dorchester, Md.; Sam, slave of Robert Tung, Charles co., Md.; Andrew, slave of William Gordon, Marion co., Mo.; George, slave of John Ronsells, Rowell, Mo.; John, slave of Seth M. Wayland, Worcester, Md.; Ephraim, slave of John Hammond, Howard, Md.; John, slave of Mrs. Ward, Marion, Mo.; Willis, slave of L. Vaughan, Hanover, Va.; Charles, slave of John Ayres, Ronsells, Mo.; Peter, slave of R. L. Gordon, Orange, Va.; Isaac, slave of Dr. Shaw, St. Mary’s, Md.; John, slave of Thos. Deralbis, Frederick, Md.; John, slave of Wm. Boesy, St. George, Md.; Lewis, slave of W. Wedington, Frederick, Md.; George, slave of W. Wall, Prince George, Md.; James, slave of Dr. Pike, Charles co., Md.; Charles, slave of A. L. Finley, Pike co., Mo.; Charles, slave of John Netewell, Kent co., Md.; Benjamin, slave of W. B. Walker, Baton Houge, La.; Robert, slave of Miss Delia Jane Warfit, Baltimore; John, slave of Wm. Roby, Charles co., Md.; Marshall, slave of Robert M. Miller, Jefferson, Ky.; John, slave of Gray Boulware, Caroline, Md.; Francis, slave of C. C. McGruder, Prince George, Md.; Miles, slave of Washington Posey, Charles county, Md.; Peter, slave of — Ross, Wakely, Tenn.; Lewis, slave of John McGran, Hemford, Md.; John, slave of Alex. Dyer, Charles county, Md.; Frank, slave of Dr. Meriwether, Jefferson, Ky.; Ashburn, slave of heirs of William Galt, Queen Anne, Md.; William Bowser, slave of William Peckham, Eastern Shore, Md.; Noble, slave of Miss Leonora Floyd, Charles county, Md.; Elias, slave of Ossian Pendleton, Culpeper, Va.; John, slave of Philip Cox, Brunswick, Va.; George, slave of C. Gardner, Prince George, Va.; Henry, slave of B. Embry, Nashville, Tenn.; George, slave of Joseph Davis, Rockville, Md.; Lloyd, slave of F. McGruder, Prince George, Md.; Yarmouth, slave of Alexander Kilga, Montgomery, Md.; Alfred, slave of John Hill. Richmond, Va.; Orange, slave of James N. Hill, St. Lawrence, Mo.; Frederick, slave of Betsy Bryding, Somerset, Md.; James, slave of John T. Wilson, Mason, Ky.; Thomas, slave of John Loates, Frederick, Md.; Thomas, slave of Mrs. Slocum, Dorchester, Md.; Edward, slave of Alfred Gordon, Charles co., Md.; Amos, slave of Mary A. Bower, Eastern Shore, Md.; Hiram, slave of B. Wyatt, Drew co., Ark.; Jesse, slave of Thos. Anderson, Marion, Mo.; John, slave of Samuel Clark, Kent co., Md.; George, slave of H. Hesth, Anderson dist., S. C.; Solomon, slave of W. Brewer, Montgomery, Md.; Calvin, slave of Abraham Gardner, Southampton, Va.; James, slave of J. F. Chaplain, Port Royal, S. C.; John, slave of George Rausser, Baltimore; Charles, slave of Miss Eva Fields, Prince George, Md.; Sam, slave of J. Green, Baltimore; Isaac, slave of James Connelly, Clarke co., Va.; William, slave of Hugh Delts, Parkersburg, Va.; Abraham, slave of Charles Stewart, Anderson, Md.; Robert, slave of Mrs. Amanda Mathews, Charles co.; Md.; Charles, slave of Mrs. S. B. Jeter, Quinely, Mo.; John, slave of Jacob Calvin, Louisville, Ky.; Samuel, slave of G. Burcher, Louis, Va.; Winston, slave of Stephen Turner, Pike co., Mo.; John, slave of Washburn Rowe, Carrol, Md.; Lewis, slave of J. Roeth, Eastern Shore, Md.; Robert, slave of A. L. Milton, Prince George, Md.; Charles, slave of A. Spead, Fairfax, Va.; Henry Lynch, a free negro.
According to the National Park Service Confederate officials approached Union General Benjamin Butler in the summer of 1864 about resuming prisoner exchanges, including blacks. General Grant was amenable to one-to-one exchanges but did not agree to paroling unexchanged prisoners because he assumed the rebels would rejoin the army while the northern soldiers would stay home.