Most upvoted comment
Did slaves actually build the white house? If not who did?(r/AskHistorians)
A number of Scottish Masons were hired for the purpose. They had developed a number of specialized techniques that were designed to protect the kind of stone selected for use from water. James Hoban, an Irish Architect, styled it after a Ducal Palace in Ireland, which now happens to be where the Irish Parliament sits. So, using masons familiar with the processes used there seemed to make a great deal of sense.
That said, there were five slaves employed as carpenters in a list of workers on the White House project in 1795, three years into the eight year process. It’s also probable that much of the quarrying and logging also included some portion of slave labor. Some records seem to indicate that the government itself owned some slaves at a quarry, the stone of which was used at both the White House and the Capitol Building.
So, the overwhelming majority of labor used on these sites were standard issue construction workers, but there was a smattering of slaves mixed in. To say that it was built by slaves might be misleading depending upon the context.
According to the documentation provided by the White House Historical Association it was a case of the government hiring slaves owned by private citizens. There were a handful of slaves owned by the Federal Government kicking about in this time period, but none of them were used in the construction of the White House.
The Invisibles: Slavery in the White House and How it Shaped America is an interesting read.
Another, but only tangentially related one that is fascinating is A Slave in the White House: Paul Jennings and the Madisons. Or maybe just read Jenning’s own book: A Colored Man’s Reminiscences of James Madison published in 1865.