William had a difficult childhood. He was the first son of Joseph Allison Tussey and Mary Elizabeth Brasington. At age three his mother and father separated. His father went to MO to live with his brother Julian as described in an 1882 letter. In 1890, Mary Elizabeth married J. C. Tice. In 1901, Joseph married Elizabeth Baily.

Joseph also had a difficult childhood. He was the third child of David Tussey and Eliza F Clodfelter. When Joseph was an infant his father David joined the Confederate Army (1862). Two months later David died "of disease" and was buried in Warrington VA.

The Tussey's arrived in North Carolina in the mid-1700s when William's ancestor William moved there from Delaware. The Tussey family arrived in America when Olof Thorsson (b. 1615) immigrated in 1641 to the Swedish colony called New Sweden in Delaware and changed the family name to Tussey. He was the progenitor of all known Tussey families in America in the 18th century. The Colonial Swedes Organization has documented the early Tussey History. The Colony began with the 1638 landing of the ship Kalmar Nyckel at the "Rocks" in current Wilmington Delaware. They founded New Swedes church, which is now Holy Trinity Church, which is the oldest church in America in continuous use.

William's ancestor Matthis Tussey (grandson of Olof Thorsson) married Sarah Lucasdotter Stedham. Sarah's grandfather Timen Stiddem was also an early New Sweden colonist. He was one of two barber-surgeons on the Kalmar Nyckel on its first voyage in 1637-38. He apparently became the very first physician in Delaware. His decedents formed the Timen Stiddem Society.

William's grandmother Eliza Clodfelter appears to be linked though Yokley and Dill to Mary White. Mary's grandfather Peregrine White is believed to have been born on the Mayflower in Plymouth harbor on 19 December 1620. Peregrine's father William White and mother Susanna Fuller (there is some controversy that it may not have been Susanna) were Mayflower pilgrims.