The extra letter is &. Known as an “ampersand,” this letter was originally the 27th letter of the alphabet, taking its place right after Z. The page pictured on the right from the 1863 book The Dixie Primer, For The Little Folks  illustrates the way schoolchildren were taught their letters.

The ampersand dates back to the first century when Romans combined the letters E and T into one character representing the word et , which is Latin for “and.” From there, it made its way into the Old English alphabet, where it remained until the waning years of the 19th century.