Gadsby looked toward Washington Hall Hotel in Philadelphia, the next stop on his stagecoach line to the north, for his next venture. The reasons behind his financial downfall are unsure. Дuring the Panic of 1819, Gadsby sold his lease and left Baltimore nearly penniless. Key finished his song the ' Defence of Fort McHenry,' now better known as ' The Star-Spangled Banner,' at the Indian Queen on September 16, 1814. Of the many notable guests Gadsby welcomed before, during, and after the Battle of Baltimore, Francis Scott Key would become the most significant. They also had a son, Charles, who died as a young child. The couple had three children in Baltimore: Virginia (1813), Augusta (1816), and Julia (1818). The patriarch of the family, Provey's father Benjamin Bradford Norris, was significant in town for being a signer of the Bush Declaration.

The Norris family was prominent in Baltimore and attended numerous events at the Indian Queen Hotel. But on February 12, 1812, Peggy passed away, and a short 11 months later, Gadsby married Providence 'Provey' (Norris) Langworthy, twenty years his junior. On May 10, 1808, Gadsby and Peggy had a son, William. The numerous bed-chambers have all bells, and the servants are more attentive than in any public or private house I ever knew. This inn is so capacious that it accommodates two hundred lodgers, and has two splendid billiard-rooms, large stables and many other appendages. We alighted at the Indian Queen in Market Street, John Gadsby in a style exceeding anything that I recollect to have seen in Europe or America. In addition to his tavern's position as a stagecoach station on four lines, this hostelry par excellence brought Gadsby both a fortune and an excellent reputation in Baltimore. The Indian Queen Hotel, at Hanover and Baltimore Streets, was a sizeable tavern that showed off new mechanical innovations, including a steam kitchen, stew stove, patent oven, and smoke jacks to move a large coffee roaster and mechanical spits. Though at that point Gadsby still had eight years remaining on his lease with John Wise, he abruptly sold it and moved to Baltimore, possibly for economic opportunity. In July of that year Ann Sophia Gadsby married Augustine Newton. First, Margaret Sarah Gadsby was married to Charles Thomas Chapman in 1807, and the couple had a son, John Gadsby Chapman, by 1808.

He then quickly married off his daughters. His most valuable 'possessions' were the enslaved African-Americans who comprised the majority of his staff.Д«y 1805, Gadsby's first wife Margaret had died, and he married Peggy McLaughlin of Georgetown. When he renegotiated his lease, Gadsby's property and possessions totaled $22,441. He made money renting his carriages and investing in a stagecoach line. The tavern quickly became the center for community events, from the George Washington birthnight ball to dancing assemblies to meetings of local clubs, particularly the Alexandria Jockey Club of which Gadsby was a benefactor. On OctoGadsby leased the City Hotel from John Wise, now the site of Gadsby's Tavern Museum in Alexandria.

Df tavern keeper