Lord Dufferin realized that Rideau Hall needed space where ceremonial events could be held, and so the government added the Ballroom in 1873. Three years later, he had the Tent Room built to balance the appearance of the building and to accommodate the increasing number of social functions held by the Governor General. Lord and Lady Dufferin organized numerous balls, concerts, dinners, theatrical performances and receptions of all kinds -- and Lady Dufferin loved to perform the lead role in the plays at Rideau Hall. Their enthusiasm increased Rideau Hall's role as a centre for social affairs. Other changes were made to the grounds. Lord Dufferin contributed $1,624.95 of his own money to build a skating rink, curling rink and toboggan slide at Rideau Hall in 1872-73, money later reimbursed by the government. These were available to the public on the condition that people "were properly dressed". The gasometer, currently known as the Dome Building, was constructed in 1877-78 to manufacturer gas from crude petroleum to supply fuel to Rideau Hall, avoiding the uncertain supply at the time from the city of Ottawa.  
The Dufferins were the first to use La Citadelle in Quebec City as a second vice-regal residence. Like many other Governors General, Lord Dufferin and his family thought the city was beautiful. When municipal officials proposed to take down the walls that surrounded the city, a remnant from its days as a garrison town, in order to have room for the city to grow, he convinced them to abandon the idea. Saving the walls helped preserve the city's historical character, which was recognized in the 1980s when Old Quebec was recognized by UNESCO as a world heritage site. In recognition of his contributions in Quebec, Lord Dufferin's last public event was to lay the foundation stone for Dufferin Terrace, a walkway overlooking the St. Lawrence River in Quebec City, in October 1878, which was built from his design.
Lady Dufferin was very active during her husband's term as Governor General. She was the first spouse to accompany the Governor General on a tour, visiting southern Ontario in 1872. By the end of their term, she had visited every province with her husband. During a trip to Manitoba in September 1877, Lord and Lady Dufferin each drove a spike in the line of what would become the Canadian Pacific Railway. And in May 1874, she presented the regimental colours to the Governor General's Foot Guards.
Throughout the Dufferin's term of office, Lady Dufferin wrote weekly letters to her mother back in Ireland. These were later published as a diary of their time in Canada, called "My Canadian Journal." In it she said that, of all her experiences, she had spent her happiest times in Canada.