You may have accidentally spent gems in a few different situations:

• on a second chance after losing a level, when the game offers you three additional guests in exchange for gems;
• to speed up the delivery of upgrades;
• when purchasing a hotel upgrade if you don’t have enough coins, as gems can be used instead;
• during an expedition to get missing resources.