In a large saucepan, combine the sugar, cornstarch and salt.
In a mixing bowl, preferably one with a spout, whisk together the yolks and half & half. Whisk the half & half mixture gradually into the sugar mixture. Set over medium heat and whisk continuously until mixture thickens. When mixture thickens, set time for 1 minute and continue whisking. Remove from heat and add butter and lime zest, whisking until butter has melted. Whisk in the lime juice. Pour into a metal bowl, set the metal bowl in a larger bowl of ice water. Let sit for about 10 minutes, stirring off and on until mixture is cool.
To assemble, Llne an 8 inch square pan with plastic wrap leaving about 4 inches of overhang.
Lay 9 graham cracker squares on the bottom of the pan, sides touching.
Spread about a quarter of the lime mixture over the graham crackers. Repeat until you have 4 layers of filling and 5 layers of graham crackers.
Bring plastic wrap tightly up and around cake and freeze for about 8 hours or overnight.
To serve, lift from pan and set on a plate. Spread sweetened whipped cream of whipped topping over the top layer of graham crackers. Cut into large squares.