Wash the limes well, and zest them. Measure out 4 teaspoons of lime zest (you will likely only need the zest from 2 of the limes for this amount). Place this lime zest into a medium bowl.
Juice the limes (including the ones you zested), straining out any seeds. Measure out 1/2 cup of lime juice and pour it into the bowl with the lime zest.
Add 1 can sweetened condensed milk to the bowl with the lime zest and juice, and stir until combined. You will notice that the milk will start to thicken. Set the bowl aside to continue to thicken.
Pour 1 can chilled evaporated milk into a large bowl or the bowl of a stand mixer. Note that the milk needs to be completely chilled to whip up properly. Mix on high speed until the milk holds soft peaks, about 3-5 minutes. Pour in the lime and sweetened condensed milk mixture, and mix again on high speed until thick and fluffy, about 2 more minutes.
Lay graham wafers over the bottom of a 8x8" baking dish in a single layer (cut or break the wafers to fit, if needed). Add 1/3 of the lime filling and spread it evenly over the crackers. Repeat with 2 more layers of graham wafers and two more layers of lime filling, ending with the lime filling on top.
Cover the dish with plastic wrap (or a lid), and place into the fridge to chill for at least 4 hours or overnight. Carefully peel off the plastic wrap, and sprinkle with crushed graham wafers or graham crumbs if desired. Cut into squares and serve.