In a small bowl, stir together the mayonnaise, sour cream and hot sauce. Set aside.
In a mixing bowl, place all the ingredients for the cakes and gently stir until fully combined. Then form into cakes, using a spoon or ice cream scooper to ensure uniform size. Roll the mix in your hand and flatten gently (about 3/4 inch thick).
In a skillet, add olive oil and heat to medium high. Add your cakes in batches and do not overcrowd the pan. Flip once, about 2-3 minutes per side, until golden brown and cooked through the center.
Serve on a platter warm, with lemon wedges and spicy mayonnaise. Add additional scallions for garnish if desired.