Cut tips off asparagus and then slice spears into 1-inch pieces. Transfer into generously salted, boiling water and stir. Wait a minute or two for the water to just start boiling again, stirring occasionally. Remove asparagus with a strainer into cold water. Once chilled, drain well and set aside. Slice ham into thin, 1-inch pieces and reserve.
Carefully add potatoes to the boiling water and cook until tender, 15 to 20 minutes. Drain well and set aside.
Add olive oil to a pan and set over medium-high heat. Saute onions and peppers with a large pinch of salt until the onions just turn translucent, and soften slightly, about 5 minutes. Let cool and reserve.
For custard, whisk eggs together in a bowl; once eggs are fairly uniform, whisk in milk, cream, salt, pepper, and cayenne.
Preheat the oven to 350 degrees F (180 degrees C). Oil a 13x9-inch casserole pan. Cut potatoes into bite-sized pieces, and arrange evenly on the bottom of the pan. Scatter over half the cheese, then half the onion-pepper mixture. Top evenly with ham and asparagus, being careful not to press down. Top with the rest of the onion-pepper mixture.
Pour egg custard mixture in carefully; top with the rest of the cheese. Jiggle the pan and tap gently a few times on the counter to settle the custard into all the open spaces.
Bake in the preheated oven until eggs are just set, 50 to 60 minutes. Let cool for 10 minutes before serving.
