Preheat oven to 350°F. Place catfish fillets in a small baking dish. Lightly oil fillets, season with salt and pepper, and cook for 15 minutes, or until cooked through.
Place catfish in refrigerator. Flake when chilled, and set aside.
Combine remaining ingredients in a bowl. Stir in flaked catfish. Spoon onto 4 slices of brioche, and top with remaining slices.
Serves 4