Make the spicy yogurt sauce by combining yogurt, hot sauce, paprika, thyme, and salt in a small bowl. Mix well to combine. Cover and refrigerate until ready to use.
Heat a large skillet over medium heat. Add ½ tbsp olive oil. Add 1 pita half, cut side up and top with a mozzarella, crumbled feta, chicken, peperoncini, sun-dried tomatoes, dill, and black pepper. Top with the matching pita half, cut-side down.
Cook until the cheese begins to melt and the bottom pita is golden, about 2 minutes. Carefully flip the pita and cook until the other side is golden, about 2 more minutes.
Repeat with remaining pitas and ingredients. Cut quesadillas into fourths and serve warm with spicy yogurt sauce.