Automatic Prompt Selection for Large Language Models

This paper proposes an effective approach to automatically select the optimal prompt for a given input from a finite set of synthetic candidate prompts. Unlike existing methods, which are usually costly and lack flexibility, the approach is designed to balance prompt generality-specificity and eliminate the need for resource-intensive training and inference. The approach consists of three steps: (1) clustering the training data and generating candidate prompts for each cluster using an LLM-based prompt generator; (2) synthesizing a dataset of input-prompt-output tuples for training a prompt evaluator to rank the prompts based on their relevance to the input; (3) using the prompt evaluator to select the best prompt for new input at test time. The approach demonstrates competitive performance on 5 question-answering datasets, including GSM8K, AQuA, MMLU, and real-world datasets from the insurance industry.
Link