The May 20, 2024 academic paper explores the **metacognitive capabilities of Large Language Models (LLMs)**, specifically focusing on mathematical problem-solving. The core approach involves developing a method for a powerful LLM, such as GPT-4, to **identify and label mathematical questions with specific skills**, which are then organized into broader, interpretable categories. This process creates a **Skill Exemplar Repository** containing skill names matched with question-answer pairs. Experiments validate that providing an LLM with these skill labels and associated examples as in-context prompts **significantly improves accuracy** on challenging math datasets like MATH and GSM8K, outperforming baseline prompting techniques like Chain-of-Thought. Furthermore, the **skill knowledge transferred effectively** to other, less powerful LLMs and different math datasets, demonstrating the utility of this LLM-generated metacognitive framework.

Source:

https://arxiv.org/pdf/2405.12205