The research proposes Parallel Scaling (PARSCALE) as a novel, efficient strategy to enhance Large Language Model (LLM) capacity by increasing parallel computation rather than merely growing the parameter count. This method reuses existing model parameters by feeding multiple parallel input streams (differentiated by learned prefixes) and dynamically combining their outputs into a single prediction. Through extensive testing, the paper develops a new scaling law, showing that scaling computation by a factor of P provides performance gains roughly equivalent to scaling parameters by a factor of O(N logP). PARSCALE demonstrates particular effectiveness in boosting performance on reasoning-intensive tasks like coding and mathematics problems. Critically, this scaling technique offers superior efficiency during inference, requiring significantly less memory and time increase than traditional parameter scaling, thereby making it highly suitable for low-resource edge deployment.

Source:

https://openreview.net/pdf?id=dEi1S731lk