AI MEETINGS

We organize weekly AI meetings on zoom. We hold these events every Wednesday from 4 pm to 5 pm unless otherwise noted. We have guests from various universities and industries all over the world. Also, our graduate students present their progress some weeks. If you want to join us in these meetings, please email us at ai-info@ku.edu.tr for the zoom info. 

Dec 1st, 2022

 Ekin Akyürek, Massachusetts Institute of Technology (MIT)

What Learning Algorithm Is In-Context Learning? Investigations with Linear Models

Neural sequence models, especially transformers, exhibit a remarkable capacity for in-context learning. They can construct new predictors from sequences of labeled examples (x, f(x)) presented in the input without further parameter updates. We investigate the hypothesis that transformer-based in-context learners implement standard learning algorithms implicitly, by encoding smaller models in their activations, and updating these implicit models as new examples appear in the context. Using linear regression as a prototypical problem, we offer three sources of evidence for this hypothesis. First, we prove by construction that transformers can implement learning algorithms for linear models based on gradient descent and closed-form ridge regression. Second, we show that trained in-context learners closely match the predictors computed by gradient descent, ridge regression, and exact least-squares regression, transitioning between different predictors as transformer depth and dataset noise vary, and converging to Bayesian estimators for large widths and depths. Third, we present preliminary evidence that in-context learners share algorithmic features with these predictors: learners’ late layers non-linearly encode weight vectors and moment matrices. These results suggest that in-context learning is understandable in algorithmic terms, and that (at least in the linear case) learners may rediscover standard estimation algorithms. Code and reference implementations released at this http link.