Abstract: In this talk, I discuss benchmarking and interpreting large language models in the context of Swedish. I present a selection of work from my PhD thesis, which analyze LLMs Swedish-specific capabilities in different areas: English-Swedish language transfer, multi-task benchmarking on Swedish NLU and targeted analysis of a specific case of Swedish linguistic variation. I outline some of the challenges that arise when trying to assess the language-specific capabilities of LLMs, some lessons I've learned throughout from my work and give a future outlook.
Bio: Felix Morger is a software engineer and computational linguist based in Gothenburg, Sweden. He recently defended his PhD thesis, titled "In the minds of stochastic parrots: Benchmarking, evaluating and interpreting large language models". His main research interests are in benchmarking and model interpretability of large language models with a large focus on Swedish.
