Ferenc Huszár

I'm interested in the principles of machine learning, with a particular focus on modern deep learning methods. My research falls into the following themes:

• Optimization, Generalization and Transferability in Deep Learning: Why do deep networks generalise? What is the role of stochastic gradient descent? What is the role of the (over)parametrization of neural networks? Can we design optimisation algorithms that find better minima? Why does transfer learning work so well in neural networks?
• Mathematical Models of Emergent Behaviours: It is clear that notions of statistical generalisation are insufficient to describe the range of "intelligent behaviours" LLMs display: in-context learning, reasoning, extrapolation of rules to unseen data. We need new mathematical models of these behaviours so we can reason about why we see them, and what aspects of training give rise to them.
• Unsupervised Representation Learning: What are the hallmarks of good neural representations of data, and how do we discover these from data without labels? How can we formalize the goals and principles of unsupervised representation learning, of self-supervised learning, of transfer learning? Can we understand why self-supervised learning works so well in vision and NLP?
• Probabilistic foundations: How should we represent uncertainty about models in deep learning? Is Bayesian inference a good principle for deep learning? Can we develop alternative ways of quantifying uncertainty that line up better with our goal? Can we develop practical inference algorithms in prediction-oriented, loss-calibrated situations?
• Causal Inference and Identifiability: The world of causal inference is full of non-identifiability: that certain causal relationships cannot be inferred from observational data alone. However, we see that humans as well as machine learning systems can often recover correct causal information in spite of this limitation. We study situations where causal relationships are identifiable and the role inductive biases play in identifying causal relationships where data provides insufficient constraints.