On the Measure of Intelligence
2019/11/05 by François Chollet, Chollet, François · 31 voices · 1 citation
#cs.AI
paper · pdf · doi:10.48550/arxiv.1911.01547
Abstract
To make deliberate progress towards more intelligent and more human-like artificial systems, we need to be following an appropriate feedback signal: we need to be able to define and evaluate intelligence in a way that enables comparisons between two systems, as well as comparisons with humans. Over the past hundred years, there has been an abundance of attempts to define and measure intelligence, across both the fields of psychology and AI. We summarize and critically assess these definitions and evaluation approaches, while making apparent the two historical conceptions of intelligence that have implicitly guided them. We note that in practice, the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs and humans at specific tasks such as board games and video games. We argue that solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to "buy" arbitrary levels of skills for a system, in a way that masks the system's own generalization power. We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience. Using this definition, we propose a set of guidelines for what a general AI benchmark should look like. Finally, we present a benchmark closely following these guidelines, the Abstraction and Reasoning Corpus (ARC), built upon an explicit set of priors designed to be as close as possible to innate human priors. We argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.
Cited by
Discussions
- The Measure of Intelligence: towards more human-like artificial systems [hn, 80 points, 12 comments]
- The Measure of Intelligence [hn, 16 points, 1 comments]
- François Chollet: The Measure of Intelligence [hn, 13 points, 0 comments]
- Among other things, Francois is the author of "On the Meaning of Intelligence," and co-creator of the ARC Challenge to test for true artificial intelligence. arxiv.org/abs/1911.01547 arcprize.org/blo [bsky, 10 points, 0 comments]
- On the Measure of Intelligence (2019) [hn, 4 points, 0 comments]
- "ARC does not appear to be approachable by any existing machine learning technique (including Deep Learning), due to its focus on broad generalization and few-shot learning, as well as the fact that t [bsky, 4 points, 1 comments]
- The ARC (Abstraction and Reasoning Corpus), introduced by Francois Chollet, is a modern-day update of what has become known as "Bongard problems" thanks to their popularization in Hofstadter's _Gödel, [bsky, 3 points, 1 comments]
- Chollet's also very clear about his definition of intelligence, which also draws on the field of psychology. Just because OpenAI has co-opted the competition for their marketing stunt doesn't mean tha [bsky, 3 points, 0 comments]
- francios chollet's On The Measure of Intelligence is a serious attempt to do just that [bsky, 3 points, 0 comments]
- I find François Chollet's definition of #intelligence as "skill-acquisition #efficiency" promising. It switches the focus from the #volume of innate, given (designed), or acquired skills to the #rate [bsky, 3 points, 1 comments]
- The Measure of Intelligence [hn, 2 points, 0 comments]
- On the Measure of Intelligence, Chollet [pdf] [hn, 2 points, 0 comments]
- The Measure of Intelligence [hn, 2 points, 0 comments]
- On the measure of intelligence (2019) [hn, 2 points, 1 comments]
- On the Measure of Intelligence [hn, 2 points, 0 comments]
- I don’t think that’s true on either count, if we are going to hew to the most classic model of “intelligence” from cognitive science. arxiv.org/pdf/1911.01547 [bsky, 2 points, 0 comments]
- The Measure of Intelligence [hn, 1 points, 0 comments]
- Esse aqui deve ser literalmente meu artigo científico favorito. LINK pra quem tem interesse: arxiv.org/pdf/1911.01547 [bsky, 1 points, 1 comments]
- This paper might help arxiv.org/pdf/1911.01547 [bsky, 0 points, 1 comments]
- the only benchmark that matters is this one arxiv.org/abs/1911.01547 [bsky, 0 points, 0 comments]
- Not true. Read arxiv.org/abs/1911.01547 page 47: "A test-taker is also assumed to have access to the entirety of the training set" [bsky, 0 points, 2 comments]
- Do you think one of the components of the ARC test should be emotional reasoning? arxiv.org/pdf/1911.01547 [bsky, 0 points, 0 comments]
- saving here for later arxiv.org/abs/1911.01547 [bsky, 0 points, 0 comments]
- The paper introducing the task: arxiv.org/abs/1911.01547 [bsky, 0 points, 1 comments]
- Now reading the ARC paper by @fchollet. https://arxiv.org/abs/1911.01547 “On the measure of intelligence” where he proposes a new benchmark for “intelligence” called the “Abstraction and Reasoning cor [bsky, 0 points, 1 comments]
- "Besides Keras, your 60 page writeup on the measure of intelligence was great. It proposed the right and pragmatic perspective to think about generalization." @x.com@shaneguML of @x.com@fchollet ... [bsky, 0 points, 0 comments]
- arxiv.org/abs/1911.01547 [bsky, 0 points, 0 comments]
- It's not an intractable problem. It's a very solvable problem and many people have attempted to do so. e.g. one of the famous ones is this. But it's not a "standard" definition or a complete one IMHO. [bsky, 0 points, 1 comments]
- arxiv.org/abs/1911.01547 [bsky, 0 points, 0 comments]
- On the Measure of Intelligence [bsky, 0 points, 0 comments]
- arxiv.org/abs/1911.01547 Surreal how this work has become even more pertinent now in 2025. Still criminally underrated in the mainstream ai community [bsky, 0 points, 0 comments]
Related