Difference between revisions of "Machine Learning/Timeline"

From Thalesians Wiki
 
(2 intermediate revisions by the same user not shown)
Line 9: Line 9:
| 1950s || Pioneering machine learning research is conducted using simple algorithms.
| 1950s || Pioneering machine learning research is conducted using simple algorithms.
|-
|-
| 1960s || Bayesian methods are introduced for probabilistic inference in machine learning.<ref>Solomonoff, Ray J. "A formal theory of inductive inference. Part II." Information and control 7.2 (1964): 224–254.</ref>
| 1960s || Bayesian methods are introduced for probabilistic inference in machine learning.<ref name="solomonoff-1964-1">Solomonoff, R. J. A formal theory of inductive inference. Part II. ''Information and Control, Elsevier BV'', '''1964''', 7, 224-254</ref>
|-
|-
| 1970s || 'AI Winter' caused by pessimism about machine learning effectiveness.
| 1970s || '''AI Winter''' caused by pessimism about machine learning effectiveness.
|-
|-
| 1980s || Rediscovery of backpropagation causes a resurgence in machine learning research.
| 1980s || Rediscovery of backpropagation causes a resurgence in machine learning research.
|-
|-
| 1990s || Work on Machine learning shifts from a knowledge-driven approach to a data-driven approach. Scientists begin creating programs for computers to analyze large amounts of data and draw conclusions or "learn" from the results.<ref name="Marr">{{cite news|last1=Marr|first1=Bernard|title=A Short History of Machine Learning Every Manager Should Read|url=https://www.forbes.com/sites/bernardmarr/2016/02/19/a-short-history-of-machine-learning-every-manager-should-read/#2a1a75f9323f|work=Forbes|accessdate=28 Sep 2016}}</ref> [[Support-vector machine]]s (SVMs) and [[recurrent neural network]]s (RNNs) become popular.<ref>{{cite journal|last1=Siegelmann|first1=Hava|first2=Eduardo|last2=Sontag|title=Computational Power of Neural Networks  |journal=Journal of Computer and System Sciences|volume=50|issue=1|year=1995 |pages=132–150|doi=10.1006/jcss.1995.1013|doi-access=free}}</ref> The fields of computational complexity via neural networks and super-Turing computation started.<ref>{{cite journal |last1=Siegelmann |first1=Hava |title=Computation Beyond the Turing Limit |journal=Journal of Computer and System Sciences |volume=238 |issue=28 |year=1995 |pages=632–637|doi=10.1126/science.268.5210.545 |pmid=17756722 |bibcode=1995Sci...268..545S |s2cid=17495161 }}</ref>
| 1990s || Work on Machine learning shifts from a knowledge-driven approach to a data-driven approach. Scientists begin creating programs for computers to analyze large amounts of data and draw conclusions or "learn" from the results.<ref name="marr-2016">Marr, B. A Short History of Machine Learning&mdash;Every Manager Should Read. '''2016'''. URL: https://www.forbes.com/sites/bernardmarr/2016/02/19/a-short-history-of-machine-learning-every-manager-should-read/</ref> Support-vector machines (SVMs) and recurrent neural networks (RNNs) become popular.<ref name="siegelmann-1995">Siegelmann, H. T. &amp; Sontag, E. D. On the Computational Power of Neural Nets. ''Journal of Computer and System Sciences, Elsevier BV'', '''1995''', 50, 132-150</ref> The fields of computational complexity via neural networks and super-Turing computation started.<ref name="siegelmann-1995-1">Siegelmann, H. T. Computation Beyond the Turing Limit. ''Science, American Association for the Advancement of Science (AAAS)'', '''1995''', 268, 545-548</ref>
|-
|-
| 2000s || Support-Vector Clustering<ref>{{cite journal |first1=Asa |last1=Ben-Hur |first2=David |last2=Horn |first3=Hava |last3=Siegelmann |first4=Vladimir|last4=Vapnik |title=Support vector clustering |journal=Journal of Machine Learning Research|volume=2 |year=2001 |pages=51–86}}</ref> and other [[kernel method]]s<ref>{{cite journal |last1=Hofmann |first1=Thomas |first2=Bernhard |last2=Schölkopf |first3=Alexander J. |last3=Smola |title=Kernel methods in machine learning |journal=The Annals of Statistics |volume=36 |issue=3 |year=2008 |pages=1171–1220 |jstor=25464664 |doi=10.1214/009053607000000677|doi-access=free }}</ref> and unsupervised machine learning methods become widespread.<ref>{{cite journal |first1=James |last1=Bennett |first2=Stan |last2=Lanning |title=The netflix prize |journal=Proceedings of KDD Cup and Workshop 2007 |date=2007 |url=https://www.cs.uic.edu/~liub/KDD-cup-2007/NetflixPrize-description.pdf}}</ref>
| 2000s || Support-Vector Clustering<ref name="benhur-2002">Ben-Hur, A.; Horn, D.; Siegelmann, H. T. &amp; Vapnik, V. Support vector clustering. ''The Journal of Machine Learning Research'', '''2002''', 2, 125-137</ref> and other kernel methods<ref name="hofmann-2008">Hofmann, T.; Schölkopf, B. &amp; Smola, A. J. Kernel methods in machine learning. ''The Annals of Statistics, Institute of Mathematical Statistics'', '''2008''', ''36''</ref> and unsupervised machine learning methods become widespread.<ref name="bennett-2007">Bennett, J. &amp; Lanning, S. The Netflix Prize. ''Proceedings of KDD Cup and Workshop 2007'', 2007</ref>
|-
|-
| 2010s || [[Deep learning]] becomes feasible, which leads to machine learning becoming integral to many widely used software services and applications.
| 2010s || Deep learning becomes feasible, which leads to machine learning becoming integral to many widely used software services and applications.
|}
|}


Line 51: Line 51:
| 1963 || Achievement || Machines Playing Tic-Tac-Toe || [[Donald Michie]] creates a 'machine' consisting of 304 match boxes and beads, which uses [[reinforcement learning]] to play [[Tic-tac-toe]] (also known as noughts and crosses).<ref>{{cite web|last1=Child|first1=Oliver|title=Menace: the Machine Educable Noughts And Crosses Engine Read|url=http://chalkdustmagazine.com/features/menace-machine-educable-noughts-crosses-engine/#more-3326|website=Chalkdust Magazine |date=13 March 2016|accessdate=16 Jan 2018}}</ref>
| 1963 || Achievement || Machines Playing Tic-Tac-Toe || [[Donald Michie]] creates a 'machine' consisting of 304 match boxes and beads, which uses [[reinforcement learning]] to play [[Tic-tac-toe]] (also known as noughts and crosses).<ref>{{cite web|last1=Child|first1=Oliver|title=Menace: the Machine Educable Noughts And Crosses Engine Read|url=http://chalkdustmagazine.com/features/menace-machine-educable-noughts-crosses-engine/#more-3326|website=Chalkdust Magazine |date=13 March 2016|accessdate=16 Jan 2018}}</ref>
|-  
|-  
| 1967 || || Nearest Neighbor || The [[nearest neighbor algorithm]] was created, which is the start of basic pattern recognition. The algorithm was used to map routes.<ref name="Marr" />
| 1967 || || Nearest Neighbor || The [[nearest neighbor algorithm]] was created, which is the start of basic pattern recognition. The algorithm was used to map routes.<ref name="marr-2016" />
|-
|-
| 1969 || || Limitations of Neural Networks || [[Marvin Minsky]] and [[Seymour Papert]] publish their book ''[[Perceptrons (book)|Perceptrons]]'', describing some of the limitations of perceptrons and neural networks. The interpretation that the book shows that neural networks are fundamentally limited is seen as a hindrance for research into neural networks.<ref>{{cite web|last1=Cohen|first1=Harvey|title=The Perceptron|url=http://harveycohen.net/image/perceptron.html|accessdate=5 June 2016}}</ref><ref>{{cite web|last1=Colner|first1=Robert|title=A brief history of machine learning|url=http://www.slideshare.net/bobcolner/a-brief-history-of-machine-learning|website=SlideShare|date=4 March 2016|accessdate=5 June 2016}}</ref>
| 1969 || || Limitations of Neural Networks || [[Marvin Minsky]] and [[Seymour Papert]] publish their book ''[[Perceptrons (book)|Perceptrons]]'', describing some of the limitations of perceptrons and neural networks. The interpretation that the book shows that neural networks are fundamentally limited is seen as a hindrance for research into neural networks.<ref>{{cite web|last1=Cohen|first1=Harvey|title=The Perceptron|url=http://harveycohen.net/image/perceptron.html|accessdate=5 June 2016}}</ref><ref>{{cite web|last1=Colner|first1=Robert|title=A brief history of machine learning|url=http://www.slideshare.net/bobcolner/a-brief-history-of-machine-learning|website=SlideShare|date=4 March 2016|accessdate=5 June 2016}}</ref>
Line 57: Line 57:
| 1970 || || Automatic Differentiation (Backpropagation) || [[Seppo Linnainmaa]] publishes the general method for automatic differentiation (AD) of discrete connected networks of nested differentiable functions.<ref name="lin1970">[[Seppo Linnainmaa]] (1970). "The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors." Master's Thesis (in Finnish), Univ. Helsinki, 6–7.</ref><ref name="lin1976">{{cite journal |first=Seppo |last=Linnainmaa |authorlink=Seppo Linnainmaa |year=1976 |title=Taylor expansion of the accumulated rounding error |journal=BIT Numerical Mathematics |volume=16 |issue=2 |pages=146–160 |doi=10.1007/BF01931367|s2cid=122357351 }}</ref> This corresponds to the modern version of backpropagation, but is not yet named as such.<ref name="grie2012">{{cite journal |last=Griewank |first=Andreas |year=2012 |title=Who Invented the Reverse Mode of Differentiation? |journal=Documenta Matematica, Extra Volume ISMP |pages=389–400}}</ref><ref name="grie2008">Griewank, Andreas and Walther, A. ''Principles and Techniques of Algorithmic Differentiation, Second Edition''. SIAM, 2008.</ref><ref name="schmidhuber2015">{{cite journal |authorlink=Jürgen Schmidhuber |last=Schmidhuber |first=Jürgen |year=2015 |title=Deep learning in neural networks: An overview |journal=Neural Networks |volume=61 |pages=85–117 |arxiv=1404.7828|bibcode=2014arXiv1404.7828S |doi=10.1016/j.neunet.2014.09.003 |pmid=25462637|s2cid=11715509 }}</ref><ref name="scholarpedia2015">{{cite journal | last1 = Schmidhuber | first1 = Jürgen | authorlink = Jürgen Schmidhuber | year = 2015 | title = Deep Learning (Section on Backpropagation) | journal = Scholarpedia | volume = 10 | issue = 11| page = 32832 | doi = 10.4249/scholarpedia.32832 | bibcode = 2015SchpJ..1032832S | doi-access = free }}</ref>
| 1970 || || Automatic Differentiation (Backpropagation) || [[Seppo Linnainmaa]] publishes the general method for automatic differentiation (AD) of discrete connected networks of nested differentiable functions.<ref name="lin1970">[[Seppo Linnainmaa]] (1970). "The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors." Master's Thesis (in Finnish), Univ. Helsinki, 6–7.</ref><ref name="lin1976">{{cite journal |first=Seppo |last=Linnainmaa |authorlink=Seppo Linnainmaa |year=1976 |title=Taylor expansion of the accumulated rounding error |journal=BIT Numerical Mathematics |volume=16 |issue=2 |pages=146–160 |doi=10.1007/BF01931367|s2cid=122357351 }}</ref> This corresponds to the modern version of backpropagation, but is not yet named as such.<ref name="grie2012">{{cite journal |last=Griewank |first=Andreas |year=2012 |title=Who Invented the Reverse Mode of Differentiation? |journal=Documenta Matematica, Extra Volume ISMP |pages=389–400}}</ref><ref name="grie2008">Griewank, Andreas and Walther, A. ''Principles and Techniques of Algorithmic Differentiation, Second Edition''. SIAM, 2008.</ref><ref name="schmidhuber2015">{{cite journal |authorlink=Jürgen Schmidhuber |last=Schmidhuber |first=Jürgen |year=2015 |title=Deep learning in neural networks: An overview |journal=Neural Networks |volume=61 |pages=85–117 |arxiv=1404.7828|bibcode=2014arXiv1404.7828S |doi=10.1016/j.neunet.2014.09.003 |pmid=25462637|s2cid=11715509 }}</ref><ref name="scholarpedia2015">{{cite journal | last1 = Schmidhuber | first1 = Jürgen | authorlink = Jürgen Schmidhuber | year = 2015 | title = Deep Learning (Section on Backpropagation) | journal = Scholarpedia | volume = 10 | issue = 11| page = 32832 | doi = 10.4249/scholarpedia.32832 | bibcode = 2015SchpJ..1032832S | doi-access = free }}</ref>
|-  
|-  
| 1979 || || Stanford Cart || Students at Stanford University develop a cart that can navigate and avoid obstacles in a room.<ref name="Marr" />
| 1979 || || Stanford Cart || Students at Stanford University develop a cart that can navigate and avoid obstacles in a room.<ref name="marr-2016" />
|-  
|-  
| 1979 || Discovery || Neocognitron || [[Kunihiko Fukushima]] first publishes his work on the [[neocognitron]], a type of [[artificial neural network]] (ANN).<ref>{{cite journal
| 1979 || Discovery || Neocognitron || [[Kunihiko Fukushima]] first publishes his work on the [[neocognitron]], a type of [[artificial neural network]] (ANN).<ref>{{cite journal
Line 74: Line 74:
}}</ref><ref>{{cite journal|last1=Fukushima|first1=Kunihiko|title=Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern The Recognitron Unaffected by Shift in Position|journal=Biological Cybernetics|date=April 1980|volume=36|issue=4|pages=193–202|url=http://www.cs.princeton.edu/courses/archive/spr08/cos598B/Readings/Fukushima1980.pdf|accessdate=5 June 2016|doi=10.1007/bf00344251|pmid=7370364|s2cid=206775608}}</ref> [[Neocognitron|Neocognition]] later inspires [[convolutional neural network]]s (CNNs).<ref>{{cite journal|last1=Le Cun|first1=Yann|title=Deep Learning|citeseerx=10.1.1.297.6176}}</ref>
}}</ref><ref>{{cite journal|last1=Fukushima|first1=Kunihiko|title=Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern The Recognitron Unaffected by Shift in Position|journal=Biological Cybernetics|date=April 1980|volume=36|issue=4|pages=193–202|url=http://www.cs.princeton.edu/courses/archive/spr08/cos598B/Readings/Fukushima1980.pdf|accessdate=5 June 2016|doi=10.1007/bf00344251|pmid=7370364|s2cid=206775608}}</ref> [[Neocognitron|Neocognition]] later inspires [[convolutional neural network]]s (CNNs).<ref>{{cite journal|last1=Le Cun|first1=Yann|title=Deep Learning|citeseerx=10.1.1.297.6176}}</ref>
|-
|-
| 1981 || || Explanation Based Learning || Gerald Dejong introduces Explanation Based Learning, where a computer algorithm analyses data and creates a general rule it can follow and discard unimportant data.<ref name="Marr" />
| 1981 || || Explanation Based Learning || Gerald Dejong introduces Explanation Based Learning, where a computer algorithm analyses data and creates a general rule it can follow and discard unimportant data.<ref name="marr-2016" />
|-
|-
| 1982 || Discovery || Recurrent Neural Network || [[John Hopfield]] popularizes [[Hopfield networks]], a type of [[recurrent neural network]] that can serve as [[content-addressable memory]] systems.<ref>{{cite journal|last1=Hopfield|first1=John|title=Neural networks and physical systems with emergent collective computational abilities|journal=Proceedings of the National Academy of Sciences of the United States of America|date=April 1982|volume=79|issue=8|pages=2554–2558|url=http://www.pnas.org/content/79/8/2554.full.pdf|accessdate=8 June 2016|doi=10.1073/pnas.79.8.2554|pmid=6953413|pmc=346238|bibcode=1982PNAS...79.2554H|doi-access=free}}</ref>
| 1982 || Discovery || Recurrent Neural Network || [[John Hopfield]] popularizes [[Hopfield networks]], a type of [[recurrent neural network]] that can serve as [[content-addressable memory]] systems.<ref>{{cite journal|last1=Hopfield|first1=John|title=Neural networks and physical systems with emergent collective computational abilities|journal=Proceedings of the National Academy of Sciences of the United States of America|date=April 1982|volume=79|issue=8|pages=2554–2558|url=http://www.pnas.org/content/79/8/2554.full.pdf|accessdate=8 June 2016|doi=10.1073/pnas.79.8.2554|pmid=6953413|pmc=346238|bibcode=1982PNAS...79.2554H|doi-access=free}}</ref>
|-
|-
| 1985 || || NetTalk || A program that learns to pronounce words the same way a baby does, is developed by Terry Sejnowski.<ref name="Marr" />
| 1985 || || NetTalk || A program that learns to pronounce words the same way a baby does, is developed by Terry Sejnowski.<ref name="marr-2016" />
|-  
|-  
| 1986 || Application || Backpropagation || [[Seppo Linnainmaa]]'s reverse mode of [[automatic differentiation]] (first applied to neural networks by [[Paul Werbos]]) is used in experiments by [[David Rumelhart]], [[Geoff Hinton]] and [[Ronald J. Williams]] to learn [[Knowledge representation|internal representations]].<ref>{{cite journal|last1=Rumelhart|first1=David|last2=Hinton|first2=Geoffrey|last3=Williams|first3=Ronald|title=Learning representations by back-propagating errors|journal=Nature|date=9 October 1986|volume=323|issue=6088|pages=533–536|url=http://elderlab.yorku.ca/~elder/teaching/cosc6390psyc6225/readings/hinton%201986.pdf|accessdate=5 June 2016|doi=10.1038/323533a0|bibcode=1986Natur.323..533R|s2cid=205001834}}</ref>
| 1986 || Application || Backpropagation || [[Seppo Linnainmaa]]'s reverse mode of [[automatic differentiation]] (first applied to neural networks by [[Paul Werbos]]) is used in experiments by [[David Rumelhart]], [[Geoff Hinton]] and [[Ronald J. Williams]] to learn [[Knowledge representation|internal representations]].<ref>{{cite journal|last1=Rumelhart|first1=David|last2=Hinton|first2=Geoffrey|last3=Williams|first3=Ronald|title=Learning representations by back-propagating errors|journal=Nature|date=9 October 1986|volume=323|issue=6088|pages=533–536|url=http://elderlab.yorku.ca/~elder/teaching/cosc6390psyc6225/readings/hinton%201986.pdf|accessdate=5 June 2016|doi=10.1038/323533a0|bibcode=1986Natur.323..533R|s2cid=205001834}}</ref>
Line 92: Line 92:
| 1995 || Discovery || Support-Vector Machines || [[Corinna Cortes]] and [[Vladimir Vapnik]] publish their work on [[support-vector machine]]s.<ref name="bhml">{{cite web|last1=Golge|first1=Eren|title=BRIEF HISTORY OF MACHINE LEARNING|url=http://www.erogol.com/brief-history-machine-learning/|website=A Blog From a Human-engineer-being|accessdate=5 June 2016}}</ref><ref>{{cite journal|last1=Cortes|first1=Corinna|last2=Vapnik|first2=Vladimir|title=Support-vector networks|journal=Machine Learning|date=September 1995|volume=20|issue=3|pages=273–297|doi=10.1007/BF00994018|publisher=Kluwer Academic Publishers|issn=0885-6125|doi-access=free}}</ref>
| 1995 || Discovery || Support-Vector Machines || [[Corinna Cortes]] and [[Vladimir Vapnik]] publish their work on [[support-vector machine]]s.<ref name="bhml">{{cite web|last1=Golge|first1=Eren|title=BRIEF HISTORY OF MACHINE LEARNING|url=http://www.erogol.com/brief-history-machine-learning/|website=A Blog From a Human-engineer-being|accessdate=5 June 2016}}</ref><ref>{{cite journal|last1=Cortes|first1=Corinna|last2=Vapnik|first2=Vladimir|title=Support-vector networks|journal=Machine Learning|date=September 1995|volume=20|issue=3|pages=273–297|doi=10.1007/BF00994018|publisher=Kluwer Academic Publishers|issn=0885-6125|doi-access=free}}</ref>
|-
|-
| 1997 || Achievement || IBM Deep Blue Beats Kasparov || IBM's [[Deep Blue (chess computer)|Deep Blue]] beats the world champion at chess.<ref name="Marr" />
| 1997 || Achievement || IBM Deep Blue Beats Kasparov || IBM's [[Deep Blue (chess computer)|Deep Blue]] beats the world champion at chess.<ref name="marr-2016" />
|-
|-
| 1997 || Discovery || LSTM || [[Sepp Hochreiter]] and [[Jürgen Schmidhuber]] invent [[long short-term memory]] (LSTM) recurrent neural networks,<ref>{{cite journal|last1=Hochreiter|first1=Sepp|last2=Schmidhuber|first2=Jürgen|title=Long Short-Term Memory|journal=Neural Computation|date=1997|volume=9|issue=8|pages=1735–1780|url=http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf|doi=10.1162/neco.1997.9.8.1735|pmid=9377276|s2cid=1915014|url-status=dead|archiveurl=https://web.archive.org/web/20150526132154/http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf|archivedate=2015-05-26}}</ref> greatly improving the efficiency and practicality of recurrent neural networks.
| 1997 || Discovery || LSTM || [[Sepp Hochreiter]] and [[Jürgen Schmidhuber]] invent [[long short-term memory]] (LSTM) recurrent neural networks,<ref>{{cite journal|last1=Hochreiter|first1=Sepp|last2=Schmidhuber|first2=Jürgen|title=Long Short-Term Memory|journal=Neural Computation|date=1997|volume=9|issue=8|pages=1735–1780|url=http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf|doi=10.1162/neco.1997.9.8.1735|pmid=9377276|s2cid=1915014|url-status=dead|archiveurl=https://web.archive.org/web/20150526132154/http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf|archivedate=2015-05-26}}</ref> greatly improving the efficiency and practicality of recurrent neural networks.

Latest revision as of 12:49, 23 December 2021

Overview

Decade Summary
<1950s Statistical methods are discovered and refined.
1950s Pioneering machine learning research is conducted using simple algorithms.
1960s Bayesian methods are introduced for probabilistic inference in machine learning.[1]
1970s AI Winter caused by pessimism about machine learning effectiveness.
1980s Rediscovery of backpropagation causes a resurgence in machine learning research.
1990s Work on Machine learning shifts from a knowledge-driven approach to a data-driven approach. Scientists begin creating programs for computers to analyze large amounts of data and draw conclusions or "learn" from the results.[2] Support-vector machines (SVMs) and recurrent neural networks (RNNs) become popular.[3] The fields of computational complexity via neural networks and super-Turing computation started.[4]
2000s Support-Vector Clustering[5] and other kernel methods[6] and unsupervised machine learning methods become widespread.[7]
2010s Deep learning becomes feasible, which leads to machine learning becoming integral to many widely used software services and applications.

Timeline

Year Event type Caption Event
1763 Discovery The Underpinnings of Bayes' Theorem Thomas Bayes's work An Essay towards solving a Problem in the Doctrine of Chances is published two years after his death, having been amended and edited by a friend of Bayes, Richard Price.[8] The essay presents work which underpins Bayes theorem.
1805 Discovery Least Square Adrien-Marie Legendre describes the "méthode des moindres carrés", known in English as the least squares method.[9] The least squares method is used widely in data fitting.
1812 Bayes' Theorem Pierre-Simon Laplace publishes Théorie Analytique des Probabilités, in which he expands upon the work of Bayes and defines what is now known as Bayes' Theorem.[10]
1913 Discovery Markov Chains Andrey Markov first describes techniques he used to analyse a poem. The techniques later become known as Markov chains.[11]
1943 Discovery Artificial Neuron Warren McCulloch and Walter Pitts develop a mathematical model that imitates the functioning of a biological neuron, the artificial neuron which is considered to be the first neural model invented.[12]
1950 Turing's Learning Machine Alan Turing proposes a 'learning machine' that could learn and become artificially intelligent. Turing's specific proposal foreshadows genetic algorithms.[13]
1951 First Neural Network Machine Marvin Minsky and Dean Edmonds build the first neural network machine, able to learn, the SNARC.[14]
1952 Machines Playing Checkers Arthur Samuel joins IBM's Poughkeepsie Laboratory and begins working on some of the very first machine learning programs, first creating programs that play checkers.[15]
1957 Discovery Perceptron Frank Rosenblatt invents the perceptron while working at the Cornell Aeronautical Laboratory.[16] The invention of the perceptron generated a great deal of excitement and was widely covered in the media.[17]
1963 Achievement Machines Playing Tic-Tac-Toe Donald Michie creates a 'machine' consisting of 304 match boxes and beads, which uses reinforcement learning to play Tic-tac-toe (also known as noughts and crosses).[18]
1967 Nearest Neighbor The nearest neighbor algorithm was created, which is the start of basic pattern recognition. The algorithm was used to map routes.[2]
1969 Limitations of Neural Networks Marvin Minsky and Seymour Papert publish their book Perceptrons, describing some of the limitations of perceptrons and neural networks. The interpretation that the book shows that neural networks are fundamentally limited is seen as a hindrance for research into neural networks.[19][20]
1970 Automatic Differentiation (Backpropagation) Seppo Linnainmaa publishes the general method for automatic differentiation (AD) of discrete connected networks of nested differentiable functions.[21][22] This corresponds to the modern version of backpropagation, but is not yet named as such.[23][24][25][26]
1979 Stanford Cart Students at Stanford University develop a cart that can navigate and avoid obstacles in a room.[2]
1979 Discovery Neocognitron Kunihiko Fukushima first publishes his work on the neocognitron, a type of artificial neural network (ANN).[27][28] Neocognition later inspires convolutional neural networks (CNNs).[29]
1981 Explanation Based Learning Gerald Dejong introduces Explanation Based Learning, where a computer algorithm analyses data and creates a general rule it can follow and discard unimportant data.[2]
1982 Discovery Recurrent Neural Network John Hopfield popularizes Hopfield networks, a type of recurrent neural network that can serve as content-addressable memory systems.[30]
1985 NetTalk A program that learns to pronounce words the same way a baby does, is developed by Terry Sejnowski.[2]
1986 Application Backpropagation Seppo Linnainmaa's reverse mode of automatic differentiation (first applied to neural networks by Paul Werbos) is used in experiments by David Rumelhart, Geoff Hinton and Ronald J. Williams to learn internal representations.[31]
1989 Discovery Reinforcement Learning Christopher Watkins develops Q-learning, which greatly improves the practicality and feasibility of reinforcement learning.[32]
1989 Commercialization Commercialization of Machine Learning on Personal Computers Axcelis, Inc. releases Evolver, the first software package to commercialize the use of genetic algorithms on personal computers.[33]
1992 Achievement Machines Playing Backgammon Gerald Tesauro develops TD-Gammon, a computer backgammon program that uses an artificial neural network trained using temporal-difference learning (hence the 'TD' in the name). TD-Gammon is able to rival, but not consistently surpass, the abilities of top human backgammon players.[34]
1995 Discovery Random Forest Algorithm Tin Kam Ho publishes a paper describing random decision forests.[35]
1995 Discovery Support-Vector Machines Corinna Cortes and Vladimir Vapnik publish their work on support-vector machines.[36][37]
1997 Achievement IBM Deep Blue Beats Kasparov IBM's Deep Blue beats the world champion at chess.[2]
1997 Discovery LSTM Sepp Hochreiter and Jürgen Schmidhuber invent long short-term memory (LSTM) recurrent neural networks,[38] greatly improving the efficiency and practicality of recurrent neural networks.
1998 MNIST database A team led by Yann LeCun releases the MNIST database, a dataset comprising a mix of handwritten digits from American Census Bureau employees and American high school students.[39] The MNIST database has since become a benchmark for evaluating handwriting recognition.
2002 Torch Machine Learning Library Torch, a software library for machine learning, is first released.[40]
2006 The Netflix Prize The Netflix Prize competition is launched by Netflix. The aim of the competition was to use machine learning to beat Netflix's own recommendation software's accuracy in predicting a user's rating for a film given their ratings for previous films by at least 10%.[41] The prize was won in 2009.
2009 Achievement ImageNet ImageNet is created. ImageNet is a large visual database envisioned by Fei-Fei Li from Stanford University, who realized that the best machine learning algorithms wouldn't work well if the data didn't reflect the real world.[42] For many, ImageNet was the catalyst for the AI boom[43] of the 21st century.
2010 Kaggle Competition Kaggle, a website that serves as a platform for machine learning competitions, is launched.[44]
2011 Achievement Beating Humans in Jeopardy Using a combination of machine learning, natural language processing and information retrieval techniques, IBM's Watson beats two human champions in a Jeopardy! competition.[45]
2012 Achievement Recognizing Cats on YouTube The Google Brain team, led by Andrew Ng and Jeff Dean, create a neural network that learns to recognize cats by watching unlabeled images taken from frames of YouTube videos.[46][47]
2014 Leap in Face Recognition Facebook researchers publish their work on DeepFace, a system that uses neural networks that identifies faces with 97.35% accuracy. The results are an improvement of more than 27% over previous systems and rivals human performance.[48]
2014 Sibyl Researchers from Google detail their work on Sibyl,[49] a proprietary platform for massively parallel machine learning used internally by Google to make predictions about user behavior and provide recommendations.[50]
2016 Achievement Beating Humans in Go Google's AlphaGo program becomes the first Computer Go program to beat an unhandicapped professional human player[51] using a combination of machine learning and tree search techniques.[52] Later improved as AlphaGo Zero and then in 2017 generalized to Chess and more two-player games with AlphaZero.

References

  1. Solomonoff, R. J. A formal theory of inductive inference. Part II. Information and Control, Elsevier BV, 1964, 7, 224-254
  2. 2.0 2.1 2.2 2.3 2.4 2.5 Marr, B. A Short History of Machine Learning—Every Manager Should Read. 2016. URL: https://www.forbes.com/sites/bernardmarr/2016/02/19/a-short-history-of-machine-learning-every-manager-should-read/
  3. Siegelmann, H. T. & Sontag, E. D. On the Computational Power of Neural Nets. Journal of Computer and System Sciences, Elsevier BV, 1995, 50, 132-150
  4. Siegelmann, H. T. Computation Beyond the Turing Limit. Science, American Association for the Advancement of Science (AAAS), 1995, 268, 545-548
  5. Ben-Hur, A.; Horn, D.; Siegelmann, H. T. & Vapnik, V. Support vector clustering. The Journal of Machine Learning Research, 2002, 2, 125-137
  6. Hofmann, T.; Schölkopf, B. & Smola, A. J. Kernel methods in machine learning. The Annals of Statistics, Institute of Mathematical Statistics, 2008, 36
  7. Bennett, J. & Lanning, S. The Netflix Prize. Proceedings of KDD Cup and Workshop 2007, 2007
  8. Template:Cite journal
  9. Template:Cite book
  10. Template:Cite web
  11. Template:Cite journal
  12. Template:Cite journal
  13. Template:Cite journal
  14. Template:Harvnb and Template:Harvnb
  15. Template:Cite news
  16. Template:Cite journal
  17. Template:Cite news
  18. Template:Cite web
  19. Template:Cite web
  20. Template:Cite web
  21. Seppo Linnainmaa (1970). "The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors." Master's Thesis (in Finnish), Univ. Helsinki, 6–7.
  22. Template:Cite journal
  23. Template:Cite journal
  24. Griewank, Andreas and Walther, A. Principles and Techniques of Algorithmic Differentiation, Second Edition. SIAM, 2008.
  25. Template:Cite journal
  26. Template:Cite journal
  27. Template:Cite journal
  28. Template:Cite journal
  29. Template:Cite journal
  30. Template:Cite journal
  31. Template:Cite journal
  32. Template:Cite journal
  33. Template:Cite news
  34. Template:Cite journal
  35. Template:Cite journal
  36. Template:Cite web
  37. Template:Cite journal
  38. Template:Cite journal
  39. Template:Cite web
  40. Template:Cite journal
  41. Template:Cite web
  42. Template:Cite web
  43. Template:Cite news
  44. Template:Cite web
  45. Template:Cite news
  46. Template:Cite conference
  47. Template:Cite news
  48. Template:Cite journal
  49. Template:Cite web
  50. Template:Cite news
  51. Template:Cite web
  52. Template:Cite web