The contributions of this work are, the clear identification and description of a fundamental problem, suggested first steps towards addressing this gap, the issuance of relevant Impact Challenges to the machine learning community, and the identification of several key obstacles to machine learning impact, as an aid for focusing future research efforts. Increasingly, ML papers that describe a new algorithm follow a standard evaluation template. After presenting results on synthetic data sets to illustrate certain aspects of the algorithm’s behavior, the paper reports results on a collection of standard data sets, However, in practice direct comparisons fail because we have no standard for reproducibility.
There are also problems with how we measure performance. Most often, an abstract evaluation metric (classification accuracy, root of the mean squared error or RMSE, F-measure is used.
Authors must demonstrate a “machine learning contribution” that is often narrowly interpreted by reviewers as “the development of a new algorithm or the explication of a novel theoretical analysis. Reconnecting active research to relevant real-world problems is part of the process of maturing as a research field. The first step in evaluation is to define or select methods that enable direct measurement, the goal is to develop general methods that apply across domains.
Finally, you should consider potential impact when selecting which research problems to tackle, not merely how interesting or challenging they are from the ML perspective. They proposed the following six Impact Challenges as examples of machine learning that matters: they do not focus on any single problem domain, nor a particular technical capability. The goal is to inspire the field of machine learning to take the steps needed to mature into a valuable contributor to the larger world.
