5:15 PM - 6:30 PM
[U10-P01] Researchers' perceptions of research and its evaluation: A comparison between the JpGU and AGU communities
Keywords:Institutional Research (IR), Impact Factor (IF), the worship of papers, creation of knowledge, ideals of science and researchers
We could answer naive questions such as "What kind of research do you think is good?" and "Why do you get research funding?" based on our personal experiences. But we could not answer these questions based on the data as our research community's opinion. We have easily used quantifiable indicators such as the number of citations and journal impact factors (JIF) as evaluation indices, without any researchers' ideal and discussions. As a result, our own evaluation has been distorted. A situation is emerging where we can believe that if we were researchers who had papers with high evaluation indexes and our organization had many such researchers, the organizations and researchers would obtain more research funds allocated by the government (we call it "the worship of papers"). However, does such a situation equals contribution to society? Starting by questioning researchers’ fundamental perceptions, the research community itself will need to create indicators to evaluate its own activities based on its own perceptions (creating Indicators of researchers for researchers by researchers) and to fulfill its accountability to society.
As a first step, to clarify the perceptions of the members of the earth science community, we notified about 6,000 JpGU members via the JpGU mailing list on June 11 and 28, 2020, and about 38,000 AGU members via the AGU mailing list on November 15 and 20, 2020, and received 292 (4.9%) and 883 (2.3%) valid responses, respectively. The results for the JpGU community were reported in the latest Japan Geoscience Letters (Yamanaka, 2021).
We asked respondents to rank the order in which they thought it was important for knowledge creation, for six items related to science in general, not just earth science, which consist of three sets of trade-offs: discovering vs. elaborating/systematizing (Set A), answering the intellectual curiosity of humanity vs. responding to the responsibilities of society (Set B), and emphasizing perfection vs. emphasizing timeliness (Set C) (Table 1). The JpGU and AGU communities are the same in terms of ranking 1 and 2 for discovery and curiosity and 5 and 6 for timeliness and perfection. But the JpGU community ranks lower in social responsibility than the AGU community in Set B and a similar ranking of timeliness and perfection in Set C.
Six questions were asked to contribute to the creation of knowledge in the earth sciences. These are the two commonly known items, "Elucidating the origin of the earth and planets" and "understanding the current state of the earth and planets," and the four items picked up from "Declaration of the Significance of Geoscience Expertise to Meet Global Societal Challenges," which JpGU signed on May 4, 2020, together with the European Geosciences Union (EGU), the American Geophysical Union (AGU), and others (Table 2). The JpGU community did not regard the four items mentioned in the Joint Declaration as contributing to knowledge creation as the two items, while the AGU community thought them to be about the same as the two items. In particular, in these four items, the significant difference is that 1/2 to 2/3 of the AGU community’s doctoral degree recipients after 2015 and graduate students answered, "Strongly Agree," while 1/4 to 1/3 of the same age group in the JpGU community answered, "Strongly Agree."
However, when asked if they refer to the JIF during selecting papers to be included in the application of the research proposal, 40% of respondents in the JpGU community were concerned about it. In comparison, only 12% of respondents in the AGU community were worried about it. This may reflect differences in the quality and quantity of research applications.
As a first step, to clarify the perceptions of the members of the earth science community, we notified about 6,000 JpGU members via the JpGU mailing list on June 11 and 28, 2020, and about 38,000 AGU members via the AGU mailing list on November 15 and 20, 2020, and received 292 (4.9%) and 883 (2.3%) valid responses, respectively. The results for the JpGU community were reported in the latest Japan Geoscience Letters (Yamanaka, 2021).
We asked respondents to rank the order in which they thought it was important for knowledge creation, for six items related to science in general, not just earth science, which consist of three sets of trade-offs: discovering vs. elaborating/systematizing (Set A), answering the intellectual curiosity of humanity vs. responding to the responsibilities of society (Set B), and emphasizing perfection vs. emphasizing timeliness (Set C) (Table 1). The JpGU and AGU communities are the same in terms of ranking 1 and 2 for discovery and curiosity and 5 and 6 for timeliness and perfection. But the JpGU community ranks lower in social responsibility than the AGU community in Set B and a similar ranking of timeliness and perfection in Set C.
Six questions were asked to contribute to the creation of knowledge in the earth sciences. These are the two commonly known items, "Elucidating the origin of the earth and planets" and "understanding the current state of the earth and planets," and the four items picked up from "Declaration of the Significance of Geoscience Expertise to Meet Global Societal Challenges," which JpGU signed on May 4, 2020, together with the European Geosciences Union (EGU), the American Geophysical Union (AGU), and others (Table 2). The JpGU community did not regard the four items mentioned in the Joint Declaration as contributing to knowledge creation as the two items, while the AGU community thought them to be about the same as the two items. In particular, in these four items, the significant difference is that 1/2 to 2/3 of the AGU community’s doctoral degree recipients after 2015 and graduate students answered, "Strongly Agree," while 1/4 to 1/3 of the same age group in the JpGU community answered, "Strongly Agree."
However, when asked if they refer to the JIF during selecting papers to be included in the application of the research proposal, 40% of respondents in the JpGU community were concerned about it. In comparison, only 12% of respondents in the AGU community were worried about it. This may reflect differences in the quality and quantity of research applications.