Is our attitude to men based on substandard research?

Professor Guy Madison & Therese Söderlund

Much of our attitude towards men comes from high profile feminist and Gender Studies scholars in academia. There is a strong association between the theorizing amongst such academics and the ideologies expressed in public debate. In fact, it is often the same individuals who write op-eds, scholarly papers, and pamphlets, and engage in political activism. It is only the type of platform that varies. Much of the ideas about male privilege, quotas, “rape culture”, and the patriarchy, for example, began in academic departments. But what is the scientific quality of publications from the world of gender studies?

We were compelled to investigate the quality of gender studies publications after some alluring lectures given at Umeå University, arguing that Gender Studies (GS) outperforms the old-fashioned, bigoted, and boring positivist methods. GS, it was argued, is an alternative and superior kind of science.

This aroused our interest, so we compiled a database of all GS publications we could find in a certain time frame, written by scholars active in Sweden. This Swedish Gender Studies List (SGSL) contains 12,414 cases, and has to date been used in three studies, all published in the journal Scientometrics (Söderlund & Madison, 2015; 2017; Madison & Söderlund, 2018). As they are already reported in detail, we provide only a brief summary of selected results here.

  • The 2015 study found that the annual growth rate of GS publications was greater than for research in general. In the period 2000 to 2010, GS publications from Sweden increased ~12% per year and ~7% internationally, while other research both in Sweden and internationally grew ~3-7%.
  • GS articles are substantially more often book chapters, dissertations, and conference contributions, and less often peer-reviewed journal articles (~20% of GS and 70% on non-GS publications).
  • Overall, non-GS articles had 2-3 times more citations than GS articles, and ~90% have 3 citations or more compared to only ~28% of GS articles.
  • 50% of GS journals were not even indexed in Thomson’s Journal Citation Reports, and the bulk of the remaining journals’  had an impact factor of less than half that of the average for non-GS journals (~1.0, as compared to ~2.0).
  • The 2016 and 2018 studies were based on ~2,800 statements culled from 36 journal articles with more and less gender perspective, and found significantly higher proportions of biased and normative statements in GS articles, and lower proportions of statements about biology/genetics and individual/group differences, than in non-GS articles. Consistent with this, non-GS articles had significantly lower proportions of statements about environment/culture and societal institutions.
  • The 2018 study found that GS texts were more abstract, less empirical, and focussed to a greater extent on societal factors. The proportion of statement of fact was 82% for GS articles but 48% for non-GS articles. Correlations were mentioned as a relationship between variables in 10% of GS statements and 37% of non-GS statements, but there was no difference in the proportion of causal relations (~5% for both GS and non-SG).
  • GS articles had less mention of limitations and earlier theories, results, and research in general, and had less support for their statements in terms of arguments or references.
  • In conclusion, GS articles appear to focus on communicating examples of different experiences and viewpoints of certain groups of people, rather than comprehensive models of the real world.

It is quite unusual that scholars comment on each other’s articles in print, but our 2015 article was challenged by ten pages listing alleged errors, asserting that it “falls well short of adequate good practice”, that we “distort” the conceptual framework, are “unreflecting”, and “lack…understanding” and ”…knowledge” (Lundgren, Shildrick, & Lawrence, 2015). We were, however, unable to find any argument as to how the many but relatively minor concerns listed might, separately or taken together, challenge the main result that more gender perspective was associated with lower scientific quality. The implication was not lost upon us that only gender studies scholars can be trusted to criticize gender studies publications (Madison & Söderlund, 2016).

From an epistemological perspective it is interesting and potentially illuminating that GS scholars find it worthwhile to launch arrows that so clearly miss the mark. The data are in. We are just saying what they look like. A comment to our 2018 article was just published (Lykke, 2018), again with a long list of grievances about definitions of concepts and our lack of competence. In essence, however, both comments argue that the differences we document are expected, and are therefore not valid critique. Maybe so, but that does not refute our argument that this field could achieve a greater impact if those differences were reduced.

Surely, it stands to reason that any communication would be more useful as a source of knowledge and guide for action and change if its scientific quality were higher? Is it not obvious that such quality is associated with more, rather than less, data, reliability, validity, non-bias, self-criticism, and most other quality indicators addressed in our studies, the very same indicators that scientific journals in general apply as criteria for peer-review? (Although perhaps not all journals: https://en.wikipedia.org/wiki/%22Grievance_Studies%22_affair).

We welcome the debate, and look forward to hear representatives of GS develop their arguments for why these criteria are undesirable for, or do not apply to, their field.

About the authors

Guy Madison is a professor of Psychology at Umeå University (since 2011) and holds a PhD from Uppsala University (2001). He has authored some 150 scientific papers, including five book chapters and about 100 peer-reviewed journal articles across diverse fields of study. His main areas of research are human timing modelling, music psychology, intelligence, personality, and individual differences, to a large extent employing an evolutionary perspective and behavioural genetic methods. He mainly teaches behavioural genetics and advanced level scientific methodology. See his publications and research projects at ResearchGate, Loop, or https://www.umu.se/en/staff/guy-madison/

Therese Söderlund has Master’s degrees in Psychology and in Worklife and Health, and Bachelor’s degrees in Swedish and English. She has been employed as a researcher and project assistant at the Department of Psychology at Umeå University since 2009.

 

References

Lundgren, S., Shildrick, M., & Lawrence, D. (2015). Rethinking bibliometric data concerning gender studies: a response to Söderlund and Madison. Scientometrics, 105, 1389-1398.

Lykke, N. (2018). Can’t bibliometric analysts do better? How quality assessment without field expertise does not work. Scientometrics, 117, 655-666.

Madison, G. & Söderlund, T. (2016). Can gender studies be studied? Reply to comments on Söderlund and Madison. Scientometrics, 108, 329-335.

Madison, G. & Söderlund, T. (2018). Comparisons of content and scientific quality indicators across populations of peer-reviewed journal articles with more or less gender perspective: Gender studies can do better. Scientometrics, 115, 1161-1183.

Söderlund, T. & Madison, G. (2015). Characteristics of gender studies publications: a bibliometric analysis based on a Swedish population database. Scientometrics, 105, 1347-1387.

Söderlund, T. & Madison, G. (2017). Objectivity and realms of explanation in academic journal articles concerning sex/gender: a comparison of Gender studies and the other social sciences. Scientometrics, 112, 1093-1109.

 

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Scroll to top