Pages

▼

Wednesday, March 17, 2021

Concerns about Methodology of Racial Diversity Report

I've posted in the past about (and cited) the Ripped Bodice's reports on racial diversity in romance publishing so I thought it was important to note that concerns have been raised about the methodologies used in their production.

Here's the abstract/summary of Nick and Ari's critique, which can be found in full here:

We offer a critique of The Ripped Bodice’s State of Racial Diversity in Romance Publishing Report. With its lack of transparency, unethical, and unclear methodology, the diversity report leaves us with more questions than with answers. Though well-meaning, a study like this does a disservice to both publishers and BIPOC authors, while also setting a dangerous precedent of allowing poor ethics and poor data practices to run rampant in the romance community. In the last couple of years, we have seen the damaging effects of allowing misinformation in the media, so why are we still uncritically accepting a report that could be spreading misinformation to be published year after year? We urge The Ripped Bodice to do better and to carefully consider a few of the alternatives presented in this article.

In further comments on Twitter, Nick adds that:

We outline the ethical, transparency, and statistical issues & offer suggestions for alternatives. We didn't *want* to do this but their resounding silence in response to our Tweets/email/requests to view the raw data led us to believe that this needs further attention. We aren't saying what they are doing is unimportant, but the study needs to be conducted appropriately.

[Edited on 23 March 2021 to add more below.]

The Ripped Bodice have responded to the criticisms in detail here. Responses to their tweet about this can be found here, there's a list of tweets which respond by quote-tweeting it here, and I'm sure there are many other responses. Here's a tiny sample of some of them: 


 

 

 

 

 

and because the following has three tweets in sequence, I'm putting it in as an image rather than an embedded link, but it came from here:

This is Nick saying (at 1:47 pm · 24 Mar 2021 "I want to reiterate that NOWHERE in the article did we dismiss the conclusions of the report. I don't understand why people are twisting our words or putting words into our mouths but I guess I'll be more explicit here. We stated that this work IS important.

Trad publishing IS a mess. And they absolutely have a long way to go to truly bring equality and diversity to the industry. Clearly this "report" has done nothing to change anything majorly in the past 5 years.

So, why not bring changes to the actual report so that the bleak numbers can be taken more seriously by the industry because my perception (or suspicion) is that they are not at all taken seriously because publishers are aware of the issues?" 


at

12 comments:

  1. Anonymous19 March, 2021 00:31

    I appreciate your posting this. There are a lot of invalid points in this critique, stemming from a pretty tenuous grasp of some pretty basic aspects of social science methodology. Not all; they do make some valid points, but largely this critique is quite weak and contains several inaccuracies. For instance, calling this research a "survey" and, at one point, an "experiment." It is definitely neither of those, so the authors of the critique need to learn a little bit more about basic methods, imo, before trying to make a case that this is such shoddy research. I've read the report in the past and think it's actually pretty good, given the limitations (which the report authors address adequately, I think).

    ReplyDelete
  2. Anonymous19 March, 2021 00:38

    Another example: critiquing the report authors for not providing details on their statistical techniques. From what I can see, they present descriptive statistics to provide a percentage of authors of color in the genre as a whole. That's just, like, division. The report authors aren't attempting to establish correlation or trying to use multivariate regression--they're calculating a percentage. Not rocket science, so that's a curious part of the critique.

    ReplyDelete
  3. Laura Vivanco19 March, 2021 00:53

    Thanks for commenting. It's interesting to me to read your critique of their critique.

    I don't have the right academic background to assess any of this: I have no training at all in statistics or how to conduct an ethical survey.

    ReplyDelete
  4. Anonymous19 March, 2021 20:47

    Well, even the ethical concerns cited in the blog post are vastly overstated. I could see a short conversation playing out about them, but honestly, this is not a survey (despite what the bloggers believe), so there are no human subjects involved; there's no one to consent. The report uses existing information such as books, author profiles--for the most part, no consent is needed, at least to the degree to which the critique describes. There are some possible issues with categorizing race/ethnicity based on how authors present (in the case where there is no information about how they self-identify), but the report authors discuss this and welcome corrections. So there are some things to talk about, but I don't see the severe ethical violations the critique claims.

    ReplyDelete
  5. Laura Vivanco19 March, 2021 21:05

    I see what you mean. It occurred to me re

    possible issues with categorizing race/ethnicity based on how authors present (in the case where there is no information about how they self-identify)

    that if the Ripped Bodice could use "white or not explicitly self-identifying as a PoC" or something like that, maybe that would avoid the problems inherent in them trying to identify race via photos? Maybe that would be problematic in its own way, but might at least be an indication of the proportion of authors who feel safe being open as an author about having a non-white identity? Though, as Sam Hirst's comments about anonymity and women authors remind me, it's probably not a good idea to make assumptions about why authors choose to present themselves in particular ways. Hirst's article's about 18th- and 19th-century authors mentions that:

    Pseudonymous and anonymous publishing were commonplace regardless of gender and were adopted for many reasons, from questions of respectability to reflections of queer identity (Vernon Lee is a salient example); from a desire to separate public and private personas to the attempt to create different brand identities (Charlotte Dacre and Mary Elizabeth Braddon used different naming practices for different types of publication). The myth that women had to publish pseudonymously ignores questions of agency in a way that never occurs when discussing male writers. (History Today)

    But yes, I can see that there's a difference in terms of consent between using information which the author has made public (e.g. on their website, in an author profile available in their books) and information which is provided privately in response to a survey.

    ReplyDelete
  6. Anonymous24 March, 2021 04:01

    Thanks for posting the update. The Ripped Bodice response is 100% on target. People who are critiquing it (still) don’t understand social science. There are limitations in every study; TRB addresses them adequately and vows to improve what they can and what’s appropriate. Their comments on not releasing the raw data because of ethical concerns is totally in line with conventional ethical practices. It does not compromise the data or analysis. This is an excellent response.

    ReplyDelete
  7. Laura Vivanco24 March, 2021 12:03

    In terms of the response from the Twitter romance community, it does seem as though a significant proportion of the problems people have with the report are to do with wanting it to be more detailed (e.g. breakdown if a publisher's imprints, distinguishing between different groups of BIPOC) or to present the data in different ways (e.g. wanting to have a better idea of a publisher's total numerical output.)

    Some are about outcomes (e.g. is it there to help BIPOC authors find an inclusive publisher? Is it there to raise awareness among publishers? Does it raise awareness in the general public?) and feeling that the report is no longer required/can only fulfil its function if it changes.

    These concerns are related to the data, of course, but they seem to be more about the purpose/extent of the report and, as you say, there are "limitations in every study."

    ReplyDelete
  8. Anonymous25 March, 2021 04:27

    These concerns remind me of proverbial Reviewer 2's rejection: "This is not the paper I would've written."

    ReplyDelete
  9. Andrea31 March, 2021 04:59

    I see issues with the critique for some of the reasons mentioned here. However, I also think TRB's response was dismissive and did not engage in good faith with straightforward requests like supplying # of titles for each publisher year over year.

    Personally, my biggest issue is with how TRB is representing what their data is able to measure based on all of the known limitations, and therefore what conclusions one could draw from the data. When you consider the context in which this report is released year after year, and see how it is wielded by TRB as a brand-building tool and then leveraged uncritically in the greater media landscape to support a particular narrative...that's where I see a lot of pushback in "romancelandia" coming from.

    In my view, the problem is not so much "this is not the paper I would've written" (although, yes there is some of that in responses) but also pointing out "what they're saying about the paper they've written is not matching the words on the page."

    And of course there's much more to say about methodology improvements (that have nothing to do with hiring statisticians to calculate %), but: does knowing this data with greater precision with more meaningful categorization help the community understand the (undeniable) problem any better? When the current report is framed specifically as a tool meant to both inspire and measure change, I think those are important questions.

    I'm not a social scientist, but I am a marketer: this report is marketing, and what I find most problematic about it is HOW it's marketing itself/the actual purpose of the "project."

    Just my 2 cents (which is 2/100).
    Andrea (Shelf Love)

    ReplyDelete
  10. Laura Vivanco31 March, 2021 05:36

    Thanks for your insight as a marketer, Andrea. I have been, and still am, pretty much completely oblivious to "how it is wielded by TRB as a brand-building tool and then leveraged uncritically in the greater media landscape to support a particular narrative..."

    What I did notice was that within the report there was a box/section listing bestsellers at the bookshop, and that didn't seem relevant to the report, so I was puzzled by its inclusion.

    Could you explain more about 'HOW it's marketing itself/the actual purpose of the "project."'?

    ReplyDelete
  11. Andrea02 April, 2021 15:44

    Laura, I have written an embarrassingly long response that is basically an essay so I guess I'm turning this into a podcast episode. Stay tuned!

    ReplyDelete
  12. Laura Vivanco02 April, 2021 15:50

    Great! I'll add a link here once it appears, then.

    ReplyDelete