Need someone to criticise your paper? Look in your references!

Receiving comments from authors you cite is associated with more citations for your paper. It is highest when you cite them, you coauthor with them and you are in the same department, too.


There are many answers to the old but classic question about how many Economists are need to change a light bulb. It’s harder to answer the question how many Economists are needed to write a research paper.

One reason why it’s hard is the small number of authors on Economics papers. In our dataset covering Financial Economics, the average number of authors is 2.1. Physicists or life scientists, where author groups are so large that not all authors know each other, would laugh at this number.

The other reason why it’s hard is that authors alone do not write papers. In Economics, a lot of collaboration is informal, as Laband and Tollison (2000) show. This is collaboration without assignment of property rights, most commonly in the form of feedback to a manuscript. Those commenters, usually acknowledged in a separte section, constitute our novel database. The average paper acknowledges 8 commenters for informal intellectual input –
with considerable variation across journals.

Here I want to show that these commenters are not (only) being put there for strategic reason, in an effort for example to signal quality. And I show that the expected increase in citations associated with one additional commenter depends on how close you are with her.

I look at three roles acknowledged commenters might take on additionally: as coauthors, as colleagues, and as referenced authors.
These roles are sufficient (but not necessary) to indicate a meaningful interaction with the manuscript. Acknowledging referenced authors for example indicates information flow targeted at the manuscript, while commenting authors and colleagues indicate a repeated interaction.

I investigate 5,320 journal articles published in 6 major financial journals (JF, RFS, JFE, JFI, JMCB, JBF) between 1997 and 2011, which acknowledge at least one commenter by name for intellectual input. It is the the same dataset used in our various papers and my previous blogposts. I exclude known referees and the journal’s managing editors, but I include discussants and PhD advisors if stated explicitly. There are 11,802 distinct commenters (compared to 5,320 distinct authors), and for 9,031 of the commenter I inferre affiliation and coauthors from their publication history as indexed in Scopus.

As many commenters are acknowledged multiple times – some 66 times in 2009-2011 alone – the total number of given comments is 46,877. Of them, 12.764 (27.64%) are by (former) colleagues at at least one of the authors’ affiliation(s).1) 2,801 (5.98%) comments are by coauthors of at least one author. 9,677 (20.65%) comments are from authors of articles referenced in the paper they commented on.

Considering the overlap, a total of 16,063 comments are by coauthors and/or colleagues and/or referenced authors. That’s 34.26% of all the comments that one can deem not predominantly strategic since a deeper relationship with these commenters is going on.2) And it’s sensible to continue with discussants for example.3)

Looking at the article level, 20% of the articles have at least a share of 50% of explained commenters, as seen from the first five bars in below figure (reversed x axis):

Are comments from different commenter roles equal in their impact on citation count?

To answer this question I proceed in two steps. First, I identify the most important roles in a simple OLS regression. Then, I estimate a Negative Binomial regression to quantify the effect of each role.

Here is the OLS regression result with the the number of commenters according to different roles:

Number of acknowledged commenters that are (1) (2) (3)
pure colleague 4.726*** 3.461** 0.565
(1.358) (1.345) (1.341)
pure referenced 7.402*** 4.712*** 2.447
(1.629) (1.642) (1.626)
pure coauthors -0.185 -1.581 -1.606
(2.775) (2.744) (2.697)
ref’d colleague 28.111*** 23.936*** 22.204***
(3.338) (3.318) (3.265)
coauthor-colleague 4.404 3.558 1.878
(2.873) (2.843) (2.797)
ref’d coauthor 28.773*** 26.258*** 25.186***
(4.301) (4.252) (4.181)
ref’d coauthor-colleague 29.332*** 23.861*** 23.925***
(4.650) (4.613) (4.533)
unexplained -0.286 -0.362 -0.537
(0.608) (0.603) (0.595)
# of authors 15.888*** 12.264***
(2.675) (2.649)
# of pages 2.124*** 1.479***
(0.238) (0.276)
Constant 23.160*** -55.115*** 33.790**
(8.095) (10.881) (13.332)
Year-FE Yes Yes Yes
Journal-FE No No Yes
N 5,320 5,310 5,310
Adjusted R2 0.105 0.125 0.156

Two most interesting findings:

  1. The number of comments received from pure coauthors is statistically insignificant
  2. The number of comments received from referenced colleagues, referenced coauthors, and referenced coauthor-colleagues is economically and stistically significant – even after controlling for basic paper characteristics and journal.

Because citations are skewed, non-negative count data, I marginal effect size of comments by commenters with different additional roles in 7 Negative Binomial regressions. Ordinary Least Squares coefficients would be distorted with this distribution. Marginal effects answer the following question: If you hold all other variables constant at their mean and increase the independent variable by 1 unit, what is the percentage increase in the dependent variable from its the mean? Because the variables are estimated at the mean of the other variables, I test one of the role variables at a time, controlling for publication year, journal, number of authors and page count. The average paper in the sample has 2.1 authors, consists of 26.4 pages and has been cited 87.56 times so far.

The following coefficient plot visualizes effect sizes and 95% error bars (all coefficients are highly statistically significant in their individual regressions):

Coefficient plot highlighting marginal effects

Acknowledging one more colleague, that you do not cite and who is not your coauthor, is associated with an increase of 3.8% more citations at the mean, given that you already acknowledge one colleague (that is the mean). Acknowledging one more author that you also cite, but who is not your coauthor and not your colleague, is associated with an increase of 5.1% more citations, given that you already acknowledge one referenced author. The respective number for a pure coauthoring commenter is 3.8% with a mean of 0.5.

It becomes very interesting when you look at interactions of referenced authors and other groups. The coefficient can be as high as 26.4%, namely for colleagues, that are both coauthors and that you cite. Part of the high coefficient however is the low sample mean of 0.02; only 802 of the 4,508 articles acknowledge commenters that fall in this category.

The tendency is clear: Receiving comments from referenced authors is associated with higher citation counts than comments from coauthors, and this effect grows when the referenced authors are also colleagues and/or coauthors. Results hold also when I regress on the intensive margin (i.e. whether someone acknowledges at least one coauthor, etc.).

Here I can only speculate about the reasons.

A compelling candidate is firstly the existence of social ties, and secondly the strength thereof.

From sociological literature we know that tacit knowledge requires a social tie to diffuse (Hansen, 1999). Simply reading a paper might not be enough to fully grasp its scope. There might be private information available only when you talk to its authors, e.g. what did not work and what did not make it into the publication. And when the paper you’re citing is pivotal to your work, it’s no wonder talking to its authors can help you.

But also the frequency of the interaction matters, not only its existence. When knowledge is very complex other things are necessary too, such as trust or hands-on learning (Hansen, 1999). This is the kind of interaction you tend to have with colleagues and coauthors. This argument however cannot explain why the association of comments from comments from coauthors with citation count is statistically insignificant. Thoughts?

  1. I define a commenter to be a colleauge if in years t, t+1, t-1 and t-2 a commenter lists the same affiliation on at least one published article as at least one of the authors of the paper under consideration, which is published in t.
  2. The share of explained commenters is likely to be higher. Scopus was not always able to extract the affiliation correctly nor could I always obtain a paper’s list of references.
  3. For example, there are are 1037 mentionings of presentations at conferences with discussants (meetings of AEA, AFA, NFA and EFA, as well as NBER Summer Insitutes)in acknowledgements. If every paper acknowledges all its discussants, that’s already 2% of all commenting relationships.