Ranking journals by the share of long-standing articles

Assessing the long-term impact of research: What fraction of articles are still cited 20 years later?

Every Economist will sooner or later have stumbled upon journal rankings. Rankings inform decisions on what to read first and where to publish preferably. Rankings determine careers and individual honours. One good thing about them is their vast number  so that everyone can pick the one with the most appropriate criteria.

It has been argued that determining impact based on journal reputation is preferable to a mere citation count: Citations do not allow us to determine whether research is novel. On the other hand, determining impact by how often someone makes it to top journals is no good alternative, too. It discourages innovative work because non-innovative work has it much easier to get published high.

Packalen and Bhattacharya (2017) have therefore recently developed a “Neophilia Ranking”. The ranking criteria is what fraction of articles mention relatively new ideas.

Here I explore the idea a ranking Journals by the long-term impact: What fraction of articles are still cited 20 years later?

To do so I look at a selection of major Economics Journals and compute the share of their 1996 articles that are cited at least once in 2016. I take all journals rated at least “A” by Combes and Linnemer (2010) in their General Economics and six field-specific rankings. This yields a total of 54 journals. I am looking at the share of articles published in 1996 and cited in 2016, according to Scopus (Elsevier’s Citation Database). Of those, 11 did not exist in 1996 (or Scopus did not index this year). For example, the Review of Economic Dynamics, ranked 35 in Combes and Linnemer (2010)’s main ranking, was founded in 1998.

The remaining 43 journals published 2375 articles in total in 2016, ranging from 170 (Canadian Journal of Economics) to 10 (Brookings Papers of Economic Activity). Of these, more than the half, namely 1230, were still cited in 2016. The ranking is led by Journal of Political Economy, with 91.11% of its 45 publications in 1996 cited 20 years later.

The following table ranks all these journals by share of 1996 publications that are cited at least once in 2016 according to Scopus, which is listed column “Share >0 citations in 2016”. The column before is the conditional citation count: Given that the article is cited in 2016, how often has it been cited on average?

Journal Publications in 1996 Cited in 2016 Average cond. citation count in 2016 Share >0 citations in 2016 (in%) Share >10 citations in 2016 (in%) Share >50 citations in 2016 (in%)
Journal of Political Economy 45 41 8.95 91.11 24.44 0.00
Quarterly Journal of Economics 41 37 10.32 90.24 34.15 0.00
Journal of Economic Literature 18 16 16.38 88.89 55.56 0.00
Journal of Financial Economics 47 41 15.34 87.23 36.17 2.13
Review of Financial Studies 37 32 6.38 86.49 16.22 2.70
Journal of Economic Growth 21 18 11.67 85.71 19.05 0.00
Journal of Finance 69 59 14.63 85.51 34.78 2.90
Econometrica 56 47 15.89 83.93 16.07 5.36
Brookings Papers on Economic Activity 10 8 6.00 80.00 20.00 0.00
Journal of Economic Perspectives 47 37 6.08 78.72 14.89 0.00
American Economic Review 149 115 8.46 77.18 16.11 2.01
Review of Economic Studies 28 20 6.40 71.43 10.71 0.00
Review of Economics and Statistics 60 42 6.40 70.00 11.67 1.67
Journal of Financial Intermediation 15 10 1.90 66.67 0.00 0.00
Journal of Econometrics 98 65 9.52 66.33 14.29 4.08
Journal of Labor Economics 28 17 3.53 60.71 0.00 0.00
Journal of Public Economics 92 55 4.16 59.78 1.09 1.09
Journal of Banking and Finance 86 50 3.60 58.14 5.81 0.00
Journal of Development Economics 63 36 6.11 57.14 11.11 0.00
Journal of Monetary Economics 57 32 6.00 56.14 7.02 0.00
Economic Policy 11 6 2.50 54.55 0.00 0.00
The Journal of Economic History 33 18 2.06 54.55 0.00 0.00
Journal of Economic Surveys 11 6 3.67 54.55 0.00 0.00
European Economic Review 111 59 5.49 53.15 9.01 0.00
Economic Journal 91 48 7.10 52.75 8.79 0.00
Journal of International Economics 41 19 5.05 46.34 7.32 0.00
Journal of Money, Credit and Banking 53 24 5.29 45.28 5.66 0.00
Journal of Economic Theory 104 42 5.19 40.38 2.88 0.96
Oxford Review of Economic Policy 30 12 4.33 40.00 3.33 0.00
European Journal of Political Economy 52 20 2.75 38.46 1.92 0.00
Economic Inquiry 48 18 3.67 37.50 2.08 0.00
Journal of Economic Dynamics and Control 77 27 2.48 35.06 0.00 0.00
Labour Economics 20 7 1.86 35.00 0.00 0.00
Economics Letters 212 63 2.44 29.72 0.47 0.00
Journal of International Money and Finance 52 15 2.87 28.85 0.00 0.00
Scandinavian Journal of Economics 35 10 3.60 28.57 5.71 0.00
Canadian Journal of Economics 170 37 2.08 21.76 0.59 0.00
Journal of Macroeconomics 41 8 3.50 19.51 2.44 0.00
Journal of Policy Modeling 27 5 1.80 18.52 0.00 0.00
Manchester School 22 3 1.33 13.64 0.00 0.00
National Institute Economic Review 27 3 1.33 11.11 0.00 0.00
Open Economies Review 22 2 1.50 9.09 0.00 0.00
North American Journal of Economics and Finance 18 0 0 0.00 0.00 0.00

Interestingly, the journals known as the “top 5” journals (The American Economic Review, Econometrica, the Journal of Political Economy, The Review of Economic Studies, and the Quarterly Journal of Economics) are not top 5 in this list. The Journal of Economic Literature, for example, is ranked third and has the highest conditional average citation count of about 16 citations. It also has the highest share of articles when we define long-standing articles as having more than 5 citations. This mirrors findings presented at this year’s ASSA conference’s Panel Discussion on Publishing and Promotion in Economics: The Curse of the Top Five. If define a long-lasting impact article by being cited more than 50 times in the 20th year, then Econometrica comes out first.

Interesting questions arise: Do these journals attract publications with a long impact? Do these articles still attract citations because they got published where they are? Or were the editors in 1996 simply good at picking the ‘right’ articles with presumably long-lasting impact?

Though the questions are not easy to answer, I would like to shed light on the differences among the journals by showing how the average long-lasting article (i.e. the one that is still cited 20 years later) performs over time. The following plot traces the average number of citations in a given year by journal to all long-lasting articles of 1996 (hover over the lines to see the journal names and zoom in by drawing a rectangle with your mouse – might not work on mobile versions).

As a further robustness check, here is above list with citations 10 years later for publications from 2006. That is, how many of a journal’s 2006 publications have been cited at least once in 2016? Again, the last column gives the average number of citations in the 10th year. The list is compiled from 2012 articles from 48 journals (only 6 did not exist in 2006 or were not indexed).

Journal Publications in 2006 Cited in 2016 Average cond. citation count in 2016 Share >0 citations in 2016 (in%) Share >10 citations in 2016 (in%) Share >50 citations in 2016 (in%)
Quarterly Journal of Economics 40 39 15.90 97.50 52.50 5.00
Journal of Political Economy 38 37 9.57 97.37 34.21 0.00
Journal of Finance 88 84 18.35 95.45 51.14 7.95
Journal of Financial Economics 88 82 16.09 93.18 36.36 6.82
Journal of Economic Perspectives 48 44 15.18 91.67 31.25 4.17
Journal of Economic Growth 12 11 6.09 91.67 16.67 0.00
Journal of Labor Economics 28 25 11.12 89.29 28.57 3.57
Review of Financial Studies 43 38 11.58 88.37 30.23 2.33
American Economic Review 191 165 9.34 86.39 22.51 1.57
Review of Economic Studies 44 38 7.92 86.36 22.73 0.00
Review of Economics and Statistics 58 50 11.96 86.21 25.86 1.72
Econometrica 60 51 12.39 85.00 20.00 6.67
Journal of the European Economic Association 61 47 4.30 77.05 4.92 0.00
Economic Journal 83 63 5.87 75.90 9.64 1.20
Journal of Economic Literature 20 15 11.27 75.00 35.00 0.00
Journal of Development Economics 67 49 7.04 73.13 10.45 1.49
Journal of Econometrics 130 92 6.84 70.77 12.31 0.77
Economic Policy 20 14 4.14 70.00 0.00 0.00
Journal of Public Economics 110 77 5.32 70.00 10.91 0.00
Journal of Economic Theory 76 53 3.58 69.74 1.32 0.00
Oxford Review of Economic Policy 33 23 5.83 69.70 12.12 0.00
Journal of International Money and Finance 66 46 4.85 69.70 6.06 0.00
Journal of International Economics 65 45 5.89 69.23 13.85 0.00
Journal of Monetary Economics 107 73 3.93 68.22 6.54 0.00
Journal of Banking and Finance 172 117 5.94 68.02 8.14 1.16
Journal of Money, Credit and Banking 99 66 4.55 66.67 6.06 0.00
Review of Economic Dynamics 33 22 2.91 66.67 3.03 0.00
Journal of Economic Surveys 29 19 4.63 65.52 6.90 0.00
European Economic Review 94 60 4.65 63.83 7.45 0.00
Journal of Financial Intermediation 24 15 4.33 62.50 4.17 0.00
Scandinavian Journal of Economics 37 23 3.09 62.16 2.70 0.00
The Journal of Economic History 36 22 2.00 61.11 0.00 0.00
International Finance 17 10 2.20 58.82 0.00 0.00
Journal of Financial Stability 14 8 5.25 57.14 7.14 0.00
Labour Economics 37 21 2.71 56.76 0.00 0.00
European Journal of Political Economy 53 30 4.53 56.60 7.55 0.00
Canadian Journal of Economics 60 30 2.00 50.00 1.67 0.00
Journal of Economic Dynamics and Control 112 55 2.75 49.11 0.89 0.00
Economic Inquiry 54 26 2.88 48.15 1.85 0.00
Journal of Macroeconomics 50 22 1.77 44.00 0.00 0.00
Journal of Policy Modeling 77 33 2.06 42.86 1.30 0.00
Economics Letters 291 122 2.61 41.92 0.69 0.00
Open Economies Review 28 11 2.36 39.29 0.00 0.00
Brookings Papers on Economic Activity 13 5 3.60 38.46 0.00 0.00
Manchester School 49 18 2.28 36.73 2.04 0.00
North American Journal of Economics and Finance 25 6 5.33 24.00 4.00 0.00
Macroeconomic Dynamics 30 7 4.57 23.33 3.33 0.00
National Institute Economic Review 57 3 1.33 5.26 0.00 0.00

Comparing both tables, we see some dynamics happening here, with the top 5 journals being closer to the top and closer together than in the ranking of 20 years-later citations. This also holds for some major field journals, like the Journal of Finance, Journal of Economic Growth or Journal of Labor Economics.

References

 

Updates
  • June 18th, 2017: Added shares for alternative definitions of long-lasting articles.

Who do we acknowledge, and why?

Being acknowledged on a research paper tells something about you and about the author: The author perceives you as important.

In the last post I reported evidence that many more researchers contribute to a research paper than just authors. The most visible difference between authors and commenters — and our only way to judge on the effort — is that authors receive all the credit while commenters are “just” acknowledged.

Working with others is that common, that it naturally rises the question: Where do we meet other researchers to get their opinion? Most likely, when it’s not faculty colleagues and friends, it is conference guests or seminar visitors to your faculty.

It’s an open question. The fact however that an author  – call him A – receives advice from another researcher – call him C – reveals information about both: For one, it means that A perceives C as important in the very field, at least important enough to get insights. It may also be because A want to signal quality by showing that he knows C. Vice versa, the fact that C gives a comment for which he is acknowledged shows that he is obviously willing to spend time on someone else’s research.

Oettel (2012) has termed this behaviour as “helpful”, saying that a helpful colleague (one that is often acknowledged) matters more to you than a productive colleague (one that publishes many papers). In a similar fashion, we created a ranking of how often financial economists have been acknowledged in six financial economics journals. Our dataset consists of more than 5,600 full research articles.

The distribution of how often someone is acknowledged is very skewed. The vast majority of commenters is being acknowledged twice or less, with very few individuals being acknowledged very often (zoom in the upper tail by dragging a rectangle):

However, we can uncover more. When A and C agree to exchange ideas, we can think think of their relationship as a tie in a network. In network lingo, a tie is being formed when both nodes (here A and C) mutually. Now take many more researchers and eventually more form a tie with C. This likely puts C in a very central position, which both displays and transfers influence. For this reason we computed the rank according to eigenvector centrality.

The idea behind this measure is: When you know important people, you are probably important as well. Importance here is measured iteratively as the degree to which you know important people. Google uses the same idea for its PageRank.

Interestingly, being acknowledged often does not automatically put researchers in central position. In our paper we explore this relationship and find that Spearman correlations for different periods never exceed 0.65.

In a related blogpost on VOX from last year we propose this ranking as a new measure of influence. This makes sense because under the assumption of mutually beneficial tie formation the market automatically puts the “right” people in the center – the network as the aggregated market wisdom on which researchers matter more for financial economics.

Please refer to our website for an ranking for the period 2009-2011 and others.

References

Informal Collaboration increases over time – But more so for top journals

Scientists do not work alone. Colleagues help them by providing feedback and commentary. Upon publication, helpful input is being acknowledged, which constitutes our novel database.

The notion of the sole scientist working endlessly in an ivory tower until a scientific contribution emerged in the form of a long book is a romantic one, but it is wrong. Science is social endeavor, and collegiate help comes in may forms. Reading and commenting on a colleague’s draft a very common one, and presenting in seminars, workshops and conferences may be even more common. In fact, collaboration is so common that tips and advices for academic presentations are everywhere.

Financial Economics is no exception to this, as we show in our papers. Financial economists commonly acknowledge helpful input by their colleagues: Whether they have commented on a draft, discussed at a conference or gave feedback after listening to an idea. Additionally, authors list conferences and universities where they presented their work so as to acknowledge the audience.

We have used 6407 full research articles from six major financial economics journals published in the 1997-2011 period. These journals are The Journal of Finance (JF; 1153articles), the Review of Financial Studies (RFS; 891), the Journal of Financial Economics (JFE; 1157), the Journal of Financial Intermediation (JFI; 278), the Journal of Money, Credit & Banking (JMCB; 816) and the Journal of Banking and Finance (JBF; 2110). The journals JF, RFS and JFE are commonly referred to as top finance journals (see for example the JF’s annual reports, which uses RFS and JFE as benchmarks).

There are three forms of informal collaboration present in Economics and its subfields: direct feedback by individuals, seminar presentations, and conference presentations. For our analysis workshops count as conference.

In 2011 the vast majority of articles report at least one form of informal collaboration all articles report any of the three forms of informal collaboration, but the share differs from journal to journal:

While virtually all articles published in the three top journals (JF, RFS, JFE) report at least one form of informal collaboration, it is usually 80% for JBF articles.

Just as the extensive margin the intensive margin increased over time as well. Of all 6407 articles in our dataset, a total of 5633 articles acknowledge at least one commenter or one seminar presentation or one conference presentation. We counted them and calculated the average per year and journal:

There are two remarkable trends: First, informal collaboration increases over time. In all journals, the average article acknowledges more commenters, more seminars and more conferences. Laband and Tollison (2002) have reported a similar increase in the number of commenters for three top Economics journal publications.

But the amount of informal collaboration in top journals (JF, RFS, JFE) has always been higher than in the other three journals (JFI, JMCB, JBF). Only the JFI tends to be become a top journal, at least in terms of informal collaboration.

While the correlations are striking, the causal effects are hard to identify. It might be quality-improvement, but it might also be a selection bias on the side of the author – i.e. only presenting the better papers -, or selection bias by the commenters, seminar venues and conferences – i.e. only accepting the better articles (Brown, 2005).

But even when there are other effects at work too: The fact that editors such as the editors of the the top finance journal editors Green, O’Hara and Schwert (2002) (among other editors) advise authors “to circulate their papers and give seminars to colleagues to receive constructive criticism before submitting to a journal” is evidence that informal collaboration has its merits.

References