Ranking journals by the share of long-standing articles

Assessing the long-term impact of research: What fraction of articles are still cited 20 years later?

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Every Economist will sooner or later have stumbled upon journal rankings. Rankings inform decisions on what to read first and where to publish preferably. Rankings determine careers and individual honours. One good thing about them is their vast number  so that everyone can pick the one with the most appropriate criteria.

It has been argued that determining impact based on journal reputation is preferable to a mere citation count: Citations do not allow us to determine whether research is novel. On the other hand, determining impact by how often someone makes it to top journals is no good alternative, too. It discourages innovative work because non-innovative work has it much easier to get published high.

Packalen and Bhattacharya (2017) have therefore recently developed a “Neophilia Ranking”. The ranking criteria is what fraction of articles mention relatively new ideas.

Here I explore the idea a ranking Journals by the long-term impact: What fraction of articles are still cited 20 years later?

To do so I look at a selection of major Economics Journals and compute the share of their 1996 articles that are cited at least once in 2016. I take all journals rated at least “A” by Combes and Linnemer (2010) in their General Economics and six field-specific rankings. This yields a total of 54 journals. I am looking at the share of articles published in 1996 and cited in 2016, according to Scopus (Elsevier’s Citation Database). Of those, 11 did not exist in 1996 (or Scopus did not index this year). For example, the Review of Economic Dynamics, ranked 35 in Combes and Linnemer (2010)’s main ranking, was founded in 1998.

The remaining 43 journals published 2375 articles in total in 2016, ranging from 170 (Canadian Journal of Economics) to 10 (Brookings Papers of Economic Activity). Of these, more than the half, namely 1230, were still cited in 2016. The ranking is led by Journal of Political Economy, with 91.11% of its 45 publications in 1996 cited 20 years later.

The following table ranks all these journals by share of 1996 publications that are cited at least once in 2016 according to Scopus, which is listed column “Share >0 citations in 2016”. The column before is the conditional citation count: Given that the article is cited in 2016, how often has it been cited on average?

Journal Publications in 1996 Cited in 2016 Avg. cond. citation count in 2016 Share >0 citations in 2016 (in%) Share >10 citations in 2016 (in%) Share >50 citations in 2016 (in%)
Journal of Political Economy 45 41 8.95 91.11 24.44 0.00
Quarterly Journal of Economics 41 37 10.32 90.24 34.15 0.00
Journal of Economic Literature 18 16 16.38 88.89 55.56 0.00
Journal of Financial Economics 47 41 15.34 87.23 36.17 2.13
Review of Financial Studies 37 32 6.38 86.49 16.22 2.70
Journal of Economic Growth 21 18 11.67 85.71 19.05 0.00
Journal of Finance 69 59 14.63 85.51 34.78 2.90
Econometrica 56 47 15.89 83.93 16.07 5.36
Brookings Papers on Economic Activity 10 8 6.00 80.00 20.00 0.00
Journal of Economic Perspectives 47 37 6.08 78.72 14.89 0.00
American Economic Review 149 115 8.46 77.18 16.11 2.01
Review of Economic Studies 28 20 6.40 71.43 10.71 0.00
Review of Economics and Statistics 60 42 6.40 70.00 11.67 1.67
Journal of Financial Intermediation 15 10 1.90 66.67 0.00 0.00
Journal of Econometrics 98 65 9.52 66.33 14.29 4.08
Journal of Labor Economics 28 17 3.53 60.71 0.00 0.00
Journal of Public Economics 92 55 4.16 59.78 1.09 1.09
Journal of Banking and Finance 86 50 3.60 58.14 5.81 0.00
Journal of Development Economics 63 36 6.11 57.14 11.11 0.00
Journal of Monetary Economics 57 32 6.00 56.14 7.02 0.00
Economic Policy 11 6 2.50 54.55 0.00 0.00
The Journal of Economic History 33 18 2.06 54.55 0.00 0.00
Journal of Economic Surveys 11 6 3.67 54.55 0.00 0.00
European Economic Review 111 59 5.49 53.15 9.01 0.00
Economic Journal 91 48 7.10 52.75 8.79 0.00
Journal of International Economics 41 19 5.05 46.34 7.32 0.00
Journal of Money, Credit and Banking 53 24 5.29 45.28 5.66 0.00
Journal of Economic Theory 104 42 5.19 40.38 2.88 0.96
Oxford Review of Economic Policy 30 12 4.33 40.00 3.33 0.00
European Journal of Political Economy 52 20 2.75 38.46 1.92 0.00
Economic Inquiry 48 18 3.67 37.50 2.08 0.00
Journal of Economic Dynamics and Control 77 27 2.48 35.06 0.00 0.00
Labour Economics 20 7 1.86 35.00 0.00 0.00
Economics Letters 212 63 2.44 29.72 0.47 0.00
Journal of International Money and Finance 52 15 2.87 28.85 0.00 0.00
Scandinavian Journal of Economics 35 10 3.60 28.57 5.71 0.00
Canadian Journal of Economics 170 37 2.08 21.76 0.59 0.00
Journal of Macroeconomics 41 8 3.50 19.51 2.44 0.00
Journal of Policy Modeling 27 5 1.80 18.52 0.00 0.00
Manchester School 22 3 1.33 13.64 0.00 0.00
National Institute Economic Review 27 3 1.33 11.11 0.00 0.00
Open Economies Review 22 2 1.50 9.09 0.00 0.00
North American Journal of Economics and Finance 18 0 0 0.00 0.00 0.00

Interestingly, the journals known as the “top 5” journals (The American Economic Review, Econometrica, the Journal of Political Economy, The Review of Economic Studies, and the Quarterly Journal of Economics) are not top 5 in this list. The Journal of Economic Literature, for example, is ranked third and has the highest conditional average citation count of about 16 citations. It also has the highest share of articles when we define long-standing articles as having more than 5 citations. This mirrors findings presented at this year’s ASSA conference’s Panel Discussion on Publishing and Promotion in Economics: The Curse of the Top Five. If define a long-lasting impact article by being cited more than 50 times in the 20th year, then Econometrica comes out first.

Interesting questions arise: Do these journals attract publications with a long impact? Do these articles still attract citations because they got published where they are? Or were the editors in 1996 simply good at picking the ‘right’ articles with presumably long-lasting impact?

Though the questions are not easy to answer, I would like to shed light on the differences among the journals by showing how the average long-lasting article (i.e. the one that is still cited 20 years later) performs over time. The following plot traces the average number of citations in a given year by journal to all long-lasting articles of 1996 (hover over the lines to see the journal names and zoom in by drawing a rectangle with your mouse – might not work on mobile versions).

As a further robustness check, here is above list with citations 10 years later for publications from 2006. That is, how many of a journal’s 2006 publications have been cited at least once in 2016? Again, the last column gives the average number of citations in the 10th year. The list is compiled from 2012 articles from 48 journals (only 6 did not exist in 2006 or were not indexed).

Journal Publications in 2006 Cited in 2016 Avg. cond. citation count in 2016 Share >0 citations in 2016 (in%) Share >10 citations in 2016 (in%) Share >50 citations in 2016 (in%)
Quarterly Journal of Economics 40 39 15.90 97.50 52.50 5.00
Journal of Political Economy 38 37 9.57 97.37 34.21 0.00
Journal of Finance 88 84 18.35 95.45 51.14 7.95
Journal of Financial Economics 88 82 16.09 93.18 36.36 6.82
Journal of Economic Perspectives 48 44 15.18 91.67 31.25 4.17
Journal of Economic Growth 12 11 6.09 91.67 16.67 0.00
Journal of Labor Economics 28 25 11.12 89.29 28.57 3.57
Review of Financial Studies 43 38 11.58 88.37 30.23 2.33
American Economic Review 191 165 9.34 86.39 22.51 1.57
Review of Economic Studies 44 38 7.92 86.36 22.73 0.00
Review of Economics and Statistics 58 50 11.96 86.21 25.86 1.72
Econometrica 60 51 12.39 85.00 20.00 6.67
Journal of the European Economic Association 61 47 4.30 77.05 4.92 0.00
Economic Journal 83 63 5.87 75.90 9.64 1.20
Journal of Economic Literature 20 15 11.27 75.00 35.00 0.00
Journal of Development Economics 67 49 7.04 73.13 10.45 1.49
Journal of Econometrics 130 92 6.84 70.77 12.31 0.77
Economic Policy 20 14 4.14 70.00 0.00 0.00
Journal of Public Economics 110 77 5.32 70.00 10.91 0.00
Journal of Economic Theory 76 53 3.58 69.74 1.32 0.00
Oxford Review of Economic Policy 33 23 5.83 69.70 12.12 0.00
Journal of International Money and Finance 66 46 4.85 69.70 6.06 0.00
Journal of International Economics 65 45 5.89 69.23 13.85 0.00
Journal of Monetary Economics 107 73 3.93 68.22 6.54 0.00
Journal of Banking and Finance 172 117 5.94 68.02 8.14 1.16
Journal of Money, Credit and Banking 99 66 4.55 66.67 6.06 0.00
Review of Economic Dynamics 33 22 2.91 66.67 3.03 0.00
Journal of Economic Surveys 29 19 4.63 65.52 6.90 0.00
European Economic Review 94 60 4.65 63.83 7.45 0.00
Journal of Financial Intermediation 24 15 4.33 62.50 4.17 0.00
Scandinavian Journal of Economics 37 23 3.09 62.16 2.70 0.00
The Journal of Economic History 36 22 2.00 61.11 0.00 0.00
International Finance 17 10 2.20 58.82 0.00 0.00
Journal of Financial Stability 14 8 5.25 57.14 7.14 0.00
Labour Economics 37 21 2.71 56.76 0.00 0.00
European Journal of Political Economy 53 30 4.53 56.60 7.55 0.00
Canadian Journal of Economics 60 30 2.00 50.00 1.67 0.00
Journal of Economic Dynamics and Control 112 55 2.75 49.11 0.89 0.00
Economic Inquiry 54 26 2.88 48.15 1.85 0.00
Journal of Macroeconomics 50 22 1.77 44.00 0.00 0.00
Journal of Policy Modeling 77 33 2.06 42.86 1.30 0.00
Economics Letters 291 122 2.61 41.92 0.69 0.00
Open Economies Review 28 11 2.36 39.29 0.00 0.00
Brookings Papers on Economic Activity 13 5 3.60 38.46 0.00 0.00
Manchester School 49 18 2.28 36.73 2.04 0.00
North American Journal of Economics and Finance 25 6 5.33 24.00 4.00 0.00
Macroeconomic Dynamics 30 7 4.57 23.33 3.33 0.00
National Institute Economic Review 57 3 1.33 5.26 0.00 0.00

Comparing both tables, we see some dynamics happening here, with the top 5 journals being closer to the top and closer together than in the ranking of 20 years-later citations. This also holds for some major field journals, like the Journal of Finance, Journal of Economic Growth or Journal of Labor Economics.

References
Updates
  • June 18th, 2017: Added shares for alternative definitions of long-lasting articles.