A referee report is not meant to be a review or a reaction to an academic article, but an assessment. Your role as referee is to provide a judgement as to whether the article has been executed well, and whether it is convincing. You are providing a recommendation to the editor, and not a decision. As an input you may find this Research Assessment helpful.
If your recommendation is REJECT you should provide:
- 2 clear paragraphs for the editor
- 5 suggestions for the author
If your recommendation is R&R you should provide:
- A clear explanation of what you consider to be the minimum requirements to get the paper over the line
- Thoughts on how to improve the article beyond this
- Note that you have the benefit of anonymity so you can be harsher than the editor and make their job easier.
- Brief synopsis – write a single paragraph (2 at most) summarising the article in your own words. Do not merely repeat the abstract. Either demonstrate that you understood the article or explain what your didn’t understand about it.
- Assessment – provide a summary of your opinion of the paper. Assess how the paper fits into the broader literature. Be clear on any points that you don’t feel qualified to judge.
- Areas/issues – provide detail on perceived weaknesses. Smaller issues (e.g. typos) shouldn’t form the bulk of the report.
Much of this advice is based on Ed Glaeser’s talk at the 2019 EEA Meetings.
“Reviewers come in numerous categories. One group, often the most gushing in their praise, show few signs of actually having read the book. A second group absorb enough of the introduction and of passages relating to their own specialty top pass resounding judgments. A third group, in a distinct minority, follow careful reading with balanced comments. And then there’s the Worshipful Company of Whingers, Carpers and Nit-Pickers, whose sole aim in life it to find fault.”
Davies, N., (2006) Europe East & West, Jonathan Cape
The “one week paper” sounds ludicrous, so it’s important I make some important caveats: 1. the paper must not require any primary empirical research (i.e. there is no data collection or analysis involved); 2. the paper is an article, therefore you’re predominantly citing/drawing upon journal articles and not books; 3. the paper fits into an area that you have already published on, and the topic is something you have been thinking about for a long time and have already collected material; 4. you have a publication lined up, so that you know the journal/edited volume it will appear in, you know the editor’s requirements, and you know the audience.
If the above apply, the one week paper is feasible. Here’s the schedule:
Monday Gather the large pile of existing material that you have been keeping over the last few months/years. Print out references and articles that you’ve been keeping (e.g. draft emails, filed emails, blog posts, RSS feeds – however you “keep” articles to follow up on later). Conduct primary desk research to assemble your literature review. Take the key references, and go through their citations. Bust Google scholar, JStor, and whatever other databases you use. Print, print, print
Tuesday Get the cafetiere on the go, find a comfortable chair, and READ. Go through the printouts and read everything. Annotate as much as possible – both in terms of comments and general thoughts
Wednesday The aim now is to settle on the structure of the paper. Do *not* switch on your computer. Collate the printed material, lay it out on the floor, and formulate the key sections. Use a blank piece of A4 to write out the structure of the paper. Label the INTRO, Sections 1, 2…, CONC. on the printouts, and pile them up (loosely) into sections. The paper should now be laid out in front of you
Thursday Have a lie in. Think about the paper as you drift in and out of sleep. Then, start writing. Consider the writing process to be similar to an oil painting. Create a base layer in terms of the structure: i.e. create the sections within the word document. Then begin populating each section by taking the pile of papers and working through them methodically.
Friday Keep writing. I find I’m at my peak from 10pm – 2am, so I try not to feel as though I’m slipping behind if I’ve not made much progress before lunch.
Sunday I don’t have a printer at home, but at this point you should print out the draft. Go to the pub, get a cold pint of lager and with a gloopy blue pen make corrections. Come home and revise the draft (crossing through each annotation with a thick black marker once it’s been adopted). Ensure spelling/grammer/formatting is sensible. Save a PDF version. Upload onto SSRN if you want broder feedback. Submit. Relax.
I’ll post links to examples of “one week papers” here: …
I’m still not sure how I managed to complete my PhD within 4 years and also get married in that time, but it must come down to:
- Pick the right school – I was immensely fortunate to go to the dynamite George Mason (that dares to be different), and it meant that knowledgeable people, who shared my passions, surrounded me. If you study in an environment like that, you can’t go wrong
- Preparing well – find out the textbooks for the core subjects and read them before you start
- Find/create a good study group and stick with it – I find that learning is a social process and therefore a stable group of close colleagues is gold dust
- Have realistic expectations – for me the first year was about survival. I’m consistently told that ambition and aspiration should exceed merely “survival”, but for me lower expectations seems to work
- Drink productively^ – if you’re going to get drunk a lot, make sure it’s productive
- Love what you do – my PhD mattered to me. I was willing to sacrifice a great deal. If it’s a chore then it’s a struggle, but I never grew tired of my subject and still want to discuss and develop the ideas I pursued.
- Change the title at the last minute – without trying to contradict the previous advice, in the final few months of intense writing/rewriting of my dissertation I changed the title. As the formatting requirements began to take priority over the content, you begin to question everything. At about 4am I had a revelation and rewrote the abstract and changed the title. It was a good decision – not only was it a more accurate title, but it reinvigorated me and breathed new life into the project
- Endure: it is a test of stamina. Pete Boettke likes to recite James Buchanan’s advice of keeping your butt in the chair. I believe that most academics are lazy, and if you work hard you will succeed. All writing is work and all work is work in progress. Just get it down and keep moving forward. If you go straight to grad school from undergrad you’ll realise that you’re open all hours. You can have a healthy work/life balance by realising that:
- Your life is your work and your work is your life
- You’re not working 9-5. Whenever you’re awake, you’re working.
Bottom line is that ideas matter.
More grad school advice:
- Grad School Rankings
- Econ Grad School
- Why Study Economics?
- Applying to Economics PhD Programs Memo, by Alvin Christian
- Recommendations by Walter Williams
- You and Your Research by Richard Hamming
- Writing, Research, Publications links from the New Economist
- Grad School Rules from Fabian Rojas
- Guide for the Young Economist comments by Pete Boettke
- Ariel Rubenstein’s advice, some of which I agree with
- Anton Howes’ thoughts.
As a warning, don’t ignore 6 bullet points on why people go into grad school in the Humanities (via Tyler Cowen). Mike Moffat has a guide here. Some funny cultural insights in grad school are here and here. Michael Munger has an excellent series of videos providing a Guide to Academic Publishing.
“If you ever go to the ballroom of a hotel where there’s been an academic conference you see men and women who roll uncontrollably to the music… off the beat…. waiting for the end so they can go home and write a paper about it.”
Sir Ken Robinson
Although I get highly anxious about presenting at academic conferences, they are an incredibly important part of academic life. In addition to receiving feedback on works in progress, they permit a survey of the contemporary state of your discipline. The best ones provide a mixture of confidence boosting rubbish and inspirational insights into elite scholar’s work. They also provide an opportunity to visit interesting places and socialise with good people.
Even if you don’t receive useful comments from the audience, the act of writing the conference talk and thinking through potential criticisms is valuable. In a strong department this would be better achieved through an informal speaker series. But for those of us without a strong department the conference does serve a useful goal as a focal point.
When I’m in the audience I place papers into one (or more) of the following categories:
- Find your focus – at least two papers in one, way too ambitious, naivety of youth, yes it’s obviously a fascinating topic, but to become a serious scholar you need to contribute to the literature, not write about what interests you
- Go do it – makes sense, but where are the results??
- Robust but trivial – what have I learnt?
- Important but flawed – yes, I’m afraid the obvious comeback does invalidate your findings, regardless of how praiseworthy your efforts might be
- WTF? – Possibly genius, but way over my head…
Increasingly I ask myself “Would the planning committee of the Soviet Union be interested in these findings?” And if so, I stop listening.
“No grand idea was ever born in a conference, but a lot of foolish ideas have died there”
F. Scott Fitzgerald
Here are my thoughts on various conferences:
- The most important is the European Academy of Management. I found the selection process to be competitive (I’ve had a submitted paper rejected) but worth it. The session that I participated in was organised with great attention and there was excellent feedback. It was quite expensive, but is usually in an interesting European city.
- The Royal Economic Society is well regarded but I’ve never been. I would also like to attend the European Conference on Entrepreneurship and Innovation.
- For several years I have been impressed by the programme for the Prague Conference on Political Economy.
- The two front runners are the Academy of Management and Strategic Management Society. Both of them have had Austrian panels but I am yet to attend. They are typically pricey and competitive.
- The most important economics conference is the American Economic Association. I haven’t submitted to attend because it’s in the first week of January, which is a crazy time to have a conference. However, it is a great opportunity to see the most famous economists, and is also a focal point for the US job search.
- The Southern Economic Association is a wonderful conference because there are many sessions organised by the Society for the Development of Austrian Economics. This offers a great combination of professional prestige and direct relevance (note that there tends to be a trade off). The SDAE is full of friends and I enjoy seeing what they are working on. There are also appointed discussants so you are guaranteed quality feedback.
- There have been some SDAE panels at the Eastern Economic Association as well. The main advantage this has over the Southern’s is that it’s more likely to be a direct flight from the UK. There’s a heavy presence of assorted heterodox economists, with a fair share of cranks.
- I regularly attend the Association of Private Enterprise Education. It alternates between Las Vegas and somewhere more exotic and has a youthful energy. The emphasis on pedagogy is excellent because I always feel that my teaching, as well as research is benefitting from attendance.
US political science
This guide is written for researchers with integrity intending to conduct industrial interviews. There are alternative means to get information, but they’re not worth sacrificing your integrity.
Personally I don’t think that tape-recording interviews is necessary, but if the interview is a long one it’s a good idea so that you’re not wholly reliant on your notes/memory. Make sure you get permission from the respondent first though.
The purpose of cold calling is to establish who has the required information. Document your journey so that it’s easier to find the right person at the next company that you contact.
- Establish common ground (i.e. explain who you’ve been introduced by)
- Explain who else you’re contacting and who else is participating
- Emphasise any confidentiality agreements (and provide in writing)
- If the interview is semi-structured explain that the questions aren’t designed for self-completion and required dialogue
Sell the interview
- It won’t take long
- The final results will be of interest
- Explain how this information helps their business needs
- Know your subject matter as much as possible
- Know your questionnaire thoroughly
- Ask follow up questions to clarify any points that aren’t clear
- Ask bold questions
- Triangulate responses against available data
- Complete questionnaire and save it
- Keep all promises and engage in any follow up work
- Leave door open for follow up questions (e.g. “would we be able to contact you again…”
Tips for getting useful answers
- “It is my understanding that…”
- “What range would you estimate…?”
- “Other interviewees have suggested that…”
Finally, follow up. Make sure you say thank you. Make sure you send a copy of the final report. Keep the relationship open. Keep your promises.
Case studies are not without controversy, and came to prominence via the University of Chicago’s Dept. of Sociology in the early c20th. It’s important to make a distinction between case studies as a research method, and sociology as a research discipline.
Choosing a Case
- A unique event
- New data that hasn’t previously been available
- Applying a theory that isn’t dominant
Collecting the Evidence
A brief overview of the key examples of what constitutes valid evidence, plus ideas on how to get hold of them:
- Letters & memoranda
- Minutes of meetings and written reports of events (e.g. from official website)
- Administrative documents (direct from the company)
- Formal studies and evaluations (Google Scholar, Library database)
- Newspaper clippings & mass media (Lexis-Nexus, The Economist, the FT)
Documents will provide the bulk of your analysis (they’re the easiest types of information to find). Rather then base your case study on what the documentation tells you, however, try to use it to corroborate evidence from other sources.
(ii) Archival Records
- Service records – Market share, profitability (company website and annual reports)
- Organizational records – charts and budgets (ditto)
- Maps and charts
- Lists of names
- Survey data – census records etc
- Personal records – diaries, calendars (from interview contacts)
Archival records are crucial to establish the factual content of the evidence you use. Not all of the above information would be accessible/useful, but the final case study will depend on the accuracy of the archival records you uncover.
- Open-ended – ask for the respondents opinions and insights
- Focused – following pre-arranged questions
- Survey – tightly defined questions used for multiple respondents
All methods are suitable, but be clear before conducting an interview which approach to take. Open-ended are very good if you have plenty of time, and the respondent is an expert on the subject. However this must be corroborated with archival records. Be careful not to ask leading questions, and use more focused interviews to double-check.
All interviews suffer from problems about bias and inaccurate information, so do not depend too heavily on interviews – use them to get advice, and to corroborate other evidence.
(iv) Direct Observation
A field trip is useful if the situation is still occurring – i.e. if the company is still trying to solve the same problem that you’re investigating. Observation can help you corroborate what an interviewee has told you about how the organisation is run, but due to the subjective nature it’s better if more than one of you is present.
This is when you take an active role within the environment you’re observing, but is usually used if the other sources of evidence are unavailable.
(vi) Physical Artefacts
Obviously you should collect as much evidence as possible, physical artefacts might include technology, tools, instruments etc
Principles for Collecting the Evidence:
Once you’ve got your evidence, there are a few things that ensure a rigorous case study:
- Use Multiple Sources. Using the classifications above, try to find information from as many different sources as possible
- Create a Database. The perfect case study should have the data/evidence completely separate from the report. Whilst you’re creating and writing up your case study bear in mind that the facts should be kept separate from your analysis (as much as possible). This can be done using two stages. Firstly, write up your interview notes, and secondly, write up your analysis. In theory the first part should be made available to other researchers who have completely different research questions, but would still find your data useful.
- Maintain a Chain of Evidence. To make your analysis reliable the reader of your case study should be able to follow the derivation of your evidence from the research question to your conclusion (or vice versa). This is done by accurate citations (whether it’s documents, interviews or observations), demonstration of the actual evidence (and the manner in which it was collected), and that all this is consistent with the original project plan.
A word of advice – if your case studies are chronological you may wish to start off by writing them backwards, i.e. have a section on the current conditions and then go back to the phenomena under investigation, and finish with the firm’s strategic decision-making.
For more, see Yin, R. (1994). “Case study research: Design and methods” (2nd ed.). Beverly Hills, CA: Sage Publishing.
- It’s not proper fieldwork, unless you become seriously ill [i.e. “you need to get a disease to be a serious anthropologist”]
- Learning a foreign language – communication vs. theory of language [the Golden Lions of the Presidential Palace]
- Three responses to “Global Prosperity Initiative”
- Jane Goodall and the Chimpanzees
- Wearing an Adam Smith tie in an interview about laissez-faire
- “Being a Westerner” (note, not an American!)
- Formal interview
- Informal conversation
- Direct/indirect observation
Local Knowledge vs. Standing Outside the Frame
- Hayek: knowledge is dispersed
- “You can’t see the picture, if you’re standing inside the frame” RS Trapp
- How to assemble knowledge of time and place?
- Equip yourself with a hunch, and a broader framework
- Note: not a theory that predetermines all your evidence
- Arrive knowing nothing of cultural context
- See what strikes you
- Learn culture from locals
- Do the history books dictate the local behaviour? No! Local culture dictates what goes into the history books. Culture is the lead variable
- Go out with the locals, learn from them
- Then fill in the gaps, and corroborate, via historical record
- Your hunch will probably be wrong, the fieldwork rests on your ability to ground it within a strong framework
These are notes for a talk that I gave on doing fieldwork. Out of context parts of them may not make much sense, but if you are interested in this topic send me an email!
A “one pager” is a succinct summary and commentary on either a book or journal article. It is intended to establish that you can grasp the key points of a particular work, and contribute constructively to scholarly dialogue (for advice on how to read an academic paper, see Peter Klein). I first encountered the one pager through Roger Congleton, and have found it to be a highly effective training device to interpret information. As the name suggests, it must be kept to one page.
There are four parts to a one pager:
- Provide an accurate citation of the book/article
- Include your own name and relevant details.
- Use three bullet points to provide a holistic summary. Each paragraph should be short, and pick up on a critical part of the thesis. If you’re reading the text with a specific reason in mind (e.g. a literature review on a particular subject), the summary can be focused on that aspect of the piece.
- Use three bullet points for constructive analysis. These might be aspects of the manuscript that you didn’t understand, sections you feel could/should be expanded, or parts you outright disagree with. The three points should demonstrate that you can critically assess the material, think creatively about how to build upon it, and draw upon a wider knowledge of the subject.
As with most skills you can develop your ability to write a one-pager with practice. It’s a method to focus your attention whilst reading an article, and therefore – I find – can drastically reduce the time it takes to absorb material, and increase the effectiveness of your reading.
Also see: Business One Pagers