Evidence Matters

because science reporting and decision-making should be evidence-based

Science: So What? Commissioning and promoting bad research in the name of science communications

with 3 comments

We were disappointed to see that The Department for Business, Innovation and Skills’ (BIS) science communication campaign Science: So What? So Everything had commissioned and was promoting a poor quality research report. This report – The Shape of Jobs to Come by Rohit Talwar and Tim Hancock [PDF] of Fast Future – has been heavily PRd, with approving quotes from Gordon Brown and Lord Drayson [PDF]. Even at a glance, though, it clearly fails to meet some basic research standards: for example, the methodologies used are not justified and the references provided are frequently inadequate.

We therefore asked a few social scientists for their opinions of the report. One of them wrote an extensive review of the report, to highlight some of the problems. The review has been sent to BIS, but they still stand by the report. The review is therefore pasted below: so that readers can make up their own minds as to the quality of the Fast Future report.

Because we wanted to give BIS time to respond, before posting, it has taken a while to blog this. In the meantime, 10minus9 has been starting to pull the report apart in excellent posts on Fast Future‘s report for Science: So What. There’s a little overlap with the review below, so definitely worth reading 10minus9 too.

The review follows in full*:

The Fast Future Report suffers from significant problems. The methodologies used are not adequately justified and the ‘future jobs’ suggested are often lacking good evidence (with some predictions rendered anachronistic by recent and current job developments). A number of unjustified and controversial assertions are made and the references provided are frequently inadequate. It is also not clear what the Report is trying to achieve. I would therefore argue that this Report is not of a suitable standard to be promoted, used or disseminated to the public by Science So What or BIS. While there are too many issues with the Report to address in full, I will go through some of my concerns below.

Methodological concerns
It is stated (p. 3) that “[t]he scope of this study was focused on presenting a representative spectrum of future science and technology developments and highlighting the kinds of jobs that could emerge as a result.” However, the Report does not justify the methodologies it uses with regard to these purposes.

The horizon scanning has not been effective, even in the Report’s own terms. The Report (p. 8 ) states that “[t]he aim was to draw on a wide range of authoritative sources citing well referenced examples of science and technology developments on the horizon.” However, the references offered in the Report suggest an excessive reliance on media sources and on relatively weak online sources. There is almost no engagement with the high-quality peer-reviewed literature on the developments being discussed (see below for more details). It is not clear why there was a focus on the sources used rather than higher-quality material. The horizon scanning also appears to miss some rather obvious points re jobs which are available now and have been for some time (see below). Moreover, it is not clear how or why the 20 jobs for the shortlist were selected from the longlist of 110.

The use of a survey is also problematic. Firstly, I am not sure why a survey was considered an appropriate methodology to use here: the Report needed to make this clear, as it is not clear why a survey of the groups asked to participate was a good way to predict science and employment trends. As things stand, the survey merely shows that some people have said some things about certain jobs: it is not clear what this is meant to prove. Secondly, even putting these issues aside, there were significant problems with the survey itself and the ways in which the survey is reported.

  • It is stated (p. 8 ) that “[a]n online survey was then run from August 12th-21st 2009”. It is not clear whether this was open access (anyone could complete it) or only open to invitees.
  • The survey went out to Fast Future’s network of 20,000, and to a number of additional networks. It attracted 486 responses. It is therefore surprising that the Report does not consider some of the implications of this response rate.
  • Respondents were overwhelmingly based in North America and Europe (pp. 42-3). Only 3% of respondents were from South America, 5% of respondents were from Africa/Middle East and 8% from Asia. There were only two respondents from China, which is problematic given the prominent role that (as the Report acknowledges) China is likely to play in future scientific and economic developments. The Report does not discuss why this geographical distribution might have been the case, nor the implications of this.
  • 65% of respondents were male; 35% female (p, 42). It is not made clear why this gender bias was present, nor whether it may have affected the results.
  • Only 4% of respondents work in scientific R&D, while 24% are in Consulting and 10% in Executive Management (p. 44). The Report does not make clear why the job responsibilities break down in this way. If this focus on those working in particular roles was thought to be appropriate, the Report should justified this decision.

Selected issues with the ‘future jobs’ suggested

  • The Report states (p. 17) that New Science Ethicist may emerge in 2015. However, numerous people are already specialising in the ethics of new technologies – and some have been working in this area for some time. An emergence date of 2015 is therefore clearly inaccurate.
  • The Report suggests (p. 31) Quarantine Enforcer as a future job. However, the Report fails to engage with the research on policing, governance and epidemic management. It seems unlikely that (if or when we do face additional epidemics) a new profession will be needed: police, the military and related services have traditionally been used to enforce limitations upon movement where necessary, and I fail to see why a new profession would be required. I’m sure that the Report could argue its position, but it does need to be argued and to do this the Report would need to engage with the relevant research.
  • Weather Modification Police is suggested as a future profession (p. 32). There have been vast amounts of research on resource conflict: a wide range of tools have been used and continue to be used in order to deal with such conflicts. Once again, I fail to see why a new profession would be created and – if the Report wants to argue this position – it would need to engage with the relevant research.
  • Virtual Lawyer (p. 33) is suggested as a profession that will emerge in 2010. However, large companies such as Amazon have long been trading internationally. They have employed senior lawyers for some time: these lawyers, and others working independently, already specialise in many of the issues associated with the so-called new job.
  • Waste Data Handler is suggested (p. 37) as a profession emerging in 2010. However, people have been employed to destroy confidential electronic data for some time: for almost as long, I suspect, as they have been computers in use. Certainly, many universities have had procedures for this in place for rather a while and it appears likely that organisations such as NSA have been destroying classified data before disposal of media for considerably longer.

Selected additional concerns

  • It is argued (p. 7) that “Governments globally are increasingly challenged by the issues of feeding a growing planet, educating our children, providing new housing solutions, delivering alternative clean energy sources, solving our need for efficient transport, ensuring our security and tackling dangerous climate change. Science and technology are seen as central to providing effective, affordable and sustainable solutions to all of these challenges and more.” However this claim is problematic: for example, many would argue that more equitable wealth distribution will be key to providing all with appropriate access to food and housing. This claim needs to be justified, not simply asserted. It is worrying that the Report then argues that “The good news is that governments are not simply stacking these challenges up at the laboratory door and walking away with hope as their only strategy for how to address them! Across the world, science and technology has been a major recipient of funding in government economic stimulus packages.” The implication that leaving problems such as access to food with scientists and technologists would be appropriate is incredibly problematic: the Report needed to engage, for example, with those social scientists who have emphasised the importance of distribution and entitlement in access to food. The Report seems to be suggesting purely technical solutions to social, political and economic problems.
  • The Report argues (p. 8 ) that “We are also entering an era where science and technology will assume far greater prominence in our lives and will play a more central role in everything we do.” This claim needed to be argued for, not simply asserted: a lot of the social science literature questions such claims. For example, it is not clear that technology today is playing more prominent roles than when – for example – it served to radically alter the ways in which wars took place: changing the shape of the world, and playing the rather prominent role of killing and wounding horribly high proportions of certain generations.
  • The “ten patterns of change” focussed on are not adequately justified as being “the key global trends and developments shaping our world” (p. 10). It is not sufficient to focus on a single project by the American Society of Association Executives and the Center for Association Leadership. There have been vast amounts of research done on global trends, across a range of fields, and the Report is remiss in failing to address any of this. It is also not clear how the Report moves (in Table 1) from these patterns to opportunities, implications and possible future jobs. Once again, this step in the argument is not adequately justified.
  • The Report suggests (p. 15) that a number of scientific innovations may be required to make cross-generational working effective. This is rather puzzling, given that cross-generational working has taken place for most of humanity’s history. While working past retirement age may appear to be a new development in Britain, there has been a long history of people needing to work into old age in order to support themselves and their families (allowing so many of the UK population to retire at around 60/65 has been a relatively recent innovation).
  • The Report (pp. 17-18 ) states that “Technology enables even the most marginalised to find a voice (e.g. Twitter in Iran)”. However, the most marginalized members of society will frequently struggle to access such communication technologies and networks. Moreover, while data are obviously relatively lacking, it does appear likely that many Iranians dissidents are not drawn from the most marginalised sectors of society: in order to participate in political action, people need to have some time and resources available to them. A lot of research has argued that revolutions seldom come from the very bottom of societies.
  • It is claimed (p. 94), based on a single link to a Virtual World News story, that “Virtual Worlds Lessens Importance of Geography”. This is a hugely controversial claim: the impact of ICTs on the role of geography and place it is hotly debated in the academic literature on globalisation, international politics, economics and geography. The Report could argue this point, but it needs to be argued and not just asserted.
  • The Report claims (p. 101) that there are ““Increasing Levels of Political Instability”. The only citations to support this are references to Aon (a “global provider of risk management services, insurance and reinsurance brokerage, and human capital consulting”) and the Global Peace Index (which, while interesting, has only been running for a bit over three years). Longer-term trends are more ambiguous – for example, the Cold War has ended and it has been 65 years since the last world war. Again, this point can be argued, but the Report would need to engage with the relevant research.

Inappropriate references

References are often inadequate. It would be too time consuming to go over all of them, so I will note a selection of the problems:

  • Footnotes 9 and 10 refer to the (reasonable) claims that demand for food, water and energy is expected to jump. However, it just offers a link to a page on the Third Eye Concept site, discussing what Prof Beddington will say at a forthcoming event. Surely a stronger source could have been found?
  • The claim that nano-medic will become a profession is supported only by a link to a blog post on ‘jobs for futurists’. It is not clear why nano-medic was chosen out of the long list of possible jobs provided. The entirety of that blog post’s description of the role is “Nano-sized machines to deliver health. ‘Nuff said.” The blog post states that some of the jobs suggested “aren’t even all that far-fetched.” I should be clear that I am not criticising the blog post linked – it is a good read. However, it is not sufficient to support the claim made in the Report.
  • Footnote 14 backs up its claim re a future profession with a link to a Time article containing under 35 words on the profession the Report is discussing. There is lots of good research on genetic engineering and farming and I’m not sure why the Report did not refer to any of this.
  • Climate change reversal is a highly controversial topic. It is therefore unfortunate that the Report’s citation for this is based on the inspiration of Fast Future staff and includes a link to only a single online article in Biomass Magazine.
  • Two of the footnotes reference only howstuffworks.com
  • Three of the footnotes reference only Wikipedia pages.

3 Responses

Subscribe to comments with RSS.

  1. Social comments and analytics for this post…

    This post was mentioned on Twitter by brainduck: http://ow.ly/ZiZ5 @James_Hayton, http://ow.ly/Zj3e @EvidenceMatters debunk stupid ‘future jobs’ report by @fastfuture for @ScienceSoWhat…

    uberVU - social comments

    January 23, 2010 at 8:11 am

  2. […] social scientist’s review of the report finds that: the methodologies used are inadequate and poorly implemented; some of the […]

  3. […] Despite positive media coverage this report has come under some criticism, as detailed by Evidence Matters, for its methodology, inappropriate job descriptions and inadequate references.  In addition to […]

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: