There's been a lot of talk about using AI and in particular large language models in evaluation and specifically in coding and processing texts. Here at Causal Map we've been working very hard on just that (and on automating interviewing too, but that's another story). And we see fantastic potential. Our Causal Map app now has a beta version of that big "auto code" button we'd always dreamed of (and feared). However, I wanted to draw attention to a really big distinction which I think is important. There's a continuous spectrum between on at the one end transparent, reproducible approaches founded in social science and the other end of the spectrum black box approaches where responsibility is shifted from the evaluator to the AI. There may be use cases for the latter, "black box" kind of approach. Maybe one day doctors will abrogate all responsibility to medical AI. Maybe one day evaluators will abrogate all responsibility to evaluation AI. But here I'd like to set out reasons why right now we should prefer transparency. Black box coding is possible today in its rudiments and it's going to get a lot more accessible and powerful quite quickly. At its most extreme, you simply say to the AI 'Here's a load of documentation from a project. You tell me if the project is efficient, effective, sustainable, draw some conclusions and make recommendations according to criteria C, D and E. This is an extreme case, but the basic idea is submitting a long text and asking for a black box judgement about what themes are present and even what conclusions can be drawn. To be sure, it's possible to say to a model 'Yes and also show your working or print out some quotes or examples to backup your findings.' But it's very important to realise that this "show your working" question is spurious because AI at the current state of development has no more insight into its inner workings than does a human being has into his or hers, and probably less so. So while it can (and will) competently bullshit about what steps somebody might have taken to reach that conclusion it doesn't mean it's actually the steps that it did take. So basically, you have no way of knowing how the AI came up with a particular finding or conclusion using this approach and it's a massive abrogation of responsibility for an evaluator to sign off this kind of output without further analysis. Now at the other, "transparent" end of the spectrum, what we recommend is using AI merely to follow established procedures of manual coding and do it faster, more reliably and more reproducibly. That's a big win. The old school way: First of all, highlighting individual sections of text according to explicit rules set by the evaluator and then aggregating and combining those codings, again according to explicit rules. As an aside, we believe that even before we get into the AI possibilities, causal mapping in particular is a really good way to summarise documents and in particular sets of documents. Obviously, there is more to documents than simply the causal claims made within them, but if you had to pick a type of content an evaluator might want to extract from a document, causal claims are pretty central and the procedure for identifying, extracting and aggregating those claims are an order of magnitude more straightforward than any other kind of useful text analysis (unless you count word clouds...). In particular, causal mapping is particularly good at making summaries from sets of documents, such as semi structured interviews with comparable respondents, rather than only the special case of making one summary of just one document. It is already possible to say to an AI, 'please read this long document and draw a causal map saying what do you think are the main causal drivers and outcomes and intermediate links and just print out the specification of a diagram'. And the job's done. That's exactly the sort of approach we are warning against because you have no way of knowing how the model has reached that conclusion. When we use AI to help code a set of documents we tell it to explicitly identify causal claims and provide the relevant quote for each individual claim, following rules we give it and in each case, it's possible to look at the actual quote it identifies and check if it really is appropriate evidence for the causal claim. Just as with human coding, in the sort of way causal mapping has been carried out for 50 years or more It's been a lot of work to develop the right set of prompts (and they are still a work in progress) embedded in our app, but the prompts we use in any given case are pretty simple and transparent: around half a page of standard prompts which are pretty much the same across use cases and another half a page or so of prompts which are specific to the use case; these themselves are 90% derived in an automated way. Nevertheless, the evaluator bears 100% responsibility for overseeing these prompts, which are plain English. They can be followed by a team of postgrads or by the AI: there is no difference in principle. There is no black box and no magic, and any human can follow every step of the argumentation. At present, the AI is much faster and more reliable and transparent than a human coder; and a human coder is much better at seeing larger connections, reading between the lines and linking up the parts of a larger story. The most interesting part of causal coding with AI is to add this human inspiration back into the AI prompt in a transparent way. In order to then aggregate, synthesise and simplify the causal maps which result, we can use the many, more or less standard, causal mapping procedures which have been developed over the years and in particular our open source set of causal mapping algorithms. So an interested outsider can follow the chain of argument right away from the original text to the final conclusion. Responsibility is the issue here. If you feed data or documents into an AI and let it come up with its own conclusions, they aren't your conclusions and as an evaluator you can't sign off on them. Maybe this will change in the future as we learn to find our way around in this new world. But right now, you need to show your working. Of course the big worry in all of this is that higher-level, black-box approaches are much quicker and easier to apply, putting together black-box approaches to get from documents to findings to (evaluative) judgements in just a few clicks, given some generic definitions of evaluation criteria. Black-box approaches could be the beginning of the end of evaluation as we know it, but they'd be really tempting for a commissioner: for a purely document-based review, who'd bother with the time and expense to commission an evaluator if you can get your report written in a few minutes? With black-box approaches, people's fears about bias are really justified.

This blog was originally posted on the Bath Social Development & Research site. We are grateful for this guest post from Michelle James. Michelle is a PhD researcher specialising in refugee and asylum seeker welfare and wellbeing in the UK. She also works as an independent research consultant in the development sector. Her particular interests include partnership models of development, community empowerment and mobilisation, and behaviour change. I am currently in the final year of a PhD in Social Policy at the University of Bath looking at how different forms of social protection impact the wellbeing of UK asylum seekers. As an experienced QuIP analyst, already impressed by the benefits of the research tool, I knew that I wanted to incorporate QuIP data collection and analysis into my PhD methodology. Collecting data from a hard to reach, linguistically diverse and potentially vulnerable population during the covid pandemic was, however, far from straightforward. My anti-oppressive research approach led me to adopt research tools that I hoped could empower participants to enact agency within the project, minimise the extractive nature of data collection, while still generating the academically rigorous data I required for my PhD. I also needed to gather data that helped me understand what impact government, community and peer-led social protection was having on UK asylum seekers without asking leading questions to minimise response bias. As such, I chose to utilise two main data collection tools, supplemented by a range of additional data to triangulate my findings. Firstly, I trained asylum seeker/refugee peer interviewers to independently undertake QuIP interviews with those in their social network. The interviews asked participants what changes they had experienced in their lives since being dispersed to their current location by the government and who/what they attributed these changes to. I hoped that the peer interviewers would benefit personally from involvement in the project through gaining work experience that they could cite when applying for future employment. In addition, evidence suggests that asylum seekers fear speaking to British institutional researchers so I also hoped that participants may have more confidence to take part and provide detailed answers if speaking with a peer in their own language. Secondly, I undertook a photovoice project with ten asylum seekers/refugees who each captured a series of images to depict what made their lives easier/happy or harder/unhappy. The images were shared and discussed in depth by all participants at a follow up workshop and the photographers collaborated with me to put on an exhibition of their work in Summer 2022. Check out the online version of the exhibition here. The QuIP interview data, photovoice narrative statements and workshop transcript were uploaded to Causal Map, alongside survey data, peer researcher and photographer feedback interview data, and exhibition feedback statements. The Causal Map app allowed these different types of data to be effectively analysed in one place, uncovering themes and causal patterns through an inductive process. Although the data were consolidated into one Causal Map project file, the software made it possible to separate and interrogate different categories of data independently when creating visualisations to understand which data were useful in answering different types of questions. This resulted in the creation of a sub-group of key informants (QuIP interviewees and photovoice participants), whose data were used to look at the breadth and depth of significant of different types of social protection on wellbeing, with the remaining data incorporated only when applicable to specific research questions. Once the Causal Map visualisations were created, I incorporated them into my thesis alongside pertinent quotes and photovoice images to offer a more rounded qualitative and pictorial description of the wellbeing changes expressed by research participants and the impact of different forms of social protection. For example, the causal map visualisation to the right shows how government-based social protection was impacting the lives of asylum seekers at the time of data collection. Causal chain diagram (Right): Trace path from formal social protection, key informants, 50 most frequent links where source count > 3 In my thesis, this was accompanied by a range of photographs and quotes to drill down on the causal links expressed in the diagram. One example photograph and quote can be seen to the left and below. “It is not easy for an asylum seeker to stay in a hotel room for months. You spend many hours alone, it is very isolating. Hotels are often a long way from support services and you have no money for bus tickets to reach them.” Photo title: Modern Jail Image and text: W, asylum seeker, Afghanistan, 2022 Finally, the causal links and themes unearthed through inductive analysis using the Causal Map were considered alongside relevant theory and literature in my thesis leading to a number of policy and research recommendation for the improvement of social protection provision for UK asylum seekers. Overall, I found the Causal Map app to be particularly helpful in combining a diverse range of data sources into one place, and the simple interface allowed for effective induction analysis by breaking up a substantial dataset into small manageable pieces of text that could be considered independently. Following helpful training by the Causal Map and Bath SDR team, I was able to interrogate the data to create helpful visualisations to answer each of my main research questions. The quantitative nature of these causal maps is helpful for top-level policy discussion, while the retention of, and ease of access to, the qualitative data that underpin the diagrams is important for research transparency and to support a more qualitative and theoretical discussion of the main causal links found in the dataset.

You start your evaluation full of enthusiasm and do your first face-to-face interviews. You learn a lot: how is the atmosphere in the office? Are staff reluctant to let you talk to project users? But it's a big project, there are potentially hundreds of people you could talk to. There are a set of questions you need to cover. Perhaps you want to trace out people's view of the causal links between an intervention and some outcomes. Wouldn't it be great if you could clone yourself and send yourself into everyone's inbox? With the latest iteration of our survey tool StorySurvey, you can do just that right now: Design an AI-driven interactive interview in any world language, share the link with your respondents and download the transcripts! Your respondents need an internet-connected device. It's free for reasonable use (up to 200 respondents). What is StorySurvey like for respondents? Here's an example survey. Try it! Notice that the link has "?survey=msc-conference" at the end of it, to direct you to a specific survey. That is the kind of link you send to a respondent. How do you use StorySurvey to design a survey? If you want to experiment with StorySurvey and design your own surveys, go straight to https://causalmap.shinyapps.io/StorySurvey3/. The way it works is you start from a script aka prompt. This is an instruction to the AI interviewer to tell it how to do the interview. It can be as simple or as complicated as you like, but basically it's just plain English (or French or any other world language). We have prepared a few example prompts. At Causal Map we are most interested in interviews which encourage respondents to tell causal stories, even printing out the individual causal links. But you can design any kind of interview. Your job is simply to copy and adapt any of the pre-prepared prompts, or create your own from scratch. Then test your survey by sending the prompt to the automatic interviewer who will then start interviewing you. Keep editing your prompt and testing again until you are satisfied. Then, you can get a link to the finished survey and send it to others to answer. Your survey link can be public or private. If it is public, the name you give your survey will be part of the link to the survey so it might be possible to guess it, but if you choose a private link, your survey URL will be very hard to guess. Your prompt is always public, so that others can adapt it to build ever better evaluation interviews. This way it would be great to build a library of evaluation interview prompts. How to get your results? You can view and test out StorySurvey, and get interviewed, without even logging in. However to save and share a survey, you need to log in with a Google account or email address. Right now, when you download your transcripts you can analyse them any way you want, but there is no "standard" way to do it. But soon, you will be able to visualise your results in Causal Map. Contact us if you need help with creating or launching a survey or visualising the results: hello@causalmap.app. Technical details StorySurvey uses GPT-4. If you've experimented with ChatGPT before, you might have noticed that the "temperature" is high, which is good for writing essays. At StorySurvey the 'temperature' is set to zero which means the conversation is more deterministic: good for social research. Chat-based interviews like this are no substitute for face-to-face key informant interviews - but they can be used to reach a much larger number of additional respondents. Obviously you can't reproduce a whole interview approach in a simple prompt! But it's interesting to try, and a simple prompt can still generate a useful survey. This site is free and experimental and we at Causal Map Ltd make no promises about what will happen to it in the future. If you want help with a survey, contact us. Privacy StorySurvey data is stored in a SQL database at Amazon RDS, which uses industry-standard security. Data is transferred using https (http over TLS). The text of the interview passes through the OpenAI servers. Data submitted through the OpenAI API is no longer used for service improvements (including model training) unless the organization opts in, which we have not. OpenAI deletes user data after 30 days. We recommend that respondents do not submit data which might identify themselves or others, and respondents have to accept this condition before proceeding with the survey. Enjoy!

Recent Projects The Causal Map app has been used for a variety of projects across many different thematic areas. Here are some which are publicly available. Voscur is a support and development agency for Bristol's voluntary , community and social enterprise (VCSE) sector. They wanted to know what factors affected organisations' access to funding, service delivery, staff training and network building. Country: UK Data: 24 interviews Year of study: 2017 Overview: Evaluation of Voscur ’s impact on the voluntary, community and social enterprise (VCSE) sector in Bristol using the Qualitative Impact Protocol (QuIP) , conducted by Bath SDR . This study was a ‘deep dive’ into changes and drivers of change in the VSCE sector. This study allowed Voscur to learn more about the impact of their projects and external factors affecting the sector. With this knowledge they were able to understand how to amplify positive forces for change (both internal and external) and mitigate negative influences. The map above, is taken from the final report, shows a causal map focusing on the factor 'identify funding sources '. See full presentation here Explore the below table for more studies Causal Map has been used for.

About Us Steve Powell has been working with Fiona Remnant and James Copestake from Bath Social and Development Research Ltd since 2019 to bring Causal Map to life. We are all interested in causal mapping in different forms, and Bath SDR had a particular need for an accessible coding and analysis solution for users of QuIP . The result of this hard work is much more; a standalone product designed to be used by any researcher interested in causal connections. In summer 2020 we set up Causal Map Ltd to further develop and promote the app. We look forward to seeing where this adventure in mapping takes us! STEVE Co-founder and Director Steve has led and contributed to research and evaluation projects in many countries around the world over the last 25 years. He has worked on a wide range of topics, from psychosocial programming after the 2004 tsunami and community resilience in East Africa to counting stray dogs in Sarajevo. Steve has expertise in both quantitative and qualitative research and evaluation approaches. He gained his PhD in psychology researching post-traumatic stress after the war in Bosnia-Herzegovina. ​ This research and evaluation work left Steve longing for a better way to collect and synthesise people’s ideas about ‘what influences what’. This inspired Steve to co-found Causal Map Ltd. JAMES Co-founder Dr James Copestake is Professor of International Development at the University of Bath. His interests span agrarian change, rural development, development finance and evaluation, poverty and wellbeing and the political economy of international development and the interactions between these diverse topics. ​ James is author of the Qualitative Impact Protocol and worked with Fiona to set up Bath SDR to continue working on the research. He is also Director of Studies for the professional Doctorate in Policy Research and Practice (DPRP), Director of the Centre for Development Studies, and author of Attributing Development Impact . James’ interest in Causal Map stems from his work on understanding the intersection between theories of change and people’s mental models of change, and the role causal mapping can play in this research. FIONA Co-founder and Director Fiona is a communications and research professional, with a special interest in the practical application of academic research in the international development sector. She has worked in communications in the private and NGO sector, in both regional and international roles, including four years in Sri Lanka at the Centre for Poverty Analysis. ​ Fiona was co-author of the Qualitative Impact Protocol whilst working at the Centre for Development Studies at the University of Bath, and co-founded Bath Social and Development Research (Bath SDR) Ltd in 2016 to promote more and better use of the QuIP. ​ Fiona works with Steve on the experience learned from causal mapping QuIP data to help with the design of Causal Map. HANNAH Outreach specialist Hannah is a project manager at Bath SDR, who works with Steve to improve educational materials and support users of the app. She also works to promote use and understanding of the tool through the creation of online content. Hannah has previously held communications and outreach roles in the charity sector and brings this experience to her role with Casual Map. They studied International Development at both undergraduate and Masters level with a particular interest in sustainability. Throughout this time she became increasingly interested in the theories and tools surrounding qualitative data analysis. SUBSCRIBE TO UPDATES Contact us to find out more: HELLO@CAUSALMAP.APP

Features When undertaking research, evaluating projects or creating policy, we need to know about people’s mental models of the world: what they think causes what. One way to find that out is to ask people, and analyse their answers. Causal Map does what no other software can: it enables you to code, organise, understand and present this information as a network or map .​ HOW IT WORKS With Causal Map you upload texts containing people’s views about what causes what. This could be interviews, emails, published documents, questionnaire answers to open questions. For more information about these functions ACCESS OUR GUIDE TO CAUSAL MAPPING Log in | Sign up You can access Causal Map for free by simply signing up in the app - click the link below. No credit card information required, all users can use Causal Map for free, up to a limit of 50 links. Or get a subscription for more. Take me to the app!