Running a digital diary study using Tumblr
Posted: May 25, 2013
I’m in the process of running some in-depth user research for a NESTA-funded project to create a mobile web app for the Royal Opera House. The brief for the research was to better understand the behaviours and motivations of mobile users when connecting to arts and cultural information online. The outcomes of this research will not only inform the design of the Royal Opera House’s new app but will be synthesised to create a toolkit of best practices for other arts and cultural organisation wanting to develop their own mobile engagement strategies.
I wondered about the best way to collect information on user behaviour and motivations: a number of studies on mobile use already exist, but they are mainly quantitative. I saw an opportunity to gather a data, more qualitative data set. I thought about running interviews or even focus groups, but one idea stuck more than others: using a diary study.
Why a diary study?
To understand how and why people use their mobile devices, you need to be where they are when they’re using them. Except you can’t–due to the very nature of mobile devices it’s hard, not to say intrusive, to naturally observe them being used “in the wild”. A diary study–asking users to keep a record of their mobile use at specific points during the day–seemed to be the natural way to go.
This kind of study isn’t without its problems of course–you’re reliant on your participants providing you with reliable data and it’s hard to verify its validity. It may be recorded some time after the event which causes some potential problems with retrospection. Also, unless you can provide a structured format for your participants to use, the data you’re collecting can be wildly variant and completely unstructured, leading to headaches at analysis time.
Despite these drawbacks, the format was the most appropriate for what I was hoping to achieve. I decided to back it up with more quantitative data in the form of a series of surveys sent out after the study was complete.
Recruitment, structure and drop-outs
Of course, it’s important to be clear about what you’re hoping to achieve before you start a study like this and to have realistic expectations about it. I decided to run my study for ten days: seven days for the unstructured study, followed by three days of follow-up surveys. Any more than this I thought would be pushing participant’s motivation to its limits, not to mention generating a whole load more data for me to analyse.
Recruitment was not a problem for this study: I was lucky enough to get support from the Royal Opera House’s internal machinery for contacting with its audience. This resulted in some enthusiastic sign-ups for the study, drawn from a cross-section of the ROH’s audience. For my part, I used the King’s College internal research volunteer machinery which was less successful. However, when it was announced via Twitter and Facebook, I was literally having to turn volunteers away. Of course, this was a largely self-selecting participant base so that will need to be borne in mind when drawing conclusions from the data.
The participants were incentivised in the form of some Royal Opera House merchandising and access to premium content on the app when it’s launched. However, most seemed to be happy to be simply helping out with the research–whether this is due to association with the ROH itself or other interests is something I’m hoping to discover.
Drop-outs in this kind of study are inevitable, but the drop-out rate seemed smaller than that predicted (10-20%) in some of the literature I read. I was surprised at this, not least as I was asking participants to record their usage at three points during the day: morning, afternoon and evening.
I didn’t have the time or resources to set up any specialist software for the study so I used private Tumblr blogs–one per participant. To set up private blogs you need to create a public one first–the private blogs are secondary. One issue I had is that the only practical way I could find to allow participants to post to their blogs was via a “submit” form URL or emailing to a dedicated email address. As owner of the primary blog, I had to publish entries before participants could see them which created a time lag between submission and appearance on the participant’s site. This caused some confusion and a little frustration for some participants who submitted entries more than once.
Also, Tumblr will set limits on the amount of secondary blogs you can create in one day (ten); quite sensibly–not least to make sure you’re not doing this for nefarious spamming purposes. I’m not sure if there’s a limit to the total amount of blogs you can set up; I certainly didn’t hit it at just under 30.
Using a third-party system has its own advantages and disadvantages. On the plus side you can outsource the running and maintenance of the software, on the downside you can do very little if people experience problems using it. Also, you might have security concerns about hosting this data on a third party system. I was largely happy with it–I wasn’t asking people to record any private data, it was behind a password and the blogs themselves will be deleted once the data has been collected.
On that last matter–data collection, there is no way to download data from Tumblr, unless you want to do things with its API. I actually found the easiest way to get the content out was to copy and paste into a consolidated Word document which I could then use as the basis of my analysis (using a print-out and pens, oh yes).
I should also spare a few words for Google Docs (or Drive, whatever). This was invaluable in allowing me to easily set up a series of surveys and to collect the results in a sensible and usable format. We can take this kind of software for granted, but it’s a great enabler, particularly if you’re in a team of one and have limited resources. So, thanks Google and Tumblr. I believe the appropriate phrase is “you rock”.
I think one the keys to success in running a study like this is regular communication with the participants. One way to do this is to monitor the entries coming in and see if there are any requests for assistance, or if there is anything you can ask the participant to expand on. However, this is naturally time consuming so I decided instead to send out an email at three intervals throughout the study: beginning, middle and end; and then an email per day during the post-diary survey stage. This was to create a sense of connection between the research team (well, me) and the participant and to try to keep them motivated.
Now begins the rather daunting process of data analysis and encoding. But, more on that later.
No comments? The comments haven't exactly been coming thick and fast recently so I've disabled them for the time being. Instead, why not drop me a Tweet at @littlednet or email me at firstname.lastname@example.org?