Events Detail

Pilot 4 Evaluations

Due to Pilot 4 being an urban discovery app for cultural heritage and this being Luxembourg we planned  to undertake our experiments before the weather turned inclement. So over the past 2 months we have undertaken a series of play tests to iteratively evaluate the App.  Our primary content has been written for the City of Luxembourg and Malta where you can discover stories related to the broad topic of migration but we also developed content about art, architecture and urban change for Belval, in the South of Luxembourg, located where the university is based. This allowed us undertake experiments to evaluate the User experience (UX), Usefulness and Usability of the APP without requiring participants to travel into the city for the purpose of our experiments only.
Here is a quick overview of what we have been doing and what we observed during the experiments.

Play test 1: Sept 15 2017 we were at the University of Luxembourg Welcome day: Location Belval, Weather: not bad - no rain and not much wind but a little overcast.  

Lots of students showed an interest in the project (we had around 40 Android app downloads). Following the experiment the major issue we observed was in the process for activating content. It was too complex and prevented people from playing and discovering stories. So we redesigned this for the next play test.  Before the next experiment we conducted a paly test with individuals to debug what we saw as improvements.

Play test 2: Sept 28 2017: we were at the LIST afterworks in Luxembourg City.  Weather: a sunny, warm autumnal evening.

Not many colleagues turned up to the afterwork drinks but 12 people download the app and we gathered 4 trajectories (traces through the city).  In this context players preferred to be social and test the app in small groups.  The new design for activating stories proved to work as people discovered more than one story.  Participants were interacting with the content stories and providing their reflections.  Main observations (1) To improve we observed that the tool needed to suit different types of learners and the individual feedback from participants indicated that some people disliked reading the texts (perhaps these are visual learners). (2) Participants struggled to add their own story.  Following this experiment we added the functionally to play text to speech and updated the design of how to add your own story.

Play test 3: We were at the LIST organised IT and Cultural Heritage Event held in Belval.  Weather: colder but not rainy.  

This event included potential stakeholders in the CrossCult project including the Luxembourg City Tourism Office, the National Center of Audiovisuel, the Fonds Belval and a number of national museums. This was the first time stakeholders got to play with the App. We received 17 (11 men; 5 women; and 1 no answer) responses to our post-test questionnaire and players in general discovered more than one story and provided reflections on the content.  We are currently analysing the participant data that was collected both with the app and via a short questionnaire about their experience. Feedback provided indicated that people liked discovering information about the city activated by GPS but that they did not know but they did not like the voice of the speech to text functionality.

Play test 4:  We were with group of students from the MA in European History at the University of Luxembourg, experimenting in Belval. Weather: Overcast, cold and 4 degrees.

The students have been set an assignment to design a tour associated to the history of mental health in Luxembourg. We are in discussion to determine if the Pilot 4 platform could be helpful for them.  So the best way is to try out the App. Students went out in pairs to rediscover the campus and We received 14 questionnaires from 10 men and 4 women. Feeback on the app was mixed with some students, stating it needs improvement whist others were very enthusiastic, stating they would recommend the app to friends because you can “learn interesting things about the place where you are at the moment”.

We are now collating all the responses and will begin processing the evaluation results in the coming weeks.

Thank you to all our play testers, we appreciate the time you have taken. Your feedback has been very valuable to us.

Pilot 4 Team

·         Pilot Specs and Experiment lead: Dr Catherine Jones

·         Pilot leader: Dr Ioanna Lykourentzou

·         Pilot 4 lead developer: Stathis Theodosis

·         Pilot 4 developer: Louis Deladiennee

·         Game designer: Antonios Liapis