Our overall design hypotheses were not supported by our experiments, and it seemed that a majority of the participants preferred the Mosaic design over Slidedeck when it came to completing the tasks and the feeling of efficiency (though there was no significant difference in speed). On the other hand, Slidedeck was shown to provide a better experience when used for the purpose of browsing for cafes. One interesting finding from the experiment was that when the participants were given less options to select from, they tended to spend more time clicking through the different cafes. It was shown that on average, participants took longer to complete the tasks when given fewer number of cafes to choose from, regardless of prototypes. It might point to the possibility that users’ behaviors change depending on the number of cafes shown. When given a smaller number of cafes, it might cause the users to want to explore more options since it would be relatively easier to go through all of them. However, when they are provided with a greater number of cafes, they would be more reluctant to want to click on the cafe unless they were really interested in it.
This project opened up some possibilities, and more research would be needed to take this concept further. There would be a need to discover whether a user’s behavior changes depending on the number of options shown on the application. If users were found to be more willing to explore all the different options, an improved version of Slidedeck may be a more suitable prototype since participants found it to be more enjoyable for browsing. Our results would indicate that our initial concept was still worth investigating, but there would be some changes to be made going forward.
7b) Reflection on Design and Evaluation Process
Initially, our prototype was based on a mosaic-like design in which a handful of cafe images were displayed on a page. Users would just simply click on a cafe image to find more information. However, after the implementation and initial trial of this prototype, we decided that the design was too simple and not appealing for users. We created another design prototype with a slidedeck approach with a more modern feel, considering that our users are students. We then decided to test out both the mosaic and slidedeck designs in an experiment to see which design users prefer. We learned that users prioritize simplicity and functionality over aesthetics as we received a lot of negative feedback about the slidedeck design being unpleasant and complicated to use, which surprised us.
Our experiment methods worked as we expected and we received quality feedback on both designs. We were able to easily distinguish why one preferred a design over another. In order to measure efficiency for a design, we originally considered the number of erroneous taps and touches. However, we decided to discard this quantitative measure since we could not reliably define an “error tap” would be. Thus, the time taken for each task became our measure of efficiency for a design. We learned through piloting that users were confused for the first few tasks but became faster on later tasks as they became more familiar with the interface. Therefore, we modified our protocol to give them time in the beginning to familiarize themselves with the interface they were about to use, before starting on the tasks. This yielded more interesting data and feedback.
Our overall design process was efficient for our project. Users were able to complete most tasks relatively easily since our interfaces did not require many steps for users to complete tasks. Some aspects of our design process that could have been improved upon were giving better signifies in our interfaces. Mainly, for our slidedeck design, some users were not able to find some features because they did not realize that they could swipe to scroll horizontally. A common first instinct was to click on anything that appeared. Another issue with our design that some users came across was the noise level preview feature. We used a “play” icon similar to YouTube’s to signify this feature, and some users assumed that a video would start playing, when our intention was to play a live recording of background sound in order to give users an impression of how noisy the cafe may be.
Blog Update #6a–Pilot Tes
We ran two sets of pilots. For the first set, the experiment was conducted internally with two group members. As a result, we updated the tasks that participants complete. Originally, there were two sets with each 7 tasks in each set. To counter learning bias and ordering effects, we created 4 sets with 3 tasks with the same level of difficulty. Instead of giving participants a list with the set of tasks, we created 4 Google Slides presentations with a clear start and end to every task. This significantly decreased the difficulties of analyzing videos for task completion times. The questionnaire was edited to better reflect our hypotheses and directly compare design A vs. design B. The interview script was changed to reflect the changes in the experiment protocol.
For the second set of pilots, the full experiment was conducted with two participants. After these experiments, we decided to introduce a minute before every task set to allow the participant to become explore the interface. In these two pilots, many features were under-observed due to the pressure of the task completion time recording. The post-observation interview was expanded to discover richer answers.
Blog Update #6b -Experiment Abstract
As a student, finding a cafe well-suited for studying in is often a struggle because important information about ambiance, amenities, and availability cannot be discovered easily. We created two prototypes to address this problem. To evaluate our designs, we conducted a 2×2 within-subjects experiment to see if efficiency and preference would vary with a different number of cafes (8 cafes vs. 16 cafes) and variation of design (mosaic vs slidedeck) within 10 participants. Our results suggest a significant preference for the mosaic design when finding a specific cafe while the slidedeck design is significantly preferred for browsing. The quantitative data collected did not provide significant evidence of an interaction effect nor a main effect of number of cafes, design on task completion time. However, the qualitative responses suggests that while both designs have positive attributes, mosaic is slightly more preferred for ease of use and efficiency.
Blog Update #6c–Revised Supplementary Experiment Materials
We decided to move away from a direct comparison between our system and Google Maps. There are various variable that would be difficult to control when evaluating our system against Google Maps, such as prior experience. Instead, we have created two separate design approaches that examine which arrangement of features creates the most pleasant experience for the user. This could depend on their approach if they are simply exploring the application or using it to find a cafe within a reasonable time. For those reasons, we had created two prototypes that we believe emphasize different aspects of the application. One design is able to better showcase many cafes at a time and requires less individual screens, whereas the other cafe provides detailed information without the user exiting the initial screen, but requires up to triple the amount of screens.
The revised goals of the experiment are:
Which design is more effective to use to find a cafe based on the time taken for users to complete specific tasks?
Which design is more pleasant to use?
Which design is more suited to exploration of cafes versus direct search based on the cafe and information layout presented in the application?
Blog Update #4b Experiment method
Participants
We plan on recruiting 10 participants from our friends and acquaintances. Participants will be screened for experience regarding the activity of studying in a café. Since the requirement is shared by many students and recent graduates, we expect that 10 participants with this experience will be relatively easy to find. With a participant pool size of 10, it can also reflect different nuances of the experience. The reason that we request this experience is that our experiment requires a certain suspension of disbelief — participants will be asked to imagine that they are finding a café to study in. It will be more realistic if the participant has the prior immersive experience of studying at a café.
Conditions
There are two main designs that we are comparing in the experiment. The first design (“Mosaic”), has two images per row on the primary navigation screen. To find more information, the user taps on an image and is led to a new page. The second design (“Slidedeck”), which includes a drop down menu and horizontal scrolling features, presents cafés in a typical vertical scroll. Each café image is expandable into a drop-down section with more information.
We will compare which design is easier to use through measures of the number of errors and time taken in completion of each given task. We will compare which design is more pleasant to use through Likert scale questions. The given tasks will reflect how a user would generally use the application.
We will also measure which design is suited for the perusal of a predetermined number of cafés such as 5 and 15. Since the strength of the Mosaic design is in the exploratory overview, we will compare it against the Slidedeck design to evaluate which design performs better in terms of time taken to complete the task and the Likert scale to measure preference.
Tasks
Participants will be given a series of task centered on the central task of finding an appropriate café to study in given certain preferences and restrictions.
Find any café that is not very busy with people speaking quietly
Find any small café in Downtown San Francisco that has a great atmosphere for studying and reading
Find any café that you believe is suitable for studying without very loud music playing and with more than only couches available
Find any café with chai mocha lattés
Find out if a certain café is busy
Find out if a certain café is small
Find out if a certain café has a cozy atmosphere
Find any café that has an outlet
Find any café that has a bathroom
Find any café that does not have a bathroom OR an outlet
Find any café that has long hours
Design
The experiment will follow two-factor design (two-way) with navigation design (Mosaic, Slidedeck; within subjects) and amount of data (8 cafés, 16 cafés; within subjects).
Procedure
The experiment will be conducted by one to two experimenters and one participant per observation
The participant will be given the experiment overview and then asked to sign the consent form.
The participant will use a mobile phone with the prototype installed. The screen will be recorded (audio and visual). The number of taps will be recorded through the prototype application, if possible.
An additional camera and tripod will be set up behind the participant to capture participant thoughts in the think-aloud and interview.
The experimenter will follow the interview script (Appendix 4c) to standardize the procedure between all participants.
The participant will be given a task list on paper. Participants are able to ask for clarification at this point, but are encouraged to accomplish tasks on their own once they have begun.
This presentation order will be fully counterbalanced by having half of the participants use Mosaic → Slidedeck and the other half use Slidedeck → Mosaic. The counterbalancing will control order effects.
The participant will follow the given list of tasks. The participant is asked to think-aloud and the experimenter will be recording down critical incidents for further probing in the interview portion. Video analysis will occur post-observation.
After completion of tasks on one design, the participant will complete the same tasks on the other design.
After the observation, the participant will be asked to complete a follow-up questionnaire (Appendix 4c) with structured questions to gauge preference and usability.
The experimenter will engage the participant in an unstructured interview (Appendix 4c). This time will be used to ask participants for information on critical incidents.
Apparatus
A phone will be provided with both designs of the application installed. This is to ensure the conditions remain consistent between participants in the experiment.
Additionally, to ensure the conditions are kept consistent the location of the experiment will be in an environment that will be constructed to emulate the actual conditions which a participant would be in when using the application. This environment will consist of following props:
Café mugs
Speaker playing background noise
Unfinished homework
If required by the interviewer, the script will be printed out. The tasks will also be printed out to ensure participants are able to reference them
Laptop will be made available to complete the post experiment interview questions.
Independent and dependent variables
Dependent variables:
Time to complete tasks on finding cafés on each design
Analyse screen recordings for time taken for set of tasks
Analyse screen recordings for total number of taps
Record the number of completed tasks
Preference of each design
Measured on a Likert scale on the post-observation questionnaire
Preference for small vs. large amount of cafés shown on each design
Measured on a Likert scale on the post-observation questionnaire
Independent variables:
Design (Mosaic, slidedeck)
Number of cafés presented (8, 16)
Hypotheses
H1: The Slidedeck design will require less time to complete tasks than the Mosaic design
H2: The Slidedeck design will be preferred to the Mosaic design for a smaller number of cafés (8)
H3: The Mosaic design will be preferred to the Slidedeck design for a larger number of cafés (16)
Planned statistical analysis
There are three factors we are measuring in this experiment: Time to complete tasks provided, numerical form of participants preference of each design, and numerical preference on number of cafés shown on each design.
ANOVA: Due to our experiment design, ANOVA will be our statistical analysis tool. ANOVA allows us to compare means between two or more factor levels within a single factor and determine whether there is a difference between the two sample groups. Using ANOVA to compare the relationships between two or more factors provides more informed results, considering the interactions between factors in the factors in the experiment design.
Expected limitations of the planned experiment
The small pool of 10 participants is limited by our resources, but also limits the amount of data we can capture and evaluate. This may lead to less significant results.
The information in the application will be mock information sourced from cafés around San Francisco, to avoid any pre-existing familiarity of our Vancouver-recruited participants with the cafés
Blog Update #4c – Supplemental experiment materials:
Our field study validated our task examples by finding that students do have a process they use to find cafés to study, though they can have different criteria for what makes any such place good, depending on individual preferences, length of the intended study session, and whether they are intending to study in a group or not. The study identified the common criteria students use to judge cafés, some of which are static (e.g. are there bathrooms), and some of which fluctuate over the course of a day (e.g. busyness). Lack of advance knowledge about these criteria was a major pain point for students in finding a spot to study in.
A clear path forward is to build a mobile app that allows students to inform each other about the properties of cafés they could study at, building a base of shared knowledge that they could then use in decision-making. Particular care will have to be directed toward designing a method of crowdsourcing input which can capture both the static and fluctuating properties of cafés, while still feeling engaging and natural for users.
Revised Task Examples
Before our field study, our task examples were focused around students as well as entrepreneurs, workers and groups who wanted to have meetings for their club in cafés. However, during our field study we decided to focus on current students as well as newly graduated students who have studied at cafés before in order to narrow our scope of users as well as have participants who will be able to help us understand how people make use of cafés for studying.
Our first task example is consistent with our field study as we were targeting students who want to utilize a café for studying and wanted to find out whether busyness and loud music are possible factors in preventing users from considering a café as a suitable study spot. We decided to make a minor revision and change noise to be the issue rather than limited seating to avoid overlap with our second task example.
However, for task example II, the user is an entrepreneur so one revision is to make the user a newly graduated student because we want a narrower scope of users and we only want users who are students or were recently students in our study. As a newly graduated student, their purpose will not necessarily be for studying so the ambient noise helping them focus on their work and as well can be removed. Based on one of our field studies, we can add that the user wants to go to a café on a cold day to relax in a cozy place as well as work on spare time personal projects rather than for the purpose of studying. Their love for chai mocha lattes are important points and are still relevant after our field study since it shows us the factors that attract the user to that café. However, the part where Taylor uses the café for interviews must be revised as our focus is only on utilizing cafés for studying.
Lastly, for task example III, the user is a software developer and would again need to be revised so that the user is a student. However, he meets with a group and they still technically use the café for studying and finding a place to sit as well as loud music are the potential issues which are still relevant after our field study. We decided to use a freshman student for this one because they are not very familiar with college campuses and part of our study is making users decide on cafés that they have never been to. This is so we can better understand their decision process since they will not just pick the café they have been to and are most comfortable with. Having a large groups and knowing whether cafés are okay with that is consistent with our field study as well as bathroom key stinginess and cool food/decor as some users in our field study mentioned how they use cafés for bathroom breaks, buying food, and just being in a nice atmosphere.
Task Example I
Jenna is currently studying at UBC and enjoys exploring the city. She heard from fellow students that there is a small café in Gastown Vancouver that has a great atmosphere for studying and reading. Jenna has a big project due on Monday so she decides to visit the café for a weekend study session. She arrives at the location, only to find that the shop is very busy with people speaking very loudly not allowing her to study well. Frustrated and unsure of what other available cafés are around this area, she wanders around until she finds another café. Upon her arrival, Jenna finds out the café is not suitable for studying with very loud music playing and with only couches available indoors. She wishes that she could know about the study availability before she took the time to go to two cafés.
Task Example II
Taylor recently graduated from the University of Bolumbia. They enjoy going to cafés on cold days to get chai mocha lattes and just relax as well as work on personal projects. They walk around town and come across a café. After looking at their menu outside the café they are happy to realize that they have chai mocha lattes but their happiness quickly turns into disappointment when they notice that there are a lot of people in there and the café is way too small. They get very frustrated that they walked in the cold weather to arrive there and will not be able to relax in a cozy atmosphere. Taylor had wished that they knew how busy this place was so that they won’t have to waste time walking in the cold.
Task Example III
Jeirm (pronounced “Germ”) is a freshman college student who is not very familiar with the college campus yet. He enjoys studying at cafés rather than libraries due to its lively atmosphere. He tends to stay at cafés for a very long time especially during midterm and final exam season. While wandering around the campus, he comes across a café and starts his study session. After about an hour, his laptop battery gets very low and he also has the urge to use the bathroom. However, he realizes that the café does not have a bathroom and does not have any outlets for him to charge his laptop. Now he has to pack up all his stuff and relocate, in a campus he is not familiar with, causing a relatively long disturbance and inconvenience in his study session.
Prioritised List of Requirements
a) Absolutely must include
All of our target personas would like to be able to search for a café that is most convenient to study in.
User is able to locate cafés on a map relative to user’s current location.
User is able to view pictures of the café
User is able to view a selected café’s available hours
User is able to view the busyness of the café
b) Should include
All of our users would like to be able to select cafés with certain characteristics:
User is able to filter cafés based on availability of washrooms, wifi, and power sockets
User is able to view café menus and price
User is able to post and leave reviews
c) Could include
Some users would like to know what cafés are most appropriate for certain activities
Users is able to view cafés best suited for reading
Users is able to view cafés best suited for studying
Users is able to view cafés best suited for group study
d) Could exclude
No users explicitly said they need these features, but we thought it would be helpful for some.
Should be able to search for cafés with positive reviews and high ratings
Should be able to view estimated wait time if café is full
This design alternative organises all the cafés in order of proximity which ensures that all the cafés are presented in order of proximity, which is a user requirement. This alternative allows for a quick exploration of cafés. The user can quickly access all options and if there is a specific café that a user is interested in, they can expand the café heading for more details. This would be useful for Taylor for instance that would need to visit several cafes, he can view an overall status of all the cafes near him. Some of the tradeoffs are that it requires additional clicks to view more detailed information, this may be frustrating, However, some users may prefer this as they would like to asses cafes close by from a high level.
The Yelp inspired alternative creates an information snapshot of a café. Users can view several café snapshots at the same time. Additionally, there are images in the snapshot for users to quickly assess a specific café’s ambiance which is a main requirement from our interviews. If there are additional details that the user would like to explore, they are able to by clicking into the café’s profile page, this may be useful for Jenna as she can browse several options at once and get a general sense of the cafes ambiance, and view some of the amenities each cafe has. She would be able to browse through a few at a time and she can also view some of the reviews people have. Some of the tradeoffs in this design are that users cannot view many cafés at once and would have to scroll to find more.
The Yelp inspired alternative creates an information snapshot of a café. Users can view several café snapshots at the same time. Additionally, there are images in the snapshot for users to quickly assess a specific café’s ambiance which is a main requirement from our interviews. If there are additional details that the user would like to explore, they are able to by clicking into the café’s profile page. Some of the tradeoffs in this design is that users cannot view many cafés at once and would have to scroll to find more.
The application design is inspired by modern matching applications. The user can view one café at a time and the cafés appear in order of convenience (location and preference). This has the advantage of providing the user with cafés that have a rich set of information in the snapshot and allow for users to asses ambience of the cafés in more detail. This would be useful for Jeirm for example because he would be able to browse through one cafe at a time and see if it has the amenities he wants. If not he can just swipe left to see his next best option until “it’s a match”. The tradeoff to seeing rich images of café to asses ambience is that users can only view one café at a time, and this may cause some frustration to some of the users like Taylor who just want a quick overview of cafes near him.
The most significant change to the project direction is that we have decided to focus on cafes and coffee shops. We believe that the users looking for library space and cafe space are very different and have diverse needs. Therefore, our team has decided that the purpose of this project is to target those that utilize coffee spaces. These users would represent our core user base. We will also focus on a way to incentivize users to enter data for the application.
Task Example I
Jenna is currently studying at UBC and enjoys exploring the city. She heard from fellow students that there is a small cafe in Gastown Vancouver that has a great atmosphere for studying and reading. Jenna has a big project due on Monday so she decides to visit the cafe for a weekend study session. She arrives at the location, only to find that the shop is full with no open tables for her. Frustrated and unsure of what other available cafes are around this area, she wanders around until she finds another cafe. Upon her arrival, Jenna finds out the cafe is not suitable for studying with very loud music playing and with only couches available indoors. She wishes that she could know about the study availability before she took the time to go to two cafes.
Task Example II
Taylor is an aspiring entrepreneur and treats Startrucks as their primary office. They find that the ambient noise helps them focus on their work and well, they simply love chai mocha lattes. With a new perspective from opening their own business, they decided they want to support local businesses. However, Taylor believes that many local cafes often have limited seats, no wifi, and really expensive chai mocha lattes. With their limited budget and specific needs, they have no clue how to start. Even losing a day to search a new cafe would be a disadvantage to their startup. Taylor wants to quickly find a local cafe to work in without sacrificing precious work time. In the future, Taylor will want to perform interviews to recruit new employees for their startup. These will be scheduled in advance, and Taylor wants to know that the cafe they’re going to perform an interview at will have a free couple of chairs for them and their candidates.
Task Example III
Jeirm (pronounced “Germ”) is a 30-year-old software developer who is trying to stay in the game by taking an online “massive open online course” about artificial intelligence. He meets every week with other people from his city that are taking the same course; the usual attendance is about six people. Every week they try go to a coffee shop that is close to most of the people that are attending that week. He needs to go in advance to grab some seats, so that everyone will have a place to sit down. He’d like to be able to see, maybe before he leaves, if the cafe he planned to go to is really full, or if there’s live music or something going on, because then he’ll try to find another cafe and notify his group members of the change. He’d also like to see if a cafe regularly kicks out large groups of guests, or is stingy with the bathroom keys, or maybe has some really cool decor or food that would make it especially worth taking his group to.
CS444 Cafe Oh Yay 