Project Part 4
Group Project 4: Prototyping & Evaluation
Introduction – prepared by Haani
In the project part 4, we will conduct a testing and evaluation with our system prototype. Our prototype is called “UTM Bus Transit System” and the final product can be accessed through an application called “UTMSmart” that is officially used by University of Technology Malaysia (UTM). Our system is the enhanced system of the current UTM bus system with some extra useful features to ensure better user experience. Additional features that we put in our system include live location of UTM buses, bus planner system, graphical route presentation for each UTM bus, etc.
The objective of this prototype testing and evaluation is to validate the function of the bus's live location, graphical bus route and also the bus planner. Therefore, this testing phase is focused on testing these functions with the user and evaluating them based on observation of the user reaction to our system.
This testing phase is important because through this testing we can find any potential issue with our system using the prototype. Through a comprehensive testing phase, we can make sure that our system meets the requirements and proved to perform better than the current system.
We are focused on three user groups to test our system. The first group is on-campus residents students and the representative of this group was handled by Brendan Chia Yan Fei. Our second user group is international students and its representative was handled by Neo Li Xin and the last group is off-campus residents students and was handled by Nawwarah Auni binti Nazrudin.
This report consists of the screenshots of our prototype, the briefing notes, the testing report of each user arranged by each task, the observations and the findings from the testing and evaluation.
Screenshots of Our Prototype
 |  |  |
Figure 1: UTMSmart Homepage | Figure 2: Shuttle Bus Page | Figure 3: Live Location Page |
 |  |  |
Figure 4: Bus Route Details Page | Figure 5: Plan Bus Page | Figure 6: Manual Bus Schedule Page |
 |  |
|
Figure 7: Automated Bus Schedule Page | Figure 8: Bus Route Details Page |
Briefing notes – prepared by Goh Jiale
The application “UTMSmart” is an official free mobile application that is used to access information and ease the process of events such as transactions and record attendance in University of Technology Malaysia (UTM) for students and staff.
This application initially consists of one feature called “Shuttle Bus”, which serves to provide a schedule of buses in UTMSmart. However, the features are difficult to access and normally students won’t know UTMSmart got these features. By improving UTMSmart with the advanced features provided by our group, it now can have some new features such as direct and clean user interface, check bus nearby and available bus, view live location of bus and plan bus schedule with few clicks.
Below is the 3 tasks that you as our user, that will be tested on through our high fidelity prototype in Figma:
Task 1: Locating the shuttle bus page from home page of UTMSmart
Task 2: Check the live location of bus B1 and route of Bus B1 through Available page
Task 3: Plan a bus schedule to reach N28a (Faculty of Computing) from KTHO
Testing with users
Below are the name of the users who participated in our user test:
User 1 (Local Student) : Tay Ching Xian
User 2 (International Student) : Huang Bo Sheng
User 3 (UTM Staff) : Norhadiyati binti Mohd Mokhtar
Task 1: Locating the shuttle bus page from homepage of UTMSmart
User 1: https://youtu.be/XygUvUanG5g
User 2: https://youtu.be/c83SpbAG0tE
User 3: https://youtu.be/vV41X3sdYYA
Task 2: Check the live location of bus B1 and route of Bus B1 through Available page
 |  |  |
User 1: https://youtu.be/sH4BDXSFtP4
User 2: https://youtu.be/u4aUixJZJ_U
User 3: https://youtu.be/ZsThnKI2ny8
Task 3: Plan a bus schedule to reach N28a (Faculty of Computing) from KTHO
User 1: https://youtu.be/ajX_hcA_fOc
User 2: https://youtu.be/ceLl3CsYeL4
User 3: https://youtu.be/71DqlZM5trw
Observations - prepared by Brendan, Auni, Neo
Task 1: Locating the shuttle bus page from home page of UTMSmart
User 1
User 1 receives Task 1 and understands it
From the UTMSmart home page, user 1 checks the UI and decides to click on the bus UTM “Now Available !!” button.
User 1 has entered the shuttle bus page.
User 1 has completed task 1.
User 2
User 2 receives task 1 and understands it.
From the UTMSmart home page, user 2 checks the UI and decides to click on the bus UTM “Now Available !!” button.
User 2 has entered the shuttle bus page.
User 2 has completed task 1.
User 3
User 3 receives task 1 and understands it.
From the UTMSmart home page, user 3 checks the UI and decides to click on the bus UTM “Now Available !!” button.
User 3 has entered the shuttle bus page.
User 3 has completed task 1.
Task 2: Check the live location of bus B1 and route of Bus B1 through Available page
User 1
User 1 receives and understands task 2.
In the Available page, User 1 clicks the “Bus B1” button in the Nearby Bus section.
User 1 able to locate the live location of bus B1.
User 1 returns to the Available page through the “Back” button at the top right corner.
Next, to find the bus route of bus B1, User 1 press the “KTR-KTHO '' link next to the bus B1 provided in the list table at Available Now section.
User 1 has entered the next page and is able to navigate the route of bus B1 through the map.
User 1 has completed task 2.
User 2
User 2 receives and understands task 2.
In the Available page, user 2 clicks the “Bus B1” button in the Nearby Bus section.
User 2 able to locate the live location of bus B1.
User 2 returns to the Available page through the “Back” button at the top right corner.
Next, to find the bus route of bus B1, user 2 press the “KTR-KTHO '' link next to the bus B1 provided in the list table at Available Now section.
User 2 has entered the next page and is able to navigate the route of bus B1 through the map.
User 2 has completed task 2.
User 3
User 3 receives and understands task 2.
User 3 clicks on the “Now Available !!” on the UTMSmart homepage.
Then, user 3 clicks on the on location button.
User 3 clicks on the “Bus B1” on the Nearby Bus section in the available page.
User 3 has entered the live location page for bus B1.
Next, user 3 returns to the Available page by clicking on the bus icon with the available label.
User 3 looks at the table provided on the Available Now section to look for bus B1.
Then, user 3 clicks on “KTR-KTHO” and bus route details of bus B1 appears.
User 3 has completed task 2.
Task 3: Plan a bus schedule to reach N28a (Faculty of Computing) from KTHO
User 1
User 1 receives task 3 and understands it.
User 1 clicks the “Plan Bus” icon at the bottom right of the app.
User 1 has two options whether to plan it manually or automatically.
User 1 decided to use the automatic schedule feature to finish the task given
User 1 fills in the “from” space with “Kolej Tun Hussein Onn (KTHO)” and “to” space with “N28a Faculty of Computing”.
User 1 clicks the “Plan now” button and the suggested routes appear on the screen.
User 1 has completed task 3.
User 2
User 2 receives task 3 and understands it.
User 2 clicks the “Plan Bus” icon at the bottom right of the app.
The user 2 saw 2 options whether to plan it manually or automatically.
User 2 decided to use the first function which is the manual bus schedule.
User 2 looks through all the bus schedules but can’t seem to navigate the bus route by himself.
User 2 decided to use the automatic schedule instead to finish the task given.
Then, user 2 fills in the “from” space with “Kolej Tun Hussein Onn (KTHO)” and “to” space with “N28a Faculty of Computing”.
User 2 clicks the “Plan now” button and the suggested routes appear on the screen.
User 2 has completed task 3.
User 3
User 3 receives task 3 and understands it.
User 3 clicks the bus UTM on the UTMSmart homepage to go to the shuttle bus page.
After that, user 3 clicks on the “Plan Bus” icon.
The user 3 saw 2 options whether to plan it manually or automatically.
User 3 decided to use the automatic schedule to finish the task given.
Then, user 3 fills in the “from” space with “Kolej Tun Hussein Onn (KTHO)” and “to” space with “N28a Faculty of Computing”.
User 3 clicks the “Plan now” button and the suggested routes appear on the screen.
User 3 has completed task 3.
Results
User 1
The time taken to complete Task 1 is 12 seconds
The time taken to complete Task 2 is 31 seconds
The time taken to complete Task 3 is 40 seconds
User 2
The time taken to complete Task 1 is 12 seconds
The time taken to complete Task 2 is 36 seconds
The time taken to complete Task 3 is 1 minute and 20 seconds
User 3
The time taken to complete Task 1 is 11 seconds
The time taken to complete Task 2 is 30 seconds
The time taken to complete Task 3 is 40 seconds
3 out of 3 users successfully completed three tasks. From the time taken to complete the tasks, we can conclude that the UTM Bus Transit System is easy to use. This is also proved by the feedback given by User 3, Norhadiyati binti Mohd Mokhtar in which the icons and labels greatly help her in navigating to different pages. Besides, user 1, Tay Ching Xian thinks that the UTM Bus Transit System will definitely benefit him as a UTM student as the live location feature will save a lot of waiting time.
Findings - Prepared by Auni, Neo
Usability Problems
User 1
After running the three test tasks on user 1, it was observed that user 1 successfully completed all three tasks and navigated through the application without any usability issues. In task 1, user 1 efficiently located the shuttle bus page from the home page of UTMSmart. For task 2, user 1 was able to check the live location and route of bus B1 through the Available page without any difficulty. Lastly, in task 3, user 1 successfully planned a bus schedule to reach N28a (Faculty of Computing) from KTHO and found the process intuitive and straightforward.
User 2
After running the three test tasks on user 2, it was found that tasks 1, 2 and 3 were completed successfully. For task 1, user 2 had no trouble locating the shuttle bus page from the UTMSmart home page. Task 2 also proceeded without any issues, as user 2 effortlessly checked the live location and route of bus B1 through the Available page. Upon completion of task 3, user 2 has a hard time reading the manual bus schedule and navigating the bus route independently. However, user 2 found the automatic bus schedule straightforward and easy to use, which greatly aided in planning the trip from KTHO to N28a (Faculty of Computing).
User 3
After conducting the three test tasks on user 3, it was noted that user 3 successfully finished all of them. In task 1, user 3 easily located the shuttle bus page from the home page of UTMSmart. Task 2 was also completed with ease, although user 3 found the “On Location” button on the Available page confusing as it did not initiate any action when clicked. Nevertheless, this did not affect user 3's ability to complete task 2 and successfully view the live location and route of bus B1. Finally, in task 3, user 3 effectively planned a bus schedule to reach N28a (Faculty of Computing) from KTHO using the automatic bus schedule.
Conclusion
Possible solution for our usability problems:
No. | Before | Issue | After |
1. |  | Some users cannot understand how to use the manual schedule. Hence, we will be adding instruction pages so they can refer to them. |     |
2. |  | The function of "On Location" confuses some users because there is no action seen when it is pressed. Therefore, we improved the feature by adding a location permission request after the user clicks it so our live location tracking can know where the user is. |   |
Through this project, we have successfully conducted testing and evaluation of our system prototype, the "UTM Bus Transit System" which is accessible via the "UTMSmart" application. This enhanced version of the current UTM bus system includes features like live bus locations, a bus planner system, and graphical route presentations to improve user experience. The testing was comprehensive, involving tasks designed to assess how easily users could navigate the new system and utilize its features.
The majority of users were able to navigate the application and utilize the new features effectively, demonstrating that our enhancements significantly improve the user experience compared to the current system. However there are also some slight issues such as users can’t easily navigate the bus route using the manual bus schedule page and the “On Location” in the Available page not showing any obvious changes to the system. By incorporating user feedback and resolving these minor issues, the "UTM Bus Transit System" can be a robust and user-friendly solution, enhancing the daily transit experience for UTM students and staff.
Comments
Post a Comment