[DG: User Experience] Usability testing results for Sakai 3 assignments

[Hancock, Kristol J]

I can set up a Breeze room for us to use next week if you would like.
I just need to know the day and time we're planning to meet.

Thanks,

Kristol

On Feb 3, 2010, at 5:55 PM, Clay Fenlason wrote:

> I plan to make it to one or some of the meetings. Keli, yes.
>
> I know that the Sakaibrary folk have also been doing a large number of
> user interviews lately - is Gaurav on this list? I wonder if there
> might be some way to synthesize the insights of your respective
> projects.
>
> ~Clay
>
> On Wed, Feb 3, 2010 at 5:47 PM,  <kamann at stanford.edu> wrote:
>> Hi
>> Wow, this thread got busy! It is nice to see this project taking
>> shape, even if everyone's still working out the kinks.
>>
>> I just posted an invitation to talk through the personas created
>> based on about 30 interviews next Monday or Tuesday. I hope I don't
>> conflict with your meeting on Tuesday--Kristol and Daphne let me
>> know if you can't make either (Clay you are also invited). I booked
>> a conference line, and, based on this conversation about the breeze
>> room being booked, will probably use Dimdim.com to do screen sharing.
>> http://confluence.sakaiproject.org/display/UX/February+8+and+9%2C
>> +2010
>>
>> It sounds like you all may be trying to pose realistic scenarios to
>> users and will be asking testers to try imagine working through it
>> with the prototype. I am wondering if the personas will be a useful
>> way to do that--I'm doing this check in about the personas
>> themselves, but the next step is to write out some context
>> scenarios for what they have to do. This might have some overlap
>> with coming up with tasks for user testers to do. I'd welcome help
>> doing this, since Jackie is out, especially if you think it's
>> related to what you're trying to acheive.
>>
>> Clay, I have been thinking about next steps for the work we are
>> doing. Part of it might be to write these scenarios around planning
>> and scheduling activities, as a way to conceptually test out
>> designs. Part of it would be to start thinking about creating and
>> evaluating activities (quizzes, tests, polls) and how that hooks
>> into this. I will continue to think about it through Monday. If you
>> can make it to either meeting or have time to talk some other time,
>> that would be great.
>>
>> Keli
>>
>> ----- Original Message -----
>> From: "Daphne Ogle" <daphne at media.berkeley.edu>
>> To: "Clay Fenlason" <clay.fenlason at et.gatech.edu>
>> Cc: "Sakai UX" <sakai-ux at collab.sakaiproject.org>
>> Sent: Wednesday, February 3, 2010 11:45:36 AM GMT -08:00 US/Canada
>> Pacific
>> Subject: Re: [DG: User Experience] Usability testing results for
>> Sakai 3 assignments
>>
>> One thing that might also be helpful for this conversation is to
>> discuss any areas in the assignments click-throughs that you hoped to
>> gain user insight into but didn't.  Concretely talking about what we
>> missed will help us make sure not to in the next round.
>>
>> -Daphne
>>
>> On Feb 3, 2010, at 11:04 AM, Clay Fenlason wrote:
>>
>>> Let me propose, then, that we start trying to devise a new round of
>>> test protocol for some 'grouping' sketches that I expect will become
>>> click-throughs before very long. I know John surfaced some
>>> earlier, I
>>> know they are a high priority for many, and I think we can begin to
>>> talk through some screens and ideas even before the click-throughs
>>> are
>>> produced.
>>>
>>> I'll try to collect everything I can get my hands on about group
>>> wireframes and the goals and requirements shaping them, get them
>>> arranged on sakaiuxframework.com, and maybe we can have an Adobe
>>> Connect session as early as Monday next week? We could start to talk
>>> through them together and then try to devise a protocol
>>> collaboratively, even while UI developers are trying to polish off
>>> the
>>> click-throughs. How would that sound? Could Berkeley host this on
>>> their server?
>>>
>>> ~Clay
>>>
>>> On Wed, Feb 3, 2010 at 1:44 PM, Daphne Ogle
>>> <daphne at media.berkeley.edu> wrote:
>>>> I agree Clay.   Real user scenarios and goals are the best way to
>>>> get
>>>> feedback about overall structure and functionality.   I wrote the
>>>> user
>>>> testing protocols for Fluid so am completely on board.  And
>>>> sometimes,
>>>> particularly with an overview like the assignments like, we also
>>>> need to ask
>>>> what they see as there is particular information we want to pop.
>>>> The fuller scenarios and activities is what I was trying to get at
>>>> with the
>>>> idea of having more context.  To use fuller scenarios, we'll need
>>>> to have a
>>>> complete workflow available to the user.  For this round we only
>>>> had small
>>>> chunks of functionality available which made it challenging to
>>>> write fuller
>>>> scenarios and tasks.  We'll need to be committed to making these
>>>> prototypes
>>>> full enough for testing.  And also to get rid of distractions, like
>>>> inconsistent designs for the same page.  Since I got much push back
>>>> on even
>>>> changing text in the prototypes, I just tried to do the best with
>>>> what was
>>>> there.
>>>> Oh, and we did capture some user information.  You can check out
>>>> the
>>>> demographic questionnaire link on any of the protocol pages to see
>>>> what we
>>>> asked.  I plan to get that posted also.  The results in general are
>>>> still in
>>>> process but my goal was to have a list of the most pressing,
>>>> frequent
>>>> patterns we saw which is where my focus has been so far.  Then what
>>>> I'd like
>>>> to do is end up with something like what we captured on Fluid [1].
>>>>
>>>> This is great that we are starting to refine our goals from user
>>>> testing and
>>>> the process we can use to gain valuable user feedback.  It's a bit
>>>> painful
>>>> via email and with so much happening so fast that we don't always
>>>> have the
>>>> communication open to make things run smoothly but I think we're
>>>> learning.
>>>>
>>>> Now, what to do with current results?  There is some important
>>>> information
>>>> about users expectations, where they had confusion and challenges
>>>> in the
>>>> results.  What are the next steps?  I see Kristol offered to help
>>>> out and
>>>> I'm also available so if there's anything we can do help iterate on
>>>> the
>>>> designs, we're here.
>>>>
>>>> -Daphne
>>>> [1]
>>>> http://wiki.fluidproject.org/display/fluid/Layout+Reorderer+User+Testing+-+Round+3+Results
>>>>
>>>> On Feb 3, 2010, at 8:17 AM, Clay Fenlason wrote:
>>>>
>>>> I didn't mean to suggest that there wasn't useful information in
>>>> the
>>>> test results, and so I'm sorry that seems to be the strongest
>>>> message
>>>> I'm getting through. It's just that after reviewing the results
>>>> it's
>>>> making me go back and think again about the emphasis of the testing
>>>> protocol - my overall impression is that the kind of question we're
>>>> tending to ask follows a pattern more of "look at this screen and
>>>> tell
>>>> me what you see" rather than "suppose you wanted to get <such and
>>>> such> done - can you try to do that starting from this screen?"  I
>>>> think I'm talking about producing scenarios and tasks like this [1]
>>>>
>>>> ~Clay
>>>>
>>>> PS It might also be helpful to record information about who the
>>>> users
>>>> were, even if it's just as crude as student, teacher, etc.
>>>>
>>>> [1] http://wiki.fluidproject.org/x/sgQs
>>>>
>>>> On Tue, Feb 2, 2010 at 8:21 PM, Daphne Ogle <daphne at media.berkeley.edu
>>>>>
>>>> wrote:
>>>>
>>>> Great.  We are on the same page as far as interaction patterns.
>>>>
>>>> A few ideas for helping users focus on interactions:
>>>>
>>>> For this round (for the instructor assignment/task list):
>>>>
>>>> 1)  I've greyed out results that are focused on visual design and
>>>> some of
>>>>
>>>> the terminology.  In some cases the terms or visual indicators
>>>> (i.e. red for
>>>>
>>>> due soon & next) were key to the user getting to their goal so I
>>>> left a
>>>>
>>>> couple in.
>>>>
>>>> 2)  I've also added the original "This test should discover"
>>>> requests from
>>>>
>>>> Sam to the top of the results and answered those questions.
>>>>
>>>> 3)  I've done some analysis of the results here but if there are
>>>> specific
>>>>
>>>> interactions we would have liked to learn about and didn't, let me
>>>> know.
>>>>
>>>> Perhaps I just haven't presented them in these results.
>>>>
>>>> 4)  I think I could also make a list of interaction patterns we see
>>>> in the
>>>>
>>>> current designs and comment on what I saw in testing.  Not sure how
>>>> well
>>>>
>>>> this would work but it might be worth it if it would make these
>>>> results more
>>>>
>>>> useful.  It could be that interactions that didn't come up in
>>>> results tested
>>>>
>>>> fine and thus aren't in the testing results on this page.
>>>>
>>>> *I do think there was a lot of good information gained through the
>>>> testing
>>>>
>>>> so perhaps it is the way the results are being presented that is
>>>> the
>>>>
>>>> problem.  They are pretty raw.  There is probably more synthesis
>>>> and
>>>>
>>>> prioritizing of issues that could be done if that would be helpful.
>>>>
>>>> For future tests:
>>>>
>>>> 1)  We can test using wireframes to alleviate visual distractions.
>>>> Or if
>>>>
>>>> testers know the visual design is not important at this point, they
>>>> can
>>>>
>>>> steer away.
>>>>
>>>> 2)  We should do a review of the prototypes before the test to
>>>> remove
>>>>
>>>> distractions such as having 2 of the same pages that look different
>>>> or
>>>>
>>>> terminology inconsistency.  Those inconsistencies grab the users
>>>> attention
>>>>
>>>> and make them wonder what they should take from the differences.
>>>>
>>>> 3)  We need to come up with a list of user goals the prototype is
>>>> meant to
>>>>
>>>> address.  We had some of this with the "what should this test
>>>> discover" but
>>>>
>>>> a list of goals will help (can come from requirements?).  Then we
>>>> can make
>>>>
>>>> sure we have tasks that will get at those goals and activities.
>>>> This was
>>>>
>>>> challenging for this round as I wasn't always sure what the intent
>>>> was, what
>>>>
>>>> wasn't there intentionally or what was just left off.  Having a
>>>> list of what
>>>>
>>>> is intended will help us focus in the right places.
>>>>
>>>> 4)  If there are specific interactions we are concerned about, we
>>>> should
>>>>
>>>> identify those too to make sure we include tasks that will uncover
>>>> any
>>>>
>>>> issues.
>>>>
>>>> Thoughts?
>>>>
>>>> - Daphne
>>>>
>>>>
>>>> On Feb 2, 2010, at 3:50 PM, Clay Fenlason wrote:
>>>>
>>>> On Tue, Feb 2, 2010 at 5:26 PM, Daphne Ogle <daphne at media.berkeley.edu
>>>>>
>>>>
>>>> wrote:
>>>>
>>>> Can you say more about what you mean by interaction patterns?
>>>> What's an
>>>>
>>>> example of a pattern we'd like feedback on?  Just want to make sure
>>>> we
>>>>
>>>> are
>>>>
>>>> talking about the same things.
>>>>
>>>> I suppose I mean more than one thing. At a basic level I'm
>>>> referring
>>>>
>>>> to the places where the user can take some action - that the
>>>> available
>>>>
>>>> choices comport well with the user's goals. Also, in some vague
>>>> way,
>>>>
>>>> that the *kind* of action (clicking on a button, making multiple
>>>>
>>>> selections, dragging things around, tagging) fits well with their
>>>>
>>>> purpose and mental models. At a different order of complexity I'm
>>>>
>>>> wondering about flows: whether the available chains of such actions
>>>>
>>>> correspond well to the flows of activity that seem intuitive to the
>>>>
>>>> user. Do they feel they have to skip around or backtrack too
>>>> often to
>>>>
>>>> accomplish common tasks? Is there something they're trying to get
>>>> at
>>>>
>>>> that seems hard to unearth?
>>>>
>>>> I agree that there is some of this in the results, I'm just
>>>> wondering
>>>>
>>>> how we might focus on it even more.
>>>>
>>>> ~Clay
>>>>
>>>> _______________________________________________
>>>>
>>>> sakai-ux mailing list
>>>>
>>>> sakai-ux at collab.sakaiproject.org
>>>>
>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-ux
>>>>
>>>> TO UNSUBSCRIBE: send email to
>>>>
>>>> sakai-ux-unsubscribe at collab.sakaiproject.org with a subject of
>>>> "unsubscribe"
>>>>
>>>> Daphne Ogle
>>>>
>>>> Senior Interaction Designer
>>>>
>>>> University of California, Berkeley
>>>>
>>>> Educational Technology Services
>>>>
>>>> daphne at media.berkeley.edu
>>>>
>>>> cell (925)348-4372
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> sakai-ux mailing list
>>>> sakai-ux at collab.sakaiproject.org
>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-ux
>>>>
>>>> TO UNSUBSCRIBE: send email to sakai-ux-unsubscribe at collab.sakaiproject.org
>>>> with a subject of "unsubscribe"
>>>>
>>>> Daphne Ogle
>>>> Senior Interaction Designer
>>>> University of California, Berkeley
>>>> Educational Technology Services
>>>> daphne at media.berkeley.edu
>>>> cell (925)348-4372
>>>>
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> sakai-ux mailing list
>>> sakai-ux at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-ux
>>>
>>> TO UNSUBSCRIBE: send email to sakai-ux-unsubscribe at collab.sakaiproject.org
>>> with a subject of "unsubscribe"
>>
>> Daphne Ogle
>> Senior Interaction Designer
>> University of California, Berkeley
>> Educational Technology Services
>> daphne at media.berkeley.edu
>> cell (925)348-4372
>>
>>
>>
>>
>> _______________________________________________
>> sakai-ux mailing list
>> sakai-ux at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-ux
>>
>> TO UNSUBSCRIBE: send email to sakai-ux-unsubscribe at collab.sakaiproject.org
>>  with a subject of "unsubscribe"
>>
> _______________________________________________
> sakai-ux mailing list
> sakai-ux at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-ux
>
> TO UNSUBSCRIBE: send email to sakai-ux-unsubscribe at collab.sakaiproject.org
>  with a subject of "unsubscribe"