END-USER SOFTWARE ENGINEERING: MIS 552 Group 4 Blog Report Assignment 4 November 9, 2011
End-user Software Engineering
This video features Dr. Margaret Burnett, of Oregon State University, speaking about problems with end-user software and attempts of the EUSES Consortium to improve the process and provide better tools and processes to end-user programmers.
To frame the problem, Dr. Burnett stated that there is a large number of end-user-created software in the world, mostly spreadsheets, but also including dynamically created web pages, simulations, email filtering, and much more.
Two problems have been identified regarding end-user software:
- Errors exist in up to 90% of production spreadsheets
- End users are overly confident about creating and modifying their programs
End-user software engineering deals with what happens after the software is created – how to support the rest of the software life cycle. The goal is to reduce errors in end-user programs. This is hindered by the fact that end-user programmers generally have little motivation, interest or training in software engineering.
EUSES Consortium – End Users Shaping Effective Software – is a collaboration among the following universities and IBM:
- Cambridge (UK)
- Carnegie Mellon
- Drexel
- Nebraska
- Oregon State
- Penn State
EUSES Research identified three areas of end-user software; where these areas overlap or intersect is where end-user software exists:
- Software Engineering and Languages group
- Education group
- HCI and Psychology group
Research areas within EUSES:
- Sources & meta-sources of faults by end users
- Static, dynamic analysis: code, data
- Incremental algorithms immediate communication
The big question is: If we build it, will they come (use it)?
Dr. Burnett discusses the research prototype developed in Forms/3 spreadsheet program. Key aspects of the spreadsheet paradigm include:
– Cells that are not locked into a grid
– Use of meaningful names
– Ability to view formulas and data at same time
One device built toward end-user software engineering is a form of systematic testing.
A test, for spreadsheet usage, is a decision whether some output is right given the relative input. Test adequacy helps determine whether enough tests have been performed, and criterion has been developed to reason when enough testing has been done.
The WYSIWYT Strategy (what you see is what you test) incrementally updates tested-ness (as per formal criterion), allowing the user to incrementally inform the system of decisions, and immediately reflect this information (new testedness) in border colors as follows:
- Red means untested
- Checkboxes are used to record when value is correct
- Blue means tested
- Purple (or any color in between blue and red) means partially tested
- Other question marks in check boxes,
- Change the inputs – if value is correct, then enough testing is done and everything turns blue.
Many Empirical studies of WYSIWYT show that participants are afforded multiple benefits. End users benefit from higher test coverage, show a functional understanding of the testing process and appropriate confidence and judgment regarding whether the spreadsheet has been adequately tested. This type of testing results in more effective and faster debugging and less end user overconfidence about the correctness of the spreadsheet.
Assertions are a very important part of this process. It provides end-user programmers with ways of saying things about what they expect of a program. Assertions provide supplemental information about a program’s properties. They also add checks and balances that continually guard correctness – which cannot be done with testing alone. The assertion process need not be all or none – even one or two assertions provide some benefit. Assertions may also be added incrementally to refine the specifications in an on-going fashion.
Two types of assertions are documented:
- User Assertion – which are provided by the user
- System Assertion – which are obtained by the system because enough insertions in the cell enabled the system to make the determination
When system and user assertions do not match, an Assertion conflict or Value Violation may be highlighted. This will alert the end-user programmer to a potential error. In this case, either the formula is wrong or the assertion is wrong. The user makes the final judgment as to which assertion is correct.
How to get end users interested: Surprise-Explain-Reward
End-users are not always interested in every tool that software offers them, or they are just novice to additional features that might benefit them. Most users are simply trying to get something done by using the software tools most familiar to them. Users tend to utilize the features they are most familiar and comfortable with.
The Attention Investment models how people make decisions about what kinds of features in software to make use of, or how to spend their time to become more productive. The following are strong considerations:
- Cost of new feature – learning and interacting with new feature
- Weigh against benefit of new feature – may not be clear without incurring the cost to learn about the new feature
- Risks – wasted cost on time learning new feature; possibility of getting the environment into a state from which they cannot easily recover
In order to get end-users interested, you must arouse their curiosity. End Users/programmers believe their programs work well enough so they don’t need to research other features. To implement change, they must be shown the presence of an information gap, which would make them curious.
Surprise – Explain – Reward Strategy
SURPRISE: show them an information gap to arouse their curiosity. Surprise delivers the end-users to the explain stage – gets them curious about enough to seek explanations.
EXPLAIN: users seek explanation to close the information gap. This is usually self-directed learning. This offers the opportunity to suggest actions we want them to take to improve their software. Explain delivers the end-users to the reward phase – make clear what rewards are and tells them what we want them to do. Empirical studies have shown that self-directed learning is very effective.
REWARD: we must make benefits of taking those actions clear early in the process.
WYSIWYT testing is accessible and easy at first, but eventually it gets harder to conjure up suitable values to cover all the test situations. This is where HELP-ME-TEST (HMT) comes in. It will help conjure up test values for the end-user. This provides an opportunity for surprises by using HMT to suggest assertions. By hovering over an assertion, the end-user can get a tooltip which explains the assertion and how to fix the potentially bad values. Tips are provided via text only explanation or video demonstration.
The explanations are delivered through tooltips because the users seek explanations from the surprise object, or source of surprise. Users typically won’t want to go far for these explanations. The suggest action portion of the explanation gets users to take the desired action and they learn something through the process. This enables the user to be more engaged in the process.
Rewards take the form of:
- Red circle around values indicate either bugs or incorrect assertions (short-term and long-term benefit)
- Improved HMT behavior on input – picks better inputs
- HMT challenges assertions on non-input cells, aggressively seeking bugs
- Computer-generated assertions might look wrong
- Red circles around conflicting assertions
- Are first surprises, then rewards
Dr. Burnett discusses a behavior study conducted to test this process. There were sixteen participants who were given no assertions in advance. The purpose was to test whether the Surprise-Explain-Reward Strategy would entice and enable users to use assertions.
The results of study indicated:
- Surprises got users’ attention eventually
- 94% used assertions at least once
- Once they discovered assertions, then kept using them
- 87% used them on both tasks
- Once discovered, assertions were used more quickly
- Entered Assertions using HMT at first for training, then entered assertions on their own without need for HMT.
- Rewards seemed to be sufficient; participants felt they contributed to the accuracy
In the process of testing, it was determined that there were gaps in the help provided. It led to asking end-user debuggers what they wanted to know that might be missing. Gaps were identified as follows:
1) Oracle / Specification – 40% of gaps – whether output is correct; concentrated on the task, not environment and its features; difficult to tie help to these gaps
2) Strategy – 30% of gaps – what should we do now – strategy hypotheses – also not about features; no way to tie help to these gaps
3) Features – only 16% of gaps – but would work well with tooltips; unfortunately it would provide only a fraction of what users want to get out of help provided
4) Self-judgment – 9% of gaps – metacognitive instances are important in learning; self-efficacy (self-confidence) – improving accuracy of self-judgments may increase debugging effectiveness
5) Big (!) Gaps – 5% of gaps – users cannot voice the specific question –time they occur in the process may help identify the problem
Implications and Opportunities:
- Information gaps do not primarily focus explanations on features
- Users’ greatest need: Oracle/specification
- Strategy outnumbered features 3 to 1; needs local and global support
- Accuracy of users’ self-judgments may impact effectiveness
Emerging Research regarding Explanation devices have identified three types of explanation devices:
1) Tool tips – continue to provide features/rewards plus links to strategy videos
2) Video explanations of strategy and self-judgments – 1 to 2 minutes, may include:
- Head shots of people discussing the problem; enable users to identify with people shown in video; self- efficacy – if people you identify with are having same problem and demonstrate that they can solve it, so can you
- Spreadsheet so you can see what is happening
3) Side panel – links to text-based versions or video versions –
- Text version has same content as the videos; doesn’t have examples, but does include hyperlinks
The preliminary results of the Empirical study indicate that the three methods of supplying explanations have closed half of the gaps identified earlier with regard to strategy, oracle/specification, and self-judgment. It also shows that videos and text are used nearly the same amount of time, depending on the tasks being researched:
- When first learning about the feature, most users chose text version
- When returning for an explanation, most users chose the video, possibly because it has more content pictorial example.
- For a quick refresher, users chose text version for direct access to the desired information
Males and females responded differently to the videos. Females looked at the head shots and were engaged with speakers, while males looked at the spreadsheet and didn’t seem to want anyone to know they needed the videos. This supported the use of both forms of explanations. Both serve the purpose of filling a certain need, by a certain person, at a given point in time.
Lessons Learned
The main goal of end user software engineering is to find ways to address such issues as looking past the creation phase of software development and resolving the issues so as to facilitate ease of flow through the remaining stages of the software development lifecycle. This requires an organized attempt to help end-user programmers reduce errors in their programs. End-user programmers need help with design and composition of elements as well as support for evolution, maintenance, and development – a complete process for software development. This requires having sufficient knowledge of the current needs of end-user programmers.
WYSIWYT (what you see is what you test) is a methodology to support a systematic testing for the end user programmers by incrementally informing the system of decisions and immediately reflecting changes.
Surprise-Explain-Reward, developed to train the end user in software engineering techniques such as testing supported by WYSIWYT, appears to be working. In tests, users found this strategy to be helpful, and the software they produced contained fewer errors.
In conclusion, the research done by EUSES identified two areas of need in an attempt to fix the problems with end-users software engineering. First, it is apparent by the error rate of 90% that users need help with learning how to use the tools / features of programs. Once they are exposed to information gaps they are more likely to learn how to use the tools available to them. This makes it possible to engage them in an effort to modify their end user behavior, which, in turn, achieves the goal of fewer errors in end user programs.
201108mis552group4blog 