Measuring User Success in Digital Library
Environments
1. Introduction
Harnad (1991), the University of Princeton cognitive psychologist, envisioned
the electronic telecommunication network as the "fourth revolution in the means of the
production of knowledge," following language (first revolution), writing (second
revolution), and printing (third revolution). He explained that the first revolution made
it possible to make and exchange ideas, the second revolution made it possible to
preserve them without the speaker (speaker-independence), and the third revolution
made it possible to preserve and reproduce easily (hand-writer-independence). Now,
we are at the threshold of the fourth revolution (digitalized information and
communication), which Harnad characterizes as "electronic skywriting." In Harnad's
view, the most powerful tool of the fourth revolution is the electronic telecommunication
network called the "Internet." The capability of the Internet to reach a global scale of
users with near real-time communications is revolutionizing the means of production of
knowledge (medium-independence). According to Massachusetts Institute of
Technology economist Klinger (1994), when we were in the third revolution, the
production cost of one page of information to reach one million people per day was
about $60,000. But now with publishing on the Internet, the cost to reach this
audience is only $2.50. The new cost is less than the five thousands of one percent of
the original cost.
The explosion of digitalized information has led to a new concept in information
resource development and use--the digital library.
1.1. Digital Library
The concept of the digital library means many things to many people. It is
applied to quite different phenomena, ranging from an actual library as an institution
(Levy & Marshall 1995) to an individual-application computerized information system
(Lynch & Garcia-Molina 1995). For the purpose of this research effort, we adopt the
digital library concept of an individual-application computerized information system,
particularly Internet-based information in the broadest sense, or Web-based information
in a more narrow sense. (The Internet is the platform; the Web is the primary
application running on the Internet.) Lynch and Garcia-Molina (1995), for example,
have defined a digital library as "a system providing a community of users with
coherent access to a large, organized repository of information and knowledge." The
Web meets this definition in terms of coherent access to an organized body of digitalized
information. The vast information resources housed on the Web currently constitute
the primary example of a "digital library," accessible to many users from many sites. The
digital library is becoming established as a component of library and information services,
and it will grow to be an increasingly more important component of future library
services as more users require Web access.
Recently, a considerable body of literature in the field of library and information
science and computer science has begun to give digital libraries serious consideration
(see Drabenstott 1993, Communications of ACM, April 1995). The main thrust of this
literature is on how to make a powerful and effective digital library. However, the
literature has focused most attention on the attributes of resources and services.
There is little attention paid to the attributes of the users and uses. Thus, there have
been few opportunities to assess user or usage variables that address the success of the
digital library, and generally accepted success measures for evaluating a digital library do
not exist. Some influential researchers call for more research in this area (Bishop et
al.1995, Lesk 1995, Peters 1995, Lopata & McClure 1995).
1.1. User Satisfaction in an Objective Output-Oriented End-User Computing
Environment
1.1.1 Definitions of User Information Satisfaction
Research on user information satisfaction (UIS) has been plagued not only by a
lack of consensus on the definition of the construct, but also by a lack of agreement on
its dimensionality. A review of studies on user information satisfaction reveals the
widely divergent definitions--from the whole information system function in an
organization (Bailey 1983, Baroudi 1987), to an information service function within
work-groups (Kettinger 1994), to an individual application information system or to
individual reports and queries (Doll 1988, 1995).
Bailey and Pearson (1983) define UIS as the sum of a user's attitudes toward a
variety of factors of an information system. They identified 39 factors as comprising the
domain of satisfaction. Melone (1990) defines UIS as a user's attitude to respond
favorably or unfavorably to a computer system, application, system staff member, or a
process related to the use of that system or application. Kettinger (1994) defines UIS
as a user's attitude toward the production and service activities performed by an
information service department. Doll (1988), defines UIS as an end-user's satisfaction
with a specific application.
Chang (1993) proposed that the various definitions of information behavior
should be categorized based on an attitudinal-behavioral continuum. One stream of
definition focuses on the attitudinal dimension of UIS. The theoretical basis of these
definitions is based mainly on Azen's reasoned action theory that links beliefs and
attitudes to usage behavior. His theory suggests that usage behavior can be predicted by
an individual's intention to use the system. This intention is determined by some weighted
combination of the individual's attitudes toward IS-related objects. When researchers
studied user satisfaction in this context, user satisfaction was defined as a weighted
attitude toward IS related objects. Once the concept was defined this way, the
operationalization procedures included two steps: (1) identify all the IS-related objects
affecting a user's attitude toward IS and (2) develop a measurement instrument to
gauge the user's reaction to those objects. Thus, user satisfaction measures in this
category included all the situational factors affecting user's attitudes toward IS, such as
the technical competence of the IS staff, the attitude of the IS staff, top management
support, etc.
At the other end of the continuum of UIS research are studies that emphasize
the behavioral dimension. This view of UIS attempts to measure a user's satisfaction
with the quality of information--the major output of IS. User satisfaction with
information system quality is typically evaluated by measuring information attributes such
as the reliability, format, and timeliness of the output. The underlying reasoning for this
approach to UIS is that there may be a direct relationship between information quality
and IS success.
Chang (1993) insists that both the attitudinal and behavioral dimensions are
useful means for determining information success. In this research, we will adopt
Pearson's definition, "the sum of a user's attitudes toward a variety of factors of the
information system," as the formative definition of UIS. This definition of UIS implies
that satisfied information system users would be expected to show the following
attitudes: (1) a strong acceptance of the information system, and (2) a willingness to
exert considerable effort on behalf of the information system.
In spite of this diversity in the definition of UIS, there has been considerable
consensus among researchers on which aspects or dimensions of UIS are most
significant for predicting UIS (See Figure 2). Building on the work of Bailey and Pearson
(1983), Ive and Olson (1983), and Baroudi (1987), most researchers have emphasized
the significance of the following three types of factors:
* the information product quality (IPQ), which refers to the technical quality of reports
and screens generated by the information systems, and includes items such as
response time, format of reports, and accuracy;
* the attitude towards staff and services (ISS), including items such as time taken for
development of new systems and relationships with staff; and
* the level of user's knowledge and involvement (UKI), which refers to issues such as
training and involvement in system design. (Baroudi 1987).
When these factors have been treated as casual determinants of overall UIS
based on a four-item measure, Ives and Olson (1983) reported the correlation between
overall UIS and a total score based on IPQ, ISS, and UKI factors as only .54. This low-
to-moderate correlation has been attributed to several methodological and measurement
problems.
A considerable number of empirical studies have been carried out over the past
ten years to improve the original UIS dimensions--the most important being Joshi 1990,
Kettinger 1994, and Doll 1988. Most of these studies have utilized a cross-sectional
design and have employed the statistical techniques of factor analysis or partial
correlation. The studies vary considerably in terms of breadth or diversity in the
types of users sampled. Numerous studies have focused on users within the same
organization (Kettinger 1994, Doll 1988). Others have included a broader range of
users but limited their samples to the IS sectors (Joshi 1990).
The cumulative results of the research to date remain in dispute. Three
researchers draw differing conclusions. Galletta and Lederer (1989) compared the
original 39-scale UIS instrument developed by Bailey and Pearson (1983) with Ivy and
Olson's a 13-scale UIS instrument by using 92 managers and executives in experimental
and control conditions. They conclude that the 13-scale UIS instrument behaves more
reliably than the detailed original scales for all groups, perhaps because of problems with
scale units and origins, and with item heterogeneity. This conclusion suggests that
researchers need more reliable measures of UIS and practitioners need to exercise
caution when collecting and interpreting UIS scores. By contrast, questioning the validity
of the conclusion of the Galletta and Lederer study (1981) leads Hawk and Raju
(1991) to conclude that "the method used by the Galletta and Lederer (1989) for
assessing test-retest reliability is not grounded in classical reliability theory. Analysis of
their data suggests that the UIS instrument has adequate test-retest reliability".
A different and more favorable conclusion is reached by Doll et al. (1995) who
conducted a confirmatory factor analysis on the original UIS construct by using a
sample of 244 respondents. Their results support the 13-item instrument as a measure
of overall UIS, and they present four components as factors for explaining the UIS
construct. The four components are (1) IS staff, (2) IS service, (3) Information Product,
and (4) Knowledge and Involvement.
Overview of UIS Research
1.1.2. Evolutionary Approach toward the Study of User Information
Satisfaction
Our own conclusion is that, despite the many types of problems cited (Iivari
1994, Malone 1990), the results of empirical studies conducted up to the present provide
a model for UIS. However, even though the model is reasonably consistent, it is not
particularly strong. This lack of strength leaves much room for improvement in our
understanding of UIS.
Several avenues have been proposed along which to seek improved models.
Some are evolutionary in orientation, seeking to broaden and deepen the UIS framework
(Joshi 1990, Kettinger 1994, Doll 1995); others are more revolutionary in their aims and
look to overturn or displace the UIS construct (Kim 1989, Malone 1990). While there
is no clear-cut division among these camps, we turn now to focus on the concerns of a
set of evolutionary researchers who attempt to modify and incrementally improve the
UIS model. In general, there are three major evolutionary or revision models for UIS: (1)
Joshi's Model (2) Kettinger's model, and (3) Doll's model. Our approach in this
research will be based on the work of the evolutionary researchers.
Joshi (1990) focused on the organizational level of the UIS concept and
argued that the UIS might be influenced not just by the original UIS factors but
also by social and political factors in the information system organization. He introduced
the equity construct borrowed from the equity theory. (As defined above, the
equity construct is "the perceived inequality in the allocation of information system
resources." Equity theory investigates the quest for fairness and equity in social
exchange.) Joshi hypothesized that, in addition to the originally identified three factors
of information product quality (IPQ), user's knowledge and involvement (UKI), and IS
staff and service (ISS), equity is also likely to be a relevant factor in determining
overall UIS. They conclude that there is a strong correlation (.67) between equity
and overall UIS. Further, the addition of the equity dimension to the previously
identified factors significantly increased the explanatory power from 48 % to 56%. These
results support the importance of the equity dimension in determining user satisfaction
at the organizational level. They recommended that IS researchers and practitioners
take into account the likely influence of equity perceptions on user attitudes and
behaviors.
Kettinger (1994) emphasized the work group level of the UIS concept
and argued that the UIS might be more influenced by the quality of service factors
provided by service group. He introduced the SERVQUAL construct developed by
Parasuraman et al. (1986) and applied it to a variety of consumer satisfaction
researches. In his research, he adapted the SERVQUAL measure to provide more
specific information about user satisfaction with the information service function. He
found that while the three original dimensions of the UIS measure remain strong
predictors of overall UIS, two aspects of IS service quality, "reliability" and "empathy,"
are also significant predictors. The result suggests that the original dimension of UIS may
not be comprehensive enough to capture the more detailed dimensions of service
quality in SERVQUAL, and that the reliability and empathy dimensions of service quality
may be needed to supplement the original UIS measure in determining user satisfaction
with information service functions.
Doll et. al. (1988, 1994) focused on the individual level of UIS in response
to the growth in end-user computing. They argued for the development of a UIS
measure which reflects ease of use and information product items in order to measure the
satisfaction of end users who directly interact with the computer for a specific application.
Using a survey of 618 end users, they conducted a factor analysis and modified the
instrument. The results suggest a 12-item instrument that measures five dimensions of
end-user satisfaction: content, accuracy, format, ease of use, and timeliness. In the
replication study using a sample of 409 users from 18 different organizations, they
validated the UIS instrument's five dimensions and determined the strength of scales and
each item's reliability. The reliability coefficient for the five dimensions were found to be
as follows: content (.89), accuracy (.72), format (.78), ease of use (.85), and timeliness
(.82). Doll et. al. suggest that researchers and practitioners can use these scales with
confidence. The scales are referred to as the End-User Computing Satisfaction
(EUCS) model.
Henderickson (1994) conducted test-retest reliability studies of the EUCS.
Whereas Doll (1994) reported relatively short-term reliability (two weeks),
Henderickson investigated a long-term stability test over a two-year interval. He also
assessed EUCS in varied application settings, using personal computer administrative
end user and mainframe administrative end users in a large public organization. The
results of the repeated test-retest using differing application platforms add further
support to the reliability of the 12-scale EUCS instrument.
We will use the EUCS scales as a formal UIS variable in this research.
1.2. User Satisfaction in a Subjective Interaction-Oriented Computer-
Mediated Environment
1.2.1. The Nature of the Digital Library and Implications for
UIS
The prior section dealt with user satisfaction in an end user computing
environment, not a digital library. Extending the concept of UIS to the digital library
domain requires some new thinking. If a digital library is comparable to the end user
computing environment for which the original measurement construct was developed,
we can adopt the EUCS measure without any modification. However, if the digital
library is not comparable to that environment, we will need to modify the original
construct to reflect the unique features of the digital library environment. In this
section, we will start by presenting the concept of the digital library and discuss some
features that make a digital library unique.
For the purpose of this research, we define a digital library as an Internet
presence (specifically Web presence). By this definition, we mean that the Internet, and
specifically the Web, provides a coherent access to an organized body of information for
a wide range of users in various locations. The users may access this information with
little or no intermediation on the part of trained information professionals.
In terms of the study of library and information science, the digital library may
be one component of the physical, more-traditional library setting. By comparing and
contrasting the digital library to other library components, we are able to identify several
features that characterize digital library resources.
Black (1995) classified information in a library into three broad conceptual
categories based on format and control. The categories are:
1) Physical items the library controls, like book and periodicals
2) Digital information the library controls, like CD-ROM and online databases;
and
3) Digital information the library does not control, like internet resources.
The last category includes the resources that we view as the "digital library." The
existence of this digital library component is having an impact on the library as a whole.
According to his estimate, the internet resources that can be characterized as digital
information that the library does not control will increase from a current 5% level to a
30% level in five years.
Significant differences exist between the traditional library environment and the
digital library environment (Levy & Marshall 1994, 1995). We identified three unique
aspects of the digital library which may have implications for the UIS model. They
are as follow:
* The digital library system has a hypermedia computer-mediated environment
* The digital library system has a multi-media information object.
* The digital library system has a personalized user.
These aspects are discussed below.
1.2.1.1. The Digital Library has a Hypermedia Computer-Mediated
Environment
Hoffman and Novak (1995) defines a hypermedia computer-mediated
environment (CME) as "a dynamic distributed network, potentially global in scope,
together with associated hardware and software for assessing the network, which allows
users to 1) provide and interactively access hypermedia content (i.e. 'machine
interaction') and 2) communicate through the medium (i.e. 'person interaction')." The
basic information access metaphor in the CME is navigation. They define navigation
as the process of self-directed movement through a hypermedia CME. There are two
important notions in hypermedia CME--telepresence and vividness.
Steuer (1992) calls CME a "telepresence view of mediated
communication, where presence is the natural perception of an environment and
telepresence is the mediated perception of an environment." When interacting with
CME, the user perceives two environments: 1) the physical environment in which he or
she is present, and 2) the environment defined by the hypermedia CME. The strength of
the experience of telepresence is a function of the extent to which one feels present in the
hypermedia CME, rather than in one's immediate physical environment.
Vividness, the representational richness of the hypermedia CME, may be
increased by the structural characteristics of breadth and depth. Breadth refers to the
number of sensory dimensions presented and is closely related to media
concurrency and media richness. Depth defines the resolution or the quality of the
presentation and is highly correlated with media bandwidth. In the hypermedia CME,
both breadth and depth are, in general, high.
1.2.1.2. The Digital Library has a Multi-Media Information Object.
Reigngold (1994) describes a digital information object as a multimedia object
consisting of three characteristics: (1) immersion, the user's feeling of being completely
surrounded by the artificial world rather than seeing it on a small screen; (2) navigation,
meaning the user's choosing a point of view and moving around in the multimedia object
instead of just seeing the point of view that a cameraman or director has chosen, and
(3) manipulation, meaning the use of a special tool to reach into that multimedia object
and change it.
1.2.1.3. The Digital Library has a Personalized User.
Who is a digital library user? Although some researchers suggest an
information analyst as a model for the digital library user, it is often not clear exactly
what this model means. Does the digital library user himself interact with the system? Or
does somebody else operate it and pass on information to him? Is the system user
accessing information which is then passed on to someone else? Who is the operator,
and who is the information consumer?
Martin et. al. (1989) described information system (IS) users by using a
typology developed to identify accurately the interaction between the user and the
system. The typology utilizes two dimensions showing the main categories of
interactions: (1) association, the extent to which the user is personally involved with the
IS, and (2) purpose, the relationship of the information to the IS user's own work role
and needs. By this typology, they identified nine categories of IS users from direct
information consumer (who is an interactive user and employs information in his own
decision role) to remote transmitter (who receives information indirectly and passes
information on untouched to another user).
The first category--the direct information consumer--is perhaps closest to the
concept of the digital library user, given the nature of the digital library technology.
The digital library is interactive, providing its user with direct access for navigation and
data retrieval. A fully interactive digital library user has discretionary control over, and
choice of, exactly how information is to be presented from moment to moment. He or
she sees and makes judgments, and selects one path rather than another.
1.2.1.4. Implications for UIS
With earlier computerized information resource technologies, such as on-line or
CD-ROM databases, the user and information source (i.e., the database) are generally
separated by a query mechanism that serves as an intermediary. This mechanism, part
of the user interface, allows input of a query in some structured format (for example, a
query language). The system validates the format of the structured information request,
processes the query, and presents the result of the query back to the user through the
intermediary. If the query is found to be incomplete or improperly structured, the user
may have the opportunity to refine and resubmit the request. (Some user interfaces
may provide prompts or assists in developing or reshaping a query.) However, each
information request (or re-request) is a distinct transaction. The individual user is not
directly exposed to the system's process of interpreting the query in order to isolate the
requested information, nor is the user able to manipulate or navigate search paths. The
user may be said to be in "search" mode, seeking a response to an information request.
Once the output provided by the system meets the user's requirement, the user is satisfied.
In a digital library environment characterized as a hypermedia computer-
mediated environment (CME), however, the user and the information source (the Web)
are not separated but rather are "mediated" by the computer. By mediation, we mean
that the person acts "through" the computer, which provides direct exposure to--and
perhaps control of--navigational techniques utilized in order to locate desired information.
The dynamic between the user and information system is changed as the user becomes
his/her own intermediary. As the user pursues a desired information result, he/she is
able to refine his/her own search process. The user may be in either "search" or "browse"
mode, or may in fact blend "search" and "browse," trying various paths, placing
bookmarks, retracing paths already traveled, and so on. That interaction "within the
medium" is perhaps the most radical departure from traditional environments.
In a digital library environment, users themselves interact directly with a
digital library system to obtain the information they want and, in the process, they
experience every interaction of navigational activity for themselves. Thus, the notion of
"experiential" satisfaction characterizing the personal interaction through and within
the digital library system is important. It may be the dimension that was missed in the
original UIS measure.
Thus, user information satisfaction in a digital library domain differs from the
concept of user information satisfaction in earlier computerized information system
environments in two interrelated dimensions--the perceived directness and variety of
experience of UIS, and the content of UIS activities. The dynamic events created by
multimedia information objects make it easier for users to experience a variety of
activities, changing the content of UIS activities. These differences in UIS arrangements
have implications for the definition of the UIS in a digital library domain.
1.2.2. Measure of Subjective Interaction-Oriented Experience
Given the fact that a digital library has some unique features that are not
reflected in the existing UIS measures, we look now to the success measure being
used in the Human-Computer Interaction (HCI) field. HCI has adopted the Flow
construct to measure the subjective human-computer interaction experience. Flow is
defined as the congruence between skill and challenge. A number of researchers have
suggested that Flow is a useful construct for describing a user's interaction with
computers (Hoffman and Novak 1995; Ghani et al 1991; Trevino and Webster 1993;
Webster et al. 1993). Thus, we believe that the concept of Flow may help us move
toward a more comprehensive measurement of user satisfaction with information system
in the digital library domain.
1.2.2.1. Definition of Flow
Csikszentmihalyi (1990) developed a concept he called "flow." He defined
it as "the process of optimal experience achieved when a sufficiently motivated user
perceives a balance between his or her skills and the challenges of the interaction,
together with focused attention." Based on this definition, Webster et. al. (1993)
developed a construct called "Flow" that measures the subjective human-computer
interaction experience. (The literature expresses the name of this model as "flow"; this
paper, however, presents the term as "Flow" in order to accentuate the name of this
research model and to avoid confusion with other uses of the word "flow.") Applying
Flow to the hypermedia CME, Hoffman and Novak (1995) note that users "1) perceive
a sense of control over their interactions in the environment, 2) focus their attention on
the interaction, and 3) find it cognitively enjoying. When in the Flow state, irrelevant
thoughts and perceptions are screened out and the user's attention is focused entirely on
the interaction."
Although UIS is measured in terms of quality of the output delivered by
the system, measuring a user's subjective human-computer interaction experience in
hypermedia CME presents different problems. Unlike information output, where user
satisfaction can be readily attributed to activities and entities, an individual interaction
experience has no direct guidelines for measurement in UIS.
Hoffman & Novak (1995) indicate that Flow can be divided into two sub-
categories; goal-directed flow and experiential flow. Goal-directed flow is "instrumental
and utilitarian in nature, extrinsically motivated, characterized by situational involvement,
and results in directed search and learning." In contrast, experiential Flow is
"ritualistic and hedonic, intrinsically motivated, characterized by enduring involvement,
and results in nondirected search and learning." They argue that goal-oriented Flow is
a necessary, but not always a sufficient, condition to achieve user satisfaction in
hypermedia CME and that overall satisfaction involves attaining experiential Flow as
well. In the context of the digital library, goal-oriented Flow is what the user is actually
left with after the output is delivered. Typically, goal-oriented Flow is evaluated based
on the assessment of whether that actual output meets both user expectation and
requirement. Because human-computer interaction experiences play a significant role in
the hypermedia CME, the goal-oriented Flow dimension alone may not equate with a
total assessment of user satisfaction, but rather may be influenced by the experiential
flow of the human-computer interaction. Experiential Flow corresponds with the
subjective human-computer interaction experience.
1.2.2.2. Subjective Satisfaction and Flow Theory.
Most research on user satisfaction in HCI has been on the basis of
Csikszentmihalyi's "flow" theory (1990), which posits that the congruence between skill
and challenge provides a frame of reference for the user's satisfaction. Accordingly,
satisfaction judgments are based on the combined effect of skill and challenge. Hoffman
and Novak (1995) describe the process by which satisfaction judgments are reached
within the skill-challenge framework.
1) Users form the level of skills prior to use. These skills may be based on
personal need, or past experience.
2) Users receive a level of challenge in their activities.
3) Perceived satisfaction may be either confirmed or not confirmed based on
skill-
challenge congruence.
Congruence provides a baseline or anchor for the level of satisfaction. If
incongruence is perceived to have occurred, then user satisfaction increases or decreases
from the baseline level. Building on this conceptualization of satisfaction, the formal
Flow construct has been defined by numerous HCI researchers as a comparison of a
user's skill with perceived challenge. If skills exceed challenge, one can expect that a user
would consider the service to be boring. If challenge exceeds skill, the user would feel
anxiety and may view the system negatively. Generally, HCI literature has treated user
satisfaction and service quality as related, but distinct, constructs. While there
continues to be debate on the causal direction between these two constructs, recent
research (Trevino 1993) seems to indicate that Flow is an antecedent of user
satisfaction.
1.2.2.3. The Development of Flow
In 1989, Webster developed the Flow instrument to measure the congruence
between user skill and the perceived challenge. Based on interviews and focus group
meetings with users from a variety of organizations, determinants of Flow were identified.
She found that, regardless of the type of activity, users used basically similar criteria in
evaluating satisfaction and that these criteria span virtually all aspects of the activity. The
scale was designed to be applied to a broad spectrum of activities, and as
Csikszentmihalyi suggests "it provides a basic skeleton through its skill/challenge format
encompassing statements for each of the four flow dimensions. The skeleton, when
necessary, can be adapted or supplemented to fit characteristics of specific research
needs of a particular organization."
On the basis of analyzing data from four independent samples, she presents a
12-item scale consisting of four Flow dimensions including:
1) Control (the congruence between the skill and challenge)
2) Focus attention (the ability to focus)
3) Curiosity (the desire to attain competence)
4) Intrinsic interest (the desire to be involved in the activity for pleasure)
Flow scales have been used in a variety of published HCI studies, and there is
a growing literature in the HCI field critiquing its use. While not all of these studies
have formally examined the scale's psychometric properties, several recent studies have
done so (Ghani et al 1991; Trevino and Webster 1993; Webster et al. 1993). In general,
the collective results from this research provide consistent support for the reliability of
the Flow instrument on the four dimensions. Support for convergent validity and
discriminate validity is mixed because several studies showed items loading on
different dimensions, and the number of factors retained has not been consistent across
the studies. Some recent Flow research argues in favor of an output only-based
measure of Flow. In an empirical study, Webster (1993) showed stronger predictive
validity for a 1992 version of a Flow instrument using only three dimensions (control,
focused attention, cognitive enjoyment) as opposed to the Flow congruence between
skill and challenge.
Arguing in favor of the congruence score-based instrument, Hoffman and
Novak (1995) state that superior predictive power of the output-only measure must be
balanced against its inferior diagnostic value to the practitioner. Additional improvement
to the 1989 Flow scales, recommended by Webster (1992) and aimed at overcoming
problems created by using negatively worded items, were made by Trevino and Webster
(1993). In sum, while versions of Flow continue to be critiqued and improved, Flow
stands as the preeminent instrument within the HCI practice of and research for the
assessment of subjective human-computer experiential quality.
Similar to the practical value of the UIS instrument, Flow is being employed as
a diagnostic tool for uncovering broad areas of user information satisfaction shortfalls
and strengths. In this way, Flow dimensions and items represent core evaluation criteria
that transcend specific domains. Flow researchers suggest that it can fruitfully
supplement UIS assessment in a specific context. Specifically, Trevino and Webster
(1993) and Hoffman and Novak (1995) note that Flow is most valuable when it is
used periodically to track service quality trends and when it is used in conjunction with
other forms of service quality of measurement.
1.2.2.4. Use of Flow as a measure of UIS
Digital libraries may now look to the HCI experience in measuring user
satisfaction. Parallels between the two fields indicate that both IS user satisfaction
literature and past operationalizations of the UIS instrument possess evaluations of
output (technical) attributes and experiential (interaction) attributes. For digital library
user information satisfaction (DLUIS), UIS might pertain to the accuracy, easy of use,
format, timeliness, and content of the information products delivered. In the UIS
instrument, the output dimensions have tended to be measured in terms of those
dimensions. The interaction dimension, on the other hand, may be evaluated not only by
what is done, but by how it is done. In the digital library context, this may be
represented by control, focused attention, intrinsic interest, and curiosity. Flow offers the
potential to contribute to DLUIS by providing this additional focus for measuring the
subjective interaction dimension of DLUIS. Since Flow has been shown to provide
specificity in measuring interaction satisfaction in the experiential flow, this study will
examine Flow's ability to improve existing measures of user satisfaction within digital
library information systems.
2. The Integrative Model and Research Questions
In this research, objective and subjective variables are incorporated to explain
UIS of Korean digital library users. In this framework, objective and subjective
variables are treated as directly or indirectly influencing UIS of digital library users. By
putting independent objective and subjective variables together, one can explain more
of the variance in the dependent variable--user satisfaction with digital libraries--than
either could have explained alone. The reason is that user satisfaction is the result
of the reciprocal effects of the properties of the information system and several
facets of psychological functioning. Thus, the complex interplay between the system and
the user in certain situational contexts may produce different degrees of user satisfaction
with digital libraries. This research addresses the challenge of measuring user
satisfaction in the digital library environment, to wit:
(I) Is subjective information satisfaction measured by Flow significantly related to
overall user information satisfaction with the digital library?
(II) What is the correlation between Flow and UIS?
(III) Does some combination of Flow and UIS dimensions better predict overall user
satisfaction with digital libraries than either of them separately?
These questions will be explored by analyzing the returns from both a
modified version of the standard Doll's (EUCS) instrument of UIS and a modified
version of the 1993 Webster instrument of Flow, administered to a sample of digital
library users in a Korean academic environment. The instrument will also include a
modified version of a four-item questionnaire developed by Joshi (1990) and used in
Kettinger (1994), which seeks to measure an overall point of view, asking for opinions
on system efficiency and effectiveness in fulfilling information needs. The responses to
the modified EUCS and Flow instruments will provide the independent variables, and the
responses to these four overall user satisfaction items will provide the dependent variable.
Research Model
3. Goal of this Research
This research seeks to address this question: How to measure user satisfaction
in a digital library environment? We argue that a combination of the end-user
computing satisfaction construct (EUCS) and the Flow construct provides a working
model for determining such user satisfaction. The goal of this research is to develop a
validated measurement instrument, based on already-validated EUCS and Flow factors,
that can be used to measure user satisfaction in the digital library environment. The
ultimate product of this research will be a validated research model and methodology,
and this research instrument.
References
Applegate, Rachel. "Models of User Satisfaction Understanding False Positives," RQ 32,
No. 4 Summer, 1993, p. 525-39.
Bailey, J.E and Pearson, S.W. "Development of a Tool for Measuring and Analyzing
Computer User Satisfaction," Management Science, 1983, 29(5), 530-545.
Baroudi, J.J and Orlikowski. "A Short-Form Measure of User Information Satisfaction:
A Psychometric Evaluation of Notes on Use," Journal of Management Information
Systems, 1988, 494), 44-59.
Billington, Library Applications of National Information Infrastructure, Committee on
Applications and Technology Report Putting the Information Infrastructure to Work
(http://nii.nist.gov/lbrstate.html)
Bishop, A.P et al. "Building a University Digital Library: The Need for a User-Centered
Approach,", (http://anshar.grainger.uiuc.edu/dlisoc/monterey.final_copy.html)
Black, Graham. "Integrating the Digital with Non-Digital: A Librarians Perspective,"
Proceedings of International Symposium on Digital Libraries'95, August 22-25, 1995
Tsukuba, Ibaraki, Japan. 138-150
Blalock, H. M. (1982) Conceptualization and Measurement in the Social Sciences.
Beverly Hills, CA: Sage.
Calvert, Philip J. and Cullen, Rowean J. (1994). "Further Dimensions of Public Library
Effectiveness II: The Second Stage of the New Zealand Study," Library and Information
Science Research, Volume 16, Issue 2, 1994. pp 87-104.
Chang, Shan-Ju and Rice, Ronald E. "Browsing: A Multidimensional Framework,"
Annual Review of Information Science and Technology (ARIST), Volume 28, 1993,
231-276.
Clarke, Sharon G. and John T. Haworth, "'Flow' Experience in the Daily Lives of Sixth-
Form Collect Students," British Journal of Psychology, 1994 (85), 511-523.
Converse, Jean M. Survey Questions: Handcrafting the Standardized Questionnaire,
Sage Publications, Inc. 1986.
Csikszentmihalyi, Mihaly. Flow: The Psychology of Optimal Experience, New York:
Harper and Row, 1990.
Cullen, Rowena, & Calvert, Philip."Dimensions of public library effectiveness: Report on
parallel New Zealand Study," Library and Information Science Research, 15, 143-164.
Delone, W. H and McLean, E.R. "Information Systems Success: The Quest for the
Dependent Variable," Information Systems Research, 1992, 3(1), 60-95.
Digital Libraries. Communications of the ACM 38(4) (April 1995):22-96
Doll , William. J and Torkzadeh, G. "The Measurement of End-User Computing
Satisfaction," MIS Quarterly, 1988, 12(2). 258-274.
Doll, William J. and Torkzadeh, G. "The Measurement of End-User Computing
Satisfaction: Theoretical and Methodological Issues," MIS Quarterly, March, 1991, 5-10.
Doll, William J. et al. "A Confirmatory Factor Analysis of the End-User Computing
Satisfaction Instrument," MIS Quarterly, December, 1994, 453-461.
Doll, William J. et al. "A Confirmatory Factor Analysis of the User Information
Satisfaction Instrument," Information Systems Research, 1995, 6(2), 177-188
Drabenstott, K. M. Analytical Review of the Library of the Future. Council on Library
Resources, Washington, DC. 1993
Ellis, Gary D., Judith E. Voelkl, and Catherine Morris (1994), "Measurement and
Analysis Issues with Explanation of Variance in Daily experience Using the Flow
Model," Journal of Leisure Research, 26(4), 337-356.
Etezadi-Amoli, Jamshid and Farhoomand, Ali F. "On End-User Computing Satisfaction,
" MIS Quarterly, March, 1991, 1-4.
Galletta, D.F. and Lederer, A.L "Some Cautions on the Measurement of User
Information Satisfaction," Decision Sciences, 1989, 20(3), 419-439.
Gatian, Amy W. "Is User Satisfaction a Valid Measure of System Effectiveness?,"
Information & Management, 1994, Volume 26, 119-131.
Ghani, Jawaid A. and Satish P. Deshpande (1994), "Task Characteristics and the
Experience of Optimal Flow in Human-Computer Interaction," The Journal of
Psychology, 128(4), 381-391.
Hanard, Steven. "Post-Gutenberg Galaxy: The Fourth Revolution in the Means of
Production of Knowledge," Public-Access Computer Systems Review, Volume 2, Issue
1 1992, pp. 39-53
Hawk, Stephen R. and Raju, Nambury S. "Test-Retest Reliability of User Information
Satisfaction: A Comment on Galletta and Lederere's Paper," Decision Sciences, 1991,
22(6), 1164-1170.
Hendrickson, Anthony R. et al. "On the Repeated Test-Retest Reliability of the End-User
Computing Satisfaction Instrument: A Comment," Decision Sciences, 1995, 25(4), 655-
667.
Hiltz, S. R and Johnson, K. "User Satisfaction with Computer-Mediated Communication
Systems," Management Science, 1990, 36(6), 739-764.
Hoffman, D.L. and Novak, T.P. "Marketing in Computer-Mediated Environments:
Conceptual Foundations," July 11 revision. (http://www2000.ogsm.vanderbilt.edu)
Iivari, Juhani and Ervasti, Irja. "User Information Satisfaction: IS Implementability and
Effectiveness," Information & Management, 1994, Vol. 27, 205-230.
Ives, B. and Olson, M.H. "The Measurement of User Information Satisfaction,"
Communications of the ACM, 1983, 26(10), 785-793.
Joshi, Kailash. "An Investigation of Equity as a Determinant of User Information
Satisfaction," Decision Sciences, 1990, 21(4), 786-807.
Kettinger, William J. "Perceived Service Quality and User Satisfaction with the
Information Services Function," Decision Sciences, 1994 25(5-6), 737-766.
Kim, K. Y. "User Information Satisfaction: Toward Conceptual Clarity," Proceedings of
the 10th International Conference on Information Systems, Copenhagen, Denmark,
1990, 183-191.
Kim, K. K. "User Satisfaction: A Synthesis of Three Different Perspectives," Journal of
Information Systems, Fall 1989, 1-12.
Klenke, Karin. "Construct Measurement in Management Information Systems: A Review
and Critique of User Satisfaction and User Involvement Instruments," INFOR, 1992,
30(4), 325-348.
Kling, Anold. "The Economic Consequences Of WWW," Electronic Proceedings of the
Second World Wide Web Conference '94.
(http://www.ncsa.uiuc.edu/SDG/IT94/Proceedings/ComEc/kling/imply.html
Levy, D. M. and Marshall, C.C. "What Color was George Washington's White Horse? A
Look at the Assumptions Underlying Digital Libraries," Proceedings of Digital
Libraries' 94, 1994: 163-169.
Levy, David M. "Going Digital: A Look at Assumptions Underlying Digital Libraries,"
Communications of the ACM, Aprirl 1995, 38(4), p. 77-84.
Lopata, C.L and McClure, C.R. Assessing the Impacts of Internet Networking on the
Academic Institution, (http://istweb.syr.edu/Project/Faculty/McClureAbstract2.html)
Lynch, Clifford and Garcia-Molina, Hector, Interoperability, Scaling, and the Digital
Libraries Research Agenda: A Report on the May 18-19, 1995 IITA Digital Libraries
Workshop, August 22, 1995 (http://www-diglib.stanford.edu/diglib/pub/reports/iita-
dlw/main.html)
Martin, C.J. "What is an End-User? Identifying Multiple Relationships Among
Information System Users," Proceedings of the 10th International Conference on
Information Systems, Copenhagen, Denmark, 1990, 265-270.
Massimini, Fausto and Massimo Carli (1988), "The Systematic Assessment of Flow in
Daily Experience,." In M. Csikszentmihalyi and I. Csikszentmihalyi (Eds.), Optimal
Experience: Psychological Studies of Flow in Consciousness (pp. 288-306), New York:
Cambridge University Press.
McClure, C.R. et al. Internet Costs and Cost Models for Public Libraries, National
Commission on Libraries and Information Science 1994
(http://istweb.syr.edu/Project/Faculty/McClure.NCLIS.Report.html)
Melone, Nancy P. "A Theoretical Assessment of the User Satisfaction Construct in
Information Systems Research," Management Science, 1990, 36(1), 76-91.
Miksa, F. and Doty, P. "Intellectual Realities and the Digital Library," Proceedings of
Digital Libraries'94, 1994: 1-5.
Parasuraman, A. and Zeithaml, V.A, "A Conceptual Model of Service Quality and Its
Implications for Future Research," Journal of Marketing, 1985, 49(4), 41-50.
Peters, P.E. "What's Needed in Future Research in Digital Library", d-Lib Magazine,
July 1995 (http://merlin.cnri.reston.va.us/home/dlib/July95/07contents.html)
Rheingold, Howard L. (1993), "Virtual Communities and the WELL," GNN Magazine,
Issue One (Oct 4).
[http://gnn.interpath.net/gnn/mag/10_93/articles/howard/howard.intro.html]
Schamber , Linda et al. A re-examination of relevance: toward a Dynamic, Situational
Definition, Information Processing & Management, Vol. 26, No. 6, pp. 755-776, 1990
Schamber, Linda. Relevance and Information Behavior. Annual Review of Information
Science and Technology (ARIST), Volume 29, 1994, 3-48.
Simon, J.H "Public Libraries and the Internet: Study Results, Policy Issues, and
Recommendations", National Commission on Libraries and Information Science, June
1995 (http://istweb.syr.edu/Project/Faculty/McClure.NCLIS/Letter.html)
Steuer, Jonathan (1992), "Defining Virtual Reality: Dimensions Determining
Telepresence," Journal of Communication, 42(4), 73-93.
Su, Louise T. (1991) An Investigation to find appropriate measures for evaluating
interactive information retrieval, Unpublished doctoral dissertation, The State University
of New Jersey at Rutgers.
Su, Louise T. (1992), Evaluation Measures For Interactive Information Retrieval,
Information Processing & Management, 1992, 28(4), 503-516.
Torkzadeh, G., and Doll, W. J. Test-retest reliability of the end-user computing
satisfaction instrument. Decision Sciences, 1991, 22(1), 26-37.
Trevino, Linda Klebe and Jane Webster (1992), "Flow in Computer-Mediated
Communication," Communication Research, 19(5), 539-573.
Webster, Jane (1989), Playfulness and Computers at Work, Unpublished doctoral
dissertation , New York University.
Webster, Jane and Joseph J. Martocchio (1992), "Microcomputer Playfulness:
Development of a Measure With Workplace Implications," MIS Quarterly, 16 (June),
201-226.
Webster, J., L.K. Trevino, L. Ryan (1993), "The Dimensionality and Correlates of Flow
in Human Computer Interactions," Computers in Human Behavior, 9(4), Winter, 411-426.