- (2012) Volume 6, Issue 1
Areti Stavropoulou1* and Martha Kelesi2
1Assistant Professor, Nursing Department Technological Educational Institution of Crete
2Assistant Professor, Nursing Department B΄ Technological Educational Institution of Athens
Aim: The paper explores the concepts and methods of evaluation in nursing education. Rationale: Evaluation of nursing education is a topic received much attention by the scientific community recently. The introduction and the background information of the paper highlight the theoretical endorsement of evaluation, the necessity of evaluation in education and the strenuous effort to find workable methodological approaches for evaluation in education. Method: Literature is reviewed from nursing, education and evaluation disciplines. Benchmarking texts on evaluation and education formed the skeleton of the paper. Systematic review search included scientific databases EBSCO, CINHAL+ and PubMed. Findings: Different evaluation concepts, methods and activities on educational evaluation are explored. Issues of methodological appropriateness in educational evaluation that communicated among scientists and received both appraisal and criticism have been traced and discussed. In particular, quantitative-qualitative debate formed the basis of significant arguments and influenced the evolution of new evaluation strategies that encompasses synthesis and triangulation. Conclusion: Methodological challenges in evaluation appear to lead equally in frustration and enlightenment and have benefits for the further advancement of evaluation science.
Keywords
Evaluation, nursing education, quantitative - qualitative methods, triangulation.
Introduction
Evaluation process is used in everyday life in order to make decisions for simple or complex matters. Every time an individual is required to make some sort of change and before choosing a particular course of action, a review of all available options is taking place. Different options are assessed; advantages and disadvantages associated with each option are under consideration. In this way the most appropriate decision can be reached, the needs of the individual are effectively met and the required change has the best chances for success. This process of assessing situations of everyday life – personal or professional – has many similarities to the process, which is formally termed evaluation in the literature. The process of evaluation, which an individual employs to reach to a decision, is fundamentally the same regardless of the area of concern, or its source or even of its importance [1]. It is then the methods used, the focus given the philosophical orientation of each individual or evaluator that provides a certain character to each evaluative activity.
Evaluation has the potential to be beneficial or harmful [2]. In other words, it might be easily misleading. It might lead to excellent or poor outcomes and decisions. It is for that reason that all elements underlying an evaluation should be carefully examined. Each evaluation has different aims and occurs in specific contexts, thus the design has to fit the circumstances, yet meet the challenges of scientific credibility. Various typologies of evaluation theory and methods exist [3] and a number of scientific discussions have been made regarding the benefits and utilization of different approaches to educational evaluation [4,5]. Although the field of research has been dominated historically by quantitative approaches [5] which were applauded for their merits, there were significant voices endorsing the qualitative approaches to evaluation, such as the influential work of Patton [6].
Specifically, Patton6 states in his seminal work on qualitative evaluation and research methods that qualitative data and information about beliefs, motivations and patterns of behaviour of the people, psychological, sociological and anthropological insights should be taken into account in each evaluation activity. This is becoming more apparent each time the evaluation activity focuses on education, a field that differs from other fields since it works towards the spirit, the intellectuality, the ethos, and the persona of people [7].
Background Information
Evaluation Concepts and Education
Evaluation in modern technologically advanced era where decisions can no longer be based on the intellectuality of particular individuals has been highly developed. Evaluation is conducted for various reasons and many scientists have provided us with their views on evaluation. A variety of definitions exist.
Evaluation is defined as an effort involving collection, analysis and interpretation of data in order to judge the achievement of a programme’s objectives [8]. Other definitions include evaluation’s role as a process of assisting decision making in a specific area of concern [1]. Rossi and Freeman [9] define evaluation as: “the systematic application of social research procedures for assessing the conceptualisation, design, implementation and utility of social intervention programmes”.
Patton [6] uses the term evaluation as “any effort to increase human effectiveness through systematic data-based inquiry” and defines evaluation research as: “the systematic examination of accomplishment and effectiveness in program and services”.
It is necessary to achieve a common understanding on what evaluation is. The word “evaluation” itself has a concrete meaning, that of judgement. In this respect, definitions are mostly concern with the process and the outcome of the evaluation rather than evaluation itself. It might be argued that not all options or attributes of evaluation are explored in detail. However, the two primary and most important concepts of evaluation, the one of the process and the other of the outcome are in the focus of definitions. After all, as Patton [6] says what else is the purpose of evaluation if not to inform action (a process- oriented activity), to enhance decision-making (an outcome-related process), to apply knowledge and to solve thus, human and societal problems?
Evaluations are undertaken for a variety of reasons. Some of them are reported by Rossi and Freeman [9] and include judgement of the worth of ongoing programmes and estimation of the usefulness of attempts to improve them; assessment of the utility of the new programmes and initiatives; increasing the effectiveness of programme management and administration and satisfying the accountability requirements of programme sponsors.
The authors further state that evaluations serve different purposes and call for different strategies at various stages in the life of programmes. For new programmes, evaluations help to determine the degree to which the programmes are effective, how successfully they are providing their intended target populations with the resources, services and benefits envisioned by their sponsors and designers.
Educational initiatives have long been targets of evaluation. Addressing “what works” questions [4], measuring and assessing growth in knowledge, stating course expectations, developing rubrics for single events and continua to measure quality of work over time, assessing students regularly, organising the assessment of data and using the feedback to make course corrections are some of the elements of evaluation in education [2]. Evaluation is becoming a necessary part of education, particularly in present times, when educational systems are subjected to continuous criticism and debate. Many educationalists view evaluation, in the same way that nurses view quality assurance as a mean of justifying their actions, gaining work appraisal and professional empowerment. It is also a way of initiating self-assessment and developing personal evaluation techniques, which are prerequisites for increasing self-confidence and personal growth.
In recent years there is a growing interest in developing evaluation approaches in education. Evidence - based education policies and evidence - based education practice contribute to this [3,4] .As Jenkins [2] and Grant-Haworth and Conrad [10] similarly stated, in the past years the reason for this increasing interest relates to the quality of education. Educators rely on evaluation in order to justify the quality of their teaching and to identify areas which need improvement. In addition policy makers attempt through evaluation to validate and sustain effective changes that educational systems have experienced. In this respect evaluation of educational programmes is viewed as a defence against the claim that existing education is not preparing adequately the individuals for the demanding challenges of contemporary society.
Evaluation of educational programmes supports and enhances the aim of education as this provided by Deming11: “to increase the positive and decrease the negatives so all students keep their yearning for learning”
Evaluation of educational programmes provides data on positive and negative aspects and supports learning experiences that have positive effects on students’ growth and development [10]. A series of collaborative evaluation efforts were reported in the field of education in order to support high-quality programmes and assist decision making at all levels [3,5,1].Within these efforts a number of different methodological approaches were applied, creating thus another broad area of debate in evaluation.
Findings
The Quantitative–Qualitative Argument and Evaluation Inquiry
In the early years of educational evaluation, the dominant approach was quantitative and experimental borrowing its procedures from the scientific method of physical and biological sciences. This was considered to be a desirable research approach and evaluation [12]. Regardless of that, qualitative approaches to evaluation have gained considerable interest as they appeared to provide the opportunity to evaluators to look further than testing hypotheses, to examine programme activities through participants’ behaviours, staff action and the full range of human interactions that can be part of programme experiences [13,14,15]. In this respect, beyond the traditional quantitative methods that can be used in evaluation, a wide range of qualitative methods are applied. These methods originated from the theoretical traditions and orientations of qualitative inquiry, such as ethnography, phenomenology, and hermeneutics. Qualitative methods in evaluation, due to the traditional argument between qualitative and quantitative research, have been exposed to criticism regarding their rigor and subjectivity [9], despite the fact that applied research in the fields of health care and education takes place in real time and in changing context over which full control will never be possible [3].
As a result of this evolvement, a series of different qualitative applications to evaluation have been developed such as process evaluation, evaluation of individualised outcomes, case studies, implementation evaluation. Furthermore, different evaluation frameworks have been developed such as the goal-free evaluation, responsive evaluation, utilisation-focused evaluation [16,17,18]. Selection of the methods and approaches to be used for an evaluation depends on the aim and the purposes of the evaluation, the researcher’s orientation, the questions, which need to be answered, and the problems which need to be resolved [14].
The quantitative-qualitative argument and the resultant scientific debate enhanced the evolution of different evaluation strategies, approaches and models. Most importantly, however, it led to the introduction and establishment of qualitative methods in evaluation. Although these methods were viewed with scepticism for many years, this scientific debate proved to be beneficial for the qualitative methods, since they have proven their utility to practising evaluators, their distinctiveness to theorists and their attractiveness to readers [19].
The starting point of the debate is a consideration of the advantages and the disadvantages of the qualitative and quantitative research methods to evaluation inquiry. Qualitative methods in evaluation research are considered advantageous in terms of depth, openness, detail of inquiry and deep understanding of cases and situations. Quantitative methods are praised for rigorous measurement, comparison, statistical aggregation of data and generalisability [6]. Qualitative research in evaluation is associated with naturalistic inquiry, inductive analysis, fieldwork, empathy and insight and holistic perspectives. Approaches in qualitative inquiry are related to naturalistic theoretical traditions and constructivism, while approaches in quantitative inquiry are associated with randomisation and experimental designs. Scriven [16] states that positivist approaches follow “classic” science principles of hypothesis-testing using quantitative data to test theory whilst constructivist approaches focus on how people make sense of their experience, using qualitative data to generate theory.
Both approaches have strong supporters as well as opponents. In addition to the texts that advocate the one or the other approach, some draw attention to the risks of this evolving argument. Sechrest and Sidani [20], state that continuing controversy over quantitative versus qualitative methods hinders the advancement of social science and programme evaluation. They consider that proponents of both approaches have exaggerated the differences. Shadish [19] believes that the qualitative - quantitative debate in evaluation has brought increased awareness to evaluators about philosophy of science, concluding that on philosophical grounds there are many errors and misunderstandings associated with the ever-lasting debate. He considers that to juxtapose qualitative methods against the “traditional paradigm” is a mistake that has to be eliminated.
The debate between the quantitative and qualitative researchers and the resultant impact on evaluation progress led many researchers and evaluators to appraise and to support new stances and methodological paradigms in an effort to resolve the positivist/naturalist debate. Attree [3] states that research designs need to combine scientific rigor with acknowledgement of context; mixed methods may be part of the solution. Within this changing era issues like pluralism and triangulation in evaluation were valued [3,21,22,5] and mixed methods were appreciated by proving to minimize bias and ensure valid and reliable results [22].
Qualitative Approaches to Evaluation and Triangulation
It is surely a matter of scientific maturity to come to a point where methodological pluralism and utilisation of different methods is clearly demonstrated and even encouraged from evaluation researchers such as Patton [6,14] and Sechrest and Sidani [20]. Patton [6,14] speaks about methodological mixes, data triangulation, theory triangulation and methodological triangulation. He further proposes a variety of different evaluation approaches using mixed forms of quantitative and qualitative perspectives. Combination of methodologies is considered as an ideal approach to strengthen a study design in programme evaluation. This is also supported by the fact that qualitative methods are no longer a new phenomenon that has to struggle for respectability against a quantitative establishment. The persistence and courage of the qualitative scientists to participate in this struggle have been respected and recognised by the wider scientific community. Today qualitative methods are included in evaluation texts; qualitative researchers receive awards and are elected in evaluation associations. Qualitative inquiry is a sophisticated field with a complex, interrelated unit of terms, concepts and assumptions.
Qualitative research is not unique to many disciplines and has a long history in some of them such as sociology, anthropology education history social work and communication. As Shadish [19] says the introduction of qualitative methods seems to be one of the most important accomplishments of the first 30 years of programme evaluation. The same author concludes that qualitative methods in evaluation can provide rich detail about what is being evaluated, can reflect the idiosyncratic constructions and worldviews of individuals, can generate new hypotheses to be investigated both quantitatively and qualitatively and are easy to relate to current issues in the reader’s work and life.
Furthermore, qualitative methods enable new topics and insights to emerge in contrast to quantitative methods that cannot provide in-depth information on the subjects’ experiences, views and feelings [21].Qualitative approach allows the study subjects to play an active role in the research and express their opinion openly. This is the reason that these methods appear to enhance a client – centered approach as O’ Cathain et al [23], state. This is more crucial in the case of evaluation of education that depends on historical backgrounds, cultures, socioeconomic developments of the existing social setting and political interactions and decisions. It involves social needs, personal preferences and ambitions, learning processes and developments. It is not certain that in the multidimensional and interactive context of education, ranking objective indicator systems would reveal all aspects of quality in an educational programme [7].
Fenton and Charsley [24] note that “qualitative sociology demonstrates a complexity and connectedness in the texture and meanings of social life that it is difficult to reproduce in quantitative methodologies”. Qualitative research is located in the world of the research participants and thus it has been characterized as contextualised research. Qualitative data focus on experiences and opinions and provide insights into the lives and understandings of the research subjects. This process produces new and different insights that transform existing understanding by challenging taken for granted assumptions about the world and the people who live and interact in it [5].
A common and major criticism of qualitative methods is that they do not yield generalisable findings. As Seaman [25] states qualitative studies even with careful description of participants and setting cannot be replicated exactly. In qualitative studies the researcher demonstrates the typicality of a phenomenon observed in a particular situation at a particular period of time. If that phenomenon has been reported in other research it maybe cautiously generalised across those situations. This criticism is probably the most important barrier to the further development of qualitative methods in evaluation. However, it should be noticed that generilisability is not the purpose of qualitative methods. The purpose is rather to elicit meaning in a given situation [26]. This is an important point for evaluation in social science, especially when evaluation methods focus on real-life settings, which are idiosyncratic and unique such as education. In real-life settings, evaluation is most probably grounded on realism where the researcher and the reader have their own unique theses and produce their own insights.
Researchers can be concerned that their qualitative findings constitute a quite biased view of the world of participants. However, the concept of bias in quantitative research cannot simply be transferred to qualitative research because the nature of the subject under investigation and the purpose of the research are different. In qualitative research the role of the researcher and any associate inherent biases is acknowledged at all points of the research process [5]. In qualitative research the focus is on the context and “there is no meaning to the idea of an independent person removed from the context of the research objectively verifying the findings because the notion of a person independent of the data has no meaning” [5].
As a contribution to verification and rigor of qualitative designs Patton [22] cited in Cheek et. Al., [5] notes four kinds of triangulation: a) method triangulation, b) triangulation of sources within the same method, c) analyst triangulation using multiple analysts to review findings and d) theory/perspective triangulation using multiple perspectives or theories to interpret the data. Method triangulation concerns with checking out consistency of the findings generated by different data collection methods while triangulation of sources within the same method concern different information sources used in the same study. The analyst triangulation using multiple analysts to review findings of a study means that all team researchers will be involved in independent review of data and this will then be constantly compared. Finally the aim of theory/ perspective triangulation using multiple perspectives or theories to interpret the data is not only looking for points of agreement but testing consistency.
In the same vein, Silverman et al., [27] underscores the importance of the subjects’ active role in research by noting that good research goes back to the subjects with tentative results and refines them in the light of the subjects’ reactions. Confirmability may also include the involvement of several members of a research team who are entitled to compare analyses, look for consistencies and inconsistencies and either resolve these by consensus or state whether consensus was not able to be achieved.
The above discussion is leading us to consider two important issues: Participant-reader and synthesis of methods. Regarding the former it seems that qualitative methods in evaluation are a mean of actively involving the reader in the process of research. Findings related to personal experiences, beliefs, attitudes, perspectives and insights are more enjoyable and easy to read for the non-specialist, at least. The reader is motivated to participate, to reflect and to contribute -either in a personal or in a public-shared level- in the enhancement of knowledge.
Qualitative methods generate a dialogue and further learning. The presentation of qualitative findings demands a response from the reader, which contributes the outcome of the research. The qualitative researcher presents his or her interpretation of findings and then the reader takes this and produces their own insight. This is the opposite of generalisation and promotes a realistic view of evaluation through the unique sight of the involved individuals.
In regard to the latter point, the synthesis of methods, it seems that qualitative approaches in evaluation, by investigating in depth concepts and fields for which little is known, can generate new questions which may be answered by using a combination of quantitative and qualitative approaches. Thus, one particular method of investigation provides researchers with the stimuli to utilise a different paradigm in their new inquiries. The ability of methodologists of one scientific school to cultivate the ground for collaborating with methodologists from a different school demonstrates a wide intellectual spirit.
Conclusion
Consideration of the advantages and the disadvantages of the qualitative and quantitative research methods to evaluation inquiry, paved the path for a more integrated methodological approach to evaluation. Issues of depth, openness, and deep understanding of cases were appraised and discussed in relation to measurement, comparison, statistical aggregation of data and generalisability. In addition major theoretical paradigms such as ethnography, phenomenology, ethnomethodology and hermeneutics were linked with notions of randomisation and experimental designs. Although the controversy over quantitative versus qualitative methods appear to obstruct the advancement of social science and programme evaluation, major advantages arise out of this debate. Increased awareness for evaluators regarding philosophical and scientific notions of evaluation, elimination of errors and misunderstandings in evaluation science and evolution of sophisticated and more integrated approaches to evaluation in education are some of them. Inspired evaluators are expected to appraise and to support new stances and methodological paradigms in the new era of the evaluation science. It remains though a scientific challenge, to view divergent methodological stances not only to coexist in evaluation inquiry but also to expand by using each others’ qualities.
3326