Abstract: This article describes the evaluation of the Neuroscience Nurse Internship Program (NNIP). The NNIP was initiated in 1988 by the National Institutes of Health Clinical Center to meet the demand for highly skilled nurses to care for persons with nervous system disorders. To determine whether the program was meeting its goals, an evaluation component was incorporated into the program. The evaluation process was based on the RSA Model of Continuing Education for nursing. The RSA model consists of four basic components, three of which were included in the assessment of the NNIP–process, content, and outcome. The evaluation revealed that the nurse-interns were satisfied with the lectures and clinical content of the program. Moreover, the nurse-interns improved their knowledge of neuroscience nursing and their confidence in performing neuroscience nursing skills. Information from the evaluation was used to modify the structure of the program and to refine lecture content and clinical requirements.

In 1988, the National Institute of Neurological Disorders and Stroke provided financial support to develop and implement a Neuroscience Nurse Internship Program (NNIP) within the Nursing Department of the National Institutes of Health (NIH) Clinical Center. The NNIP is designed for new registered nurses and registered nurses with recent clinical experience interested in beginning a career in neuroscience nursing. The dual purpose of this program is to provide advanced training in neuroscience nursing and training in the role of the nurse in a biomedical research center. As of April 1998, 54 nurses have completed the internship program. This article describes the results of the program evaluation that has been conducted on a regular basis since the program’s inception.

Background Individuals interested in the program must be registered nurses. Nurse-interns are selected each year from applicants throughout the country. Following their general orientation to the Clinical Center Nursing Department at the NIH, the nurse-interns begin a 6-month program of classroom and clinical experiences. The classroom portion consists of lectures in neuroanatomy, neurophysiology, neurological nursing assessment, and a variety of nervous system disorders. The topic areas for the nervous system disorders are pathophysiology, epidemiology, diagnostic evaluation, medical and nursing management, investigational treatments, and patient education. Additional classroom topics are rehabilitation, ethics, pain management, and dying and death. The clinical component consists of working with a staff nurse preceptor and providing direct patient care on neuroscience patient care units. The nurse-interns attend neurological and neurosurgical rounds and conferences, as well as weekly clinical seminars during which they present and discuss patient case studies. The nurse-interns also attend observational experiences both intramurally in the neuroscience outpatient clinics and extramurally at local acute trauma and rehabilitation healthcare facilities.

There are two primary objectives of the NNIP program. The first objective is for nurses to acquire the knowledge and skills necessary to function independently in caring for patients with nervous system disorders. The second objective is to prepare nurses to function as members of the biomedical research team at the NIH Clinical Center.

RSA Model To determine whether NNIP is meeting its objectives to enhance the knowledge and clinical competency of the nurse-interns, an evaluation component is incorporated into the program. The evaluation component is based on the RSA model for Continuing Education.[1] The RSA model was developed by Roberta S. Abruzzese, who believed that an effective evaluation model for continuing education should be practical, feasible, and simple. Moreover, because nurses who attend continuing education programs are adults, Abruzzese believed that a practical evaluation model should incorporate the tenets of adult learning theory. Within adult learning theory, adults are assumed to enter into learning situations as active participants with a readiness to learn. Their varied backgrounds and life experiences also contribute to the learning experience.

The RSA model comprises four levels: process, content, outcome, and impact evaluation. Process evaluation refers to the “general happiness” of participants with learning experiences in the program. Process evaluation is conducted by using self-report and observational methods to determine whether both course and student objectives are met and whether instructors are effective in meeting the learning needs of the students. Content evaluation focuses on the documentation of changes in knowledge, skills, and attitudes related to the learning experiences. For this level of evaluation, the program coordinator seeks to learn whether the students know more, whether their skills are enhanced, and whether their attitudes have changed due to participation in the program. Outcome evaluation refers to the behavioral changes in participants that can be directly related to the program. For this level of evaluation for NNIP, the program coordinator is interested in knowing whether patient care has changed due to participation in the program. Impact evaluation focuses on the effect of the program beyond the immediate participants. For NNIP, the impact of the program can be determined by observing actual changes in patient status as a result of the program. The first three of these four levels are incorporated into the NNIP evaluation plan. The fourth level, impact evaluation, requires additional resources to monitor and record changes in patient care and patient status. Because additional costs are generally associated with this level of evaluation, it was not included in NNIP as a regular evaluation component. adultlearningtheory.net adult learning theory

Methods Procedures Data for the first and second levels of evaluation–process and content–are routinely collected at four time points during the NNIP. The first data are collected during the nurse-interns’ orientation to the Nursing Department at the Clinical Center; the second during their first day of orientation to the NNIP; and the third on the last day of the NNIP. Evaluation data also are collected at the end of each lecture in the neuroscience lecture series. For the first seven classes of nurse-interns, the NNIP was 10 months. Based on evaluations conducted for the first seven classes, the program was reduced to a 6-month program beginning with class 8. For the first seven classes, the general Clinical Center Nursing Department orientation period lasted 3 months and was followed by clinical and classroom neuroscience experiences lasting an additional 10 months. For classes 8, 9, and 10, the NNIP began 1 month after the general Clinical Center Nursing Department orientation. The program continued for 6 months and included general Clinical Center orientation material as well as neuroscience materials. In order to conduct the program in the shorter time period, the total number of lectures was reduced from 119 to 104. Lectures that were eliminated were those containing general medical and basic nursing content (e.g., fluid and electrolyte balance, tube feeding techniques). Lectures related to general nursing at the Clinical Center (e.g., discharge planning) were incorporated into the Clinical Center orientation. In addition, the amount of clinical time required during the program was reduced about 30%.

Although the third level of evaluation–outcome is not conducted routinely for NNIP, three outcome assessments were conducted during the first 10 years–after the completion of the fourth, fifth, and eighth classes of nurse-interns. Following the fourth class of nurse-interns, individual interviews were conducted with three former nurse-interns–one from each of the first three classes. These participants were asked to evaluate various aspects of the program and to describe changes in their nursing practice as a result of participation in the program. Following the completion of the fifth class of nurse-interns, individual telephone interviews were conducted with nurse administrators and physicians who were asked to evaluate the program and its impact on neuroscience nursing and patient care within the Clinical Center. Following the eighth class, four focus groups were conducted with former nurse-interns. The objectives of the focus groups were to assess the strengths and weaknesses of the program, to determine how the program helped in the development of neuroscience nurses, and to determine the perceived effect of the program on patient outcomes. The individual interviews with former nurse-interns and the focus groups were taped with permission of the participants.

Sample The sample for the process and content evaluation was all nurse-interns admitted to the program since its inception in 1988. A total of 57 nurse-interns entered the program, and 54 nurse-interns from 10 classes completed the program. Most of the nurse-interns were new baccalaureate graduates ranging in age from 21 to 55 years. Two of the nurse-interns were graduates of associate degree nursing programs and had additional clinical experience. Of the 54 nurse-interns who completed the program, 29 are currently employed by the Clinical Center–15 are employed on neuroscience units and 14 on other Clinical Center units. Of those who are no longer employed by the Clinical Center, 7 continue to work on neuroscience units, 12 are employed in a non-neuroscience specialty, 3 are no longer employed as nurses, and 3 have been lost to follow-up. Of the three who did not complete the program, one took maternity leave, one applied to medical school, and one transferred to critical care nursing.

Three samples of participants were recruited for the three outcome evaluations conducted after the fourth, fifth, and eighth classes of nurse-interns. For the first outcome evaluation, the sample consisted of three nurses who had completed the program. One nurse was selected from each of the first three classes of nurse-interns. Each of these nurses was employed by the Clinical Center and was willing to participate in the interview. All were female and had recently completed the baccalaureate degree program prior to entering NNIP. Eleven individuals who were involved in the development or implementation of NNIP were interviewed following the completion of the fifth class of nurse-interns. These individuals included four physicians, three head nurses, one clinical specialist, one nursing administrator, and two research nurses. For the focus groups that were conducted after the eighth class of nurse-interns, 27 former nurse-interns representing all eight classes participated in one of four focus groups.

Evaluation Instruments Five evaluation instruments were developed to measure components of process and content evaluation. Process evaluation that assesses participant satisfaction and the extent to which the program is meeting its goals was conducted by using the Single-Lecture Evaluation Instrument and the Program Evaluation Instrument.

The Single-Lecture Evaluation Instrument was used to assess the satisfaction of the nurse-interns with lectures; data were collected immediately following each lecture. The original instrument was developed by the program coordinator and was based on similar instruments used to evaluate classroom instruction. It consisted of 15 items covering four areas: quality of instruction, usefulness of the content, quality of the learning environment, and preparation for learning (previous clinical experience in the content area and completion of assigned readings). Nurse-interns were asked to rate 13 items on a scale from 1 (strongly disagree) to 5 (strongly agree), with higher numbers indicating more positive perceptions of the lecture, content, or environment. For the preparation of learning section (two items), nurse-interns were asked to respond yes or no to queries about previous clinical experience in the content area and preparation for class. For classes 9 and 10, the scale was modified to delete five items to reduce redundancy among items and to expand the response options in an attempt to increase variability of responses. Thus, the current scale is 10 items long, covering the same four areas noted above. Eight items are each rated on a 10-point agree/disagree scale, and two items require a yes/no response. For each class, the evaluations of all nurse-interns on all lectures were combined to yield overall ranges and means for individual items on the instrument and an overall score for the items rated on the agree/disagree scale.

The Program Evaluation Instrument was used to assess the nurse-interns’ satisfaction with the program as a whole and was completed on the last day of class. The Program Evaluation Instrument was developed by the program coordinator and was based on the objectives of the program. It consists of 10 sections: program objectives, student objectives, content, assigned readings, program requirements, guest lecturers, clinical experiences, instructors, physical environment, and observational experiences. Nurse-interns rated each item on a 10-point scale from 1 (strongly disagree) to 10 (strongly agree), with higher numbers reflecting greater satisfaction with each aspect of the program. An earlier version of the instrument was used for classes 1 through 8. For this version, the items were the same; however, each was rated on a 4-point rather than a 10-point agree/disagree scale. The scale was changed to 10 points to increase variability of responses. here adult learning theory

To monitor changes in knowledge, attitudes, and skills, content evaluation was conducted by using three instruments: a knowledge assessment test, the Neuroscience Nursing Self-Efficacy Scale (NNSES), and an attitudinal questionnaire. The knowledge assessment test contained 110 multiple-choice items reflecting the content covered in the program–neuroanatomy, neurophysiology, pathophysiology, neuroassessment, and nursing care of persons with nervous system disorders. The test was administered to nurse-interns shortly after their initial orientation to the Clinical Center Nursing Department, at the beginning of the NNIP course work, and at the end of the program. The results of the test were used to determine the type and amount of neuroscience information nurse-interns know at program entry and changes in knowledge over the course of the program.

NNSES was used to assess confidence in the performance of neuroscience clinical skills. This scale was developed by the program evaluator and was based on a similar scale used in the Neuroscience Nurse Certificate Program.[2] Data for this evaluation were collected shortly after the initial Clinical Center Nursing Department orientation, at the initiation of NNIP, and at the end of NNIP. Initially, NNSES was composed of 54 skills divided into the following 10 sections: general health, alterations in consciousness, mentation, communication, protective mechanisms, mobility, elimination, nutrition, sensation, and psychosocial function. In the first version of the scale, each item was rated on a 5-point scale of perceived degree of proficiency from 1 (beginning) to 5 (proficient). After using this scale for three classes, the decision was made to change the scale to an 11-point rating scale to increase the variability of responses. Moreover, the scale was changed to measure perceived level of confidence in performing skills rather than level of proficiency. Thus each item was rated on a scale from 0 (not at all confident) to 10 (extremely confident). This change was an attempt to reduce the errors in measurement due to different interpretations of the terms beginning and proficient. In addition to these changes, six more skills were added, yielding a total of 60 skills. Mean scores were computed for individual items, for the 10 categories, and for the overall scale. Higher scores indicate a higher perceived level of confidence in performance of the skills.

An Attitudinal Questionnaire was used to assess attitudes about working in a neuroscience setting. This questionnaire was used to determine whether attitudes toward neuroscience nursing became either more positive or negative as a result of involvement in the program. For the first three classes of nurse-interns, this assessment consisted of open-ended questions to elicit attitudes about issues of importance to neuroscience nurses and persons with nervous system disorders. The responses on this questionnaire served as a basis for the development of a standardized attitudinal questionnaire that is called the Attitudinal Questionnaire for Neuroscience Nurses. This instrument consists of items divided into the following seven sections: (1) general nursing issues, (2) ethical issues, (3) likelihood of recovery for persons with various nervous system disorders, (4) effect of nervous system disorders on lifestyle, (5) professional development, (6) preference for work units, and (7) expectations of attendance at NNIP. Items in each section are rated on a 5-point scale from strongly disagree to strongly agree. The questionnaire was administered to classes 4 through 10 during their orientation to the Clinical Center and at the end of the program.

Results of Process/Content Evaluations Table 1 presents the mean ratings for the Single-Lecture Evaluation Instrument for the 10 classes. These data show that nurse-interns were generally very satisfied with both the quality of instruction and the content of the lectures. Most of the nurse-interns reported that they had clinical experience with the nervous system disorder prior to its presentation in lecture. And most nurse-interns had prepared for the lectures by reading required materials.

Table 1. Means Ratings for the Single-Lecture Evaluation Instrument for the 10 NNIP Classes

I Have Read the I Have Clinical Required Readings Overall Mean Experience in This About This Topic Class for Rated Items(*) Disorder (Percentage) (Percentage)

1 8.77 NA NA 2 9.80 NA NA 3 9.88 62 64 4 9.78 76 58 5 9.85 72 54 6 9.67 67 62 7 9.87 64 72 8 9.57 60 65 9 9.70 67 80 10 9.70 67 80

(*) Class 1 rated items on a 5-point scale, classes 2 through 6 rated items on a 4-point scale, and classes 7 through 10 rated items on a 10-point scale. Scores for classes 1 through 6 were converted to a 10-point scale for comparison. Classes 9 and 10 rated 8 items, whereas classes 1 to 8 rated 13 items.

Table 2 presents the baseline, pre-program, and postprogram average scores on the knowledge assessment test. The nurse-interns are relatively similar in regard to baseline knowledge across classes. Participation in orientation and general nursing unit activities tended to raise scores slightly, as seen in the pre-NNIP assessment. However, a dramatic increase in knowledge scores is demonstrated for all classes (and all nurse-interns) by the end of the program.