C.C. Margadant [1,5], M.E. Hoogendoorn [2,5], R.J. Bosman [3,5], J.J. Spijkstra [4,5], S. Brinkman [1,5], N.F. de Keizer [1,5]
1 Department of Medical Informatics, Amsterdam Public Health Research Institute, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands. 2 Department of Anaesthesiology and Intensive Care, Isala, Zwolle, the Netherlands. 3 Department of Intensive Care, OLVG, Amsterdam, the Netherlands. 4 Department of Intensive Care, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands. 5 National Intensive Care Evaluation (NICE) Foundation, Amsterdam, the Netherlands.
C.C. Margadant - email@example.com
Validation of the Nursing Activities Score (NAS) using timeand-motion measurements in Dutch intensive care units
Background The Nursing Activities Score (NAS) is widely used for measuring the workload of intensive care unit (ICU) nurses. However, the performance of the NAS to measure actual nursing time has not been comprehensively and externally validated. The aim of this study is to validate the NAS using time-and-motion measurements in Dutch ICUs.
Methods We measured nursing time for patients admitted to seven Dutch ICUs, between November 2016 and October 2017. The patient(s) that were under the care of a chosen nurse were followed by the observers during the entire shift and measurements were performed using an in-house developed web application. To validate the reliability of the NAS, we first converted NAS points per activity into minutes. Next, we compared the converted time per NAS item and the converted total nursing time per patient with the actual observed time. We used Wilcoxon signed-rank tests at nursing activity level and Pearson’s R and R2 at patient level for these comparisons.
Results A Pearson’s correlation of R=0.59 (R2 =0.35) was found between the total converted NAS time and the total observed time per patient. The median converted NAS time per patient (202.6 minutes) was higher compared with the observed time per patient (114.3 minutes). At NAS item level, we found significant differences between the converted NAS time and the observed time for all separate NAS items.
Conclusions The NAS overestimates the nursing time needed for patients in Dutch ICUs. Therefore, we advise revisions of the time weighting assigned to each NAS item to obtain better insight into the true nursing workload so that this information can be used for more effective nursing capacity planning.
There are concerns regarding excessively high nursing workload, both in general and ICU wards. An excessively high nursing workload can lead to burnout and job dissatisfaction among nurses and have a deleterious effect on patients. Workload has risen due to an increased turnover of patients, increased complexity of patients, together with nursing shortages. All this makes planning of nursing capacity important. In the last 30 years different instruments have been developed to measure the nursing workload to give insight into the nursing staff needed per shift and provide much needed input for capacity planning. To assess nursing workload in the ICU, Cullen et al. created the Therapeutic Intervention Scoring System (TISS). The TISS was originally developed to classify nursing workload in relation to the severity of illness of ICU patients. The TISS consists of 76 therapeutic interventions that receive 1-4 points based on the severity of illness. It appeared that nursing workload is only partly related to severity of illness, since less severely ill patients could also generate a high nursing workload. For instance, a patient recovering from a serious illness with agitated delirium would not score high in severity of illness, but could demand very intensive nursing care, up to continuous bedside care throughout the day. This made the TISS less effective in assessing nursing workload. Therefore, the Nursing Activities Score (NAS) was developed by Miranda et al. in 2003. The NAS describes activities that largely represent the work actually performed by nurses at the bedside in caring for patients and was developed to measure the nursing workload for each individual patient. The points assigned to nursing activities provide an average time consumption in caring for the patients instead of representing the severity of illness. The NAS was created by using the work-sampling approach: at random moments per shift the nurse was asked what he or she was doing at that specific moment. The researchers applied a weighting for each activity. The total NAS for an individual patient is the sum of NAS points for all activities, varying between 0 to 177 points. A score of 100 NAS points is equivalent to the amount of care that can be provided by one full-time equivalent nurse during either one shift or one day. A score above 100 points indicates that the care needed can only be provided by more than one nurse. The NAS is considered a valuable tool and is widely used for workload measurement in ICUs.[8,9] However, the performance of the NAS has not been comprehensively validated. One study showed that the NAS might either underestimate or overestimate the actual nursing time required by patients and therefore recommended revision of the original NAS because of inadequate measurement of nursing activities. The study by Stafseth et al. suggests that the reliability and validity of the NAS are good. However, this study strongly suggests more research in other countries and larger groups of patients.
Furthermore, research has demonstrated that the worksampling approach, as used for the development of the NAS, does not lead to an accurate representation of the true nursing workload. This is due to the fact that the weighting of nursing activities is based on the probability that a particular nursing activity occurred. The total amount of time in a shift is divided over the nursing activities that were carried out. When nursing activities frequently occur or take a lot of time, they would also occur more frequently in the work-sampling approach. However, this approach will not lead to precise measurements, but will only approximate the time of the different activities. Thus, in contrast to time-and-motion techniques in which every minute of a nursing shift is measured, the work-sampling approach does not measure the real amount of time spent on nursing activities, which could lead to less accurate results. Therefore, the time-and-motion technique is considered the best technique for time measurement. The aim of this study is to validate the NAS in the Dutch ICU setting using the time-and-motion technique, and to identify which nursing activities are underestimated or overestimated in the NAS.
We conducted an observational study. All 82 Dutch ICUs participate in the National Intensive Care Evaluation (NICE) quality registry. Fifteen of these ICUs are participating in the newly implemented voluntary nursing capacity module, seven of which voluntarily took part in this study. Data on characteristics of the ICUs (such as number of ICU beds) and data on patient characteristics (such as age, BMI, admission type, and mortality) were extracted from the NICE registry.
The study involved time-and-motion measurements for patients admitted to the ICU. We included different types of hospitals (academic, teaching and non-teaching hospitals) and different shifts (day, evening and night). We performed an equal number of measurements in all types of hospitals and shifts. At the start of a shift, one nurse was chosen by the observer. The patient(s) that were under the responsibility of this nurse were followed by the observer during the entire shift. A longer-term patient could theoretically be observed on different dates during different shifts and therefore could possibly be followed during more than one measured shift. The measurements took place on different days of ICU admission (e.g. first ICU admission day through to last ICU admission day) and with different types of nurses (registered and student nurses). We randomly selected nurses who took care of patients that were expected to stay during the whole shift in order to measure as many nursing activities as possible. Observers were researchers CM and MH and ten student nurses. The students were trained in performing time-and-motion measurements by oral and written instructions and one day of measuring together with one of the researchers. The observers used an in-house developed web application to record start and stop times of each performed nursing activity. The application included all activities occurring in the NAS (table 1). If two nurses were simultaneously performing nursing activities for the same patient, this was also registered by pressing the ‘two nurses button’ and multiplying this time by two in the analysis. When two different activities were carried out by two nurses, these activities could be measured simultaneously. Measurements were conducted between 1 November 2016 and 1 October 2017. Participation of the hospitals was on a voluntary basis. Seven hospitals were willing to participate. Data were processed anonymously.
The Institutional Research Board of the Amsterdam University Medical Centre reviewed the research proposal and waived the need for informed consent (IRB protocol W17_366).
Nursing activities that occurred less than ten times in the total dataset were excluded from the analysis. Most NAS items have a fixed number of NAS points but some items have different categories corresponding to different numbers of NAS points depending on the duration of that activity (e.g. bedside with hourly vital signs, bedside for two hours or more, or four hours or more). For these duration-dependent activities, we first used the measured time for that activity to assign the correct number of points. For example, a nurse performed hygiene procedures on a patient for 1.2 hours during a shift, according to our time measurements. This NAS item has three categories: performing hygiene procedures for less than two hours, for more than two hours, or for more than four hours.
Table 1. NAS activities with their points according to Miranda et al. (2003), and the median converted NAS times and observed times per NAS item
In the above-mentioned example, the activity took 1.2 hours and would therefore be assigned to the category for less than two hours, which corresponds to 4.1 NAS points. To validate the NAS, we first converted the originally assigned NAS points per activity into time. Based on Miranda et al. 100 NAS points correspond to 100% of care time provided by one nurse during a shift and hence 1 NAS point corresponds to 1% of care time provided by one nurse. As stated by the author of the NAS, a nurse is productive in 80% of the 8-hour shift; this means that one NAS point corresponds to 3.84 minutes of nursing care during an 8-hour shift ((8 hours * 60 mins)/100)*0.8).[7,15] With this information, we were able to convert the NAS scores into an estimated nursing time per patient and per nursing activity (from now on referred to as ‘converted NAS time’. Next we compared the time per NAS item and the total nursing time per patient, based on NAS scores according to the model, with observed times from the time-and-motion measurements. For the observed time, we took the sum of the times of all performed nursing activities per patient per shift in minutes (from now on referred to as ‘observed time’. The median and interquartile ranges (IQR) of the converted NAS times and the observed times were calculated. First, the difference between the total converted NAS times and the total observed times per patient were visualised by scatterplots. Second, the correlation between the total converted NAS times and the total observed times per patient were assessed with the Pearson’s correlation test. In addition, we also assessed the R2, a measure for the proportion of the variance. For each nursing activity separately, medians and interquartile ranges (IQR) of the converted NAS times and observed times were calculated and differences were tested with the Wilcoxon signed-rank test. All statistical analyses were performed using R statistical software, version 3.3.2.
Table 2 shows the ICU characteristics of the seven included ICUs compared with all Dutch ICUs; no significant differences were found between the included ICUs and all Dutch ICUs. During our study, a total of 287 unique patients were observed during 371 different shifts with time-and-motion measurements. In these patients, 46,319 nursing activities were measured. In 45% of the measurements, nurses took care of two patients per shift. In 15% nurses took care of three patients per shift. For the remaining 40%, nurses cared for one patient per shift. The patients in our study had a significantly higher in-hospital mortality rate (22.3% versus 13.0%) and length of ICU stay (3.2 days versus 1.0 day) compared with all Dutch patients in the same period (table 3). Furthermore, acute renal failure, chronic respiratory insufficiency, and cirrhosis differed between the groups, with a higher percentage in the patients in our study. For the other patient characteristics, the included patients and all Dutch ICU patients in this period were comparable.
Table 2. ICU characteristics
Excluded nursing activities The following three NAS nursing activities occurred less than 10 times in all measurements and were therefore excluded from the analysis at activity level: care of a pulmonary or left atrial catheter, cardiopulmonary resuscitation and specific interventions in the ICU (endotracheal intubation, insertion of a pacemaker, cardioversion, endoscopy, emergency surgery in the previous 24 hours, gastric lavage). Furthermore, we did not specifically measure intravenous replacement of large fluid losses and treatment of metabolic acidosis/alkalosis, since these two nursing activities usually fall under the category bedside activities.
Total patient time and times per NAS item
The median converted NAS time per patient (202.6 minutes; IQR 155.0-241.2 minutes) was significantly higher (p<0.001) compared with the observed time per patient (144.3 minutes; IQR 81.3 – 168.4 minutes), see figure 1. A Pearson’s correlation of R=0.59 (R2 =0.35) was found between the total converted NAS time and the total observed time per patient (table 1).
At the NAS item level, we found significant differences between the converted NAS times and observed times for all items. These differences ranged from -54.6 minutes (support or care of patient or relatives for about 1 hour) to 79.2 minutes (mobilisation and positioning with three nurses). For most (86%) nursing activities the median converted NAS overestimated the observed time. For four activities (support or care of patient for about 1 hour, administrative tasks for less than 2 hours, administrative tasks for about 2 hours and specific interventions outside the ICU) the converted NAS underestimated the observed time (table 1).
Our analysis showed that the NAS overestimates the nursing time needed for patients in the Dutch ICU setting. Times of most NAS items were overestimated by the NAS, except for four activities (support or care of patient for about 1 hour, administrative tasks for less than 2 hours, administrative tasks for about 2 hours, and specific interventions outside the ICU), where the NAS gives an underestimation of the observed time. This study showed that 35% of nursing time is explained by the NAS model (R2 = 0.35).
Table 3. Patient characteristics
The converted NAS time per patient (202.6 minutes per shift) in our study was comparable with the converted NAS times per patient in other studies. Bernet et al. found 150 to 156 minutes per shift and Deberg et al. found 180 to 228 minutes per shift. The different articles on the NAS give variable NAS times per shift. A full shift of work equals 480 minutes of nursing time. The low correlation of Pearson’s R and R2 (0.59 and 0.35) implicates that the NAS is not accurate enough to estimate the nursing time at patient level. However, it is currently still the best nursing workload model for quantifying nursing workload in ICUs.5 There is no clear cut-off point from which the model can be identified as ‘good enough’ based on the R2 . However, since the NAS is used for capacity planning, an R2 closer to 1 would be more desirable. Since ICU nurses also spend time on non-nursing duties in almost every shift, such as coaching a student or participating in an emergency team within the hospital, we performed a sensitivity analysis to determine whether these non-nursing duties were affecting the correlation. According to several studies nurses spend approximately 3 to 6% of their shift on nonnursing duties.[19,23] We therefore took the average of 4.5% and subtracted this from the 80% of productive nursing time, which we used in this study to calculate the converted time per NAS point. Using this approximation, the converted time would have changed from 3.84 to 3.62 minutes per NAS point. This change does not affect the results and we therefore conclude that nonnursing duties do not significantly influence the performance of the NAS. A strength of our study is that we validated the NAS with time-and-motion measurements, which is considered to be the best technique for measuring nursing workload. To our knowledge, this has not been performed before in the context of NAS validation. Measurements for nursing activities by using time-and-motion measurements are more accurate compared with the work-sampling approach, as used for the development of the NAS. Furthermore, since measurements took place in all types of ICUs, we believe that results of this study are generalisable to all Dutch ICUs. One of the limitations of our study is the fact that we excluded NAS activities because they did not occur or occurred less than ten times. Two of these activities are usually scored in other categories: the activity ‘intravenous replacement of large fluid losses’ is mostly scored under NAS item 1 ‘bedside’. The activity ‘treatment of complicated metabolic acidosis/alkalosis’ is mostly scored in NAS item 3 ‘medication’. Since these activities could be scored in other categories, we did not include them in our study. Three NAS activities (left atrial monitoring, cardiopulmonary resuscitation after arrest, and specific interventions in the ICU, respectively) and six subcategories 1b, 1c, 4b, 4c, 7b, and 8c (the nursing activities that required more than 2, 3 or 4 hours of the nurses’ time) did not happen often enough (so, ten times or more) during the measurements, which makes the validation of the NAS incomplete. Given the fact that the median time of nursing care per patient is 2.4 hours (144.3 minutes), nursing activities taking more than 2, 3 or 4 hours rarely occur in daily ICU practice so it is not likely that our results have been affected by this situation. Nurses took care of two or three patients in 60% of our measurements; we assume that nurses taking care of only one patient can perform nursing activities in a shorter amount of time. We did not specifically study this but further research could eventually point out what is the optimum time per nursing activity. Furthermore, the observed patients seem to have been more severely ill and consequently had a longer length of stay compared with all Dutch patients in the same time period, which is likely caused by our selection mechanism. In order to measure as many nursing activities as possible we probably more often choose nurses who took care of patients that were expected to stay the whole shift and these patients were probably more severely ill. This may have biased our results since our aim was to validate the NAS and check for underestimations or overestimations compared with time-andmotion measurements and it is possible that observed times in sicker patients differ from those in less sick patients. However, according to Armstrong et al. NAS scores in intermediate care patients did not differ from those in ICU patients. Finally, our study does not correct for the nurses years of experience on the ICU or level of education. In the analysis we included student and registered ICU nurses but further research in larger groups should investigate whether different groups need different weighting of NAS points. Based on our results we believe there is room for improvement in the measurement of nursing workload. The NAS could be improved by adjusting the NAS points given to the different items. The developers of the NAS did not report the Pearson’s R or R2 , but stated that the NAS is reflecting 81% of total nursing time. About 11% of the nurses’ time is spent on personal activities. The remaining 8% comes from nursing activities derived from medical interventions, related exclusively to the severity of illness of the patient not measured by the NAS. The TISS takes these medical interventions into account, such as induced hypothermia, cardiac assist device, pacemaker monitoring or ECG monitoring. For this reason, we suggest additional research towards the merging of the TISS-28 and the NAS. The models could be partly combined which could possibly improve the estimation of nursing workload. Our results on observed time per patient and per nursing activity could be taken into consideration when assigning weighting to the activities in this new model. Moreover, we think that expressing nursing activities in minutes or hours would be more informative compared with points, since it is more straight forward for ICU managers to work with.
The NAS was developed more than 15 years ago and significantly overestimates the nursing time needed for ICU patients in the current daily ICU practice. Therefore we recommend a revision of the time weighting assigned to each nursing activity to gain better insight into the true nursing workload and to enable a more effective nursing capacity planning.
The data that support the findings of this study are available from the National Intensive Care Evaluation (NICE) but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of the NICE registry. The department of medical informatics (with C.C. Margadant, S. Brinkman, and N.F. de Keizer as employees) receives funding for data processing of the NICE registry. The funding by the NICE foundation does not alter the authors' adherence to all Intensive and Critical Care Nursing policies on sharing data and materials. Four co-authors (M. Hoogendoorn, R.J. Bosman, J.J. Spijkstra, and N.F. de Keizer) are members of the board of NICE. The authors declare that they have no conflict of interests.
- Hughes R. Patient safety and quality: An evidence-based handbook for nurses (Vol. 3). Rockville, MD: Agency for Healthcare Research and Quality 2008.
- Aiken LH, Clarke SP, Sloane DM, et al Hospital nurse staffing and patient mortality, nurse burnout, and job dissatisfaction. JAMA. 2002;288:1987-93.
- Margadant C, Wortel S, Hoogendoorn M, et al. The Nursing Activities Score per Nurse Ratio Is Associated With In-Hospital Mortality, Whereas the Patients Per Nurse Ratio Is Not. Crit Care Med. 2019;48:3-9.
- Palese A, Comisso I, Burra M, et al. Nursing Activity Score for estimating nursing care need in intensive care units: findings from a face and content validity study. J Nurs Manag. 2006;24:549-59.
- Hoogendoorn ME, Margadant CC, Brinkman S. Workload scoring systems in the Intensive Care and their ability to quantify the need for nursing time: a systematic literature review. Int J Nurs Stud. 2020;101:103408.
- Cullen DJ, Civetta JM, Briggs BA, Ferrara LC. Therapeutic intervention scoring system: a method for quantitative comparison of patient care. Crit Care Med. 1974;2:57-60.
- Miranda DR, Nap R, de Rijk A, et al. Nursing activities score. Crit Care Med. 2003;31:374-82.
- Padilha KG, de Sousa RMC, Garcia PC, et al. Nursing workload and staff allocation in an intensive care unit: a pilot study according to Nursing Activities Score (NAS). Intensive Crit Care Nurs. 2010;26:108-13.
- Gonçalves LA, Padilha KG, Sousa RMC. Nursing activities score (NAS): a proposal for practical application in intensive care units. Intensive Crit Care Nurs. 2007;23:355-61.
- Stafseth SK, Tønnessen TI, Diep LM, Fagerstrøm L. Testing the reliability and validity of the nursing activities score in critical care nursing. J Nurs Measure. 2018;26:142-62.
- Urden LD, Roode JL. Work sampling: a decision-making tool for determining resources and work redesign. J Nurs Admin. 1997;27:34-41.
- Finkler SA, Knickman JR, Hendrickson G, et al. A comparison of work-sampling and time-and-motion techniques for studies in health services research. Health Serv Res. 1993;28:577.
- Burke TA, McKee JR, Wilson HC, et al. A comparison of time-and-motion and selfreporting methods of work measurement. J Nurs Admin. 2000;30:118-25.
- Dutch National Intensive Care Evaluation (NICE) foundation. Available at: https:// www.stichting-nice.nl/dd/#561 Accessed 1 June 2019.
- Interview Reis Miranda, jaarboek NICE. Available at: https://www.stichting-nice. nl/focusIC.jsp Accessed 1 September 2019.
- R Core Team. R: A language and Environment for Statistical Computing [computer program]. Vienna, Austria: R foundation. Available at: http://www.R-project.org Accessed May 1, 2019.
- Bernat Adell A, Abizanda Campos R. Nursing Activity Score (NAS). Our experience with a nursing load calculation system based on times. Enferm Intensiva. 2005;16:164-73.
- Debergh DP, Myny D, Van Herzeele I, et al. Measuring the nursing workload per shift in the ICU. Intensive Care Med. 2012;38:1438-44.
- Abbey Mphil M, Chaboyer W, Mitchell M. Understanding the work of intensive care nurses: a time and motion study. Aust Crit Care. 2012;25:13-22.
- Bosman R, Rood E, Oudemans-van Straaten, H, et al. Intensive care information system reduces documentation time of the nurses after cardiothoracic surgery. Intensive Care Med. 2003;29:83-90.
- Harrison L, Nixon G. Nursing activity in general intensive care. J Clin Nurs. 2002;11:158-67.
- Kaya H, Kaya N, Turan Y. et al. Nursing activities in intensive care units in Turkey. Int J Nurs Pract. 2011;17:304-14.
- Norrie P. Nurses’ time management in intensive care. Nurs Crit Care. 1997;2:121-5.
- Steyerberg EW. Evaluation of Performance. Clinical Prediction models: a practical approach to development, validation, and updating. Springer 2009; p. 83-99.
- Armstrong E, de Waard MC, de Grooth HJS, et al. Using nursing activities score to assess nursing workload on a medium care unit. Anesth Analg. 2015;121:1274-80.