Psychiatric training during covid-19 pandemic benefits from integrated practice in interprofessional teams and ecological momentary e-assessment

La formazione psichiatrica durante la pandemia di covid-19 beneficia della
pratica integrata nei team interprofessionali e della valutazione elettronica
momentanea ecologica



1Department of Psychiatry, South-West Yorkshire NHS Trust, Wakefield, United Kingdom

2Centre for Medical Education, University of Dundee, United Kingdom

3International Centre for Healthcare and Medical Education, Bristol, United Kingdom

SUMMARY. Objectives. During the current covid-19 pandemic, healthcare students had to stop their face-to-face attendance at medical colleges and universities. This condition has resulted in a change in how learning and assessment of psychiatric and interprofessional practice occur. The pandemic has also increased clinical mentors’ need to follow their mentees’ educational progress via virtual technology, including smartphone-based educational apps. The aim of the study is to propose a model in psychiatric and medical interprofessional practice e-assessment. Methods. In this study, 228 undergraduate healthcare students underwent a training period in interprofessional practice across specialties being assessed by Ecological Momentary e-Assessment (EMeA), and consisting of ongoing or before/after Interprofessional Education (IPE) learning evaluations with the assistance of an IPE-app linked to online surveys. The Goodness of Fit Test Chi-square and t-test statistics analyzed the data. Results. Surveys during, at entry and exit points in IPE captured increased percentages of learners, specifically, reporting high patient satisfaction with interprofessional teams (c2=22.54; p<.01), learners experiencing very good quality of care when delivered by interprofessional teams (c2=30.02; p<.01) assessed by distance technology, and learners selecting less frequently peers from the same clinical background when support was needed in patient care (c2=19.84; p<.01). Conclusions. Contextual assessment (in the real-time and real-world scenario) of IPE learning moments via EMeA shows its value and applicability during the current covid-19 pandemic when the assessment of learning cannot occur face-to-face between learners and teachers. All healthcare students, including those on psychiatric rotations, could log in their progress, self-reflective assessments, and responses to coordinated care in interprofessional teams, without needing direct contact with their clinical tutors, and while treating patients with mental and physical illnesses, also including covid-19 positive patients.

KEY WORDS: covid-19, interprofessional education, ecological momentary assessment, app, psychiatry.

RIASSUNTO. Obiettivi. Durante la presente pandemia da covid-19, gli studenti nelle professioni sanitarie hanno dovuto interrompere le frequenza in persona nei loro istituti di istruzione e università. Tale situazione ha comportato un cambiamento nel modo in cui si svolgono l’apprendimento e la valutazione della pratica psichiatrica e interprofessionale. Inoltre, la pandemia ha aumentato il bisogno per i docenti clinici di seguire l’andamento dei loro studenti attraverso l’uso di app pedagogiche accessibili da uno smartphone. Lo scopo dello studio è quello di proporre un modello di valutazione elettronica della pratica interprofessionale psichiatrica e medica. Metodi. Nella presente ricerca, 228 studenti universitari della laura magistrale in Medicina e Scienze Infermieristiche, inclusi studenti nelle discipline psichiatriche, hanno ricevuto un training nella pratica interprofessionale con una valutazione virtuale attraverso e-Valutazione Momentanea Ecologica (EMeA), consistente in un’analisi dell’Istruzione Interprofessionale (IPE) durante e all’inizio-fine dell’esperienza educativa. Tale valutazione è stata supportata da una IPE-app connessa a questionari online. I test statistici del Chi-quadrato e t-test hanno analizzato i dati. Risultati. I questionari completati durante, all’inizio e fine del corso di IPE e accessibili sull’IPE-app mostrano un aumento delle percentuali di studenti che riferiscono un incremento della soddisfazione dei pazienti quando questi sono curati da team interprofessionali (c2=22,54; p<,01), un incremento di studenti che riferiscono che la qualità di trattamento diventa molto buona quando viene effettuata da team interprofessionali (c2=30,02; p<,01), mentre studenti da varie discipline apprendono ad approggiarsi meno a colleghi con la stessa estrazione professionale durante la cura dei pazienti (c2=19,84; p<,01). Conclusioni. La valutazione contestuale (scenari clinici nel tempo e mondo reale) dell’Apprendimento Interprofessionale attraverso EMAeA mostra il suo valore e la sua applicabilità durante l’attuale pandemia di covid-19, quando la valutazione dell’apprendimento non può verificarsi durante incontri faccia a faccia tra studenti e docenti. Tutti gli studenti nelle professioni sanitarie, compresi quelli a rotazione psichiatrica, potrebbero registrare i loro progressi, valutazioni, riflessioni e risposte all’assistenza coordinata in team interprofessionali, senza bisogno di un contatto diretto con i loro tutor clinici, e trattare pazienti con malattie mentali e fisiche, compresi i pazienti covid-19 positivi.

PAROLE CHIAVE: covid-19, educazione interprofessionale, valutazione momentanea ecologica, app, psichiatria.



During the current covid-19 pandemic, healthcare students had to stop their face-to-face college and university hospitals’ attendance. This last has resulted in a change in how learning and assessment of psychiatric and interprofessional practice occur. The reduction or suspension of face-to-face encounters of healthcare professionals in training with their university teachers, mentors, and clinical tutors has resulted in a revival of distance learning skills, training, assessment, or interventions, supported and delivered by the Internet and smartphone technology.

The covid-19 pandemic has also increased mental-health professionals’ need to liaise and collaborate with other health professionals and their clinical tutors via virtual technology, including smartphone educational apps. Similarly, medical teachers and healthcare students cannot ensure regular clinical teaching attendance as covid-19 has restricted travels between geographical areas, hence stopping medical education’s major stakeholders’ learning. It is then witnessed that mental health professionals, although still active in their psychiatric or clinical wards, risk having halted the assessment of their skills from their tutors as these last are physically absent. Neither is it possible to make amendments to teaching modules as students’ self-reflective and evidence-based practice cannot be directly monitored and scored by medical teachers lacking a direct contact with students.

The authors of the current study’s propose a model of assessment of psychiatric interprofessional learning and practice consisting of an integrative model using a dedicated app for distance valuation. The distance assessment model via virtual Ecological Momentary e-Assessment (EMeA) has allowed healthcare students in psychiatry and other medicine branches to overcome educational restrictions during the Coronavirus pandemic. Besides, distance assessment can still satisfy the academic requirements without jeopardizing the pandemic lockdown and learners’ health. The current research aims to propose a model of EMeA, which was piloted before the covid-19 pandemic, then applied during covid-19 restrictions. The study aims to offer a model for psychiatric and medical interprofessional practice e-assessment. The aim is reached by utilizing an integrative paradigm and using a dedicated app and the Internet to evaluate students’ progress to overcome covid-19 educational restrictions. Hence, the present study aims measure the impact that EMeA assessment based on smartphone and Internet technology can have on a self-reflective interprofessional practice between undergraduate healthcare students from different specialties (e.g., psychiatry, medicine, surgery, orthopedics, and others) when direct evaluation is restriced, as during covid-19 pandemic.

EMA in general

The Centre for the Advancement of Interprofessional Education (CAIPE) defines interprofessional education as «when members or students of two or more professions learn with, from and about each other to improve collaboration and the quality of care and services»1. A Cochrane review on Interprofessional Education (IPE) determined that when IPE’s effectiveness is measured against no educational intervention, IPE produces positive outcomes in terms of department culture, patient satisfaction, collaborative team behavior, and management of care, and reduction of clinical error rates2. The review also showed that IPE intervention had no impact on either professional practice or patient care2. The typical methods of inquiry to collect research data in the studies included in the review were formal educational/training interventions, interviews, observations, questionnaires and audits of documents, or a combination of these. The method used in the present study takes a different approach of inquiry, namely EMA (Ecological Momentary Assessment). EMA measurements focus on information about learners’ transitory state, including emotional, cognitive, and behavioral events that are near in time to their learning event and that occur in the learner’s natural environment or standard setting3-5. EMA is performed by learners accessing desired surveys, questionnaires, or assessment tools on electronic, online platforms, and devices (EMeA).

This continuous real-time appraisal reduces the recall bias, also providing a deeper understanding of social interactions3,4,6-9. EMA also allows instant feedback to medical and nursing students and teachers who will be able to make the desired changes to an ongoing clinical program10. A Best Evidence Medical Education systematic review suggests that mobile technology (e.g., smartphones) is used by medical students to improve assessment captured in the real scenarios and to trace the development of skills11.

Interprofessional Learning in Psychiatric Care

Psychiatric patients are carriers of multiple needs requiring a coordinated effort by part of various agencies and professionals aiming to deliver integrated care12. A systematic review of 173 studies reports positive outcomes in psychiatric care when IPE is used by personnel involved in mental health care with a length of course of one to two days13. Yet, the research confirms some restrain in implementing IPE in a mental health setting, inclusive of limited involvement of medical staff in interprofessional initiatives, lack of support from senior managers, and limited fundings14. Several studies report that IPE in mental health increases knowledge and skills in collaborative psychiatric care while improving the outcomes for patients with severe psychiatric illnesses15.

In a study assessing the quality of interprofessional skills in mental health professionals, it resulted that psychiatrists and psychologists received a poorer rating in communication skills and interpersonal skills; in contrast, social workers and community psychiatric nurses received higher scores in interpersonal skills16. Service users who benefit from collaborative teams promoting interprofessional practice report personal benefits in their social life and quality of life17.

IPE for psychiatric professionals has been suggested as a solution to address several challenges in collaborative care, such as limited appreciation of the support from other professions, unenthusiastic approaches to teamwork, and insufficient knowledge and skills on how to work collaboratively18. Besides, the technology has been applied to current covid-19 impact in college and university hospitals’ attendance and reduced face-to-face assessment of learning in psychiatric care.

Implementing interprofessional practice in undergraduate study

This current study aimed to verify if EMeA of IPE improves healthcare students’ mindfulness of how the educational tasks (challenges) and their levels of expertise (skills) can affect their care quality, emotional reactions to training, and their attitudes to collaborate in teams. The self-reflective practice explored in the current research is derived from Bleakley’s19 and Hébert’s20 concept of ‘reflection-as-action,’ in which the process of reflecting on one’s learning becomes, as such, the basis of knowledge. Bleakley19 proposes that reflective practice helps learners develop mindfulness – the «ability to think about their thinking». Besides, EMeA of students’ reactions to interprofessional practice and their current approaches to teamwork were also included in the study to determine if IPE impacted them. This study was carried out in several clinical departments that have pioneered EMeA of student progress in IPE by smartphone technology21,22.

Our previous research confirmed that contextual e-assessment with mobile technology applied in real scenarios and real-time could improve medical education by reducing recall biases when self-assessment is promoted23-25. Our recent studies also suggest that continuous self-reflective learning reinforced by e-assessment is achieved by constant appraisals of students’ activities, like when learners are still acting in their clinical scenarios, performing in the patient’s room, or waiting for teachers’ tutorials inside the hospital wards26.


All cross-sectional studies (Table 3) rejected the Null Hypothesis with 59% of learners found that interprofessional practice and skills met the expected standards (c2 GFT=29.78; p<.01), 58% felt relaxed during IPE activities (c2 GFT=96.41; p<.01), 73% totally agreed that IPE skills could be transferred to daily practice (c2 GFT=235.76; p<.01), and 40% felt satisfied with learning activities (c2 GFT=51.65; p<.01). Similarly, 77% of patients were aware of an integrated approach when treated by the IPE team (c2 GFT=86.18; p<.01).

In the study on the neurophysiological variations of heart rate during IPE (Table 4), there was no statistically significant difference in the pulse rate between and after IPE activities. However, the discussion of a clinical case in the ward round had the highest mean values (74.46 ± 13.51).

All the longitudinal (entry and exit point) survey scales measuring changes in attitudes and interprofessional practice (Table 5) showed statistically significant differences between expected (before IPE) and experienced (after IPE) outcomes.

Changes at entry and exit points rejected Ho while capturing increased percentages of learners reporting high patient satisfaction with IPE (55%; c2=22.54; p<.01), claiming very good quality of care delivered by the IPE team (56%; c2= 30.02; p<.01), selecting less frequently learners from same educational background for support (13%; c2=19.84; p<.01), experiencing courtesy and humility in the team more than expected (70%; c2=2.48; p<.n.s.), and reporting an increased likelihood of conveying and communicating the own plan to the team (70%; c2=22.54; p<.01).

In the RIPLS scale (Table 6), the mean scores of those who totally agree or disagree with the statements both before IPE (57.45%; c2 GFT=71.12; p<.01) and after IPE (59.42%; c2 GFT=77.69; p<.01) rejected the Null Hypothesis individually but not combined (t=0.522; p=n.s.) similarly the positive attitudes towards IPE remained during the training. More than 70% of learners reported that studying with other students in training improved their working relations with other professionals. In contrast, joint studies helped them think more positively about different professions.


Research aim

The current research aims to assess learners’ beliefs, feelings, and approaches about the interprofessional practice and quality of care when collaborating with colleagues from other health care professions. Therefore, the current research investigates how contextual e-assessment of learning moments during IPE can improve healthcare students’ mindfulness of how IPE learning can affect the quality of their care, their emotional reactions, and their interpersonal skills in working in teams. All participating centres, training hospitals in Europe, adopted the same methodology of online collection of survey data. A flier and a website were created to direct learners to the target information.

Research hypothesis

Research hypotheses: the Null Hypothesis H01 was that in cross-sectional studies, the surveys’ answers were normally distributed. In before-after studies, the Null Hypothesis H02 was that there was no difference in the answers to the surveys at the entry and exit point, suggesting no change in collaborative attitudes before and after IPE training.

Research objectives

Objective N. 1: To assess, during, before, and after IPE course changes in students’ satisfaction in interprofessional care, expected quality of care, type of interprofessional and communication skills, degree of satisfaction, and other attitudes as expressed by the RIPLS scale. More specifically, our study’s first objective was to use EMA to measure any specific changes in interprofessional practice and mindfulness by using an ad-hoc app and electronically collect healthcare students’ answers immediately before, during, and after particular IPE training moments. In fact, in case medical teachers expect changes in student’s self-reflective practice, these transformations will be better promoted by dynamically recording learners’ emotions, attitudes, and practice via constant EMA, as when students use their educational app from their smartphone and in the targeted clinical scenarios. The current research findings will be used to make recommendations about the usefulness of continuous ecological evaluations as pedagogically suitable to implement educational programs to constantly reinforce students’ good clinical practice towards teams and improve their patient’s care.

Objective 2: To assess, along IPE, neurophysiological changes of heart-rate during interprofessional practice, academic emotions and skills in working as interprofessional teams, and degree of satisfaction in IPE activities.

Research methodology

The current study is a quantitative research approach based on online surveys to assess real-time developments in students’ attitudes, behaviors, and emotions. Some surveys are longitudinal and determine entry and exit changes in attitudes. Other surveys are cross-sections as learners reply to the questionnaires in the most crucial moments of their IPE learning.

Ethical aspects

The research was conducted according to the Helsinki Declaration for medical research. Participation to the study was voluntary as it occurred by accessing the surveys linked to the IPE-App downloadable on the personal smartphone.

Recruitment and sampling

The total population that accessed the IPE-App consisted of 228 undergraduate healthcare students undergoing a training in interprofessional education. (Table 1).

Convenience sampling was applied, as each student freely chose to participate and was already taking part in the IPE activity in the own participating centre. Each interprofessional team consisted of nursing students, medical students, student physiotherapists, student occupational therapists, and student clinical psychologists. A flier and website explained the purpose of the research and how to access the App.

IPE App and Online Surveys

The research team created an app (IPE-Education) for 3G/4G smartphones downloadable for iOS and Android smartphones. When learners provided answers to the assessments, they were instantly stored in the SurveyMonkey ( databank. The IPE app contained a link to each survey. Furthermore, learners were free to access the surveys by selecting which subsection they were more interested in and which subsection more closely matched their ongoing IPE activities, e.g., team discussion, self-reflection. The attitude study was finalized at the start and conclusion of the IPE (baseline and outcome), while the other surveys were accessed during the IPE activities at the students’ discretion. Several fliers, posters, and a website provided additional instructions regarding the research. The research adopted the following e-assessment phases:

• Ongoing e-assessments: learners accessed the app and e-surveys during or shortly after the conclusion of the IPE learning moment, or before leaving the scene.

• Before/after e-assessments: the students provided answers at the beginning and end of the entire IPE training.

Learning moments in IPE

The independent variables used for the statistical analysis represented the IPE activities characterizing significant learning moments, such as the following:

• the discussion of a clinical case in the ward round, attended by all learners and their teachers;

• the provision of care to patients as an interprofessional team;

• self-reflection in a group or ward round after daily IPE activities;

• interactions between the team and teachers.

Pulse oximeter

The learners always carried a pulse oximeter to record their heart rates before and after those IPE activities that they deemed significant, for instance, because of the stress connected with them.

Multi-item surveys

Several multi-item surveys were used to capture interprofessional challenges and skills, emotions, satisfaction, and readiness. The items were parts of the original and validated surveys, as explained below. Table 2 showed when each survey was supposed to be used during the IPE course.

Challenges and Skills in IPE. A four-item questionnaire was used with two sub-study questionnaires covering interprofessional challenges and the skills needed to deal with them. It contains three answers: ‘above-expected standards’, ‘below-expected standards’, and ‘meets expected standards’. Learners were asked to access the surveys soon after the learning moment occurred. Two concepts related to emotional intelligence, ‘challenge and skills’, were introduced by the Milan School’s Eight-Channel Model27,28. In the current context, a ‘challenge’ is interpreted as any clinical and educational activity students perform as a team, for example, treating a patient with serious clinical conditions and deciding on a coordinated care plan as a multidisciplinary team. To master a medical or didactic challenge, a student needs to prove to have sufficient ‘skills’ to reach her or his goal or challenge. Consequently, skills and challenges can be expressed in terms of what students expect to find regarding learning goals as measured against their standards27,28. According to the Milan School, feelings of self-mastery and constructive learning occur when learners face challenges and, in dealing with them, use skills that are above their usual standards27,28.

Academic emotions. A four-item questionnaire was used to measure this area with eight single-choice answers. Here, learners expressed whether they felt ‘afraid, concerned, bored, relaxed, in control, excited, happy, or glad’ during each learning moment. The theory suggests that educational events can lead to learning-related emotions, including boredom, apathy, worry, anxiety, arousal, flow, control, and relaxation27,28. Hence, during interprofessional learning, students can experience a blend of these ‘academic emotions’ which favor or hinder students’ performances in interprofessional teams and their learning inclinations21.

Student satisfaction was assessed via a four-item questionnaire with responses on a six-point Likert Scale. This questionnaire evaluated the learners’ satisfaction with each IPE component from ‘Very Dissatisfied’ to ‘Very Satisfied’.

Heart Rate was measured before and after the four IPE learning events. The American Psychological Association reports a rapid heart rate as a sign of anxiety29. With a pulse oximeter, learners, recorded their heart rates before and after the IPE learning moments and reported the app results.

The Readiness for Interprofessional Learning Scale (RIPLS) Questionnaire is a 19-item questionnaire with answers on a five-point Likert scale (from ‘Totally disagree’ to ‘Totally Agree’). The RIPLS was an adaptation of Curran et al.’s30 variation of Parsell and Bligh’s31 Readiness for Interprofessional Learning Scale, which concerns several aspects of interprofessional practice. The questionnaire was adapted to incorporate expected and experienced IPE learning practices as it was accessed at the beginning and end of the IPE course.

One-item surveys

Several one-item surveys were used to detect constructs exemplified below. Each of these surveys used its scale and were only accessed one at a time to avoid confusion in choosing the answer. Table 2 showed when the one-item surveys were supposed to be used during the IPE course:

• Patients’ awareness of the interprofessional team acting as an integrated squad.

• Interprofessional skills can be transferred from the IPE course to daily practice.

• The expected and experienced quality of care for patients treated by the interprofessional team.

• The professional role of the person to contact for advice.

• Expected and experienced patient satisfaction.

• Expected and experienced interpersonal approach during IPE.

• Expected and experienced interpersonal communication during IPE.

Statistical methods

Test statistics were cross-sectional for questionnaires accessed once and longitudinal, at entry and exit point. As all the data were reported as percentages, we used Chi-square analysis, inclusive of Chi-Square Goodness-of-Fit test (c2 GFT), Chi-square (c2), and t-test32.


The current study results have confirmed that specific changes in interprofessional practice and mindfulness can be captured by Ecological Momentary e-Assessment. The ecological assessment and the triangulation of appraisal tools have provided evidence that healthcare students can become mindful of how IPE learning can affect their care quality, emotional reactions, and interpersonal skills in working in teams. Simultaneously, medical teachers utilizing the described methods of e-assessment benefitted from accessing the current results of students’ progress collected in real-time and real scenarios while learners were performing specific tasks in their clinical training. The education technology proved to be applicable during current restrictions of direct assessment due to covid-19 lockdown and constraints in the immediate assistance by medical teachers of learners’ progress.

Teachers could access the results of the surveys on a specific SurveyMonkey App. Hence, Ecological Momentary e-Assessment emerges as a modern instrument to implement educational programs aiming to a constant and direct reinforcement of students’ good clinical practice towards teams and improve their patients’ care. The tool becomes basic during the assessment and reduces face-to-face encounters between teachers and learners during the Coronavirus pandemic. The current research results suggest that Ecological Momentary e-Assessment can help IPE training by allowing a rapid evaluation and remedial teaching more than other less ecologically structured tools. Besides, as the current study has shown, identifying specific signposts in students’ skills, behaviors, emotions, and reactions accessed via educational apps are more reliable when students can actively use smartphone technology at particular IPE training moments.

The current research findings support the theory that learners’ mindfulness frequently complements reflection about their feelings and thoughts33. Mindfulness in daily practice can be supported by distance technology of educational assessment without the need for medical teachers’ physical presence. EMA shows that it can improve the impact of learning and self-reflective interprofessional practice during social lockdowns in the covid-19 pandemic, or in any other instances where the face-to-face encounters of learners between themselves and teachers are restricted; for example, they all reside in distant geographical areas.

Additionally, mindfulness and learning are linked: learners who can self-assess their progress are better achievers than students who do not engage in self-reflective practice.34 The students accessing the surveys during and immediately after their IPE training could both apply ‘reflection-in-action’ because this last occurred while educational events ongoing, and ‘reflection-on-action,’ when self-evaluation occurred soon after learning activities were concluded35. Besides, the study suggests that E-assessments via smartphones improve medical students’ recall and self-awareness of salient events linked to their core training36 while allowing them to access the contextual experiences and emotions in IPE learning21 and to reinforce learning through reiteration37.

The findings emerging from academic emotions assessment confirm the flow theory introduced by Csikszentmihalyi38, where productive learning typically occurs when students find themselves experiencing flow emotions. Flow emotions often refer to feelings of happiness and mastery that occur when learners face challenges that require skills that are slightly above their own expected standard; these are considered optimal experiences for learning27,28. The importance of focusing on academic emotions, and the finding of positive emotions, relaxation, and satisfaction in the students assessed coordinates with Knowles’s principles of andragogy, which suggest that a learning environment should provide learners with physical reassurance, mutual trust, and esteem39. However, as indicated by the Milan School, students should have expressed to perform above their expected standards to optimize their learning. Instead, they performed at ‘expected standard,’ meaning that the learning environment did not appear to offer many educational challenges out of the expected ones in some instances.

The results of our research also show that learners awareness of the implication of collaborative practice on patient quality of care supports the theory of Van Manen40; this last designates the process of self-reflective practice as ‘pathic knowledge,’ that is, learning advances when students engage in shared skills with companions and during shared clinical activities. As also emerging from the current research, students appreciated the impact of interprofessional communication and sharing of care plans as a route to reinforce the collaborative practice. Small groups facilitate mutual assistance and problem-solving41, and new models of behaviors can be achieved through direct exposure or by observing others’ conduct42. During the current research, the students who used their smartphone-based e-assessment also emerged as more sensitive to interprofessional events and more inclined to work in interprofessional teams in the future. This finding supports the theory that frequent questions promote self-awareness, thinking, and observation43. The current study showed that during IPE training, group learning becomes ‘communicative learning’44 in which students perform progressively as a joint system, interacting reciprocally with an awareness of the rules of mutual engagement and support and communicating clinical information, decisions, and plans for the care of shared patients.

Students also indicated that interprofessional practice was beneficial in patient satisfaction and quality of care. In terms of interpersonal support, there was a shift from relying on peer support before IPE to approaching the tutor and the instructor for advice. About the quality of the interpersonal approach, after IPE, learners reported increased satisfaction in all dimensions hence perceiving their peers as courteous, empathic, humble, credible, transparent, and skillful. In terms of communication skills, learners felt that they could explore and discuss their clinical plans with colleagues and could get support from the team. The RIPLS survey captured students’ satisfaction in all interprofessional practice, with the majority of learners acknowledging that interprofessional training improved their daily working relations with other professionals and that IPE made them appreciate more the diversity of skills in interprofessional teams and more the skills to work in collaboration with other professionals.

One limitation of the current research was that the innovation of using mobile technology could account for the engagement and improvement, rather than the technology itself45. One concern is that smartphone-based education might lead to superficial learning due to the rapid availability of information and a lack of time to internalize knowledge46. Smartphones are also a distraction in class and can divert learners’ attention46. Furthermore, one should not assume that all trainees possess a suitable mobile device47.


The current research has provided a novel way of promoting the self-reflective practice in interprofessional education with the use of EMeA in psychiatric training. The focus on capturing ongoing learners’ emotions, perceived levels of skills and challenges, and the neurophysiological assessment of educational events appear to be a promising method in this direction. Its application during covid-19 pandemic has been reported. Also, the perceptions of how academic and clinical events impact on learners’ emotions and level of stress can open a new venue to improve quality of care and medical education. Besides, mindfulness of educational emotions promoted by EMA can advance safety and quality of care into the clinical practice by reducing the risk that emotional and neurophysiological events could instead impact on healthcare students’ performance.

Conflict of interests: the authors have no conflict of interests to declare.


1. Barr H, Low H. Introducing interprofessional education. Fareham: CAIPE, 2013.

2. Reeves S, Perrier L, Goldman J, Freeth D, Zwarenstein M. Interprofessional education: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2013; 3: CD002213.

3. Stone A. The science of real-time data capture. Oxford: Oxford Univiversity Press, 2007.

4. Shiffman S, Stone AA, Hufford MR. Ecological Momentary Assessment. Annu Rev Clin Psychol 2008; 4: 1-32.

5. Stone A, Shiffman S. Ecological Momentary Assessment (Ema) in Behavioral Medicine. Ann Behav Med 1994; 16: 199-202.

6. Goodwin MS, Velicer WF, Intille SS. Telemetric monitoring in the behaviour sciences. Behav Res Methods 2008; 40: 328-41.

7. Yoshiuchi K, Yamamoto Y, Akabayashi A. Application of Ecological Momentary Assessment in stress-related diseases. Biopsychosoc Med 2008; 2: 1-6.

8. Willett LL, Houston TK, Heudebert GR, Estrada C. Use of Ecological Momentary Assessment to determine which structural factors impact perceived teaching quality of attending rounds. J Grad Med Educ 2012; 4: 322-8.

9. Moskowitz DS, Young NY. Ecological Momentary Assessment: what it is and why it is a method of the future in clinical psychopharmacology. J Psychiatry Neurosci 2006; 31: 13-20.

10. Ellaway R, Masters K. AMEE Guide 32: e-Learning in medical education Part 1: Learning, teaching and assessment. Med Teach 2008; 30: 455-73.

11. Maudsley G, Taylor D, Allam O, Garner J, Calinici T, Linkman K. A Best Evidence Medical Education (BEME) systematic review of: what works best for health professions students using mobile (hand-held) devices for educational support on clinical placements? BEME Guide No. 52. Med Teach 2018; 41: 1-15.

12. Pauzé E, Reeves S. Examining the effects of interprofessional education on mental health providers: Findings from an updated systematic review. J Ment Health 2010; 19: 258-71.

13. Reeves S. A systematic review of the effects of interprofessional education on staff involved in the care of adults with mental health problems. J Psychiatr Ment Health Nurs 2001; 8: 533-42.

14. Reeves S, Freeth D. Re-examining the evaluation of interprofessional education for community mental health teams with a different lens: understanding presage, process and product factors. J Psychiatr Ment Health Nurs 2006; 13: 765-70.

15. Curran V, Heath O, Adey T, et al. An approach to integrating interprofessional education in collaborative mental health care. Acad Psychiatry. 2012; 36: 91-95.

16. Barnes D, Carpenter J, Dickinson C. Interprofessional education for community mental health: attitudes to community care and professional stereotypes. Soc Work Education 2000; 19: 565-83.

17. Carpenter J, Barnes D, Dickinson C, David Wooff D. Outcomes of interprofessional education for Community Mental Health Services in England: the longitudinal evaluation of a postgraduate programme. J Interprof Care 2006; 20: 145-61.

18. Maranzan KA. Interprofessional education in mental health: an opportunity to reduce mental illness stigma, J Interprof Care 2016; 30: 370-7.

19. Bleakley A. From reflective practice to holistic reflexivity. Stud High Educ 1999; 24: 315-30.

20. Hébert C. Knowing and/or experiencing: a critical examination of the reflective models of John Dewey and Donald Schön. Reflective Practice 2015; 16: 361-71.

21. Lachman H. Contextual activity sampling: a method to develop clinical interprofessional education (unpublished doctoral dissertation), Karolinska Institutet, Stockholm, Sweden, 2013.

22. Lachman H, Ponzer S, Johansoon U-B, Benson L, Karlgren K. Capturing students’ learning experiences and academic emotions at an interprofessional training ward. J Interprof Care 2013; 27: 137-45.

23. Lazzari C, Masiello I. Ecological Momentary Assessment and physiological self-evaluation improve mindfulness of stress-related events during interprofessional training. Meta-analysis of a pilot study. Eur Psychiat 2017; 41: S897.

24. Lazzari C, Masiello I. How satisfied are patients with interprofessional teams? Meta-analysis of a pilot study. Eur Psychiat 2017; 41: S298-9.

25. Lazzari C, Masiello I. How satisfied are undergraduate students with interprofessional training? Meta-analysis of a pilot study. Eur Psychiatry 2017; 41: S299.

26. Lazzari C, Masiello I. Flow experiences improve mindfulness of educational emotions during interprofessional training. Meta-analysis of a pilot study. Eur Psychiatry 2017; 4: S897.

27. Delle Fave A, Massimi F. The relevance of subjective well-being to social policies: optimal experience and tailored intervention. In: Huppert F, Keverne B, Baylis N. (eds). The science of well-being. Oxford: Oxford University Press, 2005.

28. Delle Fave A, Massimi F, Bassi M. Psychological selection and optimal experience across cultures. Dordrecht, Heidelberg, London, New York: Springer, 2011.

29. APA, American Psychological Association. Listening to the warning signs of stress. [Online] Available at: 2015. (Accessed: 1 August 2015).

30. Curran VR, Sharpe D, Forristall J. Attitudes of health sciences faculty members towards interprofessional teamwork and education. Med Educ 2007; 41: 892-6.

31. Parsell G, Bligh J. The development of a questionnaire to assess the readiness of health care students for interprofessional learning (RIPLS). Med Educ 1999; 33: 95-100.

32. Stangroom S. Chi-Square Test Calculator. Social Science Statistics. Available at: (accessed November 13, 2020).

33. Black P, William D. Assessment and classroom learning. Assessment in Education. 2008; 5: 7-74.

34. Andrade H, and Valtcheva A. Promoting learning and achievement through self-assessment. Theor Pract 2009; 48: 12-9.

35. Schön DA. Educating the reflective practitioner: toward a new design for teaching and learning in the professions. Jossey-Bass higher education series. San Francisco, CA: Jossey-Bass, 1987.

36. Runyan JD, Steinke EG. Virtues, Ecological Momentary Assessment/intervention and smartphone technology. Front Psychol 2015; 6: 1-24.

37. Davies B, Rafique J, Vincent TR, et al. Mobile medical education (MoMEd) – How mobile information resources contribute to learning for undergraduate clinical students – A mixed methods study. BMC Med Educ 2012; 12: 1-11.

38. Csikszentmihalyi M. Flow: the psychology of happiness: the classic work on how to achieve happiness. London: Rider, 2002.

39. Melick RR, Melick S. Teaching that transforms. Nashville: Band H Publishing group, 2010.

40. Van Manen M. Phenomenology of practice. (2nd ed.) New York: Routledge, 2016.

41. Fraser SA, Greenhalgh T. Coping with complexity. BMJ 2001; 323: 799-803.

42. Bandura A. Social Learning Theory. New York, General Learning Press, 1971.

43. Malthouse R, Watts M, Roffey-Barensten J. Reflective questions, self-questioning and managing professionally situated practice. Res Educ 2015; 94: 71-87.

44. Mezirow J. Transformative learning: theory to practice. New Directions for Adults and Continuing Education 1997; 74: 5-12.

45. Cocciolo A, Rabina D. Does place affect user engagement and understanding? J Doc 2013; 69: 98-120.

46. Wallace S, Clark C, White J 2012. ‘It’s on my iPhone’: attitudes to the use of mobile computing devices in medical education, a mixed-methods study. BMJ Open 2012; 2: e001099.

47. Bullock A, de Jong PGM. Technology-enhanced learning. In: Swanwich T (ed.). Understanding medical education: evidence, theory and practice, (2nd ed.). Chichester: John Wiley & Sons, 2014.