The goal of medical education is to produce a physician workforce capable of delivering high-quality care to our patients and communities. Remarkable advances in biomedical sciences over the past 50 years and tremendous growth in technology and medical knowledge—coupled with a greater emphasis on team-based approaches—have transformed health care delivery. Despite this, the medical education system has struggled to incorporate new technologies to personalize training. This recording discusses how aligning education with current and future needs requires new paradigms to overcome the current inflexibilities, inefficiencies, and inequities.
Sign in to take quiz and track your certificates
The AMA Ed Hub™ is a unified education portal that provides a personalized experience for physicians and their care teams to keep current, increase their professional satisfaction, claim continuing education credits and continuously improve the care they provide–leading to real world outcomes of better healthcare and better health for their patients.
Joena Chun: Welcome to the AMA Innovations in Medical Education webinar, Precision Education, a Conceptual Model for Medicine. We would like to introduce your host for today's webinar, Dr Maya Hammoud.
[00:00:14] Dr Hammoud serves as a senior advisor for Medical Education Innovation for the AMA. She is also associate chair for Education, professor of Obstetrics and Gynecology, and professor of Learning Health Sciences at the University of Michigan Medical School. I will now turn it over to Dr Hammoud. Dr Hammoud?
Maya M. Hammoud, MD, MBA: [00:00:37] Thank you, Joena, and thank you all for joining us. The goal of medical education is to produce a physician workforce capable of delivering high quality care to our patients and communities. Remarkable advances in biomedical sciences over the past 50 years and tremendous growth in technology and medical knowledge coupled with a greater emphasis on team-based approaches have transformed health care delivery.
[00:00:57] Despite this, the medical education system has struggled to incorporate new technologies to personalize training. Aligning education with current and future needs requires new paradigms to overcome the current inflexibilities, inefficiencies, and inequities. At the end of this webinar, you should be able to recognize challenges to learning outcomes due to limited personalization in our current medical education system, discuss how precision education can optimize physicians' lifelong learning by personalizing medical education, review an example of how precision education can be leveraged to impact broad high impact agendas.
[00:01:32] Now, I would like to introduce you to today's speakers and remind you that the views of the presenters are their own and don't necessarily reflect those of their institutions or of any other presenter or their respective institution. Dr Charles Prober is founding executive director of the Stanford Center for Health Education and senior associate vice provost for health education at Stanford University, Stanford, California.
[00:01:53] He is a professor of pediatrics, microbiology and immunology and an expert in pediatric infectious diseases. Dr Prober previously served as senior associate dean for medical education at Stanford. His perspectives and commentaries in the New England Journal of Medicine, Academic Medicine on the Future of Medical Educations have been widely cited.
[00:02:11] Dr Sanjay Desai is the chief academic officer and group vice president of medical education at the American Medical Association. In this role, he leads the next phase of AMA efforts to drive the future of medicine by the emerging medical education training and lifelong learning. Prior to joining the AMA, Dr Desai was Professor of Medicine, Director of the Osler Medical Residency and Vice Chair for Education at Johns Hopkins University School of Medicine, where he led the development of innovative programs, curricula, and research aimed at improving medical student and resident well-being, ensuring physicians in training are better prepared to navigate health systems.
[00:02:46] Dr Mark Triola is the associate dean for educational informatics and the founding director of the Institute of Innovation in Medical Education at NYU Grossman School of Medicine. Dr Triola's research focuses on the use of AI tools to efficiently personalize education and give new insights to programs and coaches. His lab develops new learning technologies, AI-driven educational interventions, and defines educationally sensitive patient and system outcomes that can be used to assess training. Thank you for being with us today. And now I would like to turn it over to Dr Prober.
Charles G. Prober, MD: [00:03:20] So I'm very excited to be here today.
Maya, thank you so much for your introduction. And I'd like to thank my colleague, Dr Desai, for involving me in his rather ambitious project of changing medical education, and also my accompanier, Dr Triola, who will be the last speaker. I have to say, when I first spoke to Sanjay Desai about the vision of precision medical education, he asked us who went to the initial summit that he organized, "What do you think of when you hear the term precision medical education?"
[00:03:57] And my response was, and I apologize for it, Sanjay, was I think of an oxymoron. Because in terms of my lens of watching medical education, especially at the UME level, I think one of the problems that we have is a lack of precision. I'll come back to that. I chose this cartoon for my first slide because I'm supposed to address the objective that you see listed, recognize challenges to learning outcomes due to limited personalization in our current medical education system.
[00:04:30] I scratched my head about exactly how I was going to address that particular objective and I came up with the cartoon which you see is that one size fits all is really not the right model for many things that we do. But paradoxically, or maybe it's not a paradox, I think for undergraduate medical education, we should have a size that is evident to everybody.
[00:04:55] So precision health, this is a shout out to the Dean of our medical school here at Stanford, Lloyd Minor. It's a book that he wrote about precision health. And I show this picture, 80 points, of course, of my school, But also to note that in my opinion, the end game of precision education is precision health.
[00:05:14] We really want to know how to best address the health problems that face our society in general and the patients that are sitting in front of us very specifically. Precision is an oft-used word in the health care system. I call it the pea soup of precision in health care, and I'm sure you can come up with other P's to add to the soup, but those that are oft used are, we want to be precise.
[00:05:41] Of course, in treating somebody with a specific disorder, we do want to be precise. Personalized has been in our vernacular for many years at this point, and yes, we do want to have personalized medicine for those that come to our attention, but in the context of education, it's a little bit more complicated. Predictive is very important in preventing diseases that befall the population. So those P's always come up. Preemptive is sort of part of that package. And prescriptive, of course, what else can a doctor do but prescribe? We want to be proactive in terms of anticipating the problems, and then comes the term precision medicine, which has been around for a long time. Precision cures when we're facing our patients. Precision health, which is what Lloyd termed to say that we want to be precise about keeping our population well, and then the penultimate is about precision education. So let me show my view for a moment of precision education or medical education at the UME level.
[00:06:46] I strongly believe that we have to be very prescriptive and proactive about the bricks that we put into the foundation of our educational house. I think that we do a disservice to our students and ultimately therefore to our residents and to our practicing physicians and indeed to our patients if we don't lay a foundation, which everybody understands is the necessary fund of knowledge that you need ultimately to be a practicing physician.
[00:07:16] The foundations need to be described and they need to be prescribed. Ultimately, we will build a house and I show this very simple house for undergraduate medical education. At the heart of the house is our care for patients. And I show it as a simple house because I believe that we want education to be thoughtfully and carefully designed for graduates of all of our medical schools across the country and indeed around the world.
[00:07:43] So the house we need should be prescribed very carefully. The pillars that I show on the top right-hand corner and to come to in a moment in terms of what we need to embed and imbue in terms of our educational engines. And the little people shown on the bottom half of the slide, I believe as is said in the precision medical education document that coaching our learners through the different parts of their learning journey is very important, both in the medical school, during our residencies and indeed beyond.
[00:08:17] That performance analytics, as you're going to hear more about from Sanjay and Mark, are very important in understanding our learners. In undergraduate medical education, the performance analytics are perhaps the simplest in the sense that they're measuring the concepts, the foundations, and the structure of the house.
[00:08:37] And hopefully they're looking at our learners in terms of their growing clinical aptitude. And assessment allows us to know if we're doing the right things on behalf of our learners. In my opinion, the overarching principles that we should imbue all of our education journey in is that, first of all, what we teach, what we expect our residents, our students, our practicing physicians to learn must be relevant, and it must be relevant to the improvement of health in those that they're ultimately charged with.
[00:09:13] It has to be accessible. What I mean by this is that we cannot obfuscate our messages for our students and our residents and others. We have to put them in a way which they can understand and act upon. Of course, our education has to be equitable and inclusive if we're going to have an equitable and inclusive society for health care. Our learning journey has to foster curiosity at all levels from students through residents into practicing physician and to cultivate collaboration. Medicine, I think we all recognize, is a team sport. Human-centered design is back to the idea that the material is accessible to our learners, that they understand it in the context of their educational journey.
[00:10:02] One of my views of medical education that I articulated in an academic medicine article a a number of years ago with my friend and colleague Sal Khan. Many of you will recognize Sal Khan's name from the Khan Academy, which he established many years ago to address the problems in K through 12 education.
[00:10:24] And as we became colleagues, we spent more time together. We postulated that what he was doing in Khan Academy on behalf of global learners could be quite relevant to medical education, to re-imagining medical education. And we had this picture, not in color because I couldn't afford the color version of academic medicine, I don't think it exists, but we had this concept in that manuscript. And starting in the middle bar, it's reiterating my foundational bricks and the simple house, the core foundational principles of knowledge is our responsibility to define precisely and to have our learners at our 170 medical schools in North America be exposed to so that they can have the house when they finish medical school.
[00:11:15] The arrows that point up talk about making the content reachable and impactful to the learners, not through simply reading books or watching PowerPoint presentations or watching videos, but rather by doing, by doing in the anatomy lab, which is shown on the far left, which all of us remembered our medical education journey, by doing in the simulation suite of an operating room, by doing at the bedside of our patients, and indeed by doing in the far right of the top bar in our classrooms, making the classrooms peer-to-peer interactions, not just sages on the stage.
[00:11:53] But where precision education comes to play in terms of the medical schools, again, in my opinion with Sal, is that the arrow's going down. I think that we should figure out what each of our students is passionate about and really wants to drive in addition to their core medical education, whether it's basic science or community-based learning or biomedical ethics or health policy, whatever it is, and help those students go down into those deeper dives during their medical school experience.
[00:12:27] But the foundational bricks and the foundational house still has to exist. Just to contrast that, when I think about graduate medical education, I have the same picture, so I won't go through the different parts of the picture, except the picture in the middle is of course very different. The picture in the middle is depicting all of the different kinds of houses that our medical residents and fellows will be living in.
[00:12:54] And for them, we really do need to have different targets of precision in terms of what a house for a pediatrician looks like versus an internist, versus an obstetrician, versus a neurosurgeon and so forth. So that in graduate medical education, of course, there's a differentiation at the house level that we're building on behalf of all of our learners.
[00:13:17] But the other principles remain solid in my opinion, in the realization of precision medical education. We still have to identify the foundational bricks and make sure all of our pediatricians, in my case, acquire those bricks of knowledge. We have to work under the same principles that I articulated a few moments ago.
[00:13:36] We need to help our learners by coaching them and performance analytics, which both Sanjay and Mark are going to speak about, and we have to assess them along the way. And this cartoon, as I get to the end, is similar. There's my objective, same objective on the left. This cartoon says, "For a fair selection," talking to all the animals, "everybody has to take the same exam, "so please climb that tree."
[00:14:02] Well, that's not exactly fair, and that's where learning analytics that are specific to the learner and specific to the discipline come into play. And you're going to hear more about that from my two colleagues who follow. So this is a picture that tries to summarize medical education from the little books on the left, at undergraduate medical education to graduate medical education into continuing medical education, following the same principles and have our rocket ship expand into lifelong learning.
[00:14:35] And I'll end with the last slide, which says what I believe precision education should aim to promote. It should promote that we do tailor the final product of our educational journey to the learner that has traveled from UME to GME to CME. The differentiation of that particular suit that's being measured is much greater when you get to GME because of the different pathways.
[00:15:03] And of course, when we're at our own practices, it's not specifically tailored as much for medical students. Medical students must be graduated with the core skills that are necessary for future practicing physicians. And as we go through this journey, we have to have our learners maintain their growth mindset, always pushing them to identify their passions to follow their passions and to grow in their educational journey as they live their professional lives.
[00:15:31] Maya, send it back to you for transferring with Sanjay.
Hammoud: [00:15:35] Thank you so much. Sanjay, you should be able to control it now.
Sanjay Desai, MD: [00:15:39] You have it? I got now. Thank you very much, Dr Prober and Dr Amul for the introduction. So I will jump right in and have this slide also invoking Sal Khan. I want to build maybe on the theme of what Dr Pogba introduced, which is around the idea that there's foundational elements to every stage of training in medical education. And building upon that, the realization that every learner will bring onto that journey or onto that role of building this house a different skill set and will traverse it at a different pace.
[00:16:10] And so the idea is, how do we match the educational experience and learning environment to the individual learner. And so we jumped into precision education. As Dr Prober mentioned, our first role or activity was to host a summit. And this was because we didn't really know what precision education is. And it's that elephant that Dr Prober highlighted. And so we brought together a group of colleagues and I showed this picture because it has Sal Kahneman, who I think is a pioneer in this space. And hopefully what he has done to transform K-12 is our ambition to transform medical education. And so we brought together this group of experts that you'll see on this next slide that helped us understand what the opportunity is in medicine related to precision education.
[00:16:54] And the opportunity that we sought with this group of people was to bring together experience and wisdom from people that didn't necessarily build the house that we live in. So it really was meant to bring a group of people together that have used education technology, technology or education in different spaces, not just in medicine, to help us understand what the opportunity is for us to use data and technology to create more effective lifelong learning. And with that, we were able to generate a conceptual model of precision education. So my next slide will highlight for you what that conceptual model is. And then I'll take you also through some use cases to try to make this more tangible in terms of where we are today, some leading edges related to precision education.
[00:17:35]: So our conceptual model is that precision medical education is a system that can transform lifelong learning by using data and technology. If we do this effectively, it will help personalize our educational journey, it'll help increase the efficiency of our journey, and then it will ultimately improve patient outcomes, which is the reason that we have a medical education system to begin with.
[00:17:57] The goal of medical education is to produce a physician workforce that is effective at caring for our patients and our communities and their families. And so what are the attributes of such a system? So there are many, and I will not highlight all of these right now in the interest of time, but I want to call out just a couple of them that I think are important to speak to.
[00:18:15] So the first is the second bullet here, which is prioritizing learner agency. Really what we need in medical education to transform it to to the environment that we want is to have assessments and have education occur with learners, empowering the learner in the system and not have these assessments being done to learners. And that creates an entire different model of learning. It empowers the learner in the process and it actually, hopefully will create a system where learners embrace and seek out the educational experiences that they want. And this is really a cultural shift that we would be seeking. We need to make the journey far more efficient and reduce the amount of friction that exists, the unnecessary friction that exists in learning.
[00:18:57] And so part of this is harmonizing it in the physician workflow. And if you think about the practicing physician, this is probably where it's most evident, in which there is learning that occurs too often in parallel or outside of the workflow of caring for patients. And bringing that together using data and using technology will help us become more effective.
[00:19:17] Our ambition is to ensure that a precision education system for the country is interoperable. It is great if we create a system at New York University where Dr Trill is, and we'll speak about this, or where Dr Prober is at Stanford, but that is not enough for us to achieve the ambition for the country, and so interoperability is necessary.
[00:19:37] And our hope is to catalyze pilot processes, research, and implement learning that can occur in a way that once created and demonstrated to be valuable can be reproduced and can be scaled. And then finally, one that really leverages coaching and adaptive learning, which we know are necessary to personalize training as we move forward.
[00:19:58] And many other attributes that are listed here that you might review as well. I'm going to spend the rest of my time actually highlighting some use cases that are occurring throughout the country in different universities to demonstrate really what I think the opportunity is and hopefully spark imagination for what the potential is for precision education in these spaces.
[00:20:18] First, we'll talk about medical students, then we'll talk about trainees, and then we'll talk about practicing physicians. So for medical students, I want to highlight work done by Dr Kiani and her team at Texas Tech. And this is with third year medical students where they're creating dashboards to generate data, to inform learners on an individual basis about their ability and competency in writing notes.
[00:20:38] And the cartoon is drawn here, which shows that there is in the top left corner, an interface that they created that allows students to submit notes. And then there is an interactive dashboard that gamifies this for both the educator and for the student to enable them to learn precisely where they are in the continuum of creating notes and then maps them as you see in that bottom corner along that continuum and provides them directed and personalized feedback to improve their notes.
[00:21:03] And this is all built upon the master adaptive learning framework that you can see in the middle here. This is a pilot that's ongoing. Another very creative initiative in precision education for students is actually led by Dr Chiola, and he'll speak more about this in a few moments, but I wanna highlight this as another demonstration of effective precision learning at the medical student level.
[00:21:23] There are many opportunities also at the trainee level. The first example I'd like to share is with Dr Conrad Gleiber and his team at the University of Rochester, and this is for internal medicine residents, where they've created a digital ecosystem that allows them to aggregate data about each individual resident, and then based on that data, push them with nudges into spaces where they need further development. The way that this works, it actually leverages natural language processing. The application mines notes from the electronic medical record using natural language processing. It can assess the note. It can also identify who the attending physician is for that resident. By doing so, it'll create that pair and match the learner to their coach or their trainee or their attending in this situation.
[00:22:08] And too often, evaluations are not completed that matching doesn't occur. By creating that matching, it actually very quickly and simply creates opportunities for evaluation, which are done in an efficient way and then fed right back to the learner. In addition to that, however, by reading the note in an electronic way, it's able to create a dashboard for each learner so that they know exactly what types of patients they've been caring for, what diseases they've been managing, compares that to what's required or what's used on the ABIM certification exam and demonstrates to them where they may need extra attention or where they have an overabundance of attention and then is able to, again, nudge them in specific areas of learning based on reading their data from the electronic medical record.
[00:22:50] The second one I'll highlight from trainees is a study that's been done at Hopkins, Stanford and University of Alabama, Birmingham, which helps to identify for training programs, how their residents are using their time in a hospital, and then ultimately correlates that to well-being as well as to clinical skill and then creates report cards for individual residents in terms of how they spend their time. And this leverages, again, data and technology. So instead of following people around to see where they are in the hospital, they can use high motion studies that collected about 2000 hours of observation. There's real-time location systems that collect continuous data and have well over 100,000 hours of location data.
[00:23:29] With this, you can see exactly where residents are. It confirms the yellow, which is the time at bedside. And our residents in this particular study spent about 13% of their time at the bedside. That's not actionable. But using data and using analytics, you can make this more actionable. For example, 13% is the average, but you can see there's a 10% spread. The intern that spends least time spends 9%. The intern that spends most spends almost 20%. So there's opportunity that's available for us to move the needle on this if we hypothesize that more time may be more effective. Again, this is a little bit better, but still not actionable enough. And you can create, again, more and more levels of data that are more actual.
[00:24:06] So you compare wards, for example, to specialty services. And you can see in specialty services, they spend less time at the bedside. Again, interesting, but to make it most actionable, you can see where people are when you actually manage their time. And that for medicine is during rounds. during rounds on the wards, they spend about 13% of their time at the bedside, which is the total average as well. However, if you look at specialty services, they round in the workroom. And so if you want to make an intervention, this is an opportunity to make an intervention, you collect that data real time and see if that intervention is effective or not. And then again, with individual report cards, you can see where you are compared to colleagues and then associate this to clinical skill and to well being.
[00:24:45] And then finally, for practicing physicians, I'll highlight one opportunity that is just still being developed. And it's called reconnect. And it's an application that's meant for the practicing physicians where it recognizes that physicians see patients, and they often learn in the background, they'll see a patient, they may have to go to a literature source or online education to find something that's related to help them create more effective patient care. That is not as efficient as it should be. And so reconnect is an artificial intelligence engine that's inserted in that process. So it will see which patients a physician is seeing in the next week, it will then take the meaningful data from the EMR related to those patients, find online resources in this situation right now, specifically PubMed, and then curate learning or educational content that can be delivered to the physician ahead of seeing these patients.
[00:25:35] It's patient relevant, it's efficient. Ideally, we get to a point, hopefully this is the imagination point, where we curate across far more resource than just PubMed, including other online resource. deliver in a highly consumable way. And that learning then is afforded credit for the physician all on the back end, MOC, CME, so that all of those processes, this lifelong learning becomes easy, it becomes patient relevant, and it becomes effective.
[00:26:02] Finally, actually, I'll stop right there. So hopefully I've given some tangible ideas related to precision. Actually, a conceptual model is most of my ideas of where this is now, but even more importantly, where we think it can go. And so with that, I'll turn the remainder of the time over to Dr Triola.
Marc M. Triola, MD: [00:26:18] Thanks very much and thank you Dr Desai and Dr Hammoud for the invitation to participate. And as you'll hear, there's a lot of overlap in the themes across the three of our remarks and the examples given, which I think is really exciting and shows that there is some critical masses evolving in all of this.
[00:26:37] So as we've mentioned, what we've really talked about is the fact that on the left of this diagram, which is sort of my conceptual model of what we're talking about here, where we've digitized much of the educational assessment and outcomes world for medical students, house staff, and faculty, and were able to bring together all of their data, including increasingly attributable electronic health record data, this gigantic, powerful world of clinical data, together in these data warehouses.
[00:27:08] And on top of that, what precision medical education is, is this combination of syntactic data, these new technologies such as artificial intelligence and machine learning, to create tools to help our learners and their coaches work together to derive the best path for each individual training. And so this personalization engine and artificial intelligence can help on top of these massive amounts of data that are far too great for the individual student, resident, or faculty member to sit through to understand the truths that they could follow, Whether that is data-driven coaching, as was mentioned by Dr Prober, which is such an important part of this, nudges and alerts, the examples that you've heard of, new learning pathways and plans in time variable or competency-based educational programs, and then of course, the ability for us to study all of this as it moves forward. And you can think across, this is a medical school focused set of examples, but you could could think across the entire vertical of medical school, how this could create a very bespoke experience for students who are supported through an admissions process that is increasingly both data-driven and analytics-driven, but also holistic and personalized, and informed by predictive analytics about how individual applicants may perform or where they fit into the community that's being created.
[00:28:34] In terms of self-directed learning and assessment, As we mentioned, this concept of tailoring learning and assessment to each individual, suggesting specific resources that are responsive to a person's demonstrated knowledge, need, or goals. And in the clinical learning environment, where lots of unintentional personalization and precision happens based on the randomness of case exposure, using the data from the electronic health record to drive Sadegye's last example, literature or resources, and tailor assessment that's happening, competency-based assessment that's happening in the clinical learning environment.
[00:29:11] So too could career exploration, specialty selection, professional identity formation be informed in a progressive way throughout medical school and residency with the help of these dashboards, these artificial intelligence decision support tools. The transition to GME and as Sanjay mentioned, the portability of these data and perspective from medical school into residency.
[00:29:34] And then, of course, and I'll make this point again at the end, the fact that this cannot be unfiltered recommendations to a learner, but that it is through a partnership of learners and their coaches that they can use the insights from precision medical education to set aspirational goals that are informed by evidence, informed by predictions for their own personal future, that they can come up with increasingly nuanced personal learning plans, and that we give all of of these folks the tools to maximize their success.
[00:30:05] So two quick examples on how we're using, particularly some of the clinical data, to drive precision level educational learning. So this is similar to some of the examples that were given. For our medical students, we have real-time access to our EPIC data for those medical students who are rotating at sites owned by NYU, our health system. So from this, we know which students are picking up which new patient diagnoses. can we use those routinely collected data to automatically tailor suggested educational resources? And so like some of the other examples, we use natural language processing to go through the content catalogs of these educational providers on the left who we provide to our medical students.
[00:30:48] And we indexed each of their sets of resources, their videos, and these are very similar to sort of Khan Academy where there are videos and text-based resources. We indexed each of them and mapped them using natural language processing to clinical diagnoses so that we could link the clinical diagnoses our students are seeing with the resources that are available to our students, and also created a system to identify the correct terms to search PubMed for those particular clinical diagnoses filtered for review, guideline, and consensus articles from the top journals that would be appropriate to medical students. And so now we have a system that automatically runs in the background. And as our medical students pick up new patients, they get these emails the next morning at 7 am that says, "Hi, you just saw a patient "with acute kidney failure. "Here are some learning resources "from the collections that we provide to our students. "And here are an automatically curated set "of high-quality review guideline or consensus articles "that the student might want to review."
[00:31:54] This system is automatically happening in the background. It can adapt to very different types of diagnoses. This is a mental health diagnosis as opposed to a physiological diagnosis, which we saw in the previous example. And as the new literature is published, the system is automatically querying and pulling in the latest literature. Our students like this system. They tend to prefer the literature suggestions more than the resource suggestions because they're thinking about presenting on rounds, but it's a system with no human intervention that can automatically nudge our students in an ongoing way. And much of the feedback from both our learners and our leadership has been around not only suggesting content for cases seen, but for cases not seen. It's showing the scalability and utility of all of this. Another example, which is from Dr Verity Shea, who is the Dean of our clinical clerkships here, but did this work focused on GME, was the recognition that our house staff write many notes, but get feedback on very few. And that the skill of writing a good note as a vehicle for documenting clinical reasoning and really highlighting a good assessment plan is critical.
[00:33:05] So could we create a natural language processing system that automatically reads our medicine resident authored notes to give them feedback on their quality? And you heard an example of this from Texas Tech on the UME side, this is more focused on the GME side. And they spent a tremendous amount of time creating a natural language processing system that didn't just look at brevity versus lags, but really the characteristics that made good clinical reasoning and a clearly stated diagnosis with articulated reasoning, a differential diagnosis, etc. And they created a system which is now running in the background and automatically rating every note that our Medicine House staff write. The Medicine House staff get a dashboard that looks like this. It's embedded with Epic, our electronic health record, in which they can see how they compare to their peers, their own progress over time, and the individual cases by both specific diagnosis and diagnostic group. They can click through and actually read the individual note as well. And our program directors, for the first time, are able to see, as was also highlighted before, not just the epidemiology of what types of patients our house staff are seeing, but their ability to articulate clinical reasoning and where there are diagnostic groups that are strengths or areas of potential growth. And so not only are we leveraging these, and you can see up here that there are 33,712 patients with thousands upon thousands upon thousands of notes here. So we're leveraging a tremendous amount of electronic health record data that was being used for our clinical mission, but not for our educational mission, and now is providing personalized and precision feedback at the level of the individual house officer and at the program level.
[00:34:47] So how do we get to all of this? As Sanjay mentioned, this is a big lift. Here we have an education data warehouse here at NYU and this is increasingly an approach, this syntactic view of data from a multitude of assessment, evaluatory outcomes, and clinical sources. Bringing them all together is an approach that will increasingly be one that we collectively can take across medical schools and training programs. But here we bring together all of our educational data across the top, assessments, evaluations, outcomes, simulation-centered data, portfolios, and coaching. We bring in our clinical data from Epic and our clinical data warehouse, and it is that combined view that empowers our ability to do reporting dashboards, to give basic insights about the progress of an individual student or cohorts on the whole, but also the ability to do this more sophisticated work using artificial intelligence and natural language processing to deliver precision education and nudges.
[00:35:50] And of course, as was mentioned, we can't get there, as Dr Perlman mentioned so eloquently, if we have fixed educational programs. All of this is for naught if the students and house staff can't actually make any course corrections as they go through. So flexibility in our educational programs is going to be key in order to enable insights gained from precision medical education to translate to action by students and coaches.
[00:36:14] And just to reinforce the fact that this is not a series of nudges that should be given to our learners in isolation, that it is pairings of human students and their faculty and coaches that are key. And these tools should never replace that relationship, but it should only enhance and deepen it and empower it with much more perspective on what these data all mean.
[00:36:39] This whole concept is one that is complex. Imagine every medical student having a different pathway, a different set of assessments, etc. So really, these tools need to to help make this easier and less burdensome for our students, house staff, most importantly, for our coaches and course and program directors also, and for our schools as a whole.
[00:37:01] And then I'll end with the fact that, as was mentioned, collaboration and data sharing are key in this. We're talking about artificial intelligence and predictive analytics, yet many groups are underrepresented in medicine or have been historically excluded. So we need to share data to ensure that our ability to make predictions, no matter who that student is, is one that is robust, reliable, transparent, and fair.
[00:37:26] And if this is going to work, it has to work across the UME, GME, and into practice continuum. That also really demands sharing of data, potentially very rich and robust data, so that this insight can continue to help the learner as they go through each of these different phases. So I'm really quite excited and optimistic about this.
[00:37:48] And again, I do want to thank the AMA for their invitation. With that, I will turn it back to Dr Hamoud, who will expertly facilitate our discussion on these complex topics. -
Hammoud: [00:38:00] Thank you all so much. This was great. We do have several questions. Some of them are directed towards one person. Some of them we're going to ask all of you to answer the more challenging ones.
Dr Prober, the first question is for you. I believe in your presentation, you mentioned about health system science, for example, as a place for personalization. And the question is, don't you see it as a critical foundation for all learners, like basic and clinical science? And if the answer is no, they're curious about your thinking about how to identify what's foundational versus what should be personalized.
Prober: [00:38:35] That's a wonderful question. And so I agree with the question or the posit that this should be embedded within our curricula. That happens to be one of my biases. However, I think there needs to be a much more thoughtful process about what ultimately makes up the curriculum, the core curriculum of the medical students that we train, let's keep it simple, across North America, keep it simple for the moment.
[00:39:02] There needs to be agreement upon that because everybody, virtually every faculty member has got their own individual point of view about what information they believe needs to be in the curriculum and it becomes chaos. I can attest to that as 10 years of senior dean in terms of trying to accommodate their requests. It's not possible and it's not a good idea to accommodate everybody's requests. We need, I believe, a national conversation amongst the major stakeholders, including patient groups and pick your favorite stakeholders, but there needs to be a national conversation about what really has to sit within a curriculum, again, as those foundational blocks. There will be hands raised for every topic you can possibly come up with, and a 25-year curriculum would result if you were to appease everybody. So it really has to be tailored to the needs of the community, to the country, paying attention, of course, to inclusivity and equity and everything else that we care so deeply about.
[00:40:04] So that is not an answer. that is rather a vote for a process of trying to identify this. Because right now, I feel quite strongly that we have 170 versions of that answer across the 170 medical schools in North America. And that is not fair to the learners. It's not fair to what we're trying to train people towards. And we need to develop that consensus. And one example, and then I'll stop, is that when I was a medical student, everybody has to say that, I spent, I think it was something like, Well, I spent an entire year in gross anatomy. It wasn't every 24 hours a day, but it was an entire year. And it has become more evident to the non-anatomists in the group that maybe that's too big a dose and maybe we should be teaching something like health policy in that period of time we're dedicated or whatever else.
[00:40:54] So the point is we have to recognize what the national needs and trends are and design accordingly with all stakeholders.
Hammoud: [00:41:02] Thank you. The next question is for you, Dr Desai, and I'm picking on you because as an AMA person who also sits on the table with a lot of other organizations, as you think about this, is how do we overcome licensing board restrictions for timeline-based residency training to provide licensing on a mastery-based precision education level?
Desai: [00:41:22] The easy question. So I think that this really speaks to competency-based medical education and the extension of that of competency-based time variable medical education, and that is a big lift. I think we have to get there because I think if you ask anyone that receives health care what they want, do they want someone that went to school for X years or do they want someone that's competent for caring for the conditions that they have, everyone would choose the latter.
[00:41:48] So, how do we get there? And I think that because of the encumbrances in our system that are related to workforce, that are related to process, that are related to historical things that have really solidified in our country, that process will take some time, but it takes shifts. And thankfully, those shifts are starting to happen.
[00:42:08] And one of the examples that I'll share is work that is happening with one of the grants from the AMA to Mass General Brigham, where Dr John Koh and his team are leading something called Promotion in Place. And this is an opportunity to test competency-based time variable medical education in GME. And they're confronting exactly this question, Maya, which is what happens when you start this and now we have to come up against licensing, we have to come up against board certification, examination eligibility.
[00:42:40] And thankfully this grant has prompted those discussions and they're not easy. For lots of reasons, despite everyone having the best intent they're not easy, but we're moving. So there's not an easy answer, but the movement has started. And I think it's only with these small wins that you get the, I think the big momentum that we're all seeking.
Hammoud: [00:43:00] Thank you, Dr Triola. I have a bunch of questions for you that are kind of related. So, uh, bear with me here. The first one is it's about the expense of the natural language processing analysis that you have written in the EMR. I think every time I watch you do that, I'm, I'm also jealous of what you're able to implement. What does it take to implement something like this and what kind of resources do you need?
Triola: [00:43:23] And it is non-trivial in terms of the amount of resources. One of the fascinating things is that many of the tools, the fancy artificial intelligence, natural language processing tools are free and open source, created by the scientific community.
[00:43:37] In some cases, even created by big tech companies and they're given away for free. So the expense of the tools is not a huge one. The local expertise is one of the biggest challenges. And one thing I would say is that I guarantee you that at each of your academic medical centers, there are informatics faculty somewhere doing these types of things as applied to clinical medicine.
[00:44:01] And as Dr Prober started out with, there is a real parallel between precision medical education and precision medicine. The concept that each patient is different with different genome and phenotype, so too are our students unique. So this is an opportunity for partnerships with other aspects of our health systems and medical schools, using some of the approaches that we're using on the clinical side, core medical education.
[00:44:26] But one of the most foundational things, and this was in the question, is bringing all of your data together. And I think there's another question on this as well. This is a challenge. This is something that took us well over 10 years to really do and get there, and probably has been one of the most transformative things for us to do locally something that many groups have been talking about. Groups like Mediquitous, that's part of the AAMC, have been trying to really move standards together, which will help schools connect different systems and have them talking to each other. But this is something that we as individual schools and health systems need to really address and tackle, because you can't do any of this without bringing your data together.
[00:45:07] And there too, our hospitals and health systems have been working to do the same thing with clinical data from multiple different systems about our patients. So maybe we could beg, borrow, and steal from our clinical side and leverage the tools and infrastructure that they have to help bring this here. But increasingly, whether it's education or regulatory reporting or accreditation, we do need to improve our maturity in terms of bringing all of this data together. And I think it's something that we've seen a ton of great models and experiments, lots of work across the community and great things that the AMA is fostering in terms of thinking and approaching about this.
[00:45:47] Leveling that playing field will be key. I wish I had a perfect answer for it. Maybe the AMA could help us with like meaningful use for medical education technology, but it is something that's super important.
Desai: [00:45:57] Can I just add to that very, just very quickly. I think that this is such a big challenge for us because the majority of lifelong learning occurs as practicing physicians, where there is an incredible diversity of practice setting, many of whom are not associated with the university.
[00:46:13] So as we begin this build, that's where interoperability has to be so important. And those that have resource that are building must keep in mind, how do we access this from less resource settings. And so, again, the answer isn't clear yet. But I think that that charge to us has to be known now, as we begin.
Hammoud: [00:46:31] Sanjay, while I have you, there was a related question to that to talk about, has there been any experience with implementing this on a community training sites? Because when the medical school owns the hospital, it's probably easier to do some of these. How do you actually do it when that's not the case?
Desai: [00:46:47] Yeah, so I can just speak to the practicing physician side. So the pilots that we're now exploring partnerships with are related to clinics, not necessarily non-university based clinics, but clinics in the outpatient setting. And so the goal is to pilot and implement in a way that can be scaled in those settings. And I'll ask Dr Prober or Dr Triola if they have other examples in the community.
Triola: [00:47:10] This is challenging. We're not good at sharing data across health systems. I think that there are some interesting examples. So for example, thinking on the GME side from the ACGME, the requirement that house staff have access to their patient panel has helped the situation.
[00:47:26] And there's a recognition that empowering learners and their educational programs, whether UME, GME, or doctors in practice, and the specialty societies are thinking the same way, with zoomed out data about their practice that respects the anonymity of the patient is key. But we absolutely need much more data exchange.
[00:47:45] We have interoperability on the clinical side, technically, but not politically, per se. And on the education side, we have very little. And in fact, I would argue that it's challenging to get data from the clinical site, but we could do an even better job sharing data from you and me to GME within the same institution in furtherment of helping our individual learners.
[00:48:06] So this is going to be something where each individual school can't do it by themselves. themselves. We're going to need the help of the AMA, of all of the other alphabet organizations to create an environment where this is an expectation and value comes from this rather than continued silos and walls.
Hammoud: [00:48:25] Dr Prober, there are a couple of questions that I want to go back to you. There was one follow up unless you want to make comment about this before I ask you the next question.
Prober: [00:48:32] No, I was actually going to make a go back comment if I can even preempt your because I'm looking at the Q&A as well. So I think going where you're going.
[00:48:39] So Dr Kaufman raised the question about breadth of topics in UME education and how you convince licensing boards and others to actually do that. And then a very insightful response actually from Dr Lisa Howley, who was at the AAMC and senior director of the Transform Medical Education Project, talking about harmonizing or at least addressing between different bodies. I lost her note. The answer to that very question. So the good news, Dr Kaufman, is that many are thinking about this, including the AAMC, working with the AMA and others to try to address that point. Another question related to this says, "Well, if we spend too much time talking about precision medical education and not enough time talking about what needs to be foundational, will we lose ground?" I actually changed the way that question was asked because I can't remember the words, but it is a very important point. We cannot lose sight of the end game and the end game at the UME level is to produce physicians who have the core competencies that this group that I mentioned and Dr Hawley mentioned, define what is needed for a person to go on to the next stage of their education.
[00:49:57] There's a comment about communications and professionalism. from my point of view, we should be recruiting students, this is back to admissions, who actually could communicate. That's not exactly a novel thought, but it's also not a universally embraced thought. So those are some of the comments I'd have for those, Maya, you may have others.
Hammoud: [00:50:16] Yeah, that was the questions I wanted to ask you because there was multiple questions about, it seems like there's a lot of focus on content and not on process, how actually learners learn. And I think, Mark, also there was, kind of like, for example, look at your dashboards, you can tell about communication skills, for example. Can you comment on that, please?
Triola: [00:50:33] Yeah, I mean, the process could be so much more responsive using this precision medical education approach that from the data, from explicit assessments we do from our learner, we can really understand each of them as unique individuals and create actionable insights from that.
[00:50:52] Right now, even if we do understand that, it still is, as Dr Prober showed those pictures, It still is one size fits all in terms of much of the educational content that we're delivering and is difficult to be responsive to the needs of each individual learner or what specialty they're going into what they want to be, how they want to structure their career.
[00:51:13] So I think that these tools, in a way that needs to be efficient and "easy," I think these could really help us do a much better job of both content and process, and truly meet the needs of the individual learner, so that each one of them can hit all of their aspirations as efficiently as possible.
Desai: [00:51:34] I can add to that. I think just on the content process part of this, I agree that process is ultimately what we're seeking, and I would argue even more than process, we need cultural change so that the culture of learning has to be different. And this is such a big challenge because it doesn't start in medical school that we have this cultural problem.
[00:51:54] This cultural problem starts in preschool. And it is all about, you know, how do I get to the next step? This deficit mindset of how we assess people as opposed to the words that we, that I've learned more recently is this psychology of abundance where everybody has an opportunity to grow. And we all embrace, conceptually, the growth mindset, but how to practice that and implement that into a learning environment—and this is, again, why one of our attributes was learner agency—is challenging. It can happen in a microenvironment. I think that's where it has to start. And then it grows, and hopefully it takes over at least one bigger space and then a bigger space. And then, ultimately, what we really need is this cultural shift. And I think that, hopefully, we believe—and I believe I'm speaking for everybody on the panel that precision educational advances will catalyze that cultural shift. It will help implement those shifts that we want culturally for learner agency and growth mindset.
Hammoud: [00:52:49] Thank you. We're almost out of time. One more, a last question, one word. What is the biggest barrier to getting there? Sanjay, start with you.
Desai: [00:52:57] Historical encumbrances. Culture. I think really culture is the problem, in my view, the biggest problem.
Hammoud: [00:53:03] Charles?
Prober: [00:53:04] I think a lack of collaboration, either across schools, either or across faculty, or across the different organizations that are trying to work together to solve the problem. So more collaboration is desirable. - Thank you. Mark, last word to you.
Triola: [00:53:22] In one word, I would say it is momentum, that we need to really embrace disruption in this and realize what's possible when empowered with these data, these tools, and fantastic students and passionate, dedicated faculty.
Hammoud: [00:53:36] Thank you so much. Again, you gave us a lot to think about. I'm sorry we couldn't get to all the questions that were in the chat. Really appreciate everyone's engagement. And thank you again so much for all our speakers and for all our attendees. Have a wonderful evening, everybody.
Disclosure Statement: Unless noted, all individuals in control of content reported no relevant financial relationships.
If applicable, all relevant financial relationships have been mitigated.
Participation Statement: Upon completion of this activity, learners will receive a Participation Certificate.
You currently have no searches saved.
You currently have no courses saved.