[00:00:00] Sanjay Desai: All right, good morning everyone. It's great to see everyone here. thank you for making it to day three with us. We hope that, actually my own experience, I'm just so impressed with the level of engagement that everyone has had for these last few days. The goal, as we said, when we started the conference was to learn and to grow, to share and to contribute. And also to meet people, build community, and be inspired, and I'm hoping that you've, been able to experience all of that in the last couple of days, and we hope we have a day set up today where that can continue until the afternoon. So again, thank you all for being here. It's very exciting for us to have you and to have these discussions.
We're going to start today with a discussion around precision education. We have a couple of breakouts after that, again, and we really hope that you take this opportunity to ask lots of questions. Again, meet, grow, and, continue to inspire others and become inspired. So thank you for being here.
A few housekeeping announcements before we jump into Precision Education. First is the Solutions Showcase is still open today. It's in the exhibit hall from 10:15 to 12:30. We hope you have an opportunity to go and enjoy that. If you are seeking credit, CME credit for this conference, it is available and if you need to learn the mechanism how, there is a booth on the second floor at the top of the elevator. So please, visit the Ed Hub desk there and they can give you details. And very importantly, we want this conference to continue to get better and better, and that requires your feedback. I don't think I have to explain the value of feedback to this audience, but we really hope you provide it for us. And so after the conference is complete, you will receive an email. It should just take a couple of minutes to complete, and if you could do it today—and we all know, again, that timeliness is important—so if you could do it today or on your way home, it would be very much appreciated by us. And we will use that to make this conference more and more valuable for you in the future, so thank you in advance for completing that as well. And I'll repeat this last one which is important at the end as well. There is Garrett popcorn available upstairs, during lunch, and after, as the day continues. So please, go and enjoy that as well.
So with that, we're very excited to have an expert panel here today. We, our goal… I think throughout the last couple of days, I hope that you've also seen that Precision Education, the word has come up a bunch. We've talked about it a bunch, it's been written about, and what we wanted to do was have a panel able to share with you their experience, their thoughts, and their vision for precision education. We want this panel to be interactive with you as well, so each person will have an opportunity to share some words with you, but then will be opportunity also for all of you to ask questions, so please take advantage of that with this expert panel here today.
And so let me, I'm going to quickly just introduce everyone and then, as they come up in the order that they're seating, they'll provide their thoughts with you. So we have Mark, Dr Mark Triola from NYU. I think actually most people up here, all of but I will, again, announce them. So Dr Mark Triola—thank you for being here—has led innovation at NYU and, will share his experience with precision education. Stephanie Sebok-Syer, from Stanford University, again, master educator, and will share her thoughts, as well. Dr Kim Loomis from here at the American Medical Association, and Martin Pusic from ABMS as well as Harvard Medical School. We look forward to hearing their thoughts.
I will kick us off with a conceptual model of precision education and I'm just going to step away from—we created a little four foot by four foot square so that we can walk around a little bit while we talk with you. What I wanted to do was just set the stage a little bit for the conversations that you will hear. And I wanted to start by providing some context on lifelong learning and specifically what are the barriers that we see in lifelong learning, and how precision education may be a valuable tool for us to address those barriers. So we'll talk about lifelong learning. I'll then transition into why precision education, and then conclude really with what is precision education. And we've built a conceptual model around this, which continues to evolve. And then our expert panel will come up and give examples of how this is being used and their experience with precision education, as well. And again, our goal is to keep this interactive as soon as the, as soon as the, presentations are complete.
I have no disclosures or conflicts of interest for this talk. So lifelong learning. The goal, as all of you have created in this country is this model for lifelong learning. We have pre-med all the way up to continuing practice and aspirationally, we want a lifelong learning model where a person enters it and between each of these compartments there are gates of entry and we hope that these gates and these structures and processes are designed in a way that recruit, develop, and promote learners all along and, throughout the entire system so that they are ideally competent at the very end of this. And that competency continues to grow at every stage that they are on this journey. And we know, unfortunately, this is not reality in this country or any other country. So how does it, deviate from this?
So the first thing is that the line is not smooth. The learner's journey is actually an undulating journey. There are moments where they are absolutely accelerating development, but there are many moments also where their development starts to decelerate. And part of this is because of the one-size-fits-all structures that we have and also the randomness that you might have based on rotation, based on where you match, based on a variety of other conditions that are not related to your development needs that leads to this undulation that occurs. And has been the conversation throughout many of the breakouts at this conference, we have these plateaus that occur—these predominately occur at the transition from one department to the next, and that's because we're unable to transfer information efficiently and with fidelity so that somebody can continue to grow, and so that learning begins to pause.
Perhaps the most important is there's a lot of variability in these journeys. And so, this is a typical—I'm going to go through a few archetypes. There's a predominant journey. This is the learner that has that undulating path, plateaus between compartments, but ultimately makes it, thankfully, to a level where the country expects, where they are able to deliver competent care to our patients and to our families.
There are a number of other archetypes, and I think that you guys, hopefully some of these will resonate with many of you. There's the person that achieves in the traditional classroom setting. Overachieves in those settings and really stands out for that but then when they enter the clinical environment starts to decelerate and that slope of acceleration and learning doesn't occur as effectively. There's an inverse of that, of course, the late bloomer that is not able to demonstrate that they excel in these traditional assessments that we've used in structured environments, but they hit the bedside and it's absolutely magical. And in my decade of being a program director this was a common archetype that we saw.
There's certainly the person—and this is, this happens I think well beyond medical education in K through 12 as well—the person that gets just pushed along. We don't have a system that's able to really spend the time and the resource to bring everybody up and as you get pushed along, the disparity between where you should be and where you are grows further and further.
We unfortunately have compartments where people find themselves in an environment where they're not supported for development and when that occurs, then they often will plateau and in worst case scenarios, they exit the system altogether. That can obviously happen anywhere along this path, in medical school, in residency, before that, or certainly after that.
And the final one that I'll describe is the person that gets all the way through, again, to continuing practice, but then finds himself in a clinical practice environment that doesn't facilitate lifelong learning. I think all of these have, all of these occur. I think if we spend even two minutes, you guys will describe a thousand other paths that you could see, in lifelong learning. And the problem is not just the variability in journey. The problem is that variability leads to variable outcomes in competency. And I think if we were to explain to our patients and our families in this country that in the setting of spending four trillion dollars on health care, we have in the end a physician workforce where we're not sure sometimes about the reliability of the competency that's achieved at the end of this, and I think they would find that as unacceptable as we do.
So how might we begin to improve this? And we talked earlier about the four priorities related to, change med ed at the American Medical Association. We think each of these four have an opportunity to improve this. So competency based medical education, if we can assess all along the path, then we might be able to improve and increase the slope of this learning and reduce the variability at the end. If we can improve transitions and transfer information more effectively about each person from one compartment to the next, then we can start to reduce these plateaus. If we can embed equity in the structures and processes that exist in our medical education journey, then we would again increase the slope and prevent people from falling out from this journey all together. We write fix the pipes here because it's the analogy related to water, where if you have a house with a leak, then focusing on the water is not going to be an effective solution. We have to fix the pipes through which that water flows.
And finally, the focus for the rest of this morning session is precision education. This is the idea of bringing the right education and the right training to the right person at the right time. If we're able to do that by leveraging data and technology, then you can start to reduce these decelerations. If we reduce and eliminate those decelerations, we can start to smooth out this path again and elevate the slope. If we're able to do these, make advances in each of these different domains, then the goal aspirationally is that you're left not with a system like this, but you're left with a system in which the outcomes are more reliable. And, very importantly, it's not that we are trying to create the same person out of everyone that goes through this. We just want to ensure that there is a reliable achievement of competency through the system. There will continue to be, we hope, all of the diversity in the learning that occurs, but the reliability of competency is what we're trying to achieve in this process.
So why precision education? So we believe by leveraging data and technology and using that to individualize and make the learning and make the education more efficient, then we can actually start to affect these other, these other levers. So it's a tool that we believe, if you're able to have that precise data, then you can actually assess more precisely and you can start to achieve better implementation of competency based medical education. If you're able to develop these data sets and then have platforms that can transfer them, then the transitions will be improved. If we're able to understand how to assess with equity and measure outcomes that eliminate and reduce, at least certainly reduce bias and noise in the system and personalize the journey, then you're able to embed equity in the system, as well.
So that's why we believe precision education is so important. We also believe that we are at an inflection point in the world in terms of data and technology. AI is obviously the easiest example to describe and to put your mind towards where we're able to do things that we could not do before. In fact, I would argue that we expect this level of precision in every other aspect of our life. When you pick up your phone and you're interested in a sofa, somehow Wayfair has a, an ad for you. Why is it so easy there, but we don't do that in edu? So we're, we are at this inflection point where not only do we have the technology and we have the data, but we also have, I believe, educators and learners who are impatient with the current system. They are ready to receive this type of journey. And so we think this is the opportunity that we are here to take advantage of.
So what is precision education? I think that question I received a lot, uh, and I think probably everybody on this panel hears a lot. You're going to hear different answers. So what my goal is to explain where we are in this journey of defining it, and I think we will continue to define this as we move forward. But the way that we approach this is that we started about a year and a half ago, June of 22 around thinking about precision education. And we brought together not only educators and people in medical education in the traditional, I think people that you might imagine in medical education, we actually started to talk with people, as we described it, that didn't build the house that we live in, to really get an outside view and a comprehensive view of what we think the opportunity is here. So you can see, the types of industries that we included and Sal Khan, who I believe is an inspiration of thinking of how you reimagine education and using data and technology to do. We then had a design sprint earlier this summer, and then have started writing about precision education, as well.
And this is the definition that we've, that we've come up with, which is that these are systems, precision education are systems of education in medicine that use data and technology to transform lifelong learning, and they do this by personalizing it, making it more efficient, and very importantly, they are able to transfer agency of learning to the learner. And this is an important part of the process, we believe, because of the opportunity of a learner or a program or an organization taking over and taking ownership of their own development, which we think is important to actually having effective journeys. And importantly also, we say this can happen, I think we had some conversation even over breakfast today, at the individual level, which you'll hear more about today, at the program level, which you'll hear more about today, and also at the organizational level. So I encourage you to ask questions about that as well. And remembering that this always touches back to, why do we do any of this? The goal is to have a physician workforce capable of meeting the needs of our patients. If we do this effectively, then we will improve patient outcomes, and that's really the motivation behind, I think, all of what we do.
What is the model that we've come up with? I think, I want to share, and this will build a bit, but this is the model. And I think those of you that, look at this, I think, in this room will immediately see two things. One is that this builds off of the Master Adaptive Learner model. I know Bill is here, along with others in this room that have come up with that model. The additional feature here is that top arrow: analytics. The use of analytics inserted into this engine of learning that we think is unique to, towards the precision, the systems of precision education. The other feature that, that's here is in the middle, which is that there's a learner, there's the program, and there's the organization and this builds off of Mark's work and Jesse Burke Rafel's work who have published on precision medical education. And this builds on that in terms of being, learning how we apply that to organizations and at the meso level for programs as well. And again, this is like everything else building on the work of our community as we develop these ideas.
The center of this is a field of data. The fabric of data. And again, this is the critical part in terms of the precision. So data are collected. There is such an abundance of data available. Data that we deliberately collect about learners. But there is an enormous abundance of data that is not deliberately collected just for learning and development, and that is available if we can use it to inform learning and development. So it's this abundance of data, and these data, as all of are currently held in a myriad of fragmented sources and areas in the country. So, whether they're local, at institutions, at governments, in associations, in entities and products, which I'll talk about next, these data exist, but we interact with them through entities and products. And easy examples of entities are some of what I just described. So the American Medical Association, the AAMC, the AOA, ACOM, NRMP, these are associations obviously, but there's also, every health system owns data. Government owned data and collect data. So these are fragmented and they're throughout the system. In addition, there are products. So we consume data through products. Easy examples of products are electronic medical records, learning management systems, and all of the apps that we usually use to help figure out how we might educate somebody more effectively.
So there's a field of data. We consume it and interact with it through these entities and products. So this is the engine that we think about and what we believe needs to happen is that we need to define what are the principles and standards by which we interact with these data? And how are they consumed? And, this is a lot of work. And this is work we hope to catalyze again in the now and in the years ahead. So we have to set up principles. We have to set up standards. And we can talk more about that this morning if there are questions about this. But I've given some examples of why this is so important as we move forward, and the types of questions that we have to answer.
Finally, I'm just going to conclude with the last couple of slides here where one of the opportunities is to break down this picture. So this picture is UME GME continuing practice, but it is what we have today, which is there are these very strong and wide lines between these compartments. There are barriers between going from one compartment to the next, as we've described, and we believe that precision education…what we showed is the cross-sectional view of that model, but if you turn that model and you stretch it out longitudinally, we can start to eliminate those barriers. So the idea here is, again, this is the same model, but it's stretched out over time and, trying to convey that there's a spiral and longitudinality to this product, or to this process. And in the center of it is our data and entities and products and the, what I want to draw your attention to here is that the data arrows are multi-directional. We think this is critical to setting up a model for lifelong learning. that eliminates this barrier that we experience today between compartments.
And so if we can share data across—we have to set up the rules for how that happens—but if we are able to do that effectively, then we can start to improve the journey of a person starting all the way through to retirement.
And then How: I'm going to leave that to my colleagues to describe how we do this. But as I think you're aware and hopefully you've learned in the last couple of days, there's, there, this innovation is occurring. And it's occurring rapidly in different spaces. What I'm sharing here is just a bit of a scatter plot of innovation that we wanted to call out. This is a subset of what's happening. We know that there is more innovation occurring. And I've put in bold, so just to orient you to this, there's the compartments across the top, the stages of journey, and then the macro, meso, and individual level across the first column. And the ones in bold are the ones you're going to hear more about. So I won't go into those today, but these are just examples of really trying to convey that that this field, it's moving. Although it's nascent, it is moving and it's important for us to think about it, and that's why we're excited to have you all think about it with us.
And the last slide I have here is about what we see as immediate needs. We think that this is, as I described, a nascent field, but we need to cultivate this community and this ecosystem and nurture it. And again, that starts with discussion like this. We need to write about it. We need to think more about that model that we just shared, and test that and build on it even as we move further along.
Principles and standards. I just want to really highlight this one more time because As we talked about, innovation is occurring, but the default future, if we don't start to think about how we set up principles and standards to ensure that what we create is a public good. So, health care in this country is a public good. The physician workforce is an effector arm of that public good. And the default future with innovation in this space is that you will have institutions that have resource develop these really exciting models and systems of precision education that benefit a very small subset of learners and physicians in this country, and I think the charge to all of us to, is to ensure that we build systems that can be replicated and can be scaled. We cannot have the impact in CBME and transitions and equity that we want to have in this country unless they can be scaled. So that requires us to develop these standards and develop these principles.
So, we're here today for the last one, which is to build, share, and scale, and start that with discussion. With that, I'm going to turn it over to Dr Triola to lead us through some of his experience in the space.
[00:22:02] Mark Triola: Thank you. Thank you so much, and thank you to Sanjay and the AMA leadership team for this vision and this direction as we move into all of this. I'm Mark Triola from NYU. I'm here representing a team of innovators, many of whom are here, including Jesse Rafel, Abby Winkle, and Colleen Gillespie, whose work, really is the genius behind, much of what we're going to be talking about.
So, my disclosure's that I work at NYU, it's a great place, I highly recommend it. I'm a new board member at the AAMC and we're very proud that the AMA has been so generous in their funding all of this. So, Jesse and I recently published a paper on this topic of precision medical education, and this was our definition. It aligns very nicely with the AMA's definition that it's a systematic approach that integrates longitudinal data and analytics to drive precision educational interventions and to address each individual's learner's needs and goals in a continuous, timely, and cyclical fashion. And as was mentioned, that it's not about necessarily these proximal educational outcomes, but it really is about, system- and clinical-level outcomes as these tentpoles of impact.
And we're not implying precision medical education has not been happening. It has been happening for a hundred years in medical schools in this country in a variety of ways. But to scale this, to pair it with the changing, rapidly changing complexity of our health care system and the nuance and complexity of our medical schools, we need new solutions.
We really see that there are a spectrum of opportunities. I'm going to focus mostly on UME, talking about this. And particularly at our medical school, we're taking analytics and informatics approaches to drive precision medical education across the whole spectrum. On the admissions side, we have shifted to our initial application screening being an artificial intelligence-based screening of medical school applications that complements holistic review. I think holistic review is a great example of a difficult-to-scale precision medical education intervention. Using a learner's performance, those, that sort of spaghetti hurricane plot that Sanjay was showing, to tailor the assessment and instructions that our learners are getting, both in the preclinical and clinical learning environments, and, in particular, giving suggestions to learners and perhaps, equally importantly, their coaches about educational pathways as they go through this. Maybe that's which electives to take, maybe that's which individual patients they should follow as they go through their clinical training. And then, we mentioned that the transition to GME, which is a, an amazing amount of innovative work here, is one that is It's ripe for customization and precision at the individual level, given the opportunities for variability. And coaching. And we really see coaching as an effective and essential component of making precision medical education work. That it is not the, it's not quite the ad on your phone about the couch—it is the coach working with the medical student or trainee to understand the insights and recommendations.
And as Sanjay said so eloquently, all of this requires a huge amount of data. We have been working for a decade on this Medical Education Data Warehouse, where we bring together absolutely every piece of data we can get, more than they're comfortable with often, on our medical students, residents. and faculty. We bring that together in our education data warehouse and we've partnered with the AMA to bring in things like master file data, Jesse's work on CMS claims data so that we can track our graduates out into clinical practice, and since we're an integrated academic medical center, we merged our education data warehouse and our EPIC clinical data warehouse so that we can integrate the diagnoses and clinical outcomes of our medical students and residents as they're going through their training. These, this data drives our dashboards, our artificial intelligence, and our interventions.
And so pragmatically what this really looks like is that on the left, just like we have at all of our places, we have our learners generating and experiencing education mediated through applications, whether it's the electronic health record, whether it is the learning management system, whether it's computer based testing or a simulation center. We have all of that data coming together in our education data warehouse, and in the middle, we have the precision medical education engine, which is personalization and AI. And then delivering to the learner through an education portal data driven coaching, facing both the students and their coaches, nudges and alerts, action and learning plans as they go through these multiple pathways. And then all of this feeding back and driving a research mission, a scholarship, and a feedback loop about what works and what doesn't.
So a few examples of this, we have a series of precision medical education tools that are now in everyday use at our medical school. The first I'll talk about is called Dx Mentor, where we have access in real time to the EPIC data of our medical students and residents. So we know what diagnoses our students are seeing as they're writing H&Ps and admission notes and picking up new patients. We pull from that the diagnoses and through a complicated series of informatics, we map those diagnoses to a series of medical education resources. And the student gets automatic nudges that suggest educational resources and self-study questions based on the cases that they've seen, as well as relevant review and guideline articles from the medical literature. This is just delivered seamlessly to them in morning emails, no human intervention required. And we're using things like AMBOSS and Osmosis, who's here, the up-to-date content catalog, and slightly different content for residents as well.
So the medical students are, feel free to click on or ignore these things. We have also added the ability, since we have a private instance of ChatGPT, for them to click a button and get an AI generated summary of the abstracts of all of the articles that we're suggesting to them, including the stuff that they cared about, like areas of unidentified research that they could talk about on rounds. But our, this is not meant to change the way that they learn, but to make it seamless and frictionless.
We're rolling this up to residents, and since this is precision medical education, we have the ability to tailor not just what it is the individual gets, but whole content catalogs. So now our internal medicine interns will be getting these same type of educational nudges, but the literature is adjusted for their level of training. They're getting things more like medicine-specific podcasts, and in this case the attendings are getting the real-time notifications as well to close the loop and create team-based learning around precision medical education.
As we've built all of this and created an internal tool set around this, we're now thinking about how to really penetrate across the full thickness of our curriculum. And so our newest tool is Study Buddy, which can take any set of clinical text—whether it's a single diagnosis, a patient note, the transcript of the lecture that a faculty member gave—and then map that to a large content catalog of medical education resources, of USMLE type self-study questions of flashcards. All of this data, by the way, is now freely available because companies like OpenAI packaged it all to train the large language models for chat GPT and we can use that effort to bring it down. So now StudyBuddy, when you put in a text, can generate a set of learning resources that are relevant to the individual diagnoses, custom curated USMLE study questions that are automatically pulled and given to the learner, and flashcards, which our learners love so much, and tracks it all as they go through this and it can adapt to the individual learning of that learner as well as, as the level of learner as they go through the spectrum that Sanjay showed.
It's not just about medical knowledge, we also now have a medical student coaching application within which we've integrated ChatGPT and are curious how much precision medical education can be applied to things like goals and coaching. So we have a system where our students enter goals as they go through medical school. This is an example of a real goal that a student entered that they want to learn about rehab medicine, physiatry. Can they stay up to date, read journals, become familiar with the literature and books. We have a button there where they can say, hey, ChatGPT—this is a private version of ChatGPT, they're not sending it to the public ChatGPT—give me some advice on this goal. And ChatGPT gives remarkably good advice, to our medical students. This is the actual response that CHAT GPT generated. Not only does it understand what that book was, it knows what the right journals were in that particular specialty, and it recommends specific facilities and clinics in our medical school and hospital system. We didn't tell it any of this stuff. It already knows everything about all of us because it's read every webpage that, that we've got. This is generic advice. So this doesn't really understand who this student is, but it's complementary to what that student's coach, who knows them very well and is definitely going to give them better advice. This type of information is complementary and can help further the precision of that coaching relationship.
And the last thing I'll end with is that we have also been experimenting with rolling this up to graduate medical education where the focus is more on learning in the clinical learning environment. So we have a system called NoteSense, which Jesse and Verity Shea have developed, that does natural language processing on every note that our house staff are writing and gives them feedback on the quality of clinical re reasoning that are present in each of these notes. So as an individual house officer, you can see how you're doing over time by individual diagnostic category, and tailor in theory, the curriculum or individual patient exposures based on the velocity and progression of your clinical reasoning. And our program directors now have a sort of precision medical education dashboard for clinical reasoning among their house staff, both by individual PGY cohort, by clinical domain, something that was a goal of theirs always, but impossible to do before. So this precision medical education approach has enabled this and made it scalable, and is really helping us transform how we can efficiently customize medical education and progression for all of these learners as they go through their program.
So thank you. I'll stop there, turn it over, or back to Sanjay.
[00:33:16] Sanjay Desai: A quick, a quick comment in terms of orientation, I just, so you'll, we're trying to talk across the, describe things across the continuum. And so you saw Mark really focused on UME, Stephanie will focus largely on GME, Kim on continuing practice, then Martin on organizational level, just again to orient the discussion.
[00:33:30] Stefanie Sebok-Syer: Thanks so much for coming today. I'm really excited to be talking about precision education and graduate medical education. I don't have any disclosures, but I want to take a moment to thank all the organizations that have funded a lot of the work that I'll be talking about with you today.
So when I was asked to talk about what precision education looks like in graduate medical education, this was the image that came to my mind here. We've got this enormous topic, and we've got a lot of great people at all these different institutions, and we're all working really hard to try and figure out what's going on here. And of course, there's many more, there's only so many people you can put on the elephant. But what I think is really interesting and really exciting in the GME space is because we've got all of these people working on all of these different things, what we're actually starting to see now are these collaborations forming. So when you think about how we could use electronic health record data to support resident assessment, you can see Stanford, the University of Cincinnati, NYU, the AMA, we've gotten together and we've said, how can we do this together, right? How can we make this better together? And that's how Tracers came about and we've been really working on how do we take what we've learned and learn from each other so that we can move things forward.
And so what I'm going to talk to you about today is one of the newest collaborations that I've been a part of. This is with Stanford and the University of Michigan. And there I am with my co PIs, Dr Brian George and Dr Andrew Crum. And earlier this year, we were awarded $4.92 million to transform surgical training using precision education.
And so before I get into what exactly we're doing with that, I want to just take a moment to talk about what I mean when I say precision education and assessment. And when I talk about that, I think of an educational strategy, I think about robust measurement, I think about population models, and I think about personalized curricular interventions. And I'm putting this out there not to add another definition and confuse the landscape of precision education—and if you want to know a really good definition of that, Mark already mentioned that him and Jesse have a paper on that, and I encourage you to read it if you haven't already. But what I'm hoping to do here is offer you the lens to which we have approached the work that we're doing.
And so our goal with this was to implement an integrated, unified, precision education system that assesses competence and tailors teaching to actually truly deliver on this idea of competency based medical education. And so we've developed, our process for how we're going to do that. And I want to outline sort of three key features that we think are necessary for this to actually work.
And the first one being, we need a very robust data infrastructure. We often talk about data in relation to precision education, but we don't always talk about the infrastructure that is needed. And so this is where we really started, was we leveraged the simple collaborative and the data that it has to say, okay, what's already in there? What sort of things do we have? So we have workplace based assessments. We have patient outcomes data. We have simulation data.
And then we started thinking, what are the outputs from that? What are the things that we, want to be able to have? And then the added layer on this was thinking to what our Keynote speaker yesterday talked about of not just what do we have now, but where do we want to go in the future? So we started thinking about what are the other types of data? the motion sensor data from Carla Pugh's work at Stanford, what does that mean? How does that relate to competence and where would we put that within this data infrastructure? And then also thinking again of those sort of futuristic outcomes of what do we want going forward? wouldn't it be great if we had automated, credentialing? my personal favorite, The provider contribution measure where we can turn around and actually represent all the things that each individual is contributing in the context of team based care. So thinking about not just what we have now, but where we want to go in the future and the infrastructure that's needed for us to get there.
And so really why we started with infrastructure and why data is so important is because data is really driving these predictive analytic models and it's really informing the measurement approaches that we're taking and so for those of you if you're not aware of this paper I highly recommend it. It came out earlier this year in JAMA surgery and basically, we often talk about learning curves and we talk about, where is the individual on a particular learning curve, but this is really a lovely paper because it outlines the different ways in which we can be more precise.
So what they did was they had learning curves for each of the procedures and then they were also able to break it down by the PGY level. So when we have the curves for all the tasks and activities that we have our residents doing, and then we turn around and can break it down by the PGY level, then we can take that time aspect away and just be able to say, where are you on this curve? this is one of the things that I'm like most interested in and I think is probably the coolest and most exciting thing that, that we're doing with this project and similar to what Mark mentioned earlier, being able to say, okay, what's the next thing? Being able to recommend. Often when you talk to trainees, they don't know, they're like, no news is good news, right?
As long as I'm going along and nobody's telling me that I'm not doing anything bad, I think it's fine, but we could really, be doing better and there are opportunities to do that. And if we know that somebody knows the basic anatomy and they know their port placement, what would we recommend to them next? But the other thing that we've really been doing, there's two things related to this project that are like changing our thinking and shifting the way that we've approached this, is we've taken all of the procedures in surgery and we've looked at, are you a core procedure? Are you an advanced procedure?
And then within that, we've mapped out what are the components of each of those procedures, right? So that it's not just, you've done something basic, now we'll give you an advanced task, but if you do a lap coli, What do we know about the skills and the components that go into being able to successfully do that? Because when you do that, you can then say, Okay, we know that you can do something with the gallbladder. We know you can do something with the pancreas. What, if anything, do we know about your ability to do a Whipple? so this idea of getting down to the nitty gritty and getting down to the components of what we're asking people to do.
And finally, the last thing, and I just, I think this is so amazing, is we're really hitting this bang on, so when you look at these performance curves, there's an area on the curve that presents the largest amount of, uncertainty, and that's really the steepest point of the curve, where we don't know, it's like Maybe you could do this, maybe you can't. And we're really honing in and focusing on that, using the principles of computer adaptive testing from the educational measurement world. We've brought those, that idea and this idea of we're not, we don't have to give the same test to everybody. If we know where you are on the curve, and we want more information about that precise spot of where you are to know, if your next performance activity is going to move you up or down. we really got to focus in this area and so that's what we're doing now is really figuring out how can we hone in on the places that are less precise, less accurate and we're more uncertain with our measurements and really target our nudges or our interventions or our coaching there.
And so when you put this all together, you can see how, having the data infrastructure and the predictive analytics and the coaching and how those activities come through and really having this be able to have some sort of continuous benchmarking so that this is being done in real time. That's one of the huge limitations in the work that we're doing is that we can't get this stuff back to people fast enough. And so when we start to do this, you can see how this could exist along this continuum from UME to GME and right into CPD. And so our hope with this is really that, back to the elephant, the hope with this is really that we're really trying to, by focusing on precision education at a systems level, at a specialty level, that we're really able to raise the tide and have everybody benefit from the, what can come out of taking this precision education approach.
So thank you to the funders and thank you, we got a wonderful team that's working on this and I just want to take a second to acknowledge them.
Kim Loomis: It's a pleasure to be part of this conversation. There's so much exciting work going on. It's fantastic. I am presenting also on the part of a team. I want to acknowledge that other AMA actually initiated this project and that's a little bit of the point of my story in this contribution. Stephanie started to talk a little bit about... the importance of getting this to people in their workflow and making this meaningful. And we were approached by colleagues in our publishing division who had this great idea about how to use technology to elevate current literature that's most relevant to a physician's practice. I was like, that's fantastic, that's really cool. What if they don't have time to read the article that you're sending them? Maybe we should go talk to them. And so we were lucky to partner with the health care system and get face to face with physicians in their system. We also used our network. Some of you participated in these conversations and did some virtual interviews and really tried to get a sense of how do physicians in practice cobble together their continuing professional development.
So we were fortunate given our network to work with people at different stages of careers, different disciplines, etc. And, as you would imagine, it is truly cobbled together. We will not go through this in detail. We don't have time to cover it. This was my attempt, as they were describing the different strategies that they use. how would we organize this? How are they talking about this? And so there's things that aren't really that relevant to their practice, but they're required to do from some external body. There's things that are highly relevant, but maybe too specific to the needs of an individual. patient to really stick broadly, there's things that take a lot of time and effort and things that are quick fixes.
So we put this together and start to think through what are they describing. Another strategy was the notion of a pull. I have a question, I know where to go to find that answer. Or the idea of a push. I sign up for something to be sent to me on a routine basis. And so none of these solutions obviously were perfect and we would expect people to use a great deal, different modalities.
But what I was most struck by was this tone of moral distress. Repeatedly physicians said, I know I should be doing more to keep up. I always try to set aside time in my schedule to go read more articles and it just doesn't happen. And they were actually quite grateful that we came in with this notion of inquiry and not judgment and that we acknowledged the difficulties of their flow. what we, something just jumped ahead, sorry. to get to the actual product, now we understand better what it might contribute. And the idea of this technology is to think about how do we reconnect the physician with primary literature in a way that is most relevant to their needs. It seems to be just running on its own, I'm going to set it down. it, this is an AI engine that essentially is... is trying to do the notion of a push and pull together. So it is anticipatory in that it looks ahead at a physician's upcoming This is not a one-to-one clinical support tool. This is looking at your practice patterns in the panel and saying a deep dive into the electronic health record about those patients to dig up comorbidities and other factors so it's not just the headline of what they're presenting with.
And then uses that to do a query to, right now it's pointed at PubMed, see I didn't do that. You touched it. I didn't do that. I didn't do that. It's funny. It's probably got some kind of auto play going. It's fine.
Sebok-Syer: Yeah. I'm not keeping up.
Loomis: It's actually Martin. He's you're using my time. I'm coming up next.
So what I like about this is it does a deeper query than the physician themselves could think to ask because it's embedding so many components from the electronic record. And then it does this probe of the literature and then what comes back, it's another screening by the algorithm to elevate what's truly most relevant and to weight things differently.
And so if it's a problem that you see frequently, it's going to feed you the most cutting edge. research, new ideas. If it's something that you don't see often, it's going to elevate more secondary resources and review articles. essentially, there's this integration with the EHR. An important element to note here is that none of the data comes to the AMA.
This is all done in a way that the patient data is protected within the system that, that it lives. and then it creates this panel relevant elevation of specific articles that present as those cards that you saw on that initial slide. And so the physician gets before going to clinic a panel of cards that says you might want to look at these articles, and they're able to see the abstract and then click on it if they want to go ahead to read the article.
And as I said, what I think this is filling is two things. When we heard people talk about their patterns, there really wasn't a way. to know what you don't know. There wasn't a way to do informed self-assessment and because this asks deeper questions than we would ask and it probes a broader breadth of journals than you yourself may look to, it can feed you things that wouldn't even occur to you to read in the first place, and it is anticipatory.
It's not reactive learning. It is building up your skills before you step in to the room with your patients that week. And so in a limited pilot, we had a small number of users in this health system, and they actually ended up, we didn't tell them how much time to spend, they only spent about ten minutes a week on the platform, and we were initially a bit worried about that, but then we saw what was happening.
Despite that short engagement, in the clinic, they were recalling two to three pieces of literature to which they had been exposed to through the app. And it influenced one to two of their clinical decisions each week. And so with a very small investment of time, we're getting what we perceive to be pretty high yield.
The other thing that was an important factor, we think this will contribute to being, is that they walked into the clinic with an enhanced sense of preparedness after this brief refresh of, hey, let's think about the literature, let's look at what we might need to do. And as I mentioned, they were grateful that someone is trying to help them figure out how to make this work in their day to day.
One of the things that really was critical to this is obviously we're integrating with an EHR, so we are partnering with the health system. And we know that there's sometimes this degree of adversarial relationship between the health system and the physicians, right? Physicians are overworked, they're always asked to do more and more.
So this is an opportunity for the system to put in place, some tools that can help the physician do the work and tackle something that's pretty big and scary, and do it in a way that's safe. And that's all. I'm happy to take questions later.
Martin Pusic: Hi everybody, my name is Martin Pusic, I'm a Pediatric Emergency Doctor at the Boston Children's and I'm the Director of the Research and Education Foundation at the ABMS and I'm pleased to be a co-investigator on one of the Reimagining Residency grants, the Promotion in Place grant. I forgot to create a disclosure slide, which is not atypical and, but all of my dark money flows in Canadian dollars, so it doesn't really matter. What I'd like to do is take an organizational level look at this, and so that, so that, so that we'll talk about goals, principles, cultures, vague stuff that, that seems inimical to, to the word precision, but what I'm going to argue is that with these examples, CBME, the Master Adaptive learner and Deliberately Developmental Organizations that we can bring precision at the organizational level using conceptual ideas to layer onto this tsunami of data that has been harnessed by the panel members.
And so that so let's start with goals. And so that so as. Because somewhere in the precision education definition, we have to take into account that we need a shared conceptualization of where we're driving all of this to. And CBME, it turns out, is a great example of that. And so that, here's the VanMell 5 components of CBME and what needs to happen for us to have a truly competency-based program. And if you read these things, they tie in beautifully to each of the interventions that Mark, Stephanie, and Kim have described, sequenced progressively. Meaning that we're careful about the sequence of experiences that people have, and we think about way, what their progression is, and then what experience they need to have in terms of that.
We clearly articulate what's going on so that word precision can also apply to what we're trying to do. Learning experiences facilitate, teaching practices promote, and assessment practices support and document a developmental progression amongst the, amongst our trainees. And this is built around GME, but you can see it mapping onto UME and CPD in the modern notion of a physician who not only survives at some minimal level, but thrives and is ever improving in terms of the way they develop.
And so this is from our basic research, in which what you're seeing here is, is a bunch of emergency medicine residents who practice reading ankle x rays with feedback, and they read 250 ankle x rays. And on the y axis is an index of goodness. And, and so that dark black line there is, a beautiful summarized learning curve across all of them.
And so they did progressives really nicely and that was the standard model and the model we taught to before. But in a competency-based precision education notion, we look at the spaghetti ness of this plot and the individual lived experiences of our learners and embrace that variability and say we own this.
And, and so that, we can support people and what is a gradual, sort of diminution in terms of the variability. That's our job as educators and the fact that there's still variable at the end is something that you and I have to grapple with and continue to soak up in our systems.
And so that in a time-based setup, there on the left are two individual learning curves, we standardize, how much, what you do, and we're left with a variable outcome. But in a competency-based notion, we draw this line of where we're headed to. And we're more precise about that line, and so that now, instead of on the left treating both of those people exactly the same, we are on the right individualizing and noticing that these two people with their individual learning curves need completely different approaches at the 150 x-ray mark as we go along. And so the competency based medical education framework really owns that variability and says that not only is it a good thing for each individual learner that we meet them at it, but that it's part of our job and part of the way we should set up our organizations in order to take that on.
In terms of principles, there, I think the idea is a shared conceptualization of how we will achieve these goals. And the example here would be the Master Adaptive Learner. And so that, so in, looking at an organizational level around quality of care, we have boiled down a tremendous variability in terms of the research literature into evidence-based practice and shared conceptualizations of how that works. And similarly, for quality improvement, we have the PDSA cycle, and so the tremendous amount of data that flows around quality is, in a way... funneled through PDSA cycles in which everybody knows what we're talking about when we go to improve a clinical process. And similarly to a PDSA cycle, the MAL adaptive learner model, the master adaptive learner model suggests a way in which we can have a shared vocabulary around what we're going to do to learn. And so that, so clinician learning, if we socialize this model and the words around it and the edu speak that you and I are guilty of in general, and we make it translatable to the average learner, the average clinician, the average person, then we can get to a more precise notion of what we're trying to do.
And then the third one I'd point out is culture. A shared conceptualization of who we will be, and here the example is the Deliberately Developmental Organization that celebrates coaching throughout the organization, and again creates a shared model of who we are. So here's a typical setup in an academic health center in which you have a clinical health system, you have a community health system, and we have our system, the educational institution.
And what's supposed to happen in this model is some Kissinger intermediary shuttles back and forth between the two, carrying knowledge of education to the clinical health system and back. And so that the argument that I would make is that we should storm the ramparts and with, with our ways of thinking, and so that a health system leader has as part of their mission, education leadership.
right there, fused into the way they go, a la the NYU IT model in which EPIC and the Educational Data Warehouse are inside the same firewall and interacting on a microsecond-by-microsecond basis instead of being over at the NYU main campus in some different learning management system and so on. And so that, so the fusion has to happen and that's what each of the speakers have argued for. If you don't know the notion of the deliberately developmental organization, it's a utopian vision of what an organization can be, it's admittedly. And yet knowing, what they're, what Bob Keegan and Lisa Leahy have tried to put forward is really helpful in terms of, helping us strive towards, towards an organization that has a shared idea of how we're going to get better.
So what they say is in an ordinary organization, in businesses large and small, in government and agencies, schools, hospitals, for profits, nonprofits, all over the place, most people are doing a second job that no one's paying them for. They are spending time and energy covering their weaknesses. Says, managing other people's favorable impression of them, showing themselves to best advantage, playing politics, yada, yada.
So that, so this is what they're saying is that we are forced into a fixed mindset by our organizations when we all want to be growth mindset in terms of the way we bring things forward. And so that, so this. conversation about what an organization is and how you develop your human capital has all sorts of potential to then leverage all of the infordances that precision medic education, precision medication would allow.
And so the so here and again, this gets into a lot of organizational arcana, but principles, practices, communities, these sorts of words have to feed into and cross pollinate the I, The Dazzling IT Infrastructure that we're proposing as well. And so that in summary, kind of precision can operate at an organizational level. And what the notion is to bring clarity to what we're all trying to do and that these conceptualizations need to layer into the way we deal with the tsunami of data that's coming forward. And so these mechanisms, goals, principles, culture aren't just soft words that are out there, but rather are really ways in which we can narrow down the undesirable variability to then maximally take advantage of the affordances that have come through.
Desai: Thank you. So there are microphones here. I want to thank all of our speakers for their comments, but we do welcome the audience to come up to microphones and we'll have about ten minutes for question and answer.
And I'll start while you guys come up. So one, actually for Martin and for… First of all, it's not lost on me that the theme of revolution has come up I think every day that we've been here, but for Martin and for Kim, you both comment on culture. We know that the technologies we're describing can only be successful when embraced by the culture of the organization. So if you have just thoughts on how we, and how do we cultivate the cultural change that's necessary? And I'm going to ask everybody to be brief because there are people lined up and we have limited time.
Loomis: Sure. So I think how we do it is going to be something we all collectively try to unpack. I think the need to do it is something we need to keep high and in the center. I think there's an… It's easy to get enamored by the glittery, shiny things that are happening, you look at some of the examples that were given. If you create a structure and a process by which everyone is in learning, everyone's improving, that starts to shift the culture right to, and so I think the more this is normalized—that each individual, it's not just a remediation issue, it's optimizing the training for each individual, now it becomes more and more open and transparent.
Pusic: It's a deep question, right? How do you change culture? And it has to be from top-down, it has to be from bottom-up. And so each one of us advocating for education within a clinical mission is super important. So as the master adaptive learner, the conceptualization goes through what Bill Cotrer and I, and all the others who've been advocating for it have said that the identity, a portion of the identity of a clinician that we're trying to build is a person who's expert at learning. And so that gives us entree to socializing education all across the organizations and expands the idea of what it is to be an educator, because you're no longer about June 30th of their final year, you're about the whole system and how this person's going to fit into the culture of the system.
Desai: Lisa, and we'll just go back and forth. Thank you.
Lisa Egbert: Lisa Egbert, I am a member of the AMA Board of Trustees, and I want to first thank Sanjay for inviting us to enjoy this amazing, few days that we've had here at Change Med Ed.
I am here because I am not a person who lives in the ivory tower that I think the majority of you guys probably live in. I'm just a solo practitioner. Yes, you heard that word: solo. and I am amazed and enthralled at the steps that you have taken to make education better for students, for residents, and even for those of us out in practice. I'm just really concerned that people like me who are in small practices and solo practices and rural practices aren't connected to an ivory tower, are going to not have access to these types of things. And I hope, I want to say this to all of you who are thinking about these things: Please remember us because we're dying on the vine. And as you conceptualize these things, find a way to make it accessible for those of us who don't work for somebody else. Because we really want to stay in practice and we really want to do the best thing for our patients. So I'm challenging all of you to think about that and let me know where you think that could go.
Desai: Thank you, Lisa. Any comments?
Loomis: Thank you for the reminder. As a team, thank you, we are very conscious of that at the AMA, not even just at the solo practice level, but even among the organizations that probably are represented here. Schools that are lesser resourced or community-based programs, we're very thoughtful about how will we democratize this process and how do we leverage the creativity of those who are in a position of privilege and then create perhaps more open resource accessible tools that we can transfer. But we are thoughtful about that in terms of specifically the ReConnect team. In theory, their tool can connect to any electronic health record. It's going to be a matter of once we've tested it and learned how to do those integrations, doing that outreach, and making it tenable for individual practices to do that. But I think that the technology offers tremendous potential if we're willing to share.
George Mejicano: Hi everyone, George Mejicano, Carle Illinois College of Medicine, that was great, thanks. I want to look back to Wednesday when Phil Hanson showed us the thing where we all turned, we made the screens black and then we went to an eraser mode. And the question is, does precision education actually, what is the role of subtraction? In other words, what not to teach, or what can we skip? Because I think if it's just an add on, the cognitive load and the organizational load will be immense. I think we have to reduce some pieces. Especially Mark, I'd love to hear your comments on this.
Triola: We've thought a lot about this because our medical school has now shifted to a three-year curriculum for all students, which means, by definition, less time. And we all know that there's a tremendous amount of waste and inefficiency in the current medical school curriculum. It has been one size fits all by design. That was the point of the Flexner Report, to ensure that the public knew that they were getting a consistent product as students graduated from medical schools. Whether they came to med school with a PhD in biochemistry or a bachelor's degree in music theory, they took the same exact biochemistry course. Precision education will allow us to take advantage of giving the right education to the right person at the right time. Not duplicating things, giving them the right dose, and maybe that's not only curriculum, but also significant differences in assessment, which I think will equally challenge us talking about cognitive load. If you're a coach or administration, if your students are having significant amounts of assessment data, that's something we'll have to deal with.
Regardless, granted it's challenging, we have to embrace this. The medical education, our medical schools and our training programs have to evolve to be more efficient. Our students are more diverse in every aspect as they come to us. And our health care system is changing faster than ever, meaning we need to evolve and we feel, I, I think the whole panel here feels that this is a key way to do it.
Desai: I just want to…we are up against time. We'll take one more question and then I'm going to ask our speakers to stay, if you don't mind, during the break course, and we'll invite everyone else to come up. Abby.
Abigail Ford Winkel: Hi, I'm Abigail Ford Winkle. I'm at NYU. I'm obviously, I've already been sold on this vision and I think it's so compelling. And I guess I'm reacting to this idea of this tsunami of data. And looking at that wheel that you showed at the beginning, Sanjay, I think that we have a little bit of a problem in terms of those inputs. That the assessment data we are getting and are available now at that patient level or system level are really imbalanced towards knowledge and skills and that other side of professional development. I feel like, yes, we need to look for the eraser and we need to figure out what's, how to select the most important things, but I think our current skill as a community is that we're much better at assessing certain kinds of things that doctors need to learn than others. And I actually think we need to work hard to get inputs on how people are doing in developing those health system science competencies in their personal and professional development. What does it mean to develop your integrity and your reliability and your ability to process stress? And we don't get data that's at the same quality in those kind of skills that we all know are essential to glue it together.
So as much as we have to deal with these mountains, these tsunamis of data, I also think we need to be working hard to make sure that our data paints a whole picture so that our learners can really follow that path.
Desai: Thank you. Stephanie, if you have a quick reaction, then we can, we can break.
Sebok-Syer: Yeah, no, I think that's right. I think that's right and that's where a lot of you know this futuristic thinking like being able to take the time to map out all of these things and say what is missing and what have we not tackled yet and put the pieces together. I think a lot of times we've taken the data that is like opportunistic and that it's available and things like that. And so being able to go, okay, what is actually happening? What are all the components that go in? What are we missing? What are the things, to George's question of: what can we outsource? As soon as we know that, Mark's algorithm can do it, I don't really want a physician doing it. I want your time used on something that is going to be more beneficial and maybe that is coaching and maybe that is getting at more of those other constructs or other measures that we're just beginning to tap the surface on.
Desai: Thank you. I'm going to ask the AV team to put the last slide up and, while they do that. Big round of applause again for our speakers who are helping us lead this space. This slide, I just want to draw everybody's attention. There is a, series called the National Health Equity Grand Rounds, which a collaboration between the American Medical Association, the ACGME, IHI, and others, and it relates to the next, in the series relates to data and accountability. So there's a QR code if you would like to join us. please do. It's virtual. and we welcome your attendance.
So thank you again. Thank you again. There's a break now and I believe the breakout start at 10:30. Thank you.
CME Disclosure Statement: Unless noted, all individuals in control of content reported no relevant financial relationships.
If applicable, all relevant financial relationships have been mitigated.