Bonus Episode: Exam Prep Edge
Exam Prep Edge contains opportunities for students to go through actual lessons in the topics where they feel they need it and get tailored feedback based on how they're progressing.
About This Episode
This is a very special roundtable episode telling the story of the creation of The CE Shop's newest product, Exam Prep Edge. Live now in 45 states with more on the way, Exam Prep Edge expands The CE Shop's product offerings beyond online regulatory courses.
Featuring Michael McAllister (CEO and Founder), Mike Matoush (Director of Curriculum), Allie Nickle (Brand Manager), and Robin Downing (Curriculum Product Manager), we discuss the origins of the product as well as how the monumental task was tackled by the Curriculum team.
Learn more about Exam Prep Edge here.
Subscribe to our podcast and listen every monthView All Episodes »
JON: Hello and welcome to Shop Talk: The Real Estate Show. I'm Jon Forisha and this is a very special bonus episode in which we'll be talking about everyone's favorite subject: Exam Prep.
JON: The CE Shop just launched Exam Prep Edge in 45 states and we felt like there was a bit more to the story than just what's on the website. This is our first round table episode with several people here in the office with me as well as one attendee all the way away in San Antonio. Let's start by going around and introducing ourselves, including what your role was in creating Exam Prep Edge.
MICHAEL: Thanks for having us Jon.
JON: Thanks for joining me.
MICHAEL: I'm Michael McAllister, CEO and Founder of the company here. And I'm just excited about this new product launch and not just the value that we bring to our customers, but the fact that everybody in the company collaborated on bringing this together. So thank you in advance for the time.
MIKE: Hey Jon. I'm Mike Matoush, Director of Education and Curriculum. I oversee the entire Curriculum department, including the group that worked on this particular product. Right.
ALLIE: Thanks for having us Jon. I'm Allie Nickle, the Brand Manager and I am a co-product sponsor on Exam Prep Edge. I was involved in everything on the front end, including user interface and user experience.
ROBIN: And I'm Robin Downing, Curriculum Product Manager and one of the curriculum stakeholders on the project. So I did a lot of the back end and managed a lot of the back-end stuff. So the team that developed the content components for exam prep edge, I manage that team and was involved in the decisions related to functionality and how the product was built in the back end and also built all the products that are available.
JON: Awesome. Sort of a dream team here. We've assembled the Avengers team of exam prep.
JON: The CE Shop offers online real estate courses, and exam prep feels like a little bit of a departure from what we normally do. Why did you decide it was worth the effort?
MICHAEL: You know, I'll jump in on that. It really rounds out the whole life cycle of education. So our brand promise at The CE Shop is to deliver innovative and results driven learning experience. So we wanted to leverage the innovation that we had in our learning platform to help our students be more successful. So obviously if you want to become a real estate professional that starts with Pre-Licensing, once you complete what can be in many, many, many hours of content, you really needed our students see that help honing in on very specifically what they needed to be successful in their licensing exam. That's why we developed Exam Prep Edge.
MIKE: I would add that one of our main promises to our students is to give them the very best chance of passing their licensing exam and ensure that our own Pre-Licensing programs are aligned with their licensing exam requirements. And going through the process of developing Exam Prep Edge really gave us an opportunity to look at all of our existing Pre-Licensing products. At the same time we were developing exam prep to make sure that they were fully aligned and contained all of the right information.
JON: So how is it different from existing exam prep products?
ALLIE: It's very different actually. Most of the exam prep products out there, at least in the real estate industry are just practice exams. A lot of them also don't have state specific information. They're all just national and are generally based on questions and they don't really give the student anything to go off of. You just take up to about 17 exams to see how you do and then hope for the best, not necessarily understanding the content behind them.
JON: So how much did you look at competing products when you were developing this?
ALLIE: We looked at a lot of competing products. We definitely had a few that we kept our eye on. Won't name names there or anything. But for the most part we saw a huge opportunity. And just looking at it from a student perspective, every product out there, there was something missing from it. And so it became very obvious very quickly what kind of opportunities we had in the market. And again, it was just a lot of general practice exams. There wasn't a lot of thought put into developing an actual product.
JON: And what were some of the problems that you saw with those existing products that we were trying not to replicate?
ALLIE: A lot of generally based information, not state specific. Um, a lot of kind of glitchy, uh, platforms that you took the practice exams on. There wasn't a ton of feedback either, so you just saw whether you've got a question right or wrong, but also there was a lot of outdated contents. There was the, there were things in there that you felt like you were doing crossword puzzles or something, something that wasn't necessarily geared towards professionals in the industry. It was more of just a weird study tool.
JON: Yeah. Seems like a lot of them were also pretty flat, sort of a pdf.
ALLIE: They are. A lot of pdf downloads that look like they're 10 years old.
MIKE: Yeah. And I would add, one of the things that we saw as an opportunity and wanting to make sure that we captured was content and actual opportunities for students who engage in content where they need the additional learning. So Exam Prep Edge contains opportunities for students to go through actual lessons in the topics where they feel they need it and get coaching or tailored feedback based on how they're progressing through that content.
JON: So what were some of the challenges when you first started development?
ALLIE: Oh there were a few. I would say one of the biggest things is figuring out the best user experience for the students. We had so much content and opportunity that we wanted to put in to helping the student learn that we almost got in our own way of making it too complicated and we had to simplify a lot of the process in order to make it a real useful study tool. Do you agree Mike?
MIKE: I would agree. I would also add the volume. As Jon already mentioned, you know, we're already live with 45 states, so just keeping track of all of the different states and all the different products that had to be developed was the challenge. I think Robin would probably echo that since she managed that process.
ROBIN: Yes, yes. A lot of spreadsheets to keep it all in line. And make sure we were putting the right products with the right states and all that.
JON: Wow. Yeah, that sounds like a lot. So the decision was made to include Exam Prep Edge in Pre-Licensing course packages for free. Why?
MIKE: So as I said earlier, one of our commitments to our students is to give them a product that's going to help prepare them for their licensing exam. We already lead the lead the industry in that area with our Pre-Licensing programs. But that said, we're also committed to improving on those. So we saw it as a great opportunity to give our own Pre-Licensing students yet another advantage going into their licensing exam.
JON: And why does that matter so much? Why are pass rates such a focus?
MIKE: Well, when a student enrolls in a Pre-Licensing program, they have a couple of major goals. They're looking at it changing career or entering a new industry. So obviously they want to be armed with the information and knowledge that they need to be successful. But also important is that they want to be able to pass that licensing exam, which is notoriously challenging across virtually every state.
ALLIE: Well, and as Michael said, we're results driven, focused. So if we can't help a student pass their exam and actually get to the point that they want to be at, which is starting their career in real estate, then we need to reevaluate what our products are doing.
MICHAEL: Yeah. I mean, the ultimate goal of someone interacting with our products at all is that they want to get in the industry. So they don't, most students, I would imagine, don't know the number of steps and the amount of work that it takes to actually get it, see the goal. So this product really stays focused on really the ultimate goal and that's passing that licensing exam.
JON: All right. So how did the process start?
MIKE: Well, I guess the process really started with looking at what the demand was, what the requests were, kind of evaluating what the overall need was. Um, and using that information that didn't start coming with a design and straight from what's the product gonna look like, what are the most critical features, um, how can we make it as student friendly as possible. And so there was a lot of work that went into the initial interface design, including, you know, working with focus groups and a lot of different individuals to make sure that we were on target with what was going to be a value to the students. And then after the design was complete, there's obviously a technology component where the systems have to be created to allow, um, for the system to run and for everything to function. And then you get into curriculum development, which in itself is a pretty gigantic project, which I'm sure Robin will speak to.
JON: So Allie, as the lead designer on this, how did you decide what the product would look at? Feel like?
ALLIE: Well in that sense, my job was made actually really easy because earlier last year we had actually just relaunched our LEAP 2.0 online learning management system. We gave it a remodel, I guess you could say, and made it completely mobile responsive and just a better experience for the students. So we used that as a starting point. We knew that exam prep would be in there and rather than starting from scratch on it, we just figured out how to optimize it and make it even better and add new functionality for Exam Prep Edge.
MICHAEL: Yeah, we were really excited about the, the ability to put something so powerful in the mobile device of our customers. Mobile enabled is an absolute must in this industry. You know, most, most students probably don't complete their, you know, 40 to 200 hour Pre-Licensing courses via their mobile device. But we knew that students would use their phone, their mobile device, to study exam problem. So it's a beautifully designed interface. Allie and her team did a fantastic job.
ALLIE: Thank you. Yeah, we had a lot of opportunity looking at competitors. I'm seeing what was out there and what kind of opportunities we had to make it better, as I said earlier. But also there's so many other online learning systems out there and applications that there's a lot of inspiration to pull from, see what other people are doing, what other students are using right now outside of real estate exam prep and uh, we just saw a lot of opportunity in what we designed here.
JON: So how long did the design process take?
ALLIE: I think that design process took about eight months, 12 by the time we really narrowed it down into what we wanted, figured out how the content would work with it. And uh, we definitely had a lot of back and forth. Yeah. Mike, Robin and I got to know each other very well.
JON: I can imagine.
MICHAEL: I think that iterative process though really helped serve the end product. You know, take the time, prototype, get some design, get it in the hands of those that would use it. That iterative process was really, really demonstrated well by this group. And I think the amount of time that it took, although you know, it's always, there's always an urgency, at least from my seat. I always feel a sense of urgency to get the product to market. But I think the amount of time that we took with it really is paying dividends.
JON: So where did the name come from?
ALLIE: Dang it, I didn't want to answer this one. So the name came from knowing that we want it to be able to brand his product as a standalone product. It technically fits outside of the real estate education lifecycle where we have Pre-Licensing, Post-Licensing, and Continuing Education as the regulatory part. But this made sense to have its own identity because it is a standalone product that does not have regulatory content in it. In terms of actually determining the name, Exam Prep Edge, we a had a lot of brainstorming about it. Our marketing team is very good at coming up with some fun ones. I can tell you there were some very good suggestions out there. Jon, I know you've had one or two suggestions as well.
JON: Exam Prep Choo Choo.
ALLIE: Yep. Absolutely. Get on the exam prep train. Yeah. But in the end we really liked Edge because it is simple. It flows with the name exam prep really well, but also our product does give our students an edge on passing their licensing exam and we wanted to make that well known and keep it simple.
MICHAEL: Yeah. Real estate professionals need a competitive angle. Yeah. They need to find their niche. They need to find the way in which they compete in their market place. And we really feel like this product gives our students an edge, competitive edge and a variety of ways.
JON: So once the look and feel was decided on how'd you determine what the next step was going to be?
MIKE: Well, once the interface was decided and we knew kind of the general functionality, then it was about getting the content in place and actually designing all of the curriculum. And that's really where Robin and her team of instructional designers, subject matter experts, editors, lots of different groups that were involved in that. But that's kind of where they took over and started developing the actual content and putting it into the system.
ROBIN: Yeah, we had a lot of conversations about what is it that students are asking for, what materials are going to be most beneficial in their studying for the exam. So what do we want our study sheets or resources to look like, the types of questions that we need to include on the exam. So a lot of good conversations about how that was gonna, how that was going to work and how to get the information students needed to them through those materials, the exam questions and the lessons in the product.
JON: How different was the process from creating a course?
MIKE: Well, in a, in a standard Pre-Licensing course, students progress in somewhat of a linear fashion, it's very structured. They go from one section to the next and it's a real logical flow. The major difference is that Exam Prep Edge is more of a learning experience. So students won't necessarily go in the same order. We also recognize that some students who are using Exam Prep Edge will have two days to prepare for their licensing exam and others will have months. So we wanted to create a product that works well for both of those scenarios in the way that that works is um, students are initially presented with a, with an initial assessment, um, that covers all of the various topics that are going to be presented on their licensing exam. And then once they complete that initial assessment, they're going to get a read out that identifies their greatest areas of opportunity in their greatest areas of strength. So, um, then students are able to determine their own path for learning and how much time they're going to spend on each individual topic. So they can go right to the ones where they're struggling the most with and participate in learning components, lessons, take additional topic exams until they're comfortable in that area. Whereas with some of the ones where they're already strong in, um, they may not need to spend any time in those.
MICHAEL: It's really creates an efficient learning experience for the students.
JON: How many man hours did the process take?
MIKE: Well, it was thousands of hours. Um, I can tell you that and it was spread out over, over a year. That's just the content development, not even counting the design elements and the IT requirements that went into it. But from a curriculum development perspective, it was tens of thousands of hours.
JON: Wow. That's a big project. How many curriculum writers were involved on compiling all of this?
ROBIN: Yeah. At any given time, we had about half of our designers working directly on exam prep state. So somewhere in the realm of like six to eight designers. Um, but since we're able to leverage a lot of our Pre-Licensing information and exam questions and creating the exam prep materials, really every single member of the curriculum team has contributed to exam prep in one way or the other. But it's been the exam prep life for about half our team for more than a year. And the work continues on more states and not includes for our designers six week I think about six designers still working in exam prep. But also as was mentioned earlier, we've got editors, subject matter experts, quality assurance reviews and, and all those members continuing to contribute to exam prep.
JON: So Robin, you mentioned it before, but you said you had spreadsheets and spreadsheets trying to keep track of all this data. I mean being live in 45 states now, how did the curriculum team breakdown all the necessary pieces?
ROBIN: Yeah, so in the states where we've launched so far, they all have a couple things in common. They use one of the major exam providers and the licensing exam is made up of two portions and national and a state portion. So each exam provider has a Candidate Handbook that's available to test takers and it outlines all the topics and subtopics that make up each portion of the exam. And our product is built around those outlines and for example, in Texas and Delaware, both of which use the same exam provider to administer their exams. The national portion in Texas for a test taker is the exact same as a test taker in Delaware is going to see. So our national portions align to that outline, they're far reaching, we can use them in any state that uses that exam provider. That national outline where we have the variation is with the state portion and this is where a bulk of those tens of thousands of hours have been been spent because every state is unique, has their own rules, their own practices. And we have to dig deep into those items, work with the subject matter experts in each of those states to learn that information and then convey it to the important information to those students in those states. So in this NCE exam prep process is very similar to Pre-Licensing development. It's critical that we understand the laws, the requirements, the practices, so that we're getting it right in each state and providing the accurate information to our students so that we can contribute to them being successful in the course. And then on the licensing exam and then eventually in the real estate careers.
JON: Wow. A lot of research went into it. So unlimited unlimited practice exams are included in exam prep. How, how does that work? I mean, how can you say there are unlimited? You must have a massive bank of questions.
ROBIN: We wrote a lot of questions. Yeah, I think as a, as a project, we've got over 11,000 questions built into the products, but touched so many more. And depending on the exam, there's a bank of either dozens or hundreds or even thousands of questions that feed into those individual exams. And each of those questions get tagged. So they're tagged with a topic and a subtopic as well as a cognitive level. It's not all just recall and memorization with exam questions on the licensing exam, you need to be able to recall information, but you also have to be able to apply it at those deeper cognitive. So the application level and the analysis level. So our exam questions are, I'm tagged based on which cognitive level there they're testing. So with each exam we're able to specify a number or percentage of questions based on topic and based on level. And that's especially helpful on like the national and the practice in the comprehensive practice exams where we can set those data points so that the exam that the student is seeing actually aligns a lot with the actual licensing exam. So if they're going to see 13% of their questions on agency, our national practice exam will also present them with 13% of the questions on agency and also in the right pocket of levels. So it's, it will have equal difficulty as the actual licensing exam will be.
JON: Yeah. So I'm sure we whole new people in school who would just study and study and study even though it was a sure thing they were going to pass their exam. How, how does exam for edge let you know when you reach that point?
MIKE: Well, Exam Prep Edge has several indicators to identify where you're ready in certain areas. And specifically that's going to be on the home page if the product and what, what those gauges are? You have one for completing the initial assessment, which is the first step everybody takes regardless if their own custom study plan. And then you also have an indicator by topic of how well you've mastered each of the individual topics. And we have an internal recommendation to students if they master each topic, um, by passing five topic level exams per topic each. So that would show readiness there. And then also passing the state practice exam and the national practice exam with at least 80% or higher three times. So all of that is laid out very cleanly and it's easy to see from the course home page where you sit with that. But ideally if somebody has enough time to contribute, if they're able to move all those progress gauges over, um, then that should indicate readiness.
JON: Yeah, sounds like it. So how much did learning methodologies play into the layout of content? I know that's a big part of how we create courses, but how did it play into exam prep?
ROBIN: You know, as adult learners, our brains just naturally group and categorize the information that's coming in. So it's only natural that the exam prep layout of the product that has the same thing. And even in the Candidate Handbooks, you see this in the outlines, they've categorized it into your national versus state portion and then your topics and subtopics to categorize it for a way that's going to be easier for a test takers too, just to study and to grasp that information. So that's what we, we followed. But even though we set this overall layout of the product, as was mentioned earlier, we still want students to have that control. As adults, we need control over what we're, how we're studying and in the places that we want to go to study. So that flexibility of the product to be able to go to whichever topic you want to go to or just take practice exams over and over again. And that was important to have for our first students to, to have, but the adult learning traits, they extend beyond the layout of the content and into the content itself. So on the exam, a student again has to do more than just recall the information they need to be able to apply it to a situation or analyze why a certain actually I might be right or wrong and we have to prepare our students to do this. So it makes it important to not only just presents the facts but also explain the information and you scenarios and examples throughout our study materials and in our exam questions and explain that all important. Why? Because we need that why to help ingrain that information in our brains so that we can apply it to those deeper cognitive levels that are going to be required to pass the licensing exam.
JON: Kind of mentioned in the beginning, but again, why does somebody need this? Why is it not enough for them to take their Pre-Licensing education?
MICHAEL: You know Jon, the Pre-Licensing coursework is a regulatory requirement and you have to complete that course work before you can sit for a licensing exam. This product is absolutely vital for anybody who wants to actually pass that licensing exam. It's really difficult for students to take all those potentially hundreds of hours of content and to know very specifically what they need to know for their exam exam. Exam Prep Edge leverages the content and technology and adult learning science and really boils that down for our students in a really efficient manner so that they can have the confidence that they need walking into the exam cause you can complete your coursework, you can study, but unless you really have confidence heading into that exam, it's going to be challenging. This product gives our students that competitive edge to be successful on their licensing exam.
MIKE: I would also add that it can be a real time and money saver for students. I mean, it's expensive to retake the licensing exam over and over again and uh, spending the time in this product to make sure that they're ready allows them to reduce that expense and the time that it takes in the logistics behind having to reschedule those exams.
ALLIE: Well not only that, but hopefully it'll alleviate some stress for those who didn't necessarily pass it the first time. This is a brand new product that they may not have used before that can help them really figure out where their competencies line, where they don't so they can really hone in and pass it again.
JON: So is this only available to students who have taken Pre-Licensing with The CE Shop?
ALLIE: Absolutely not. What kind of marketing team would we have been if that was the case? Uh, no, it's available as a standalone product on our website along with in pre-licensing packages. So if you did take your pre licensing elsewhere or you're just looking for an additional study tool, um, or you didn't pass your exam where you only passed a portion of your exam the first time, you can come by, uh, the product that you need from our site. We do have an available, um, depending on the state, we have just the national portion available for your state, the combined national and state, or just the state offering as well. And they range anywhere from about $59 to $99, depending on the state and what products you're really looking for. I also want to point out, we also have a free assessment option. So if you aren't sure if you need another study tool or something, you can come take that initial assessment that Mike mentioned for free and really see where your competencies lie and you may not need to buy it. If it says that you scored a 90% on it, there may be a reason that you don't have to buy it at all. And we want to make sure that we support our students in that and help them out and they don't need to buy a product that they may not need.
MICHAEL: Yeah. We've always felt that our students should have an opportunity to get a sneak peek at it before they make an investment in it and that stands true with this product as well.
JON: Alright. Do you want to add anything else?
MICHAEL: I just wanted to give a big shout out to the entire team and the people circle it for this interview, Jon, that you pulled together, clearly led the project and did a phenomenal job, but this was a company wide effort. Our brand promise is, as we mentioned to this deliver an innovative and results driven learning experience. We're really proud to bring this to our industry, to our many partners of students out there that are entering your career that we believe is one of the most powerful and important professions that anybody could serve. So thank you all for your efforts, your commitment, and the thousands of hours that you've put into bringing this product to market.
JON: Congratulations on the launch, it's an exciting new product, and thanks for joining me.