Putting Our Test to the Test: Evaluating the Am I Called? Assessment

Putting Our Test to the Test: Evaluating the Am I Called? Assessment

by Yaacov Petscher, Ph.D. 

The Am I Called Assessment is designed to help you evaluate your pastoral strengths and weaknesses related to a calling to pastoral ministry. This test is built around the six organizing questions from Dave Harvey’s book:

  • Are You Godly?
  • How is Your Home?
  • Can You Preach?
  • Can You Shepherd?
  • Do You Love the Lost?
  • Do Others Agree?

Each question on the test is scored on a scale of 1-5, with 1 representing a significant weakness and 5 representing a pronounced strength. Let’s be clear from the get go, assessments are not the litmus test to determine if you are called to be a pastor. Scoring high on this assessment does NOT mean you are necessarily called to be a pastor. Scoring low on this test does NOT necessarily mean you aren’t called. Rather, this assessment is simply a tool to help you think more deeply about your own strengths and weaknesses, and what those strengths and weaknesses might reveal about your potential call to ministry.

So, are the results you get from taking the assessment reliable and valid? 

A good assessment carries a lot of weight. If you’re contacted by your child’s teacher and told that he or she scored poorly on your state’s reading assessment, you’re likely to ask the teacher about what the score means, whether there are areas of strengths, and what next steps are appropriate given areas of weakness. In the coming weeks and months, you are likely to help your child by reading more with them, possibly hiring a tutor, and monitoring whether there seems to be improvement. You’ll watch your child’s reading grades more closely over the next few months, and you’ll keep a keen eye out for next set of reading scores to see if skills have improved. All of these actions are taken with an underlying assumption by the teacher and parent. Whether explicit or not, it is assumed that the assessment has been rigorously tested to make sure that the scores mean what they’re supposed to mean. After all, what good is a reading score if we don’t somehow know that a low score means the child is a poor reader or a high score means the child is a good reader?

We hope that if you’re taking the Am I Called assessment that it carries a lot of weight for you. But we only want it to carry that weight if it’s a good assessment. So, after using our assessment for several years, the Am I Called team began to ask whether or not the results produced by our survey are consistent and valid from both a ministry content and a scientific perspective? After all, what good is a score from this assessment if we don’t somehow know that a low score might indicate an area of weakness or that a high score might indicate a pronounced strength?

How then might we use the scientific method to address this question? As a data researcher, I love this stuff, and worked with the team at Am I Called to put the test to the test. Here’s how the project went:

  • First, we measured the frequency with which each answer was given. Because the questions on the AIC assessment are multiple choice, I asked, “Do respondents tend to pick one of the choices more often than the others?” Answering this is important. If a large majority of respondents are choosing the same answer for one of the questions, that question really isn’t giving helpful information. It’s a bad question.
  • Second, we measured the average score for each test question. I asked, “Are the average scores on the test higher than expected?” With questions that use a scale of 1 to 5, you’d expect the average answer to be about 3. On average for this assessment, guys rated themselves to be “above average”. To some degree, this is expected. Typically, the men taking this assessment are already involved in some process of discipleship and development. You’re discerning a call, and you’ve evaluated yourself as men who are growing to be equipped for that call. But something else may be in play as well. In self-assessment tests, scientists regularly see respondents score themselves above average. Scientists call it the halo effect. The Bible calls this pride. In other words, you boys have a tendency to think you are angels! And that’s why we think it’s wise to have several other people in your life such as your wife and pastor grade you on these criteria after you’ve taken the self-assessment for yourself.
  • Third, I measured the reliability of responses to the questions. I’ll admit right up front that the math on this one is more complicated (Google “Cronbach’s alpha” if you’re dying to know). In short, I was trying to gauge the extent to which the questions in each category “hang together”. We’d expect, for example, that all of the questions that relate to preaching would be correlated with one another. If you’re a good preacher then you should score more highly on all of the preaching questions and not abnormally high on just one. If the answers for one question are an outlier for the overall category, it’s probably a bad question for that category. Overall, the reliability of scores from the Am I Called assessment conform to acceptable levels in the scientific community. After measuring the frequency of answers and reliability, we have thrown out five or six questions and replaced them with a few more we hope will be better measures.
  • Finally, I measured the validity of the categories. What we like to look for in validity is whether what we’re measuring in one area seems to align to what we’re measuring in another area. For example, in some types of evaluations, you’d expect an inverse relation between what you’re measuring—the more time spent playing video games is associated with lower grades. For the New York Knicks higher payroll tends to be associated with fewer wins. But I expected something different with the AIC assessment. I looked for a strong positive correlation between our six categories. Becoming stronger in loving the lost, for example, should be related to a higher shepherding score. A strength in shepherding should be associated with better preaching. That’s exactly what I found. With one notable exception, Dave’s categories hang together like the chapters in a book. The one place where reliability and validity did not bear out as strongly was with the category How is Your Home? At AIC, we believe that single men can serve effectively and fruitfully in church planting and eldership. But our original assessment questions were developed based on the requirements for ministry outlined in 1 Timothy 3 and Titus 1 where being married and having kids is assumed as the norm. As a result, the original version of the assessment contained a number of questions in this category that were not applicable to many who filled it out. So, in the revision of the assessment, we created a separate set of questions for men who are single. And, for men who are married, we took out all the questions that related to having kids.

What do these findings tell us? Just as the teacher and parent may have confidence in how they approach helping a child in their reading skills because of confidence in what the score means, we believe that you may have confidence in what the Am I Called assessment scores means.  The goal in doing this evaluative work was to serve you, potential pastors, by making a good assessment even more helpful. If you took the assessment before the website and assessment redesign, you might consider taking it again. If you’ve never taken it, you can take it right now. I commend Dave’s work to you as a trustworthy guide. Our hope and prayer is that it continues to help you think more deeply about your own strengths and weaknesses as well as what those strengths and weaknesses might reveal about your potential call.

 

Yaacov Petscher, Ph.D. (twitter: @yaacovp) serves at Four Oaks Community Church in Tallahassee, Florida and works as the Director of the Quantitative Methodology and Innovation Division at Florida State University. His primary work is in using statistical models to understand how children learn to read. He has written books and papers that serve as excellent source material for putting children to sleep or kindling for fireplaces.