Let’s apply science and common sense to a business challenge.  Imagine that you advertised a pizza delivery job and received 10 online job applications each accompanied by a resume.  The resumes seem equally impressive.  The applicants are all high school graduates.  They like to interact with people, have valid driver’s permits, and are over 18.  Knowing that your budget allows you to hire only one person at this time, what would you do to select the best employee for your business?  

Let’s say that you decide to use an interview as a part of your process.  

The interview  

1. Should every applicant get the same set of questions?  Why or why not?  
2. Are there benefits to creating a set of questions before you start interviewing?  
3. During the interview, will you ask probing questions to follow-up on the job applicant’s responses?  Why is this a bad idea?  

Shall we apply science and common sense to answer some of these questions?

Should every applicant get the same set of questions?  Why or why not?  

If you want to compare job applicants in a fair manner, you need to use the same “yardstick.”  Thus, your interview must be made up of the same set of questions.  Anything other than the same set of questions prevents a fair comparison.  

Are there benefits to creating a set of questions before you start interviewing?  

When you need to consistently ask each job applicant the same set of questions, it is practical and useful to prepare your list of questions in advance of the interview.  Thus equipped, you can ask each job seeker the same set of questions in the same order.  This helps you to keep the interview administration conditions consistent.  

As it pertains to interviews, what does the term “structured” mean?  

In the field of personnel selection, interviews are a type of assessment.  Interviews can be “structured” vs. “unstructured.”  In “structured” interviews, the same questions are consistently asked in the same order with no probing.  Thus, each job applicant gets the same exact assessment.  

In this way, the interview’s structure facilitates “comparability” of the results or scores because each job seeker has equal opportunity to answer the same set of questions. Structured interviews confer a sense of fairness to your process.  

Apart from consistently asking the same set of questions, what else is needed to create scores that can be genuinely compared?  

We need to score job applicants’ responses in a consistent manner.  To do this well, you need a well-constructed rubric or scoring guide to score the job applicants’ responses.  A proper scoring guide contains sample responses and the number of points you will assign for each part of a response.  A good scoring guide reduces subjectivity in the scoring process and lets you score responses in a consistent manner.  Thus, a well-designed scoring guide enables you to score responses of different job applicants reliably and objectively.  

The Goal

When you use an interview as a part of your selection process, the scores must assist you in making accurate personnel selection decisions.  This occurs when candidates who score higher on the interview also perform better on the job.  The psychometric term for this property of assessments is “predictive validity.”  Check out our blog articles that specifically address predictive validity.  

Summary

In interviews, when job applicants are consistently asked the same questions in the same order with no probing, the interview is said to be “structured”.  Such structure gives each job applicant an equal opportunity to respond.  This means that you are essentially using the same yardstick with each candidate.  

To ensure that the scores from your interview can be compared we highly recommend that you employ a scientific process to develop the set of questions and the scoring guide.  Prepare a structured interview questionnaire.  Then, administer the interview and use the scoring guide in a consistent manner.   

Scoring responses to the set of questions that comprise the structured interview, using a well-constructed rubric, produces results that are comparable.  There is an additional benefit.  A well-constructed rubric confers objectivity to the scoring process.  This means that responses scored using a well-constructed rubric, are less influenced by the subjective judgments of different evaluators or hiring managers.  

At the end of your process, scores from the assessment must assist you in making valid personnel selection decisions.  This occurs when candidates who score higher on the interview also perform better on the job.  The psychometric term for this property of assessments is “predictive validity.”  

Note:  Predictive validity answers the question:  Do people who score higher on the interview outperform those who score lower?  To ensure your structured interviews and psychometric assessments have predictive validity, we recommend that you employ a scientific process to develop the assessment.  

We hope you found this blog article helpful.  For more information or assistance, please contact us at info@psychometric-solutions.com 

Related Blog Articles

1. How do I accurately and objectively measure an Individual’s Job Performance?  
2. What data do I need to be able to calculate predictive validity myself?  
3. When is the best time to calculate predictive validity?

About the Author

Dr. Dennison Bhola has more than 25 years of experience working in the field of psychometrics creating validated instruments such as assessments, structured interviews, and inventories.  With his teammates, he has developed more than 1000 assessments for use by organizations around the globe.  His expertise includes evaluating new hire quality, assessment effectiveness, source channel efficiency, time to hire, cost per hire, and other such metrics that are of critical importance to Human Resource Professionals.  He also specializes in fairness analyses and auditing compliance with Equal Employment Opportunity guidelines, such as the 4/5ths rule.  

As an HR professional, you might be daunted by the jargon spouted by some psychometricians.  This blog article intends to dispel fear, demystify the jargon, and provide solid information in language that’s as simple as possible. 

One of the most basic questions a consumer of assessments (and assessment results) asks is: “Is this assessment working well?”  In the Human Resources (HR) context:  The key question is: “By using scores from this assessment or interview, are we selecting people who will be able to do the job well?”   

In psychometric jargon, to answer the above questions, we need to evaluate “Assessment Effectiveness.”  To do so, we look at the 3 most essential psychometric quantities:

1. Validity
2. Reliability
3. Fairness

Validity

Regarding validity, to start, we need evidence of the assessment’s content validity.  To do this your assessment provider provides a document describing how they decided what to measure, how the questions were created and how they established relevance to the target job or job cluster.  All worthy assessment providers have such a document already prepared and available for you.  If you do not already have it, all you need to do is ask for it. 

Then, you need to see a number that indicates the strength of the relationship between the assessment scores and actual job performance for people who were hired using the assessment.  In psychometrics jargon, this number is a predictive validity coefficient.  To calculate this number, the assessment provider will need real job performance data from people that you actually hired.  Make this a priority as it will enable you to answer questions such as:

1. Is my “New Hire Quality” improving?
2. Do people who score higher on the interview outperform those who score lower? 
3. Is the assessment empowering me to make more correct selection decisions?

If you do only ONE thing, ensure you get the predictive validity coefficient computed with at least 100 of your own new hires.  Do this as soon as possible.  Only renew your contract if the predictive validity coefficient, computed using your own data, is suitably high. 

Suitably high depends upon the number of people involved in the research study.  The predictive validity coefficient can range between -1.00 and +1.00.  When, you provide good performance data on 100 or more of your own people, you can expect the coefficient to be positive and in the +.40 to +.60 range.  For more detail about setting realistic predictive validity number expectations see articles by Robertson & Smith (2001), and McDaniel and his colleagues (1994).  References for both articles are provided at the end of this blog. 

Until you have sufficient data from your own people, and BEFORE YOU MAKE A PURCHASE DECISION, request predictive validity coefficients from other clients (names are not needed as proprietary clauses in agreements usually prevent this) where the assessment has been used to select people for jobs with the same or similar responsibilities to your “target” job or job cluster.  Get at least 3 predictive validity coefficients each based on at least 100 people, and ensure that these people match the demographics of your job applicants.  If the demographics of samples used to calculate the predictive validity coefficients do not match the demographics of your job applicants, then reject the numbers as unsuitable for you to consider when making such an important purchase decision and significant financial investment. 

Reliability

Psychometricians can compute any of several reliability coefficients.  One of these numbers is an Internal Consistency Reliability coefficient which indicates whether job applicants responded to questions in the interview or assessment in a consistent manner. This consistency measure is sort of like a “lie detector” embedded within the assessment.

The most common Internal Consistency Reliability coefficient is Cronbach’s Coefficient Alpha.  This single number is on a scale from 0.00 to 1.00, and numbers above 0.80 are desired although numbers above 0.70 are acceptable.  Thus, a Cronbach’s Coefficient Alpha of 0.85 indicates that job applicants responded to questions in the assessment in a consistent manner.  Researchers interpret this number as an indicator of whether the set of questions in the assessment were understood and interpreted by most job applicants in the same way. 

It is smart to always ask for the reliability number to verify that it is close to or above 0.80.  Once assessment scores are available an Internal Consistency Reliability coefficient can be readily computed.  So, request it.  Your assessment provider should have this number readily available for you based on existing data from the research conducted to develop the assessment.  Get Internal Consistency Reliability coefficients from at least 3 relevant samples of people. 

Useful guideline: When observing numbers from other organization’s data, ensure that the people in the research samples match the demographics of your job applicants.  If the demographics of the samples used to calculate the Internal Consistency Reliability coefficients do not match the demographics your job applicants, then reject the numbers because they are of unsuitable quality for informing your decision. 

Other reliability coefficients may require more time and additional data, so be understanding, but feel comfortable always requesting an Internal Consistency Reliability coefficient for the assessment.  Again, as soon as possible ask your assessment service provider to calculate the Internal Consistency Reliability coefficient using data from your own job applicants. 

Fairness

Fairness is also called equity or the absence of bias.  The Equal Employment Opportunity Commission has specific guidelines that assessment developers follow to establish and document that scores from an assessment are fair.  We will not delve into these details here.  You can assume that the professionals on the assessment developer’s team are conversant with the Uniform Guidelines on Employee Selection Procedures (1978).  Simply request the fairness numbers, and graphs if they are available.  The graphs will bring the Fairness Indices to life, and help you see how similarly groups (e.g., males and females) who are matched on job performance, score on the assessment.  You simply need a document that says: scores from this assessment are fair to females and males, people 40 or older and those under 40, and people from different racial/ethnic classifications.  In America, when doing fairness calculations, we often use the following racial/ethnic categories: White; Black or African American; Hispanic or Latino; American Indian or Alaska Native; Asian; and Native Hawaiian or Other Pacific Islander. 

Before putting a product on the market, a responsible assessment developer would have completed several studies to establish fairness.  An integral part of these studies is flagging individual questions or statements that show statistical bias and repairing or replacing them before doing a new study.  By the time the assessment gets to you, its fairness should be well-documented and beyond reproach. 

Your professional responsibility is to have the service provider, use your data from your new hires, to calculate fairness statistics for your organization.  Failure to do so is to neglect your duty.  It is normal to conduct the fairness analyses at the same time the predictive validity study is being attended to. 

If you proactively conduct return on investment analyses, it is likely that you already utilize your applicant flow data to calculate metrics such as time to hire, cost per hire, new hire quality and source channel efficiency.  At the same time, it is a good idea to use your applicant flow data to ensure that your 4/5ths rule statistics are in compliance with guidelines from the Equal Employment Opportunity Commission (EEOC).  If you need help with this, please reach out to the psychometric-solutions.com team. 

Data Requirements

• Content Validity – at minimum, requires a job description and job analysis to establish the job relevance of questions used in the assessment. 
• Reliability – requires assessment score data. i.e., the scores each person’s response received when he/she responded to each question. 
• Predictive Validity – requires job performance and assessment score data. 
• Fairness – requires job performance data, assessment score data and demographic data (e.g., gender, age and race category). 

This concludes our brief tour of the criteria you need to use when evaluating “Assessment Effectiveness.”

Summary

When evaluating Assessment Effectiveness pay attention to validity, reliability and fairness. 

Validity pertains to the accuracy of the decisions you make on the basis of assessment scores.  Effective assessments produce scores that empower you to make more correct than incorrect selection decisions.  When evaluating an assessment’s content validity, job relevance is essential.  Pay significant attention to the predictive validity coefficient computed using your own data.  Only renew your contract if the predictive validity coefficient, computed using your own data, is suitably high.  Expect numbers in the +.40 to +.60 range. 

Reliability is usually well-established and documented.  Simply ask for it, as your assessment provider will have it ready and available for you.  Prior to your purchase decision, ensure that the Internal Consistency Reliability coefficients provided for similar jobs, from other clients, are consistently above 0.80.  Then, ask your assessment provider, how soon they could calculate this coefficient using your own data. 

Responsible assessment developers complete several studies to establish fairness before delivering their products to clients.  As a responsible client, seeking to maximize the Return on Investment for your company, ask the service provider to use data from your new hires, to calculate fairness statistics for your organization. 

We hope you found this blog article helpful.  For more information or assistance, please contact us at info@psychometric-solutions.com

Future Blog Article

1. The correct data and metrics to use to evaluate each individual’s job performance.  What are the keys regarding appropriate and accurate Measurement of an Individual’s Job Performance? 

References

Equal Employment Opportunity Commission, the Office of Personnel Management, the Department of Justice, the Department of the Treasury, the Department of Labor’s Office of Federal Contract Compliance, the Civil Service Commission and the Commission on Civil Rights (1978).  Uniform Guidelines on Employee Selection Procedures.  Washington, DC: Author.

Equal Employment Opportunity Commission, the Office of Personnel Management, the Department of Justice, the Department of the Treasury, the Department of Labor’s Office of Federal Contract Compliance, the Civil Service Commission and the Commission on Civil Rights (1978).  Uniform Guidelines on Employee Selection Procedures.  Washington, DC: Author.

McDaniel, M. A., Whetzel, D. L., Schmidt, F. L. & Maurer, S. D. (1994).  The validity of employment interviews: A comprehensive review and meta-analysis. Journal of Applied Psychology, 79 (4), 599-616.

Robertson, I. T. & Smith, M. (2001).  Personnel selection.  Journal of Occupational and Organizational Psychology, 74, 441-472.

About the Author

Dr. Dennison Bhola has more than 25 years of experience working in the field of psychometrics creating validated instruments such as assessments, structured interviews and inventories.  With his teammates, he has developed more than 1000 assessments for use by organizations around the globe.  His expertise includes evaluating new hire quality, assessment effectiveness, source channel efficiency, time to hire, cost per hire, and other such metrics that are of critical importance to Human Resource Professionals.  He also specializes in fairness analyses and auditing compliance with Equal Employment Opportunity guidelines, such as the 4/5ths rule. 

In this article, we provide crucial questions to ask when choosing an assessment or interview for employee screening or personnel selection on behalf of your organization.  These questions pertain to psychometrics, equal employment opportunity law and common sense. 

In order to make your purchase decision, you should be comfortable finding out whether your New Hire Quality will improve if you use the assessment results to make decisions as prescribed. And, within what time frame will the supplier be able to use your actual New Hire information to evaluate the “correctness” of your hiring decisions. 

Here are some smart questions to ask your assessment provider:

1. Will people who obtain higher assessment scores be more productive?
2. Was the assessment developed with samples of people like your job applicants in terms of age, gender, race, prior job experience, and highest level of education attained? 
3. Will your job applicants understand the language used in the assessment?

Note: Interviews are one type of assessment.   If your interview does not provide reliable numerical scores, it should be replaced with one that does. 

As you do your research, you can also use your specific job description to ask more detailed questions such as:

1. Will people who obtain higher assessment scores treat our customers with greater courtesy and respect?
2. Will people who obtain higher assessment scores collaborate more productively with teammates? 
3. Will people who obtain higher assessment scores be more creative?
4. Will people who obtain higher assessment scores be better problem solvers?
5. Will people who obtain higher assessment scores be more efficient time managers?
6. Will people who obtain higher assessment scores consistently show up for work when they are supposed to?
7. Will people who obtain higher assessment scores be able to stay on task when there are distractions in the work environment?
8. Will people who obtain higher assessment scores work hard without constant supervision?
9. Will people who obtain higher assessment scores proactively seek out opportunities to learn on their own time? 

Your assessment or interview manufacturer has answers to all of these questions because they are critical to you making an intelligent purchase decision.  If your assessment provider cannot answer all of these questions, then you have valid reasons for putting their product quality under intense scrutiny.  As the customer and leader investing in the future of your organization, you need answers to these simple questions in order to make intelligent purchase decisions. 

Summary

When purchasing an off-the-shelf assessment for employee screening or personnel selection, please ask your assessment provider:

1. Will the interview/assessment improve my New Hire Quality?
2. Do you have documented evidence that scores from the interview/assessment are fair to male and female candidates, people 40 and older vs. people under 40, and people from different ethnic groups or race categories?
3. Will my job applicants be able to understand the language and examples used in the interview/assessment?

We hope you found this blog article helpful. For more information or assistance, please contact us at info@psychometric-solutions.com

Related Blog Articles

1. Do I purchase the assessment outright or do I license the product? 
2. Is a custom assessment solution better? Why?
3. How to accurately measure an Individual’s Job Performance. 

About the Author

Dr. Dennison Bhola has more than 25 years of experience working in the field of psychometrics creating validated instruments such as assessments, structured interviews and inventories.  With his teammates, he has developed more than 1000 assessments for use by organizations around the globe.  His expertise includes evaluating new hire quality, assessment effectiveness, source channel efficiency, time to hire, cost per hire, and other such metrics that are of critical importance to Human Resource Professionals.  He also specializes in fairness analyses and auditing compliance with Equal Employment Opportunity guidelines, such as the 4/5ths rule. 

The term “Psychometric” comes from the place where Psychology, Measurement, Research Methodology and Statistics overlap. 

Psychometrics helps HR professionals answer questions such as:

• Is my hiring process efficient?
• Is my process getting me high quality employees?
• Which part of my process is most effective?
• Are we getting people with the attitudes our organization needs to be successful in the long-term? 
• What is the predictive validity of each part of my process?
• What is the ROI from each part of my process?

As an HR professional, you’ve probably used the terms Psychometric Assessments and Metrics during communication with your colleagues and clients.  The terms assessments and metrics both come from the field of Measurement in which numbers are scientifically assigned to quantities.  Most people are familiar with test scores that they obtained in high school from teacher-made assessments, college entrance exams or pre-employment interviews.  The numbers from these interviews and assessments are possible because of the field of Measurement. 

What are psychometric assessments?

In Psychometric Assessments, measurement involves

• Asking the right questions
  • Scoring the responses with an algorithm to create a score that has predictive validity

Predictive validity answers the question: Do people who score higher on the assessment ACTUALLY perform better on the job?

Job Performance is defined in the context of the job, and could include answers to questions such as are they more productive, do they communicate well, do they collaborate well, do they apply common sense to solve problems, are they creative?

• In Psychometric Science we apply rigorous Research Methodology and appropriate Statistics to develop customized psychometric assessments by studying your people and using their job performance data. 
• We are therefore able to develop psychometric assessments which help you to identify more people whose attitudes, work ethic and skills are like those of your top performing employees, and that differentiate them from your other employees. 
• To be in compliance with Equal Employment Opportunity Law, Psychometric Science addresses:
  • Validity
  • Reliability
  • Fairness

For details about validity, reliability and fairness, see our blog titled: Psychometrics 101 for HR Professionals: Evaluating Assessment Effectiveness.

Summary

Psychometrics provides information for evaluating:  New Hire Quality, Assessment Effectiveness, Sourcing Channel Efficiency, and compliance with Equal Employment Opportunity Law. 

Psychometrics answers essential questions such as:  Do job applicants who score higher on an interview actually perform better on the job?  Are all of our assessments of high quality?  Is our selection process getting our organization employees with the attitudes, knowledge and skills we need to be successful in the future? 

We hope you found this blog article helpful.  For more information or assistance, please contact us at info@psychometric-solutions.com

Related Blog Articles

1. Smart Questions to ask when Purchasing an Assessment for your Organization.
2. Psychometrics 101 for HR Professionals: Evaluating Assessment Effectiveness.
3. How to accurately measure an Individual’s Job Performance. 

About the Author

Dr. Dennison Bhola has more than 25 years of experience working in the field of psychometrics creating validated instruments such as assessments, structured interviews and inventories.  With his teammates, he has developed more than 1000 assessments for use by organizations around the globe.  His expertise includes evaluating new hire quality, assessment effectiveness, source channel efficiency, time to hire, cost per hire, and other such metrics that are of critical importance to Human Resource Professionals.  He also specializes in fairness analyses and auditing compliance with Equal Employment Opportunity guidelines, such as the 4/5ths rule.