A Common Question

Human Resource (HR) professionals are often asked: “Which recruiting strategies are bringing our organization the greatest returns on investment?”  To be able to answer this question you need to be collecting data on an ongoing basis.  Leaders need this type of information to make important decisions.  

Essential Metrics

HR leaders who are on top of things share valuable information with your organization’s CEO, CFO, COO, VPs, and other executives.  To make crucial decisions, an organization’s leaders demand the following metrics from the HR Department: cost per hire, time to hire, source channel efficiency, assessment effectiveness, and new hire quality.  To make your HR leader look good at the C-level, the HR Department needs to collect the relevant data and have these numbers available upon demand.  

System Audit

There are many functions in an organization as well as many systems.  Organizations that practice strategic management frequently review all their operating systems to ascertain what is working well, what is lacking, and what can be improved or re-engineered.  The HR system is vital to an organization’s success.  

A system audit is defined as:

“… a disciplined approach to evaluate and improve the effectiveness of a system. Audits are carried out in order to verify that the individual elements within the system are effective and suitable in achieving the stated objectives. The system audit also provides objective evidence concerning the need for the reduction, elimination and most importantly, prevention of non-conformities”

(System audits and the process of auditing.  https://www.ispatguru.com/system-audits-and-the-process-of-auditing/ Retrieved 24/02/2020 at 10.04 a.m.).  

HR Audit 

A human resources audit (hereinafter HR Audit) is a type of system audit.  It is a systematic process used to gather objective data to ascertain the “people” needs of an organization in keeping with its mission and functions.  

“Like any audit, the Human Resource Audit is a formal process, which is designed to examine the strategies, policies, procedures, documentation, structure, systems and practices with respect to the organization’s human resource management. It systematically and scientifically assesses the strengths, limitations, and developmental needs of the existing human resources from the larger point of view of enhancing organizational performance (Concept of Human Resource Audit.” (http://www.whatishumanresource.com/Human-Resource-audit Retrieved 24/02/2020 at 11.07 a.m.).”

Apart from ongoing data collection, HR Audits are recommended every three to four years.  During HR Audits it is common to find two or three jobs with different job titles but with identical job descriptions.  The longer the interval between audits the larger the number of problems that need to be solved.  

How is a HR Audit done?

A HR Audit examines the entire human resources management system and assets.  It looks at the existing human capital and the specific processes that affect it.  Specifically, a HR Audit evaluates the efficiency and effectiveness of the following HR functions: recruitment/sourcing, screening tools, selection decision making procedures, and placement criteria.  A HR Audit includes analyses of new hire quality, assessment effectiveness, source channel efficiency, time to hire, cost per hire, and other such key performance indicators used by the specific organization.  

Being a system audit, a HR Audit also includes analyses of 

1. job descriptions with recommendations for improvements to meet the organization’s future needs
2. employee relations management mechanisms
3. grievance and disciplinary procedures
4. human resources management policies, e.g., promotion criteria
5. compensation, pay scale, salary grades
6. the human assets and database management system (Human Resources Information System)
7. the performance evaluation system. 

The HR Audit incorporates the use of both qualitative and quantitative data; hence it uses a mixed-methods research design.  At the start of the audit, basic text data from conversations with business leaders, interviews with executives, and documents such as job descriptions and job advertisements, are collected and analyzed.  

These data answer questions pertaining to: the business of the organization; how it conducts its business, i.e., what are the functions, operations and processes; its employees, i.e., the number of employees, positions in the organization and inter-relationships among them (includes review of organizational  chart); the hiring system, the employee relations mechanisms, health and safety, security, the sourcing channels, the screening process, the number and quality of the personnel selection assessments, the performance appraisal tools and process used with each job cluster,  the frequency of observations for coaching and skill improvement, the frequency of summative performance evaluations, and other data of interest to the HR leaders in the specific organization.  

The audit team also observes how business is conducted; who does what, when they do it, how they do it, what tools or equipment they use, the operating environment, the use of the chain of command, as well as employee interactions with each other, subordinates, superiors, and customers/clients.  These observations give insights into customer satisfaction/loyalty, the organization culture, work ethic, communication effectiveness, quality of collaboration, accountability, and productivity.  

The HR audit is a strategic intervention that requires the sponsorship of top leadership.  It also requires the sanction of the governance body (Board) and worker representatives (Union).  It must obtain cooperation of middle managers, supervisors, and the frontline professionals (production and service personnel).  

Communication with everyone in the organization must be clear.  Special care must be invested to avoid sending mixed messages or a message of impending organization demise or downsizing.  Even though downsizing may become a recommendation after the data is analyzed, a HR Audit is not designed with downsizing as a predetermined goal.  For the HR Audit to be successful, stakeholders must be comfortable to share their opinions honestly without fear of reprisal.  

Audit Process

As shown in Figure 1, the audit process begins with open, effective communication and then moves on to the technical procedures.  The numerous procedures start with the collection of data relevant to the business of the organization, what it does, who does what and the mechanisms that support personnel in completing their assigned duties.  

Research methods such as on-site observations, focus groups, interviews and surveys are used to collect qualitative data.  Numerical job performance measurements and productivity metrics facilitate quantitative analyses such as comparisons of production and targets or quotas.  In terms of production comparisons, the audit team may conduct gap analyses of the quantity and quality of goods produced per day and compare that data to the daily quotas.  During analyses data could be aggregated by month, quarter, or year.  

Once the data collected satisfies quality assurance standards such as reliability, the data can then be analyzed as required.  Results from these analyses are interpreted to formulate recommendations.  These recommendations are included in a written report and discussed with the organization’s leadership.  Later, it is a good idea to share findings with a cross-section of the stakeholders, particularly those who were involved in the focus groups, interviews, and surveys.  The outcome of the discussions with leadership as well as with other job clusters, will inform initiatives and proposed actions to be captured in an action plan for moving the organization forward.  

The duration of an audit depends on the size and complexity of the organization, the use of technology, and other relevant factors.  Typically, a HR Audit can be accomplished within three months to four months.  If the audit takes longer than six months then that is a signal that major organization re-engineering may be required.  

References

Cascio, W.F. 1992. Managing Human Resources: Productivity, Quality of Work Life, Profits. New York: McGraw-Hill.

Cummings, T.G. and Worley, C.G. 1975. Organization Development & Change. 6^th ed. Soth-Western College, Cincinnati. 

About the Author

Dr. Kerry Sumesar-Rai is a Management and Organization Reengineering Consultant.  He leads program advisory services, audits, and workshops.  Armed with proven results and wisdom, he helps clients manage changes in public policy, leadership/management doctrine, and cultural re-engineering.  His record of improving internal systems in the public, state and private enterprise sectors demonstrates his ability to facilitate a collaborative approach among leadership teams. As author and life-long learner, his scope of interests which span Strategic Planning, Policy Analysis, and Operations Project Management, include completing a Leadership Ph.D.

Let’s apply science and common sense to a business challenge.  Imagine that you advertised a pizza delivery job and received 10 online job applications each accompanied by a resume.  The resumes seem equally impressive.  The applicants are all high school graduates.  They like to interact with people, have valid driver’s permits, and are over 18.  Knowing that your budget allows you to hire only one person at this time, what would you do to select the best employee for your business?  

Let’s say that you decide to use an interview as a part of your process.  

The interview  

1. Should every applicant get the same set of questions?  Why or why not?  
2. Are there benefits to creating a set of questions before you start interviewing?  
3. During the interview, will you ask probing questions to follow-up on the job applicant’s responses?  Why is this a bad idea?  

Shall we apply science and common sense to answer some of these questions?

Should every applicant get the same set of questions?  Why or why not?  

If you want to compare job applicants in a fair manner, you need to use the same “yardstick.”  Thus, your interview must be made up of the same set of questions.  Anything other than the same set of questions prevents a fair comparison.  

Are there benefits to creating a set of questions before you start interviewing?  

When you need to consistently ask each job applicant the same set of questions, it is practical and useful to prepare your list of questions in advance of the interview.  Thus equipped, you can ask each job seeker the same set of questions in the same order.  This helps you to keep the interview administration conditions consistent.  

As it pertains to interviews, what does the term “structured” mean?  

In the field of personnel selection, interviews are a type of assessment.  Interviews can be “structured” vs. “unstructured.”  In “structured” interviews, the same questions are consistently asked in the same order with no probing.  Thus, each job applicant gets the same exact assessment.  

In this way, the interview’s structure facilitates “comparability” of the results or scores because each job seeker has equal opportunity to answer the same set of questions. Structured interviews confer a sense of fairness to your process.  

Apart from consistently asking the same set of questions, what else is needed to create scores that can be genuinely compared?  

We need to score job applicants’ responses in a consistent manner.  To do this well, you need a well-constructed rubric or scoring guide to score the job applicants’ responses.  A proper scoring guide contains sample responses and the number of points you will assign for each part of a response.  A good scoring guide reduces subjectivity in the scoring process and lets you score responses in a consistent manner.  Thus, a well-designed scoring guide enables you to score responses of different job applicants reliably and objectively.  

The Goal

When you use an interview as a part of your selection process, the scores must assist you in making accurate personnel selection decisions.  This occurs when candidates who score higher on the interview also perform better on the job.  The psychometric term for this property of assessments is “predictive validity.”  Check out our blog articles that specifically address predictive validity.  

Summary

In interviews, when job applicants are consistently asked the same questions in the same order with no probing, the interview is said to be “structured”.  Such structure gives each job applicant an equal opportunity to respond.  This means that you are essentially using the same yardstick with each candidate.  

To ensure that the scores from your interview can be compared we highly recommend that you employ a scientific process to develop the set of questions and the scoring guide.  Prepare a structured interview questionnaire.  Then, administer the interview and use the scoring guide in a consistent manner.   

Scoring responses to the set of questions that comprise the structured interview, using a well-constructed rubric, produces results that are comparable.  There is an additional benefit.  A well-constructed rubric confers objectivity to the scoring process.  This means that responses scored using a well-constructed rubric, are less influenced by the subjective judgments of different evaluators or hiring managers.  

At the end of your process, scores from the assessment must assist you in making valid personnel selection decisions.  This occurs when candidates who score higher on the interview also perform better on the job.  The psychometric term for this property of assessments is “predictive validity.”  

Note:  Predictive validity answers the question:  Do people who score higher on the interview outperform those who score lower?  To ensure your structured interviews and psychometric assessments have predictive validity, we recommend that you employ a scientific process to develop the assessment.  

We hope you found this blog article helpful.  For more information or assistance, please contact us at info@psychometric-solutions.com 

Related Blog Articles

1. How do I accurately and objectively measure an Individual’s Job Performance?  
2. What data do I need to be able to calculate predictive validity myself?  
3. When is the best time to calculate predictive validity?

About the Author

Dr. Dennison Bhola has more than 25 years of experience working in the field of psychometrics creating validated instruments such as assessments, structured interviews, and inventories.  With his teammates, he has developed more than 1000 assessments for use by organizations around the globe.  His expertise includes evaluating new hire quality, assessment effectiveness, source channel efficiency, time to hire, cost per hire, and other such metrics that are of critical importance to Human Resource Professionals.  He also specializes in fairness analyses and auditing compliance with Equal Employment Opportunity guidelines, such as the 4/5ths rule.  

I am a senior citizen and grandpa.  Because I have over 45 years’ experience studying top performers in the world of work, people sometimes ask me: “What do I have to do to be successful in my new job?”.  Of course, there are many answers to this question, but to me, it seems that six universal truisms are necessary for success.  Two of them can be viewed as precursors for success while the last four are on-the-job actions.  

The Two Precursors 

1)  Find what you do best, your Talent, and 2) Work hard to develop and grow your Talent.  As children we engage in free play and eventually find things we enjoy doing.  This gradually leads to more structured activities in which we home in on our preferences.  We find things that interest us and do them more often than we do other things.  After a while we become pretty good at doing them.  The more things we are exposed to help us become aware of our strengths and give us more options when we need to make important choices for an occupation.  

Thus, if we listen to our yearnings, we understand ourselves better and make choices that lead us to an occupation that interests us and that we are good at doing.  When we think of talent, we usually think of movie stars, athletes, singers, authors, artists, sculptors, designers, entrepreneurs, etc.  These people’s accomplishments are usually very visible and well-documented.  What most people fail to recognize is that it requires talent to do a so-called “ordinary job” very well.  A few years ago, I was consulting with a seafood restaurant chain and I met a world class shrimp breader.  This person knew more about breading shrimp, than I believe, anyone in the world AND he really liked his job.  Plus, he produced delicious results every time.  

Congrats 

Congratulations on landing your new job!  You must be elated, and if you are reading this, you must be the type of person who likes to be prepared in advance.  This is an essential quality for success in the world of work.  Regarding advance preparation: We’ve observed that you can increase your effectiveness by routinely making a checklist of action items, identifying your top three priorities for the day, and ensuring that you complete those first!

The On-The-Job Actions 

Now, that you are starting that new job tomorrow, the last four ideas become of paramount importance for your success.  The next point is about your attitude.  Be open to learning, 3.  learn, learn, learn and be a sponge.  Find out everything you can about your new company; it’s customers, what it does best, what its challenges are, it’s products, your co-workers, and your manager’s expectations for you.  Part of this point leads us to the fourth and fifth ideas which are 4.  become productive as fast as you can which yields to the notion that 5.  you are paying for yourself and more.  Figure out what you need to do to pay back the company’s investment in you.  This means becoming productive quickly and continually finding ways to continue that productivity into the future.  It also means passing your learnings and knowledge on to those who come later to the company.  Become the person that others approach for advice and to share ideas.  

The last and most important point, 6.  Is never lie, cheat, or steal.  This point is obvious to most people.  A few years ago, we asked people in various organizations that if they could choose one word which would become synonymous with their name or reputation in their company, what word would that be?  Many words were chosen such as cooperative, mentor, productive, friendly, etc.  When asked about their choice, many used the word honest as part of their narrative.  And, many actually choose the word honest, integrity or ethical as their word.  Therefore, this point becomes the basis or bedrock for the other five.  This one underlies all the above.  

In closing, I’d like to address one last issue regarding a phrase you hear almost daily: “If you want it bad enough, you can get it.”  That is partially correct.  We would add to it as follows:  if you want it bad enough AND are willing to work hard enough AND you have the talent you can get it.  Our definition of talent includes attitudes to work and covers dimensions such as integrity, responsibility, courage, empathy, self-discipline, positivity, resourcefulness, openness to learning, and diligence.  To shine in your new job, build on your foundation of talent, work hard to acquire the knowledge and skills, and apply them for the benefit of your customers and employer.  

Remember that talent exists for all jobs.  I worked on a construction crew the summer after my first year of college and by the end of the day, I would be a dirty, sweaty mess, covered in mud while my coworkers would be very clean.  That was part of their talent in doing the same job I was doing.  Each person doing a job well, knows something about that job that is not apparent to the rest of us.  And it does not matter what that job is.  There is world class talent in every job!  We know that great athletes, singers, artists, authors, etc., spend thousands of hours perfecting their performances as do housekeepers, teachers, nurses, firefighters, maintenance workers, baggage handlers, shrimp breaders, and the high performers in every job function.   To shine in your career, grow your skills, wow your customers, and be helpful to your teammates.  These tips work for jobs where you are in the spotlight as well as those where you are a part of the supporting cast, behind the scenes.  Best wishes for much success in your new job!  

About the Author

Dr. Richard Harding has more than 45 years of experience working in the field of psychometrics creating validated instruments such as assessments, structured interviews, and surveys.  With his teammates, he has developed hundreds of interviews and surveys for organizations in the US and around the globe.  His expertise includes evaluating new hire quality, assessment effectiveness, source channel efficiency, time to hire, cost per hire, and other such metrics that are of critical importance to Human Resource Professionals.  He teaches statistics and research.  He also specializes in fairness analyses and auditing compliance with Equal Employment Opportunity guidelines, such as the 4/5ths rule.  

When your goal is to remodel, change or develop an enterprise, you are interested in Organization Re-Engineering.  It starts with a strategic intervention and culminates with a new or re-engineered organization. 

A re-engineered organization is one that has undergone some combination of transformation, reformation, and sustaining.  The extent of the changes is based on the organization’s needs and goals.  Transformation is about significant change whereas reformation pertains to change in some aspects while the fundamental business, product and services remain unchanged.  The parts of the business that remain unchanged, are described as being sustained.

For instance, a transformation would be changing your organization from producing playing cards to producing cell phones.   In transformation the product as well as the target market changes.  Reform is like moving from the production of bread to boxed cereal.  The fundamental business and the targeted consumer remain the same.

You may ask: “Why include sustaining in a conversation about re-engineering?”  While re-engineering suggests widespread change, that goal is not always desirable nor achievable.  Re-engineering speaks to changes, modification, alterations, and even rationalization of processes, some systems, and operations. 

Changes may include the introduction of new technology and production systems or new skillset requirements for employees.  All of these affect the ways things are done and the outcomes produced.  Modifications and alterations may be realised in the work flows and resources used.  

Rationalization may come in truncating systems and processes, thereby reducing time in the movement of resources and inputs or reduction of the number of human resources involved. Rationalization includes what some call “right sizing.”  It is about producing the same or more output with less inputs. 

Re-engineering may not see every facet of production changed, some aspects of the organization’s operations, some functions may remain the same: e.g., procurement, financial accounting, records storage, communications and even some logistics.  The flow of resources to enable production and service delivery may remain the same or change.

However, it’s expected that a re-engineered organization does things differently, is more efficient, and is more effective in the use of resources, decision making, and the delivery of goods and services.  What the organization chooses to transform, reform, and sustain depends on the purpose of the organization (see Figure 1). 

The purpose of the organization defines the business and operations strategies, the objectives, and goals.  Organization re-engineering is geared towards changing or modifying outcomes, such as productivity metrics, and requires deliberate measurement and studies to evaluate the impact of the targeted changes.

The attempt to re-engineer an organization requires considerations of institutional strengthening and capacity building to enhance the capabilities, efficiency, effectiveness, and profitability.  An organizational diagnosis needs to be done so that the issues and problems can be defined and the organization development thrust can be properly positioned, focused, and validated. 

The organization development itself will be comprised of several initiatives accompanied by relevant studies and analyses.  These analyses and studies will require data capture. The data collected is then analysed to obtain an understanding of what exists and make decisions about what requires change.  The research may include gap analyses to discern issues and problems to be addressed.  The research tools used for this part include both qualitative and quantitative methods. 

Crucial to effective analyses is the capture and modelling of the existing processes and work flows.  Flow charts, maps and resource data are essential for the functional and task analyses.  Upon completion of the data collection and after the analysis phase the operational and other processes are remapped through redesign to properly orient the systems and methods of production or services.  Remapping requires documentation of the work flow or the production process: input, throughput, and output.  The analyst then seeks to rearrange or reduce the inflow and outflow steps to achieve greater efficiency and reduce use of resources.  

This phase is followed by another series of strategic actions (even inaction if such is warranted).  Job analyses are done, new jobs are identified, new job performance standards are developed.   The new job performance standards are used to set training performance outcomes, develop training materials, and provide professional skill development so employees are equipped to do their re-engineered jobs. 

These strategic actions lead to improved accountability, the development of improved assessments for personnel selection, and the creation of improved methods of performance appraisal.  For the organization to benefit from re-engineering endeavours the culture and production qualities must change to match the thrust and focus of the new entity. 

Summary

The re-engineered organization may have elements that were transformed, some that were reformed and yet other parts that were sustained.  Organization transformation is defined by a radical change in most aspects of an organization.  It can be the organization’s systems or business focus or both.  Reformation constitutes changes to parts of the organization but the raison d’être remains the same or slightly modified.  Sustain means that the systems or processes or some features and modus operandi remain the same.

References

Champy, J. 1995. Reengineering Management. London: HarperCollins.

Hammer, M. and Stanton, S.A. 1995. The Reengineering Revolution Handbook. London: Harper Collins.

Handy, C.  1993.  Understanding Organisations. . 4^th ed. London: Penguin Books.

Manganelli, R. L. and Klein, M. M. 1994. The Reengineering Handbook. New York: American Management Association.

About the Author

Dr. Kerry Sumesar-Rai is a Management and Organization Reengineering Consultant.  He leads program advisory services, audits, and workshops.  Armed with proven results and wisdom, he helps clients manage changes in public policy, leadership/management doctrine, and cultural re-engineering.  His record of improving internal systems in the public, state and private enterprise sectors demonstrates his ability to facilitate a collaborative approach among leadership teams. As author and life-long learner, his scope of interests which span Strategic Planning, Policy Analysis, and Operations Project Management, include completing a Leadership Ph.D.

As an HR professional, you might be daunted by the jargon spouted by some psychometricians.  This blog article intends to dispel fear, demystify the jargon, and provide solid information in language that’s as simple as possible. 

One of the most basic questions a consumer of assessments (and assessment results) asks is: “Is this assessment working well?”  In the Human Resources (HR) context:  The key question is: “By using scores from this assessment or interview, are we selecting people who will be able to do the job well?”   

In psychometric jargon, to answer the above questions, we need to evaluate “Assessment Effectiveness.”  To do so, we look at the 3 most essential psychometric quantities:

1. Validity
2. Reliability
3. Fairness

Validity

Regarding validity, to start, we need evidence of the assessment’s content validity.  To do this your assessment provider provides a document describing how they decided what to measure, how the questions were created and how they established relevance to the target job or job cluster.  All worthy assessment providers have such a document already prepared and available for you.  If you do not already have it, all you need to do is ask for it. 

Then, you need to see a number that indicates the strength of the relationship between the assessment scores and actual job performance for people who were hired using the assessment.  In psychometrics jargon, this number is a predictive validity coefficient.  To calculate this number, the assessment provider will need real job performance data from people that you actually hired.  Make this a priority as it will enable you to answer questions such as:

1. Is my “New Hire Quality” improving?
2. Do people who score higher on the interview outperform those who score lower? 
3. Is the assessment empowering me to make more correct selection decisions?

If you do only ONE thing, ensure you get the predictive validity coefficient computed with at least 100 of your own new hires.  Do this as soon as possible.  Only renew your contract if the predictive validity coefficient, computed using your own data, is suitably high. 

Suitably high depends upon the number of people involved in the research study.  The predictive validity coefficient can range between -1.00 and +1.00.  When, you provide good performance data on 100 or more of your own people, you can expect the coefficient to be positive and in the +.40 to +.60 range.  For more detail about setting realistic predictive validity number expectations see articles by Robertson & Smith (2001), and McDaniel and his colleagues (1994).  References for both articles are provided at the end of this blog. 

Until you have sufficient data from your own people, and BEFORE YOU MAKE A PURCHASE DECISION, request predictive validity coefficients from other clients (names are not needed as proprietary clauses in agreements usually prevent this) where the assessment has been used to select people for jobs with the same or similar responsibilities to your “target” job or job cluster.  Get at least 3 predictive validity coefficients each based on at least 100 people, and ensure that these people match the demographics of your job applicants.  If the demographics of samples used to calculate the predictive validity coefficients do not match the demographics of your job applicants, then reject the numbers as unsuitable for you to consider when making such an important purchase decision and significant financial investment. 

Reliability

Psychometricians can compute any of several reliability coefficients.  One of these numbers is an Internal Consistency Reliability coefficient which indicates whether job applicants responded to questions in the interview or assessment in a consistent manner. This consistency measure is sort of like a “lie detector” embedded within the assessment.

The most common Internal Consistency Reliability coefficient is Cronbach’s Coefficient Alpha.  This single number is on a scale from 0.00 to 1.00, and numbers above 0.80 are desired although numbers above 0.70 are acceptable.  Thus, a Cronbach’s Coefficient Alpha of 0.85 indicates that job applicants responded to questions in the assessment in a consistent manner.  Researchers interpret this number as an indicator of whether the set of questions in the assessment were understood and interpreted by most job applicants in the same way. 

It is smart to always ask for the reliability number to verify that it is close to or above 0.80.  Once assessment scores are available an Internal Consistency Reliability coefficient can be readily computed.  So, request it.  Your assessment provider should have this number readily available for you based on existing data from the research conducted to develop the assessment.  Get Internal Consistency Reliability coefficients from at least 3 relevant samples of people. 

Useful guideline: When observing numbers from other organization’s data, ensure that the people in the research samples match the demographics of your job applicants.  If the demographics of the samples used to calculate the Internal Consistency Reliability coefficients do not match the demographics your job applicants, then reject the numbers because they are of unsuitable quality for informing your decision. 

Other reliability coefficients may require more time and additional data, so be understanding, but feel comfortable always requesting an Internal Consistency Reliability coefficient for the assessment.  Again, as soon as possible ask your assessment service provider to calculate the Internal Consistency Reliability coefficient using data from your own job applicants. 

Fairness

Fairness is also called equity or the absence of bias.  The Equal Employment Opportunity Commission has specific guidelines that assessment developers follow to establish and document that scores from an assessment are fair.  We will not delve into these details here.  You can assume that the professionals on the assessment developer’s team are conversant with the Uniform Guidelines on Employee Selection Procedures (1978).  Simply request the fairness numbers, and graphs if they are available.  The graphs will bring the Fairness Indices to life, and help you see how similarly groups (e.g., males and females) who are matched on job performance, score on the assessment.  You simply need a document that says: scores from this assessment are fair to females and males, people 40 or older and those under 40, and people from different racial/ethnic classifications.  In America, when doing fairness calculations, we often use the following racial/ethnic categories: White; Black or African American; Hispanic or Latino; American Indian or Alaska Native; Asian; and Native Hawaiian or Other Pacific Islander. 

Before putting a product on the market, a responsible assessment developer would have completed several studies to establish fairness.  An integral part of these studies is flagging individual questions or statements that show statistical bias and repairing or replacing them before doing a new study.  By the time the assessment gets to you, its fairness should be well-documented and beyond reproach. 

Your professional responsibility is to have the service provider, use your data from your new hires, to calculate fairness statistics for your organization.  Failure to do so is to neglect your duty.  It is normal to conduct the fairness analyses at the same time the predictive validity study is being attended to. 

If you proactively conduct return on investment analyses, it is likely that you already utilize your applicant flow data to calculate metrics such as time to hire, cost per hire, new hire quality and source channel efficiency.  At the same time, it is a good idea to use your applicant flow data to ensure that your 4/5ths rule statistics are in compliance with guidelines from the Equal Employment Opportunity Commission (EEOC).  If you need help with this, please reach out to the psychometric-solutions.com team. 

Data Requirements

• Content Validity – at minimum, requires a job description and job analysis to establish the job relevance of questions used in the assessment. 
• Reliability – requires assessment score data. i.e., the scores each person’s response received when he/she responded to each question. 
• Predictive Validity – requires job performance and assessment score data. 
• Fairness – requires job performance data, assessment score data and demographic data (e.g., gender, age and race category). 

This concludes our brief tour of the criteria you need to use when evaluating “Assessment Effectiveness.”

Summary

When evaluating Assessment Effectiveness pay attention to validity, reliability and fairness. 

Validity pertains to the accuracy of the decisions you make on the basis of assessment scores.  Effective assessments produce scores that empower you to make more correct than incorrect selection decisions.  When evaluating an assessment’s content validity, job relevance is essential.  Pay significant attention to the predictive validity coefficient computed using your own data.  Only renew your contract if the predictive validity coefficient, computed using your own data, is suitably high.  Expect numbers in the +.40 to +.60 range. 

Reliability is usually well-established and documented.  Simply ask for it, as your assessment provider will have it ready and available for you.  Prior to your purchase decision, ensure that the Internal Consistency Reliability coefficients provided for similar jobs, from other clients, are consistently above 0.80.  Then, ask your assessment provider, how soon they could calculate this coefficient using your own data. 

Responsible assessment developers complete several studies to establish fairness before delivering their products to clients.  As a responsible client, seeking to maximize the Return on Investment for your company, ask the service provider to use data from your new hires, to calculate fairness statistics for your organization. 

We hope you found this blog article helpful.  For more information or assistance, please contact us at info@psychometric-solutions.com

Future Blog Article

1. The correct data and metrics to use to evaluate each individual’s job performance.  What are the keys regarding appropriate and accurate Measurement of an Individual’s Job Performance? 

References

Equal Employment Opportunity Commission, the Office of Personnel Management, the Department of Justice, the Department of the Treasury, the Department of Labor’s Office of Federal Contract Compliance, the Civil Service Commission and the Commission on Civil Rights (1978).  Uniform Guidelines on Employee Selection Procedures.  Washington, DC: Author.

Equal Employment Opportunity Commission, the Office of Personnel Management, the Department of Justice, the Department of the Treasury, the Department of Labor’s Office of Federal Contract Compliance, the Civil Service Commission and the Commission on Civil Rights (1978).  Uniform Guidelines on Employee Selection Procedures.  Washington, DC: Author.

McDaniel, M. A., Whetzel, D. L., Schmidt, F. L. & Maurer, S. D. (1994).  The validity of employment interviews: A comprehensive review and meta-analysis. Journal of Applied Psychology, 79 (4), 599-616.

Robertson, I. T. & Smith, M. (2001).  Personnel selection.  Journal of Occupational and Organizational Psychology, 74, 441-472.

About the Author

Dr. Dennison Bhola has more than 25 years of experience working in the field of psychometrics creating validated instruments such as assessments, structured interviews and inventories.  With his teammates, he has developed more than 1000 assessments for use by organizations around the globe.  His expertise includes evaluating new hire quality, assessment effectiveness, source channel efficiency, time to hire, cost per hire, and other such metrics that are of critical importance to Human Resource Professionals.  He also specializes in fairness analyses and auditing compliance with Equal Employment Opportunity guidelines, such as the 4/5ths rule. 

In this article, we provide crucial questions to ask when choosing an assessment or interview for employee screening or personnel selection on behalf of your organization.  These questions pertain to psychometrics, equal employment opportunity law and common sense. 

In order to make your purchase decision, you should be comfortable finding out whether your New Hire Quality will improve if you use the assessment results to make decisions as prescribed. And, within what time frame will the supplier be able to use your actual New Hire information to evaluate the “correctness” of your hiring decisions. 

Here are some smart questions to ask your assessment provider:

1. Will people who obtain higher assessment scores be more productive?
2. Was the assessment developed with samples of people like your job applicants in terms of age, gender, race, prior job experience, and highest level of education attained? 
3. Will your job applicants understand the language used in the assessment?

Note: Interviews are one type of assessment.   If your interview does not provide reliable numerical scores, it should be replaced with one that does. 

As you do your research, you can also use your specific job description to ask more detailed questions such as:

1. Will people who obtain higher assessment scores treat our customers with greater courtesy and respect?
2. Will people who obtain higher assessment scores collaborate more productively with teammates? 
3. Will people who obtain higher assessment scores be more creative?
4. Will people who obtain higher assessment scores be better problem solvers?
5. Will people who obtain higher assessment scores be more efficient time managers?
6. Will people who obtain higher assessment scores consistently show up for work when they are supposed to?
7. Will people who obtain higher assessment scores be able to stay on task when there are distractions in the work environment?
8. Will people who obtain higher assessment scores work hard without constant supervision?
9. Will people who obtain higher assessment scores proactively seek out opportunities to learn on their own time? 

Your assessment or interview manufacturer has answers to all of these questions because they are critical to you making an intelligent purchase decision.  If your assessment provider cannot answer all of these questions, then you have valid reasons for putting their product quality under intense scrutiny.  As the customer and leader investing in the future of your organization, you need answers to these simple questions in order to make intelligent purchase decisions. 

Summary

When purchasing an off-the-shelf assessment for employee screening or personnel selection, please ask your assessment provider:

1. Will the interview/assessment improve my New Hire Quality?
2. Do you have documented evidence that scores from the interview/assessment are fair to male and female candidates, people 40 and older vs. people under 40, and people from different ethnic groups or race categories?
3. Will my job applicants be able to understand the language and examples used in the interview/assessment?

We hope you found this blog article helpful. For more information or assistance, please contact us at info@psychometric-solutions.com

Related Blog Articles

1. Do I purchase the assessment outright or do I license the product? 
2. Is a custom assessment solution better? Why?
3. How to accurately measure an Individual’s Job Performance. 

About the Author

Dr. Dennison Bhola has more than 25 years of experience working in the field of psychometrics creating validated instruments such as assessments, structured interviews and inventories.  With his teammates, he has developed more than 1000 assessments for use by organizations around the globe.  His expertise includes evaluating new hire quality, assessment effectiveness, source channel efficiency, time to hire, cost per hire, and other such metrics that are of critical importance to Human Resource Professionals.  He also specializes in fairness analyses and auditing compliance with Equal Employment Opportunity guidelines, such as the 4/5ths rule. 

The term “Psychometric” comes from the place where Psychology, Measurement, Research Methodology and Statistics overlap. 

Psychometrics helps HR professionals answer questions such as:

• Is my hiring process efficient?
• Is my process getting me high quality employees?
• Which part of my process is most effective?
• Are we getting people with the attitudes our organization needs to be successful in the long-term? 
• What is the predictive validity of each part of my process?
• What is the ROI from each part of my process?

As an HR professional, you’ve probably used the terms Psychometric Assessments and Metrics during communication with your colleagues and clients.  The terms assessments and metrics both come from the field of Measurement in which numbers are scientifically assigned to quantities.  Most people are familiar with test scores that they obtained in high school from teacher-made assessments, college entrance exams or pre-employment interviews.  The numbers from these interviews and assessments are possible because of the field of Measurement. 

What are psychometric assessments?

In Psychometric Assessments, measurement involves

• Asking the right questions
  • Scoring the responses with an algorithm to create a score that has predictive validity

Predictive validity answers the question: Do people who score higher on the assessment ACTUALLY perform better on the job?

Job Performance is defined in the context of the job, and could include answers to questions such as are they more productive, do they communicate well, do they collaborate well, do they apply common sense to solve problems, are they creative?

• In Psychometric Science we apply rigorous Research Methodology and appropriate Statistics to develop customized psychometric assessments by studying your people and using their job performance data. 
• We are therefore able to develop psychometric assessments which help you to identify more people whose attitudes, work ethic and skills are like those of your top performing employees, and that differentiate them from your other employees. 
• To be in compliance with Equal Employment Opportunity Law, Psychometric Science addresses:
  • Validity
  • Reliability
  • Fairness

For details about validity, reliability and fairness, see our blog titled: Psychometrics 101 for HR Professionals: Evaluating Assessment Effectiveness.

Summary

Psychometrics provides information for evaluating:  New Hire Quality, Assessment Effectiveness, Sourcing Channel Efficiency, and compliance with Equal Employment Opportunity Law. 

Psychometrics answers essential questions such as:  Do job applicants who score higher on an interview actually perform better on the job?  Are all of our assessments of high quality?  Is our selection process getting our organization employees with the attitudes, knowledge and skills we need to be successful in the future? 

We hope you found this blog article helpful.  For more information or assistance, please contact us at info@psychometric-solutions.com

Related Blog Articles

1. Smart Questions to ask when Purchasing an Assessment for your Organization.
2. Psychometrics 101 for HR Professionals: Evaluating Assessment Effectiveness.
3. How to accurately measure an Individual’s Job Performance. 

About the Author

Dr. Dennison Bhola has more than 25 years of experience working in the field of psychometrics creating validated instruments such as assessments, structured interviews and inventories.  With his teammates, he has developed more than 1000 assessments for use by organizations around the globe.  His expertise includes evaluating new hire quality, assessment effectiveness, source channel efficiency, time to hire, cost per hire, and other such metrics that are of critical importance to Human Resource Professionals.  He also specializes in fairness analyses and auditing compliance with Equal Employment Opportunity guidelines, such as the 4/5ths rule.