May 29, 2020

Remote Hiring – A Paradigm Shift

Remote Hiring

Covid-19 pandemic has pushed many companies beyond the tipping point to adopt remote working as the new normal. Remote working is quite easy and natural for the knowledge workers – especially those working in the field of software development. This article focuses on how remote working has impacted hiring software engineers and the demands that it puts on the hiring tool being used by the companies.

Accountability and transparency

Talent acquisition managers would no longer have their teams of recruiters next to them. They would feel disconnected and helpless due to the lack of visibility. This feeling would be even more acutely felt while working under time pressure. They will, on a daily basis, need to know what their teams of recruiters have been doing. There has to be a proper measure of performance. E.g. if a recruiter gathers a lot of missing information which results in rejection of the candidate, then the recruiter should be get recognized for saving the time of all the stake holders and for preventing a potential bad hire. Companies should use tools that have proper metrics to measure the performance of their remote teams. In addition to the conventional quantitative metrics such as number of candidates sourced, screened and tested, companies should focus on qualitative measures such as average suitability score. The tool must judiciously assign importance to various selection criteria while determining suitability.

Removing bias and getting feedback

Personal biases introduced by likes and dislikes get discovered in face to face interactions. Decisionmakers subconsciously apply corrections to arrive at an unbiased decision. We need to have a mechanism to remove biases and look at one version of truth in a remote team. A tool that has a metric associated with each selection criterion lends itself well for such a mechanism.

Such a tool would also allow companies to assign relative importance to these selection criteria by assigning weightages. The rationale behind assigning numerical weightages to these selection criteria while calculating the suitability score can be shared with the hiring managers. In fact, hiring managers should be encouraged to fine tune the requirements by refining these weightages. Changing weightages for a reason could be a way to capture and convert verbose feedback into meaningful numbers.

Technical Assessment

Most companies administer technical tests to assess programming skills. Some of the companies are using on-line technical tests. The pandemic situation has made it virtually mandatory to conduct the technical assessments on-line. Technical assessment can be a quick test with a few MCQs or a grueling code sharing and pairing session. Given the time and attention needed by an expert interviewer to do the later, it might make sense to use the former as a filter. Mobile based gamified tech quiz can make it easy and light for the candidates. There is a bit of a resistance from the senior candidates to undergo online tests. One way to get them to agree is to explain the shortage of technical panels to do a deep dive coding session with each and every candidate. Most of them see the merit of a quick and light quiz as compared to a telephonic round by a non-technical or a semi-technical interviewer.

It is highly desirable for the tool to ensure full video proctoring to avoid cheating. Also the questions that get asked should be picked from a large question bank to avoid repetition. Also the tool should automatically delete questions that everyone can answer or those that no one can answer.

Overcommunication and Collaboration

There’s no watercooler or cafeteria in the virtual world. Information that used to flow unhindered through these social interactions needs to flow even more rapidly to compensate for the lack of attention and context.

There are many tools that capture the first level of information. E.g. A candidate was rejected in an interview or another candidate submitted his quiz. However it lacks the next level of detail or the color. Why was the candidate rejected? Was he not as good at SQL server as we had assessed? Did the other candidate score well? How long did he take to complete the quiz?

There are many questions which don’t get asked because of the social and physical distance. E.g. a TA Manager won’t bump into a hiring manager and casually ask how he is finding the quality of candidates? Or there is no opportunity for a recruiter to ask why a candidate was rejected? Companies should acquire a tool that proactively answers such questions. Every piece of information needs to be supported by next level of data- ready to provide the detail should someone ask. The tool must provide a reason why a candidate should be considered for a job. The tool must tell why the candidate who was top ranked suddenly went down. The tool must encourage the hiring managers proper feedback and advice to the rejected candidates.

Companies should acquire a hiring tool that captures important events and alerts all the interested stakeholders using popular channels like WhatsApp and Skype. It’s also important to make these events actionable by providing links from the message to the appropriate page in the application.

There are multiple people collaborating to get work done. The agency sponsoring the candidate, the hiring manager, competing candidates and the recruiter should be alerted when the candidate successfully completes and submits the technical quiz. The interviewee candidate, the recruiter and the agency would be interested when the interviewer submits her feedback. The prolific alerts for all events would keep all the stakeholders abreast of the situation.

Employer Branding

The hiring tool should more than make up for the absence of handshakes and smiles, by providing a world class candidate experience. E.g. even rejected candidates should get proper feedback and valuable career advice. There’s nothing more frustrating than lack of communication that follows the interview. Providing proper feedback with current standing of the candidature in terms of the rank would make your company look very professional and enhance the employer brand value in the larger candidate population.

Conclusion: What has changed and how hiring tools should adapt?

Here are some challenges that hiring tools will need to address

Remote interviews have already become the norm. Now companies need to get more information about the candidates from all possible channels to make up for the lack of face to face interaction.
Companies need to identify gaps in the information shared by the candidate in her resume and get all such missing information from the candidate
Remote hiring is not only about remote interviewing. Companies need to remotely collaborate with all the stake holders including the recruiters, talent acquisition managers, hiring managers, interviewers, vendors and candidates.
Our tool must present a consistent view of not only the jobs and applicants – but also the status of each applicant in terms of her suitability, rank and interest to all the stakeholders.
Stakeholders should find easy answers to questions like why candidate X is more suitable than candidate Y and what Y should do to improve her ranking. This will help them drive the process to accurately identify the suitable candidates to minimize waste and disappointment.
To make up for the lack of warmth afforded by in person interviews, Companies need to go an extra mile to provide not only proper feedback but also some advice to the rejected candidates.

You could request a demonstration of Rezoomex- a hiring tool that promises to address all these challenges

January 28, 2020

AI spells better times ahead for the subject matter experts

It is commonly believed that AI will empower the employees who are willing to learn – others will find it hard to keep their jobs. But there’s an even bigger opportunity for the subject matter experts. They will not only learn the changes being brought about by AI , but permanently enrich their jobs by becoming the trainers and administrators of the AI apps.

Firstly, let us understand why a human being – no matter whether she is a subject matter expert or not, is needed to train the modern AI systems.

Human-in-the-loop systems

Humans will be instrumental in building what are known as human-in-the-loop systems.

As per this article by Mothi Venkatesh human-in-the-loop system combines “Supervised Machine Learning” and “Active Learning”. I am reproducing the definition from the article.

HITL=SML+AL

Supervised ML, curated (labelled) data sets used by ML experts to train algorithms by adjusting parameters, in order to make accurate predictions for incoming data.

Active Learning, the data is taken, trained, tuned, tested and more data is fed back into the algorithm to make it smarter, more confident, and more accurate. This approach–especially feeding data back into a classifier is called active learning.

It is obvious that we need a human being where the confidence level in the algorithm is low due to insufficient training data or the risk associated with an inaccurate inference is very high. But why would we need a subject matter expert. They are not just another “human” augmenting the algorithm’s artificial intelligence

Subject matter experts as trainers and administrators of AI

This article by Daniel Faggella explains the need for subject matter experts. I am reproducing three ways as written in the article.

Determining the Business Problem an AI Solution Can Solve. For example, let’s say an insurancecompany wants to become more competitive. A high-level employee at the company, who has been in the industry for more than 20 years, may know from experience that clients often complain about the time it takes to process claims.
Determining How AI Can Solve That Business Problem. For example, a marketermight tell a data scientist to make sure the model they build can predict the best price to bid for ad space on Google ad Network without realizing that such an application is nascent in practice and technically challenging. A data scientist would be able to explain this to the marketer, preventing them from promising their team and stakeholders too much.
Maintaining and Updating the System Once It’s Built. Take for example a manufacturingcompany with a large number of heavy machinery to maintain. The subject-matter experts collaborate with data scientists to develop an AI system that gathers data from IoT sensors on the machines to predict the optimal time to do preventive maintenance on them.

MLTrainAdmin

To understand this better we need to take a closer look at how the AI apps work.

Every AI app needs training data that needs to be prepared by a human expert. E.g. we need to have a lot of data about the films and the people who watch those films to intelligently recommend a bunch of films to a new user.
After preparing the training data a data scientist could use this data to train and test an algorithm. He will repeat this step till he finds a suitable algorithm to generate a model that can predict or infer as precisely as the human expert who has prepared the training data in step 1. This step could take several iterations.
Once the algorithm which can most precisely predict , the same can be used to create a model. This model is handed over to the subject matter expert to be used for predictions or inferences.
The SME uses this model for day to day predictions and inferences. He applies his own judgement to decide if the predictions and inferences are precise enough by comparing what the algorithm has predicted or inferred with what the SME would have done.
If the model is not accurately inferring or predicting the SME would override the algorithm and manually assign predicted or inferred values.
This edited data is added as new input data and the same is used to retrain the algorithm. This retraining will ensure better results going forward. Step 4 to 6 are repeated by the SME with no intervention from the data scientist. The SME would really make the application more and more precise. As it becomes more precise it will more closely mimic the expert’s inferences and predictions.

Who can be a subject matter expert.

You don’t have to be highly placed or academically well qualified. There’s nothing better than years of expertise. E.g. a security guard knows how to spot an intruder in a crowd. He can help a data scientist build an automated threat detection system

January 5, 2020

IT hiring as seen by interviewers/hiring managers: Part 2

In this part, we will propose a solution to the problems perceived by the hiring managers as concluded in part 1 of this article. Ideally, there’s no need to have such an elaborate solution if the hiring manager did everything from writing the job description, determining and communicating the selection criteria to conducting the interviews. Practically as we have learned, only 50% of the hiring managers write the job descriptions. Only 64% of them are engaged in determining and communicating the selection criteria and most of the interviews are conducted by interviewers who are not the hiring managers. Hence the need to have a solution that facilitates collaboration and communication.

The diagram below describes the proposed solution. Here are the steps of the solution.

ProposedSolution

Step 1– Hiring manager creates the job description and the software will help him decide the selection criteria based on the JD.

Step 2– A software service applies the selection criteria to the resumes uploaded by the recruiters and computes the suitability scores.

Step 3– Top-rankers based on the suitability score are identified. Recruiters follow up to gather any missing information and administer a technical quiz to each one of them.

Step 4– The technical quiz is a quick, in-camera assessment, conducted on the candidate’s smartphone. The candidate can take the short 20-minute test at her own convenience. Recruiters can ensure authenticity by checking the video recording.

Step 5– Only those few top-rankers who score well in steps 3 and 4 are presented to the interview panel.

Step 6– Feedback from all the interview rounds is processed by the hiring manager and any obvious disconnects between the recruiters and the interviewers can become visible to the hiring manager on the dashboard. E.g. The top-rankers as per steps 3 and 4 get rejected and bottom-rankers get selected.

Step 7– Based on the results as seen on the dashboard the hiring manager can tweak the selection criteria and the technical quiz to align the recruiters’ and the interviewers’ understanding of the requirement.

Let’s take a closer look at the problems identified in the conclusion section of part 1 of this article to understand their magnitude and severity. Let’s also examine if the proposed solution addresses those problems.

Problem 1 : The interviewer is not completely plugged in and it results in a lower hit rate. On an average 3 out of 5 candidates are found to be unsuitable in the face to face interviews. This may be the tip of the iceberg as we are assuming that those who are selected are perfectly suitable. Bad hiring decisions are rarely acknowledged.

Solution 1: Sharing the job description, and the selection criteria with all the interviewers will improve everyone’s understanding of the real requirement. Assigning weights to the selection criteria would further refine this understanding. It will bring cohesion between the job description, the selection criteria, the quiz questions and the interviews in steps 2, 3 and 5.

Problem 2: As already seen in part 1 of this article majority of the interviewers think that they are wasting a lot of their time in the hiring process. On average an interviewer spends 45 minutes per interview.

InterviewTime

Solution 2: Justify to the interviewer the time spent by her by sharing the reason why the candidate deserves to be selected, with specific reference to the weighted selection criteria as stated in steps 2 and 3. This will positively orient the interviewer going into the interview, and keep the preliminary screening process honest.

Problem 3: The recruiters are unwilling to reject candidates at their level. It’s obvious that the recruiters need better tools than preliminary phone screens and tests. Technical tests are inconvenient and time-consuming. Some candidates cheat while appearing for the online tests.

Solution 3: Improve overall quality in the preliminary screening process by doing the following

Recognizing fact-finding work resulting in the rejection of candidates by the recruiters as their value addition. Presently, in most companies, the recruiter gets no credit for rejecting a candidate at her level. We should stop looking at the number of interviews scheduled as the measure of the recruiter’s performance. Recognizing the value of rejection will bring quality consciousness and reduce the stress placed on the interviewers by mindless overcrowding of the recruitment pipeline.
Having some kind of metric to measure the suitability as depicted in step 2. Such a metric will encourage the recruiters to focus their attention on sourcing more suitable candidates. Recognizing average and total suitability scores as measures of the recruiter’s performance would improve the choice and quality of candidates.
Having a lightweight and efficient technical screening in step 4. This would eliminate candidates who can’t answer simple questions on topics that are important.

Problem 4: The formal feedback process is geared to capture the overall impression of the interviewer, but doesn’t help to improve the next batch of candidates. We need to learn and improve by a better understanding of the reasons for rejection.

Solution 4: The interview feedback should require the interviewer to indicate his acceptance or rejection against the commonly shared weighted selection criteria and the reasons for proposing the candidate as suggested in solution to problem 2. This will enable the recruiters and the interviewers to change the selection criteria as needed and capture the reasons for such changes in the interview feedback forms.

Problem 5: Interview feedback is not shared with all the stakeholders. As the purpose of the feedback is more for documenting the reasons for selection, the same is shared with HR and other managers who are supposed to take further action upon the selection of the candidate. The need to share it with the recruiters is not felt. Recruiters often don’t have access to the detailed feedback provided in the system. They have to remain content with short “Selected” or “Rejected” status that shows up in the system. Part of the problem is also because the recruiters rarely state the specific reasons why a candidate deserves to be selected, hence they forfeit their right to know the reasons for rejection.

Solution 5: In step 6- the interview feedback process the interviewer could provide some tips on what the recruiter could have done to reject the candidate at her level. This information can be used by the recruiter to improve her hit rate. With some additional work, the recruiter can also pass on a part of the feedback to the rejected candidates. Useful for the candidate to improve herself. Maintaining this level of transparency will go a long way in building the employer’s brand.

December 6, 2019

IT hiring as seen by interviewers and hiring managers: Part 1

In a recent survey conducted by the author, the majority of the interviewers said that the hiring process was wasting a lot of their time. More than 20% felt the need to complain as they could use that time for more important project-related work. About 50% didn’t complain, although they felt that their time was being wasted. TimeSpentRating

Why do they get involved?

Are the interviewers a willing party to this waste of time? Why do they get involved? A short survey showed that only a few times the hiring manager interviews candidates for a project in which he or she is directly involved. This means he or she is merely an interviewer working for some other hiring manager. The majority of the time the interviewer is roped in by HR or by his superior for his or her expertise. Organizations do want their best experts involved in the selection process. There are a few times when the interviewer gets called in at the last moment because the pre-appointed interviewer is unavailable. Would the interviewer serve as a good proxy for the hiring manager? Would he be as familiar with the job for which the candidate has applied as the hiring manager? One wonders what impact all this has on the quality of hiring.

Broken communication starts with the job description

The hiring manager is supposed to write the job description and decide the selection criteria. The interviewers are supposed to select candidates based on the selection criteria and the JD. Very often the JD is not written by the hiring manager. Some hiring managers said that they don’t look at the job description, many said that they believe in a short one-line description of the requirement.

Does everyone know when the Job Description changes?

Given this initial low level of participation in the creation of the JD by the hiring managers, it’s not a wonder that only 20% of the respondents are able to always keep using the initial job description. In the remaining 80% cases, the job description changes. When it does, ideally, the hiring manager should change the selection criteria.

Surprisingly 36% of the respondents didn’t see any need to communicate this to the recruiters and “Almost Always” kept the selection criteria secret! It must be quite frustrating for the recruiters as they are left guessing the real job requirements.

How do you communicate changes to the selection criteria?

selectionCriteria

Impact of broken communication on the hiring process

We asked the hiring managers whether they noticed some gaps in the way the hiring process identifies and short-lists candidates. They were unequivocal in their feedback – “Sometimes it works, sometimes it doesn’t” (see the table below). There are some obvious facts that the recruiter can check at her level even before sending a resume to the hiring manager. E.g. Whether the candidate has relevant experience? Whether the candidate’s pay expectations are within budget? Has the candidate been a rolling stone?

What are some of the obvious weaknesses in the resumes that are overlooked in the pre-screening stage?

RecruiterMisses

When we talked to some recruiters, we discovered that they were maximizing the number of candidates interviewed. As a result, there is an overall resistance by the recruiters to reject a candidate. They would like to maximize the chances of selection by recommending as many candidates as possible. As a result, the interviewer’s available time is quickly consumed. This has two downsides

With so many candidates to be interviewed, multiple interviewers conduct interviews for the same position. This makes objective comparison difficult. Hiring standards become inconsistent and the quality of hiring suffers.
Interviewers are overworked. Fatigue sets in. The communication between the interviewer and the recruiter becomes tenuous. Recruiters feel that their hard work is not rewarded and the interviewers are frustrated by interviewing unsuitable candidates.

Are we using the interviewers’ feedback to improve the success rate?

We discovered that the same reasons for rejection appeared multiple times in the feedback provided by the interviewers. Does that mean that the recruiters are not using the feedback to improve their selection? In which form is the feedback provided by the interviewers?

We asked this question and please see the responses we received in the table below.

What is the mechanism for providing interview feedback?

IntFeedback

It is evident from the table that there’s no commonly followed standard process. Each company follows its own process. The majority (17 out of 25) seem to be using some kind of form. It appears that the primary objective of the form is to put on record the impressions of the interviewer.

We are wondering whether the feedback reaches the recruiter and whether the recruiter is able to change the selection criteria based on the feedback. These two are topics for further research.

Reasons for rejection would reveal what changes we need to make to the selection criteria. Informal conversations with the hiring managers led us to believe that the feedback forms are mainly meant for justifying the selection and not enough importance is given to the reasons for rejection. Also as seen in the table, about 40% of the respondents “frequently” send emails justifying selection, but find no need to justify rejections. Without this information, one wonders if the interviewers’ feedback is used for reducing the number of rejections.

Conclusion

We can conclude that hiring for software engineers is a broken process

Not every interviewer is “plugged in” to have a good understanding of the job.
Most interviewers and hiring managers feel that their time is being wasted- even though majority don’t complain.
Recruiters are averse to the idea of rejecting candidates. They often overlook easily detectable weaknesses evident in the resumes. They try to maximize the number of interviews. This results in overwork, fatigue and tenuous relations between the interviewers and the recruiters.
The established feedback mechanism doesn’t serve the purpose of improving selection rates. More stress is given on the reasons for selection than those for rejection.
Selection criteria do change but the links carrying the communication back to the recruiters are weak.

We will take a look at the impact of these shortcomings on the organizations and suggest some probable solutions in part 2 of this article.

September 22, 2019

There’s an elephant in the room- let’s build a pragmatic, objective and transparent compensation policy! PART 2/2

As stated in part 1, this is an experience report from the author’s company where a more pragmatic, objective and transparent model is being used to determine compensation. For most skills having a sufficient number of jobs and job-seekers, a market mechanism becomes operational. Ruling market rates are the most impartial and objective way to determine compensation.

Boundaries that define the “Job Market”

There are 4 factors that define “market” for the purposes of employee compensation:

Geography- Inflation rates vary from country to country depending on the economics of that country. Fast-growing economies such as China and India tend to have higher rates of inflation.
Skills- Certain skills tend to be more in demand. Recently “Data Science” skills are in demand, which has resulted in better compensation for Data Scientists

Experience- A professional becomes better at his job as he gains more experience. This trend tends to taper off after 7 years. At which point one can think of up-skilling to get a better job. The graph above shows how compensation reaches a limit after a few years in a job. The only way to break out to the next compensation level is by up-skilling to acquire a better job.
Performance- If we do A/B/C grading for employee performance on the job, A graders tend to do significantly better and C graders tend to do significantly worse than the average.

How to determine “market” compensation

We can practically determine the ruling market compensation by taking a small randomly selected sample. The sample should be large enough so that it results in a bell-shaped frequency distribution. We sampled candidate data sourced from job portals, employee and agency referrals for certain job skills and experience ranges in certain cities. Please refer to one such frequency distribution in the graph below. We observed that most samples resulted in a slightly right-skewed bell shape for sample sizes as small as 50!

Model for determining compensation

The compensation model

We used these frequency distributions to build a compensation model that passes the following tests –

Can the model be shared with all employees to bring transparency and objectivity
Can the model manage non-uniform expectations based on skills and experience?
Can the model lend itself to hire talent by offering good pay hikes to the new hires, without upsetting the loyal incumbents?
Can the model account for special rewards for the superstars, thus avoiding ad-hoc negotiations?

Our company conducts quarterly performance appraisals in which employees are appraised of their last quarter’s performance by their immediate superior in an hour-long meeting. They can openly discuss and understand what OKRs they need to achieve in the next quarter to achieve an improved grade. We have successfully managed employees’ expectations by clearly linking percentile of pay level with grade achieved in an appraisal. (Refer to the table above) E.g. All “B” graders are paid at 75th percentile. There is total transparency as any employee can study the frequency distribution based on her technology skill and experience level. She can then decide whether to achieve better pay by up-skilling to a hot technology or to build deeper expertise in the same technology. As this model is based on the frequency distributions it makes it much simpler to uniformly relate performance to percentiles across multiple experience levels and technology skills.

Here’s a link to the frequency distributions we have compiled based on samples taken from 28 technologies and 5 experience ranges. We have made this data accessible by writing a web-based application called Paywatch. Our company has been Paywatch for building our compensation model as described above. You may want to use this data which is updated on an annual basis as it’s open for all to use and provide feedback.

We have carefully drawn samples across various experience levels to compile the Paywatch frequency distributions. The report is categorized by technologies because hot technologies like AI and Big Data tend to have higher pay levels as compared to older technology categories like Java or Dotnet. Your company can be easily benchmarked for each of these categories and experience brackets. Companies can achieve transparency by using Paywatch to formulate a rational compensation model. The model can be made transparent as employees can verify the pay levels by using the Paywatch frequency distribution relevant to their technology skills and experience levels.

Results

This model is broadly known to all employees in the company. Those recommending pay raises have been briefed in meetings and via e-mail. We have been using Paywatch for the past 3 years for determining employee compensation. We have quarterly appraisals. A few (less than 1%) employees request reviews of the raises recommended by their superiors. Though Paywatch is open to all, not many employees actually use it to cross-check. This article is also an attempt to get our employees to use Payatch and provide their valuable feedback.

Initially, we observed that some managers were more lenient in grading and generous in recommending raises. These anomalies stood out as they were clearly violating the Paywatch frequency distribution. We had to educate those individuals by asking them to justify their actions in writing. Some of these justifications were not valid. E.g. 50% of the team members can’t be top 5% performers. When we shared the feedback with all the managers, it resulted in a common understanding of the model. This helped us to bring uniformity and remove any individual bias. In fact, the model works so smoothly that recent raise recommendations needed no moderation or intervention by the HR department. There is so much transparency that no one is afraid of discussing her pay with others. In fact, a glitch in our payroll software had resulted in many employees getting pay-slips of many of their colleagues which was very embarrassing for the accounts and the HR folks, but it didn’t result in any dissenting or attrition. There was no surprise for anyone there!

We use the same model for making offers to the new hires. That is the mechanism to keep us honest. The fact that most offers were accepted, and less than 1% of the offers were rejected on the grounds of low pay, validates that our model is well synchronized with the market. It also proves that we have successfully avoided having double standards in determining pay for the new hires vs the loyal incumbents. New hires are smoothly on-boarded and accepted by the incumbent teams as there is no secrecy about the pay offered.

Is this model the silver bullet?

We aren’t suggesting this as a silver bullet to address all the issues related to determining employee compensation. This is a sincere attempt to take on this big challenge instead of turning a blind eye. This is just a modest beginning. Employees, HR, and top management have to engage in a meaningful open debate using popular formats like fishbowl or breakout groups to discuss this topic and come out with a solution that is acceptable to all in the given context. Only then would we reach a decision to accept-

Solutions or partial solutions
“Workaround” in the absence of a solution
The overall agreement that there is no solution or “workaround” likely in the immediate future
Long term changes that need to be implemented to avoid some of these problems

Is this the only way?

There are many consultants who would charge a fat fee to do a benchmarking study and recommend pay structures. Paid studies are good for getting a one time snapshot of pay levels – but it would be unviable to get your company benchmarked every year.

Popular platforms like Glassdoor and Payscale crowdsource the compensation data. Even if we assume that the information volunteered by the users is accurate, these platforms have no control over the sample – making it hard for them to draw statistical inferences.