September 11, 2017

Is Automation Eating Jobs or Creating New Ones

-based on Atul Jadhav’s views in a tweetchat hosted by Rezoomex

Historically technology has always created new jobs by automating repetitive/mundane work with predictable expected results. Human brains evolve as they learn new skills to solve challenging problems. This leads to creation of new job types which are more evolved yet challenging . e.g. Delivery by drones may take up delivery jobs however this automation itself opens up a number of specialised jobs with higher expertise. E.g. GIS experts, geospatial data analyst/scientist, algorithm designer/ developers, control centre operators, security experts etc.

shutterstock_152563265

But hundreds of delivery boys can’t learn these skills – they would lose their jobs unless they evolve. Change is hard especially for the unprepared. But we need to learn from experience. We have come a long way from postcards to emails.

We often come across cases of individuals like a manual tester known to the author. This tester has spent all the 10 years of his career doing manual testing but now he is unable to learn automated testing-as a result he is losing his job. Evolve or perish, that’s the mantra! Some people can’t evolve- either because of inaptitude or inertia.

Not all jobs will have a clear evolution path like typists who evolved by learning word processing as typewriters were being replaced by personal computers. It can be concluded that automation will impact low skilled jobs and create new jobs that use human intelligence and creativity. Upskilling is the way forward for those affected.

Another angle to consider is that we need not work as much as we used to because automation will give us more free time to undertake hobbies or creative and innovative pursuits. But looking at 140 years of data automation has always created more work.

Here are some points to ponder-

1) Examples of jobs that have been made obsolete by automation

In manufacturing robots have replaced workers with significant increase in productivity. Bank teller job is automated, check out clerks in the supermarkets are being replaced by automated point of sales machines. Telephone operators, travel agents , timekeepers, typists, film projectionists, packing labour, stockroom/warehouse managers, farmers, data entry operators, call center operators, weavers, knitters the list can go on.

2) Jobs that are not yet obsolete but are at risk

Research says by 2021,four out of ten jobs will be lost to automation. Tasks needing low skills that are highly transactional can be automated using AI & ML. (Artificial Intelligence and Machine Learning)

Like delivery boys losing their jobs to the drones, taxi and truck drivers too will follow suit if Daimler, Tesla & Google succeed in their plans to make vehicles driverless. The list can go on- nursing and health care, insurance underwriters, geographers, nuclear technicians, financial advisors, news reporters, loan officers, accountants and auditors. Many of these jobs will have their next version, different set of OKRs to serve much more evolved customer demand and business models. About 40% of the jobs will go away in the next 5 to 10 years. We can’t say for sure if your job is one of those. Hence its safer to assume that all jobs will become obsolete in the next 10 years.

3) New IT jobs that are recently created by automation

Technology is evolving faster than ever,IBM Watson, Google’s Deep Neural Network (Tensorflow) and several other AI & ML platforms require humans with skills to implement various business models and use cases on those platforms. New breed of systems integrators called Robotic Process Automation (RPA) implementers are building systems with optimised work flow. Skills like DevOps are high in demand due to the need to further automate the service offerings via CI & CD (Continuous Integration and Continuous Delivery). CD results in effective use of infrastructure resources by reducing the “Time To Market” for new products or services in addition to improving the efficiency and reducing the cost. Also in demand is the skill to convert old monolith applications to containerised microservices. Containerised microservices with container orchestration are taking the discipline of DevOps to the next level.

4) Industry verticals most likely to be Impacted by automation

Businesses today are striving for enhancing their customer experience. They are building products and services which will help them to achieve the same results more quickly and effectively. At the same time businesses – both B2C and B2B – are collecting lot of data about behaviour, productivity and performance issues.

E.g. your car goes to refuel itself from the nearest gas station while you sit in your office focusing on work or spending time with family without worrying much about gas level or spending that extra 10 minutes at the gas station. The data required is sent to the service station 24/7. When the car goes to get the servicing done by a fully automated service robot, it already has the data about all the vitals of your car along with exact issues. And this scenario isn’t as far fetched as it sounds and the day isn’t as far as is commonly believed.

Almost all industries are going to leverage the power of AI & ML. To name a few – healthcare, agriculture, manufacturing, transportation, customer service, finance and defence are going to be benefit from these technologies.

As a result new jobs will come into existence. Some of the tasks for almost all the existing jobs will need new skills. All of us have to keep learning new skills. We are all running on a treadmill. Running hard to stay where we are! And the speed is increasing continuously. e.g. some years back DevOps skills were in demand . But today developers are writing containerised microservices. They can both build and deploy without any help from DevOps. This way of working called NoOps is making DevOps skills redundant.

5) Opportunities for IT Professionals in Non-IT Sectors

IT departments in non-IT organizations will benefit from the advances in technology. Presently every company does have a technology department however they will be needing serious updates to the latest technology which will be a challenge in terms of costs to be incurred and new skills to be acquired. Consequently technology would become central to the future strategies of these companies.

To build the smart systems mentioned in point 4 above, we will require skilled IT professionals who are not only capable of developing solutions based on intelligent platforms but are also quick learners. As Non-IT sectors start being heavily driven by IT more techies will be needed to maintain existing systems, to cater to new business cases, to alter existing business cases while ensuring the data security. Governmental rules and regulations will evolve based on various learnings resulting from automation initiatives such as driverless vehicles. This will open up altogether new careers for IT professionals in law enforcement departments.

Right now every company is a software company. Ford sells computers-on-wheels. McKinsey hawks consulting-in-a-box. Fedex boasts a developer skunkworks. IT is employed in every industry today, from manufacturing to agriculture, from logistics to education.

The new breed of millennials joining the workforce will be focusing on analytics and research, data interpretation and insights, intelligent assembly lines, IoT platforms for process automation, CAE & CED , 3 D printing for building prototypes, streaming data collection and ingestion, cloud based machine learning algorithms.

6) Evolution of Non-IT Jobs Complemented by ML,Chatbots & AI

Product Designers, Data Scientists, Digital Marketers, Actuaries, offline auditors and validators, e-tailors, health care professionals, legal advisors. Almost every job in existence today will undergo the next level of evolution. Automation technologies like AI and ML will bring actionable facts or research applicable to a situation or an individual. This information will be leveraged for doing relevant cross-sale / up-sale of products or services. Customer support/ service jobs will be assisted by intelligent chatbots delivering a delightfully engaging and constructively interactive end user experience. In many cases final decisions still have to be made by humans. But these will be very well informed decisions.

These are the non-IT jobs that involve creative skills, emotional and social intelligence, smart networking that will get complemented by automation. Recruitment and selection process too will change with automated parsing, evaluation and shortlisting of resumes based on priorities.

7) How an Individual Should Evolve

There are ample resources available on the internet, free and paid, in the form of video, text, guides which can be leveraged by an individual to re-skill or upskill. No doubts are left about learning opportunities open to anyone when you see a thirteen year old delivering a key note address on IBM Watson and actively contributing to AI & ML. Professionals need to start being more curious about the purpose of what they are doing and how it is impacting the businesses they are working for.Anything that is happening anywhere in the world that is likely to bring about the next level of disruption in their jobs. Use this awarenesss to chart the course of skill acquisition well before the disruption happens. It is not very different when we do financial investment to expect certain amount of returns in future. Investment in oneself has to be a continuous process. People have to work towards building creative skills and being versatile. Those having T-shaped skills would find it easier to learn neighbouring skills. Above all thinking creatively about your future is important. Try making the change instead of coping with it. Finally one has to overcome inertia which is the biggest problem. Unlearning old habits is difficult. Old habits die hard!

8) Jobs or Roles that will Prosper

Roles having more cognitively demanding tasks will be valued more than ever. Here’s an example to put it in perspective. Traditional tellers have almost been replaced by ATMs. The individuals who used to be Tellers are now more focused on customer relationship selling products which are relevant to retail/corporate banking customers. This needs more time to be spent to understand the intricacies of businesses that are served by the banks. The new role is to provide designer solutions to cater to the business needs.

This does increase customer satisfaction and business for the banks. This ultimately results in the increase in number of branches and subsequently in the number of jobs.

This logic of automation creating new jobs applies for many roles which are under threat due to automation (Construction, Healthcare, Services etc..). Improvement in tools and automation, increases the effectiveness of emotional and social skills, creativity and judgments. Consider the example of the automated voice response (AVR) system used for handling lost baggage complaints by airlines. If used without any human intevention; it can put anyone off- it simply lacks the empathy an aggrieved passenger needs.

Having said this, let’s not ignore the fast pace evolution going on in AI and ML and cognitive capabilities been built around them making them more and more self conscious. Synerzip was involved in building an iPad app in which an animated nurse could actually recognize facial expressions. We had to use hundreds of thousands of images of happy, sad, painful and smiling faces to train it. It just goes to show how difficult it is to build an emotionally and socially intelligent bot.

9) Management Help for Employees to Evolve with Automation

If automation is a challenge for employees then it is definitely a challenge for businesses who are employing them. Management has to invest on fostering the culture of multi skill acquisition by their employees. Businesses should proactively introduce the disruptive technologies into work environment and provide training to their employees. Recently Webonise got into the business of implementing Robotic Process Automation(RPA) and IoT solutions. This put demands on the employees to acquire new skills required to develop, implement and maintain these solutions. Thus creating learning opportunities for the employees of Webonise. It was uncertain times for everyone involved. In these times of uncertainty management should develop an atmosphere of fearlessness- do gap analysis and find training needs. Once the training needs are ascertained; they should conduct classroom training followed by hands on reskilling programs. Its an opportunity for the company to develop a learning culture and to motivate every employee to perpetually keep learning and upgrading their skills.

As a part of strategic planning management should tell every employee what his/her next job will be like. Spare no one. There is no one whose job is future-proof. Fearlessness is a pre-requisite for creating a learning atmosphere. Management should measure performance and allow employees to do course correction by providing learning goals and performance objectives. Management should avoid replacing current employees by hiring laterally. Dynamic re-teaming and refitment of jobs would allow cross pollination of ideas and better learning across teams. Finally any good management should set the right employee expectations by making every one aware of the risk of obsolescence and impress the need to continually assess the way automation is impacting our work.

August 10, 2017

Blockchain and Smart Contracts Will Change the Game From Big Upfront to Lean Experiments.

We often make big upfront commitments , decisions, investments and assume the risk of not achieving the desired outcome.

Screen Shot 2017-08-10 at 4.18.31 AM

We hire a software developer or a team of developers at the beginning of the project. Often we are required to define the technology stack and expected level of complexity up front. We wouldn’t be able to hire the team without doing so. Our choice of technology may turn out to be a bad one. Or we might end up committing to a higher budget by hiring a team with higher than the needed level of seniority. But we can’t go back on our commitment and switch to a better team of developers with appropriate level of seniority and expertise in a new technology.

Welcome to the world of smart contracts and blockchains. In not too distant a future it might be possible to enter into a short 2 week contract with the team. (Please read my earlier post) Each developer gets automatically paid in cryptocurrency as their code passes all the automated tests and is deployed in production. At the end of the sprint we have the freedom to suitably modify the team composition by entering into a new set of smart contracts. This gives us the freedom to experiment and evaluation a few options.

What changed is our ability to enter into smaller , more frequent transactions without the need of a central aggregator such as a Bank or the Corporate HR department. Aggregators can’t economically work with shorter contracts and smaller payments because there is a fixed overhead associated with each contract and payment.

The table below shows a few more examples where such a change can occur.

Big Upfront	Lean Experimental
Buying a car	Sharing a ride- No aggregator such as Uber or credit card needed.
Buying Servers	Virtual servers on the cloud can be rented on pay as you use basis- going forward there will be no need to use credit cards. Cloud provider will automatically receive their payment in cryptocurrency.
Buying heavy machinery	Renting made possible by more details about usage, wear and tear and damage if any will get communicated by data streamed by IOT sensors
Launching a company to enter a new business with all the associated overheads	Assembling a team loosely bound by smart contracts. Blockchain ensures compliance without the protection of corporate law
Group health insurance- healthy members pay more and those susceptible to health issues pay less. Claiming insurance is fraught with legalese embedded in fine print.	Wearable devices communicate the current health status frequently enough that makes it possible for a healthcare provider to directly enter into a smart contract with the individual .

This new way of working has become even more possible because of recent advances in IOT which will enable digitalization of lot of information about the physical world. Broader acceptance of “shared economy” is pushing individuals to “rent & try” instead of “buy & regret”. All this works very well for those who think and work in the lean and agile way. It gives them the freedom to conduct experiments to validate their hypotheses before making a large commitment. It lowers the barrier to entry into businesses which were a territory exclusively reserved for the big corporations.

July 13, 2014

Catch-22 of Hiring

There is an inherent conflict at the point of basic information acquisition in the process of hiring. The question is how much information should the candidate be required to fill up while uploading his resume? Too much information increases his/her work. On the other hand if minimal information is acquired ; hiring managers are left with a whole lot of resumes and very little information. Its frustrating for hiring managers to read a number of unsuitable resumes before getting one that is suitable.

Catch-22 Situation in Hiring

To elaborate this situation let me take the example of my company. We were getting hardly any interesting resumes from our website. We decided to do away with the lengthy process and made it very simple. Now we have an apply button in front of each opening on the careers page; all that a candidate needs to do is to upload his latest resume. But simplifying this process resulted in a whole lot of resumes being uploaded. Now our HR executives are spending significant amount of their time managing resumes. Our hiring funnel in the chart below shows more than 99% of resumes being filtered out to make less than 1% offers.

Talent Acquisition Funnel

Should we switch back to our old “elaborate” process? Will the “elaborate” process and form filling ensure that hiring managers get what they want? The reality as we learnt from our experience is quite the opposite. Really interesting candidates don’t bother to go through the ordeal of “registering” and uploading their resumes . And those who do are not really interesting and those who appear to be interesting are just that. They “hype up” their resumes to make themselves appear interesting.

I had this problem on my mind when I attended a day long event focused on applications of machine learning

Inspiration

Of all the talks I was most inspired by this talk by Nilesh Phadke of BMC Software. He demonstrated applications developed for the IT support – far away from the world of hiring. However I felt that the problem I had on my mind could be solved by applying the machine learning approach.

Information Extraction for Filling up Forms

To automate any workflow ; one needs to enter long forms about entities – be it a support ticket or new candidate for a company. Long forms demotivate users and introduce an element of delay. There also is a tendency to skip non-mandatory fields even though the information is available.

Nilesh demonstrated “Formless Incident Creation” where the user was allowed to type a complaint in one Text Field. As he filled in details, based on the words that he was typing the fuzzy matching algorithm in real time matched the correct entities to those words. Not only did it complete the form needed but also searched and found similar past incidents from a myriad of templates of typical incidents.

Machine Learning Approach to Catch-22 of Hiring

I was immediately reminded of the catch-22 situation in hiring. Most of the information is already present in the resume that is being submitted. Can we not use information extraction algorithm to automate this tedious task? Can we also have different templates for Developers, QA Engineers, IT Support Engineers and Project Managers? Can we use the information extracted to synthesize micro-resumes or short summaries of less than 500 characters to help hiring managers quickly read resumes on their mobile devices?

This gave a new direction of thinking to resolve the catch-22 situation. Here are the set of tools that we plan to use.

Fuzzy Matching – Solr

Search- Lucene

Natural Language Processing – OpenNLP

Analytics for unstructured text-UIMA

Please stay tuned for updates on how we use machine learning for formless resume acquisition and more efficient search by hiring managers.

September 25, 2013

SMAC Social-Mobile-Analytics-Cloud is more powerful than sum of its parts

Social Media , Mobile Apps, Big Data Analytics and Cloud computing are four distinct technology areas that are seen rapidly evolving. I had an earlier posting about this Four Way Confluence. There are many products that leverage one of these- but the real power is derived by products that use two or more of these in combination. These areas are very interdependent and in fact consumer grade enterprise applications and apps can only be delivered by using all four in tandem as shown in the video. Industry leaders have coined a short single acronym SMAC to indicate the synergy and cohesiveness of these four areas. Let us examine how leveraging one technology area without the other would be limiting.

Synergy between Big Data back end And Social Media interface

Facebook users population exceeds 1.1 billion. Other social media are behind but all of them are generating huge data in text, photo and video form which can be mined by marketers, researchers , recruiters and providers of numerous services – only if we have tools to handle such unprecedented volumes and variety of data that is coming in faster than we can process them. Big Data or NoSQL databases like Hadoop, MondoDB and Cassandra use massively parallel processing to deal with Big Data. The explosion of data can be harnessed for proper use by these tools.

Also social media like Facebook, Twitter and LinkedIn use big data analytics when they promote the usage by suggesting friends, trending topics and building social graphs. When I follow someone on twitter ; I instantly get more suggestions which are a result of quickly mining all the tweets for lexical similarities – which is done at lightning speeds by massively parallel processing afforded by big data tools.

Synergy Between Cloud Storage And Big Data back end

Its unimaginable to have large volumes of data without proper management of the same. The need to have backup and disaster recovery process can’t be overemphasized. Cloud offers virtual storage as in the case of Amazon S3, which can grow on demand. No need to worry about installing large hardware and configuring the same as the existing infrastructure deals with data overflow. Cloud computing paradigm allows you to expand or contract storage by merely running a few commands to manage clusters of servers that process big data in massively parallel processing. Cloud based services are now evolving further to offer not only storage but ingestion and processing of big data . Big Data can’t be moved due to its volumes and it needs to be consumed at its source. Ubiquitous nature of cloud storage allows one to capture data near the source no matter where on earth it is.

Synergy between Mobile Apps And Social Media interface

More than 40% of social media updates are originating on mobile devices today. In fact there are many people who experienced internet for the first time on their smartphones. Be it pictures or videos or location updates; mobile devices will soon be used as the default way to interact with social media. One can see the reason why all major social media players like Facebook, Google+, Twiter and Linkedin have developed native apps for android and iOS.

Mobile Apps are even more dependent on social media interfaces because users have started moving from email , text messaging and voice calls to Facebook, Twitter, WhatsApp and Hangouts. Telecom providers are now feeling the heat as they see their traditional business under threat and they are monetizing Apps by selling valuable marketing insights by selling data captured by free apps enriched by demographic and geo-location context. Marketers can now sharply focus on their target customers using this information.

Synergy between anytime anywhere Cloud Service And Mobile Apps

Besides sharing photos and videos on social media ; mobile users also want to store them for sharing them across multiple devices. Storing them on the cloud provides a mechanism to have the private and shared photos, videos and documents accessible from anywhere at anytime using any device- be it smartphone, laptop or tablet. Cloud storage also serves as a backup in case someone loses his smartphone. Cloud application providers are also providing ways of sharing content selectively on social media. Solutions like GoogleDrive, iCloud, DropBox and SugarSync are already allowing people to upload files from their desktops to view them on their mobile phones and tablets.