Data Scientists -Thought Patterns

This article is based on the concepts explored by Daniel Kahneman, a psychologist and a Nobel Prize winner in economics. I took them and applied to data science.

Before we start understanding data, lets try to see how we interpret things in real life. Evaluate the executive in this video and write down what you think:  http://hbr.org/video/2363621363001/use-values-to-make-work-life-decisions. Then evaluate your thought patterns after your read this article. If your decision changes, that’s good; if it does not change, that’s good too. That is how we make decisions!

 

Hal Varian, Google’s Chief Economist aptly describes the role of data scientist: “I keep saying the sexy job in the next ten years will be statisticians. People think I’m joking, but who would’ve guessed that computer engineers would’ve been the sexy job of the 1990s? The ability to take data—to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it—that’s going to be a hugely important skill in the next decades… Because now we really do have essentially free and ubiquitous data. So the complimentary scarce factor is the ability to understand that data and extract value from it… I do think those skills—of being able to access, understand, and communicate the insights you get from data analysis—are going to be extremely important. Managers need to be able to access and understand the data themselves.”

 

 

Some think it’s just a new nomenclature and others think the required skill set is different.  I rather think Psychology plays an important role in data scientist’s day to day activities. Let me explain why……

 

A data scientist is both an engineer and innovator:

They are able to look for problems with Engineers hat, and convert them into advantages with innovators hat. Even by its very nature, if a company has a problem then its competition will have a similar problem. If a data scientist is able to overcome the problem, then it becomes an advantage for the company by its absence.

 

Defining slow thought:

Slow thought refers to conscious thought, but is often referred to as “Non-singular” thought. If you make a snap judgment such as being instantly afraid of a spider when you see one, then that is a singular thought. It is not a slow thought because if it were, you would reason that you could outrun the spider, which it is smaller than you are and therefore you would win if you were locked in mortal combat.

 

Slow thought takes more time because the conscious mind needs a little longer to process information because it also involves things such as deeper reasoning and causality functions.

If you were in an office with one door and no windows and had two keys, but were only allowed to try one in the door, what are the chances that you will escape? 1:2, 2:3, or 1:1. There is no obvious solution to this puzzle, so you would need to use slow thought to reason your way out of it and come up with ideas. Many believe the answer is 2 out of 3, as you can try the door first to see if it is unlocked and then try one of the two keys. Again here you will arrive at 2:3 if you apply slow thought.

 

Defining fast thought:

Fast thought is unconscious thought, but this is not restricted to instinct, such as pulling a hand away from an open flame. It refers to any thought where your subconscious has the first say. Fast thought occurs quickly and with no voluntary control. It has been linked to a survival instinct that monitors our environment for patterns.

For example, if you had a bad experience with a particularly aggressive client, then you may develop an unconscious fear of anyone who bares that clients logo or someone who has good relationship with the client. This fear would be on a subconscious level, leaving you with a feeling of apprehension that you cannot explain. So-called “First impressions” are fast thought, such as being moved to look away from people with disfigurements. Your conscious mind knows that the disfigured person is just another person, but your first impression is one of disgust/repulsion.

If you were told that your Statistician/data analyst was called “Mr Chen” then what nationality would you think he was? Fast thought looks for patterns and links them to experience, and since a common stereotype is for Chinese people to be good at math, it takes little effort to link the name “Chen” and “Math” to come up with the idea that he is Chinese. If on the other hand your HR department had just hired a receptionist called “Mr Chen” or “Mr Daniel Chen”, may you not have guessed that he was from Korea, Israel, Taiwan as much as from China?

 A data scientist is supposed to live by slow thought:

Any variety of scientist should not jump to conclusions, and any bias or pre-conclusion should be weeded out using techniques such as double blind experiments. Slow thought is linked with looking for causes beyond the fast thoughts first connection. For example, fast thought would suggest that a swan is identified by the fact that it is white. This would be fine if the Australian black swan had not been discovered. Slow thought would have suggested that “most” swans are white because slow thought would have led them to realize that evidence of absence is not possible. In other words, you cannot prove something does not exist; only that it exists.

 A data scientist is supposed to find evidence to support a conclusion:

This is a function of slow thought, as fast thought would conclude upon evidence and then stop. Based on slow thought, scientists are supposed to find evidence, conclude, and then find evidence that they are wrong. If rigorous testing cannot prove their conclusion wrong then it is slowly accepted as a scientific fact.

 A good data scientist cannot live by fast thought:

It is unlikely that your data scientist is going to rely on first impressions. He or she will have instinctive first impressions, but will be mindful enough to exclude them. Having this intellectual function is very important. For example, both Atari and HP rejected Steve Job’s attempts to make them interested in Steve Wozniak’s personal computer. Both directors of Atari and HP were suffering from severe fast thought problems, where they allowed their first impressions to control their decisions. If the directors were wearing the hats of data scientists who used psychological thought patterns, they would have investigated and tested the idea before throwing it out.

 Data scientists must be aware of the halo effect:

Have you ever strongly disliked a person because they did something? Maybe they took your most profitable client. Did you notice how you disliked that person for more than just the fact that they took your Client?  How did you feel about their strategies, marketing products, clothing style,  brand name ? You may have even noticed yourself taking a natural dislike to their adverts in the same publications as yours. Generally, you dislike everything about that person, even if that person is remarkably similar to yourself. This is part of the halo effect and data scientists cannot be party to it.

Data scientists are going to take part in, and be aware of, very many failed projects. Each time a similar project comes up, the data scientist may be influenced by experience on a quantitative level, but should not be influenced on an emotional level. He or she should not be influenced by the halo effect, in that he or she should not be predisposed to rejecting an idea because it bares similarities to a past failed project.

In essence, it would seem like all the above is common sense – well, it is. However, data is data until you put a context to it. Humans put context around data and that’s where psychological aspects kick in.

Now