When the “data” and “experience” to fight, you should listen to who?

author introduction: Steven Hillion yes co-founder, responsible for leading advanced analysis platform for enterprise development. Before joining Alpine, he has been in Siebel, companies such as Greenplum management data scientist and engineer team.

now, the data is more diversified than before, and rapidly changing. To analyze data effectively, without advanced software and machines. But with the rise of big data analytics, our intuition and use? If the results of data analysis and the sixth sense of business manager, that she should how to choice?

I can say such a thing, and maybe some surprising, after all, I itself is a data and scientific researchers. But I firmly believe that only by making based on the analysis of data and business knowledge intuition guide data work, to achieve the real value.

some people think that, as long as enough of the mathematical analysis and machine performance is applied to the database, you can get the best model. However, smooth by crunching the data you want to obtain the answers, needed to promote the business progress so silly. Because, in data science, intuition and data analysis, complement each other, inspire each other.

first of all, the intuition to guide analysis. Analytical insights rarely appear out of thin air. They are numerical method is applied to test assumptions and ideas derived from intuition and observation results. Intuition can guide researchers to test these imaginary method. What data is relevant? Which variables and shift is reasonable? The relationship between cause and effect may be what? Which model is suitable?

at the same time, analysis inspired by intuition. Non supervised modeling technology can recognize the relation between data and model, and we are through the observation of surface or small data samples is often difficult to find these relationships and patterns. In a nutshell, is the exploration analysis can bring surface observation doesn’t give way to enlightenment, and perhaps even counterintuitive.

if you don’t let the data wise leader of the team and business team to guide the data analysis process, according to the work experience and professional knowledge of intuition to balance, problem is produced.

here just to name a few.

there was ever a consumer finance team want us to do a customer churn model, helping Banks predict which customers are most likely to unsubscribe. However, we didn’t get anything of value. Analysis of deposits, loans, and credit card data, and there is no obvious cause factors of customer sales. After creating a new account, consumer spending and pay customs basically no difference.

however, bankers did not give up, they are more carefully study the data, review the team to make the customer segmentation of data. One analyst with her instincts suddenly has a valuable new discovery. She realised that there is a specific customer group showed an unusually high value loans, long-term customer value and a few other special factors. They are likely to belong to the small business owner. To check the personal account, she guess indeed.

she also speculated that those accounts open the business owner may not be aware of it, more than a credit card or ordinary better financing loan account. So, the team will be towards the high value customers, provide them with more suitable products. Further, the team access to historical data on the user’s behavior, in order to recommend to other customers of the corresponding products, now began to provide customers tailor-made product strategy, so as to improve the customer lifetime value.

so, data have important insights that alone is not possible. The data analysis business insight is priceless.

intuition is of vital importance in data analysis, but the odd thing is that the business team is often left out in the process of data analysis. In fact, the business analyst should be invited to participate in the process from the start. I have changed the operation process, let the whole team involved in the initial model assessment, or even earlier raw data review process. The effect is very good.

in another case, one of our clients, big beer company, to predict its future sales in the Japanese market. We built a model to study the next year, sales in different markets and pricing will react under pressure. Our customers told us, they think its beer sales directly affected by the economy, if Japan’s slow economic recovery, people on the soft drink consumption will increase.

they required us to in the model with the nikkei index variable, as a kind of trend. The index in the initial does improve the accuracy of the model is so – or surface. But in the next year, some of the model’s predictions become very unusual. Because the Japanese economy began to rebound, but the nikkei index are beyond the scope of training data, the original model fitting “excessive”.

if more experienced some modelers, introduce the variables, they may not. Sometimes more on sixth sense, but in this case, the data science experts recommend caution point, and realize the modeling process limitations and pitfalls. We changed the model, to suppress the influence of stock market index. Later, on the guide new propaganda plan and forecast effect, we played a good role model.

data often arise contradiction between scientists and business people quarrel – especially when the data seems counterintuitive, the effect of the new development plan seems to be insignificant. So you can often see marketing researchers questioned “the data come from?” Data scientists are not afraid to fight back.

but I think that saving is a good thing. Math and science should be able to live the good question. Sometimes, the data can prove that intuition is wrong. But also sometimes, is the result of experienced intuition will find out the defects in the process of data analysis. Ideally, everyone can benefit.


You may also like...