the author weibo @ Internet super, WeChat SuperSofter
baidu is China’s most technology gene Internet companies, to a certain extent can say technology is the foundation. Chinese search, knowledge map, big data, natural language processing, intelligent advertising has become a segmentation technologies such as baidu unique advantages, the cornerstone of all these technologies? The answer may be NLP: natural language processing.
NLP is the technology of search engine
search engine is the most basic mode to automatically aggregate enough “content”, to parsing, processing, and organization and response of users’ search requests to find corresponding results back. Each link, the need to use natural language processing.
crawl the web to analyze HTML pages and web page content, dealing with the crawl to the need to parse, participles, transformation and so on, this is very dependent on natural language processing technology. Search engines want to understand the user’s “people”, the user’s search request is strange, keyword combination way less and less, direct use of natural language search has become the habit of many people, people are more inclined to search “guangzhou to Beijing flight information” rather than “guangzhou flight to Beijing”, so the search engine must understand user intent can help them to find accurate answer.
every link of natural language processing is very important. This is NLP technology is committed to solve the problem. NLP (Natural Language Processing: NLP) is a cross subject of computer science, artificial intelligence, and linguistics, aims to solve the problem of interaction between computer and human Language, including the analysis of Natural Language, understanding and generation, retrieval, transformation and translation, etc.
itself, not just search search engine derivatives metropolitan application to NLP technology. Such as baidu, Google to provide translation services is the most classical natural language applications, it can intelligently translation itself, mutual translation between support multiple languages. And baidu knows, it can intelligently identify similar problems to avoid repeating questions users. And Siri’s voice assistant wave finally by baidu, Google and other search instead players force, because the search engine more effective in NLP technology, speech aides and baidu Google Now in the field of Chinese and English speech performance is more prominent than Siri.
all in all, the search engine to connect people with information, it can help people to obtain information more efficiently will have to learn like a human to read the information, it must be able to understand the demand of information to the people you need to understand the “people”, this is doomed the NLP technology will be the search engine technology.
the ubiquity of NLP let baidu more intelligent
baidu since the first day of dealing with the user first search began to application of NLP technology. More than a decade, baidu users has exceeded 500 million, baidu company employees more than 30000 people, baidu’s market value is close to $100 billion. Along with the rapid development of baidu, baidu NLP team in synchronously, even faster development, shifting in NLP technology itself, but on the whole, it is to help baidu to become more intelligent.
in the spring of 2010, the international top natural language processing experts to join baidu, Dr Hai-feng wang to reframe baidu NLP work and planning, in addition to word segmentation, proper noun recognition and the query requirement analysis, query rewriting, such as traditional direction is enhanced, expand the machine translation, machine learning, semantic search, semantic understanding, intelligent interaction, depth q&a, understand new technology such as direction. NLP team has now had only 10 people developed into more than 200 people today, the size of the team, traders is Dr Hai-feng wang. Because theoretical attainments and achievements in baidu’s engineering practice, hai-feng wang to become the world’s most influential in natural language processing international academic organization ACL (Association for Computational Linguistics), the only Chinese President in the history of more than 50 years.
now NLP group technology is applied in every aspect of baidu.
search engine is the most place of NLP technology. Compared to when the user needs to learn how to use the search engine, today’s users can directly input any needs or questions, think even broadcast to ask baidu voice assistant directly. For example, the user can search “weather”, “calendar” and “tickets”, “exchange rate” fuzzy demand, baidu will present the results directly in the search results. Again such as user can search for “how much is hk $5000 yuan” baidu results can directly present a calculator. Users can also search played a return zhugege “Taiwan actor” “grandfather who is the son of Cecilia cheung” such a complicated problem, baidu can accurately answer.
based on the technology of knowledge map “close” baidu also use NLP technology everywhere. Knowledge map is to make the search engine results can think out more structured information technology related, baidu results page structured information is on the right side of the knowledge map the result set. Users to search, for example, the tiger sniff network search engine on the right side will show a similar other media such as “36 kr” science and technology. Users’ search ‘plateau’ and ‘serena Williams’ close right will recommend’ li ‘, but the former is recommended by the singer li na, the latter is recommended by tennis star li na, this is the entity disambiguation technology based on NLP.
depth q&a is opportunity NLP technology and q&a of deep learning technology products. Baidu’s intelligent q&a robot in jiangsu satellite TV “open sesame” program has made remarkable achievements, problem solving accuracy than human contestants, thanks to profound knowledge precipitation, and baidu baidu to user accurate understanding of the problem. But the technology is not only used in games, voice assistant in baidu, baidu knows, and baidu search products, such as the depth of the technology are widely used, it can accurately understand the user’s complex problems, and can be combined with context, combined with the scene to make a different understanding to the problem.
, for example, when we are in the baidu search “germinated peanut can eat?”, the answer is no longer on the Internet a variety of original page answer, but let a person be clear at a glance the summary of the type the answer in this paper. “Question and answer” in addition to this, the user can also talk to baidu, ask it a question of another, NLP eventually allows users to use natural language searching, and adopt the way of several rounds of interaction to gradually clarify and meet the demand, thus accomplishes the search depth decision task. Baidu NLP team research and development of “interactive search technology” can be several rounds of questions and answers, and “multimode” (multi – modal) interaction can let users by text, voice, picture, LBS input in the form of more humane, more simple and easy to use search, future expansion to through the video as input is likely to search.
technology can also be applied to many fields of vertical depth. Such as “bully” baidu test combined with education became high school students fill in the volunteer’s right-hand man, it graduates this year, a total of up to 5.5 million times, help them more efficient, simple and comprehensive understanding of the colleges and universities and professional. Combined with health care can be achieved for instance again sought treatment, we can search for “have a headache how to do” baidu will be given a structured list of suspected cases and even medical advice. On the “search” generic demand baidu is a good way to understand the fuzzy demand of users, in baidu search “home cooking”, search engines will return more than 30000 kinds of home cooking in accordance with the “cuisine”, “efficiency” and “taste” categorize, distinct, be clear at a glance.
both search engines and depth of knowledge map or question and answer, the key technical points is NLP. NLP ready will form the barriers, because although opponents can make similar search channel or product, but on the intelligence, accuracy and ease of use to catch up with the baidu is not easy. NLP technology, need data accumulation on the accumulation of algorithm, the team’s precipitation takes time, others are difficult to match.
NLP future on baidu will be increasingly important
NLP (natural language processing) technique for baidu will only become more important.
search engine has not only duty is to help users to find the answer, but to help users find prayer, between people and the physical world service, become the user’s intelligent assistant. Infinite close to the wisdom of human is the ultimate goal of human beings give machines. For baidu, one is to rise from the search engines to the service connection platform, the second is to develop the artificial intelligence, big data, deep learning, for the future of emerging technologies such as these goals are inseparable from “understanding user needs, intelligent to organize information and service”, which are very dependent on NLP technology.
NLP technology itself is the upgrade. From the name it is a language processing technology, to understand the text seems to be enough. Characterization, in fact, now it is going to understand is the language of human needs. In addition to text, voice, image is the carrier of human demand expression. At the same time, the user’s environment (LBS), the user’s personal characteristics (user), the user request the time will be assigned the same request of different purposes. Therefore, NLP technique in dealing with the user language itself, still need to combine the above factors to understand behind the customers’ needs. NLP is upgrading to become a new look “human understand technical information identification and demand”.
is baidu genes, technology and data accumulation of users, product and brand advantage is the moat of baidu. NLP is the cornerstone of baidu technology, it gives the “smart” genes, baidu technology in the intelligent search for baidu more and more useful for the Chinese search market laid a king; Intelligent in the future between the user and the service as baidu and tencent giants such as ali’s largest segment, NLP techniques will only become more important. So it is no exaggeration to say that NLP is baidu’s technology foundation.