फीचर वैक्टर से परे सीखने के प्रतिनिधित्व के मूल सिद्धांतों में गोता लगाएँ

एआई से एमएल

All intelligent life forms instinctively model their surrounding environment in order to actively navigate through it with their actions. In Artificial Intelligence (AI) research, we then try to understand and automate this interesting ability of living systems with machine learning (ML) at the core.

• Generally speaking, deriving mathematical models of complex systems is at the core of any scientific discipline. Researchers have always tried to come up with equations governing the behavior of their systems of interest, ranging from physics and biology to economics.

• निष्पक्ष होने के लिए, "मॉडल पहचान" नामक एक समान दृष्टिकोण भी नियंत्रण सिद्धांत का एक पारंपरिक हिस्सा रहा है , जहां अंतर समीकरणों के पैरामीटर, अंतर्निहित सिस्टम की गतिशीलता का वर्णन करते हुए, इनपुट-आउटपुट डेटा से मापा जाता है, अक्सर भी जटिल, दायरे में सिस्टम। एक सटीक मॉडल के साथ, सिस्टम के विकास को वांछनीय लक्ष्य उपायों की ओर ले जाने के लिए इष्टतम नियंत्रण क्रियाएं प्राप्त की जा सकती हैं, जो कि अब एआई (कम गणित और अधिक प्रचार के साथ) की भावना में है।

While the approaches to the problem of mathematical modeling of complex systems evolved in various, largely independent, ways, one aspect remained almost universal — the data representation. Indeed, while the mathematical forms of the hypotheses and models have traditionally varied wildly, from analytical expressions and differential equations used in the control theory, all the way to decision trees and neural networks used now in ML, the input-output observations have traditionally been limited to the form of numeric vectors.

This only seems natural. Since the advent of computers, we have become very accustomed to turning any property of interest into a number, ranging from physical measurements, such as force or voltage, all the way to color, mood, or preference of ketchup over mustard.

ऐसे कई इनपुट ( एक्स ) और आउटपुट ( वाई ) वेरिएबल्स को देखते हुए जिन्हें ब्याज की प्रणाली पर मापा जा सकता है, बिजली संयंत्र से मानव तक कुछ भी होने के कारण, ऐसा प्रत्येक माप तब संख्याओं के एक विशेष वेक्टर को कम कर देता है, जिसे आमतौर पर संदर्भित किया जाता है एक फीचर वेक्टर के रूप में ।

लेकिन एक और अच्छा कारण है कि फीचर वैक्टर अत्यधिक आकर्षक क्यों हैं। प्रत्येक डेटा नमूना (माप) को एक एन-आयामी अंतरिक्ष में एक स्वतंत्र बिंदु के रूप में मानते हुए, रैखिक बीजगणित की मानक मशीनरी को सीधे अपनाने की अनुमति मिलती है, जो अन्य डोमेन से पूर्ववर्ती इंजीनियरिंग के सैकड़ों वर्षों से सिद्ध होती है।

कुछ संयुक्त Pxy वितरण से स्वतंत्र रूप से समान रूप से खींचे गए (iid) माने गए नमूनों के इस प्रतिनिधित्व के लिए धन्यवाद , मशीन सीखने का अनुसंधान भी संभाव्य एकाग्रता सीमा (जैसे, Hoeffding) से ज्ञात सांख्यिकीय परिणामों पर सीधे मानक एमएल सिद्धांत के साथ आने के लिए निर्माण कर सकता है। "शायद लगभग सही" (पीएसी) सीखना। नतीजतन, अधिकांश क्लासिक मशीन लर्निंग, कम से कम जब ठीक से अध्ययन किया जाता है, तो बहुभिन्नरूपी आँकड़ों के अंतर्गत आता है ।

As a result, any classic ML method now expects input data in the form of a table, where each column corresponds to a feature X or target variable Y, and each row corresponds to a single example measurement. The most general task is then to estimate the joint probability distribution Pxy that generated the observed data or, more commonly in supervised ML, to estimate just the conditional Py|x. These again are tasks that have been commonly studied in statistics for a long time now.

The Need for Relational Representations

अब हम अपने डेटा को संख्याओं की ऐसी तालिका (या टेंसर) में प्री-प्रोसेस करने के लिए इतने अभ्यस्त हो गए हैं, जो लगभग किसी भी एमएल लाइब्रेरी में इनपुट प्रारूप के रूप में अपेक्षित है, कि यह कल्पना करना भी मुश्किल हो सकता है कि यह संपूर्ण डेटा नहीं है। प्रतिनिधित्व। हालांकि, बस यह देखने के लिए चारों ओर देखें कि वास्तविक वास्तविक दुनिया का डेटा कैसा दिखता है। यह संख्यात्मक वैक्टर/टेंसर में संग्रहीत नहीं है, लेकिन इंटरनेट पेजों, सामाजिक नेटवर्क, ज्ञान ग्राफ, जैविक, रासायनिक, और इंजीनियरिंग डेटाबेस इत्यादि के अंतःस्थापित संरचनाओं में संग्रहीत है। ये स्वाभाविक रूप से संबंधपरक डेटा हैं जो स्वाभाविक रूप से उनके संरचित रूप में संग्रहीत होते हैं ग्राफ, हाइपरग्राफ और रिलेशनल डेटाबेस।

लेकिन रुकिए, क्या हम इन संरचनाओं को फीचर वैक्टर में नहीं बदल सकते हैं, और सब कुछ सामान्य हो जाता है?

ठीक है, लोगों ने निश्चित रूप से उपरोक्त (सुविधा) कारणों के लिए किया था, और हाल ही में, इन डेटा संरचनाओं के साथ एमएल करने का यह प्राथमिक तरीका था, जिसे अक्सर प्रपोजलाइजेशन कहा जाता है । उदाहरण के लिए, कोई संरचनाओं पर विभिन्न आँकड़ों की गणना कर सकता है, जैसे कि एक ग्राफ से नोड्स, किनारों, या सबग्राफ की गणना करना (और इन पर संचालित विभिन्न कर्नेल विधियों का भी उपयोग करना)।

और एक व्यावहारिक दृष्टिकोण से, संबंधपरक संरचनाओं से सुविधाओं को तैयार करने में कुछ भी गलत नहीं है, लेकिन यह महसूस करना अच्छा है कि इस तरह के दृष्टिकोण में एक सामान्य संज्ञानात्मक पूर्वाग्रह है:

"यदि आपके पास केवल एक हथौड़ा है, तो सब कुछ एक कील जैसा दिखता है।"

तो, क्या हम रिलेशनल डेटा से फीचर वेक्टर निर्माण के इस चरण को छोड़ सकते हैं? यदि आप अब "बचाव के लिए गहरी शिक्षा" के बारे में सोच रहे हैं, तो यह महसूस करना महत्वपूर्ण है कि, हाल ही में, सभी क्लासिक गहन शिक्षण विधियों को भी निश्चित आकार के संख्यात्मक वैक्टर (या टेंसर) के रूप में प्रतिनिधित्व के लिए प्रतिबंधित किया गया था। . गहरी शिक्षा के पीछे का विचार "निम्न-स्तर" (इनपुट) प्रतिनिधित्व वैक्टर से "उच्च-स्तरीय" प्रतिनिधित्व वैक्टर के मैन्युअल निर्माण को छोड़ना "केवल" है, लेकिन आपको अभी भी बाद वाले को शुरू करने की आवश्यकता है!

और, एक सैद्धांतिक दृष्टिकोण से, संबंधपरक डेटा को वेक्टर प्रतिनिधित्व में बदलने के साथ एक गहरी समस्या है, क्योंकि इस प्रीप्रोसेसिंग (प्रस्तावीकरण) के दौरान जानकारी के बिना (अवांछित) नुकसान के बिना किसी भी निश्चित आकार के प्रतिनिधित्व में एक अनबाउंड रिलेशनल संरचना को मैप करने का कोई तरीका नहीं है। ) कदम।

इसके अलावा, भले ही हम खुद को निश्चित आकार की संरचनाओं तक सीमित रखते हैं, एक संख्यात्मक वेक्टर या टेंसर के रूप में एक उपयुक्त प्रतिनिधित्व को डिजाइन करना अभी भी बहुत ही समस्याग्रस्त है। उदाहरण के लिए, केवल ग्राफ़ डेटा लें, जो संबंधपरक डेटा का एक विशेष रूप है। यदि एक (निश्चित आकार) संख्यात्मक वेक्टर (या टेंसर) के मानक सीखने के रूप में ग्राफ़ को मैप करने का एक निश्चित तरीका था, तो यह ग्राफ आइसोमोर्फिज्म समस्या को मामूली रूप से हल करेगा ।

• चूँकि यह जाँचने के लिए कि दो रेखांकन समरूपी हैं या नहीं, उन्हें ऐसे सदिशों में बदलना और इसके बजाय समानता के लिए इनकी तुलना करना पर्याप्त होगा। बेशक, हम आगे मानते हैं कि ऐसे वैक्टर बनाना कुशल होगा (यानी एनपी-पूर्ण नहीं)।

जीएनएन पर इंटरमेज़ो। बेशक, अब तक आपने ग्राफ़ न्यूरल नेटवर्क्स (जीएनएन) के बारे में सुना होगा, हाल ही में ग्राफ़-संरचित डेटा से निपटने के लिए प्रस्तावित - और चिंता न करें, हम इन्हें एक अनुवर्ती लेख में प्राप्त करेंगे! अभी के लिए, बस ध्यान दें कि जीएनएन संबंधपरक प्रतिनिधित्व (एक ग्राफ) के एक रूप से निपटने का एक विशिष्ट तरीका है, जो ग्राफ आइसोमोर्फिज्म समस्या ( वीसफेलर-लेहमैन ) के लिए एक विशेष (बहुत अच्छा) अनुमानी में निहित है।
आइए अब व्यापक परिप्रेक्ष्य के साथ जारी रखें।

रिलेशनल मशीन लर्निंग

Much of the recent deep learning research was then about discovering models and learning representations capturing data in various forms of sets and graphs. However, it is only rarely acknowledged that these structured learning representations have for long been studied (as a special case) in Relational Machine Learning.

A relation, as you might recall, is a subset of a cartesian product defined over some sets of objects. Every set is thus simply a degenerated case of some (unary) relation. Every graph can then be seen as an instantiation of a binary relation over the same set of objects (nodes). Tabular data, with more than 2 columns (objects), then correspond to relation of a higher arity, also known as a hypergraph. Add multiple such relations (tables) over the objects, and you have a relational database.

Much of the real-world data is then stored in such relational databases, which you have certainly encountered before. Now imagine that your learning samples are not prepared nicely as rows in a single table, but spread across multiple interlinked tables of the database, where different samples consist of different types and numbers of objects, with each object being characterized by a different set of attributes. This situation is actually far from uncommon in practice, but how on earth are you going to fit something like that into your favorite SVM/xGBoost/NN model?

While these data representations inherently fall outside the standard vector (tensor) formalism, there is actually another representation formalism that covers all these formats very naturally. It is relational logic.

Indeed, relational logic¹ is the lingua franca of all structured (relational) representations. In practice, many of the standard formats (e.g. ERM & SQL) designed for the structured data, ranging from sets to databases², follow directly from relational logic (and relational algebra).

And while you are probably already familiar with the relational logic/algebra formalism from your CS 101, it is quite likely that you have never heard of it in the context of machine learning. However, apart from being a great data manipulation and representation formalism, relational logic can also be used to directly tackle complex relational machine learning scenarios, just like the one outlined above.

Learning with Logic

Much out of the lights of the machine learning mainstream, there has been a community of Inductive Logic Programming (ILP), concerned with learning interpretable models from data with complex relational structures.

As outlined, ILP exploits the expressiveness of the relational logic formalism to capture these data structures (including relational databases and more). However, interestingly, relational logic here is also used to represent the models themselves. In ILP, these take the form of logical theories⁴, i.e. sets of logical rules formed from the used logical relations.

Moreover, ILP introduced the fundamental concept of background knowledge which can be, thanks to the logic-based representation, elegantly incorporated as a relational inductive bias directly into the models.

For decades[3] this, rather unorthodox, relational ML approach was then the premier venue for learning with data samples that do not succumb themselves to the standard form of i.i.d. feature vectors. This allowed ILP to explore some very general learning problems of manipulating structured data representations, involving variously attributed objects participating in relationships, actions and events, beyond the scope of standard statistical ML.

Example. For illustration, let’s see how the recently explored graph-structured learning problems can be approached with the relational logic. To represent a graph, we simply define a binary ‘edge’ relation, with a set of instantiations edge(x,y) for all adjacent nodes x,y in the graph. Additionally, we may also use other (unary) relations to assign various attributes to the sets of nodes, such as ‘red(x)’, etc.

The models, i.e. the logical rules, then commonly express relational patterns to be searched within the data. This covers all sorts of things from finding characteristic substructures in molecules to paths in a network. Thanks to the high expressiveness, declarative nature, and inherent use of recursion in relational logic, the learned models are then often very compact and elegant. For instance, a (learned) model perfectly capturing paths in a graph, such as your subway connection from X-to-Y, will commonly look like

path(X,Y) <= edge(X,Y).
path(X,Y) <= edge(X,Z), path(Z,Y).

Interestingly, this is in direct contrast to, e.g., tackling the same problem with a Differentiable Neural Computer — one of the recent Deepmind’s highlights, which required a lot of examples and additional hacking (e.g., pruning out invalid path predictions) to tackle the task with an “inappropriate” tensor (propositional) representation (emulating a differentiable memory).

Statistical Relational Learning

While substantially more expressive in representation, learning with logic alone is not well suited for dealing with noise and uncertainty.⁵ To tackle the issue, many methods arose to merge the expressiveness of relational logic, adopted from ILP, and probabilistic modeling, adopted from classic statistical learning, under the notion of Statistical Relational Learning (SRL)⁶ which covers learning of models from complex data that exhibit both uncertainty and a rich relational structure. Particularly, SRL has extended ILP by techniques inspired in the non-logical learning world, such as kernel-based methods and graphical models.

Generally, there have been two major streams of approaches in SRL — probabilistic logic programming and lifted modeling, which will serve as a foundation for our further exploration of the deep relational learning concept.

Lifted models

As opposed to standard (aka “ground”) machine learning models, lifted models do not specify a particular computational structure, but rather a template from which the standard models are being unfolded as part of the inference (evaluation) process, given the varying context of the relational input data (and, possibly, also the background knowledge).

For instance, the (arguably) most popular lifted model — a Markov Logic Network (MLN) [7] may be seen as such a template for the classic Markov networks. For prediction and learning, an MLN is combined with a particular set of relational facts, describing the input data (e.g., a database), and unfolds a classic Markov network. Let’s take a closer look at that.

Example. For example, such an MLN template may express a general prior that “friends of smokers tend to be smokers” and that “smoking may lead to cancer”. The learning data may then describe a social network of people with a subset of smokers labeled as such. The lifted modeling paradigm of MLNs then allows to induce the smoking probabilities of all the other people based on their social relationships, as if modeled by a regular Markov network, yet systematically generalizing over social networks of diverse structures and sizes!

Importantly, this also allows the lifted models to capture the inherent symmetries in learning problems⁸, such as the regularity of the friendship relation across all the different people in the network, by tying their parameters.

This parameter sharing can then significantly reduce the number of weights that have to be learned, and allow the lifted models to convey a highly compressed representation of the problem, since all the regular relational patterns (symmetries) are parameterized jointly by the single template. This in turn allows for better generalization.

• Additionally, exploiting the symmetries with this lifted modeling approach can also significantly speed up the evaluation itself, which is commonly known as “lifted inference” in SRL.

Consequently, their usage in real-life applications is far from where we see deep learning these days. The neural models, on the other hand, have still been vastly limited to the fixed-size tensor (propositional) representations which, as explained in this article, cannot correctly capture the unbound, dynamic and irregular nature of the structured learning representations, for which the relational logic formalism is the natural choice.

In the next article, we will then browse into the history of “Neural-Symbolic Integration” aimed at combining symbolic logic with neural networks. This will provide some further background for our path towards the desired unification of modern structured deep learning models, such as Graph Neural Networks, with relational logic into an expressive “Deep Relational Learning”.

1. Often also referred to as predicate or first-order logic (which additionally also introduces logical function symbols that will not be needed here).

2. Relational logic is not even limited to the relational databases, but further allows to cover all the rich knowledge bases and fancy deductive ontologies.

[3] Cropper, A., Dumančić, S., Evans, R. et al. Inductive logic programming at 30. Mach Learn (2021). https://doi.org/10.1007/s10994-021-06089-1

4. These form the basis of logic programming in languages like Datalog and Prolog, for which this ML approach is conceptually close to the field of program synthesis (due to the high expressiveness of the learned models).

5. The uncertainty in relational learning naturally arises from the data on many levels, from the classic uncertainty about the values of attributes of an object to uncertainty about its type(s), membership within a relationship, and the overall numbers of objects and relations in scope.

6. Some other terms for this research domain include multi-relational learning/data-mining or probabilistic logic learning. Structured prediction models can also be seen as an instance of SRL.

[7] Richardson, Matthew, and Pedro Domingos. “Markov logic networks.” Machine learning 62.1–2 (2006): 107–136.

8. Later also extrapolated (by Pedro Domingos, again) to neural networks in:
Gens, Robert, and Pedro Domingos. “Deep symmetry networks”. Advances in neural information processing systems 27 (2014): 2537–2545.