http://divcom.otago.ac.nz/COM/INFOSCI/SMRL/people/andrew/publications/faq/hybrid/hybrid.htm Hybrid Systems Frequently Asked Questions

Hybrid Systems FAQ

Frequently Asked Questions: Hybrid Systems

Version Dr= aft 1.00
Last Updated June 23, 1997
Copyright 1997 by Andrew = Gray and Richard Kilgour

[0] Information about the FAQ
[1] What is a hybrid system?
[1.1] Sequential Hybrid
[1.2] Auxilary Hybrid
[1.3] Embedded Hybrid
[2] Why use a hybrid system?
[2.1] When not to use a hybrid system
[3] <= A HREF=3D"#Types">What types of hybrid system are there?
[3.1]= Neural Network-Statistical Hybrids
[3.2] Neural Network-Fuzzy Logic Hybrids
[3.3] Neural Network-Genetic Algorithm Hybrids
[3.4] Fuzzy Logic-Genetic Algorithm Hybrids
[4] Recommended Literature (including on-line material)<= /A>
[4.1] General References
[4.= 2] Neural Network-Statistical Hybrids
[4.3] = Neural Network-Fuzzy Logic Hybrids
[4.4] Neural Network-Genetic Algorithm Hybrids
[4.5] = Fuzzy Logic-Genetic Algorithm Hybrids
[5]= Relevant Web Sites
[5.1] General
[5.2] Neural Network-Statistical H= ybrids
[5.3] Neural Network-Fuzzy Logic Hybrids=
[5.4] Neural Network-Genetic Algorithm Hybrids=
[5.5] Fuzzy Logic-Genetic Algorithm Hybrids
[6] Acknowledgments

<= A NAME=3D"Information">Information about the FAQ

My research h= as focused on building models using hybrid systems for the past few years= and I've yet to find an introductory reference that I could recommend wi= thout qualification. I like the idea of hybrid systems. After all it is = my Ph.D. topic and I'm not getting any extra funding for its trendiness! = But, and this is a big but, they have to be used for a reason. Not unde= r the assumption that they are automatically better, but because there ar= e genuine reasons for their use. In fact, there are many tasks that can = only be realistically accomplished using some form of hybrid.

Again= , in case this is starting to look like the Hybrid Systems Bashing FAQ, I= 'll state that I like hybrid systems but that they have to be deve= loped and used correctly. Exaggerated claims of technique's effectivenes= s will only backlash against the field later.

This FAQ is my attemp= t to put together a document that will provide someone just starting out = in AI, or more specifically starting out with some form of hybrid system,= with some introductory knowledge and references with as little bias a= s possible although some bias is unavoidable especially in areas wher= e my Ph.D has taken me. As an advance warning most of my work on hybrids= in the past have been neural network-fuzzy logic systems which I now fin= d questionable, and I'm now emphasising neural network-statistical hybrid= s. As well as helping answer some introductory questions on hybrid syste= ms I'm hoping that more established researchers will also find some value= in this FAQ, especially in terms of the software, references, and links.=

In some ways this document is also intended to help me with my wor= k so my goals are hardly entirely humanitarian. This document with be up= dated at least each month with new material added each time. I wo= uld appreciate any comments, suggested FAQs, answers to FAQs, etc. Pleas= e send me suggestions, requests, links, etc using either email or this = form.

What is a hybrid system?

There are many different definitions for a hybrid system. My de= finition is one that uses more than one problem-solving technique in orde= r to solve a problem. This immediately leads to defining a technique. I= t's hard to precisely formulate what is meant by a technique, but as far = as I am concerned each of the following are individual techniques:

Supervised neural networks

Unsupervised neural networks

Regression= techniques

Data reduction techniques

Fuzzy logic

Genetic algori= thms

Case-based reasoning

This list is certainly not intended to b= e complete. It is just a list of the techniques that I am interested in.= Other techniques that can be used in hybrid systems include:

Exp= ert systems
Regression and decision trees
Clustering techniques
= Artificial life
Simulation techniques

If anyone would like = to contribute more information about these techniques and how they can be= used as part of a hybrid system then I would really appreciate hearing f= rom you.

There are, in general, three ways for two paradigms to be = used:

Sequential Hybrid - The fi= rst paradigm passes its output to the second
```
 INPUT --> PARADIGM =
1 --> PARADIGM 2 --> OUTPUT
```
An example of this paradigm combination= would be a statistical pre-processor that passes its output based on fac= tor analysis to a neural network.
This is the weakest form of hybridization, and some would argue that it i= s not, in fact, a hybrid system at all.
Auxillary Hybrid - The second= paradigm is called by the first paradigm and returns some information
```
INPUT --> PARADIGM 1 --> OUTPUT
             / \
              |
          PARADIGM 2
```
An example of this paradigm combination would b= e a neural network that calls a genetic algorithm module to optimise its = structure.
As with the Sequentail hybrid, two distinct systems can be identified. Th= e level of hybridization is higher, as the second (auxillary) paradigm is= intimately involved with the first. Paradigm 1 can exist without the aux= illary system.
Embedded Hybrid - The first p= aradigm contains the second paradigm
```
INPUT --> PARADIGM 1 + PARAD=
IGM 2 --> OUTPUT
```
An example of this paradigm combination would be a neural network-fuzzy l= ogic hybrid where the neural network simulates some characteristics of th= e fuzzy system.
Here the hybridization is absolute. In the extreme case, neither paradigm= can be defined without the other.

For more than two parad= igms these can be considered building blocks out of which larger system c= an be built.

Why use a hybrid system?

There has been enormous interest in hybrid systems (especially = neural-fuzzy, neural-genetic, and fuzzy-genetic) in the past ten years. = Almost every conceivable problem has been approached using some form of h= ybrid system. Why? Is this because hybrid systems are universally bet= ter than conventional approaches?

One claim is that hybrid systems are intrinsically better. They allow for= the synergistic combination of two techniques with more strengths and= less weaknesses than either technique alone.

WHen not to use a hybrid system

Although useful for many types of problem, hybrid systems provide even= more opportunity for misuse than single techniques. Although motivated b= y combining the strengths of the system, the hybrid will, in the worst ca= se, contain none of the strengths and all of the weaknesses of the compon= ent systems. While hybrid systems have great potential for solving some v= ery difficult problems, they can also be used inappropriately. As a tech= nique becomes more complex, the opportunities for misuse become greater, = and hybrid systems are intrinsically more complex than single techniques.= Many researchers are still making gross misuse of neural networks and f= uzzy logic as single techniques, and you can expect that this will carry = over into hybrid systems as they become more and more accessible.
The main focus of hybrid systems in the past has been on combining data-d= riven learning techniques (such as neural networks) with knowledge-driven= techniques (such as fuzzy rules). As an example, while the idea of a ne= ural network-fuzzy rule hybrid is very appealing since it would enable th= e use of existing knowledge, fine-tuning with data, and then extraction a= nd validation, it is often forgotten that many criticisms can be levelled= at fuzzy logic in terms of the ability of experts to specify useful rule= s (especially given different t-norms and t-conorms). For this reason, t= he validation on extracted rules can also be challenged. The middle stag= e of data driven refinement of the initial rules can be attacked by argui= ng that if a suitable learning algorithm is used then the benefits of fas= ter training and avoiding local minima (for the gradient descent cases) m= ay not be worth the extra cost of the hybrid approach, if these benefits = even exist! Most of the training time for a neural network is spent on t= he fine-tuning in any case. So even if correct rules are specified such = that the network is closer to the minima in its weight space at the start= of training the benefits may be minimal. In fact if poor rules are spec= ified then the network may learn at an even slower rate.
The mindset that exists seems to be one of if one technique is good th= en a hybrid must add something more to that and make it even better. = The concept of techniques having niches, in other words being best suite= d for certain classes of problems, has been used to justify using hybrids= since surely they will have larger niches and perform better. This line= of reasoning ignores the fact that hybrids can also fall into the jac= k of all trades and master of none trap. By this I mean that the hyb= rid technique may be good for a wide range of problems, but one or more o= f the component techniques may have been even better for certain types of= problems. Since for many applications, especially medical and financial= , the performance of the model is the single most important criterion for= success a specialised technique may perform better.
Because they are not always the best technique for a given problem, somet= imes the motivation appears to be to develop hybrids for hybrids sake.Thi= s seems to be especially evident in neural network-fuzzy logic hybrids. = The research seems driven by achieving some hybrid technique with little = consideration given to why it is needed.
Research is sometimes more co= ncerned with funding and publications than with the quality of the resear= ch itself. The literature abounds with new attempts to create some previo= usly unheard of hybrid, even to the extent of combining probability, poss= ibility and fuzzy logic into a single technique (I recently heard this su= ggested in all seriousness). Funding and publications are easier to get w= ith new research ideas, and this may be a motivating factor in this area.=

What types of hybrid system are there?

Theoretically any two or more techniques can be combined to form a hyb= rid, but not all are equally useful. The table below shows subjective ra= tings for the potential of various hybrid combinations. The ratings are = on the scale of 1 being useless and 10 being ideal. I'll stress again= that these are subjective ratings. Any arguments in favour of diffe= rent ratings are most welcome.

<= td>59<= td>Fuzzy Logic

	Unsupervised Neural Network	Regression	Data Reduction	Fuzzy = Logic	Genetic Algorithms	Case-Bas= ed Reasoning
Supervised Neural Network	6	8	4	4	3
Unsuperv= ised Neural Network	-	4	8	4	4=	3
Regression	-	-	7	6	6
Data Reduction	-	-	-	7	7	7
-	-	-	-	6	6
Genetic Algorithms	-	-	-	-	-	6

Below some of = the more common, and those that are not so common but I like, combination= s are discussed. Again, if you would like to contribute anything on area= s that I'm skipping over, please let me know.

= Neural Network-Statistical Hybrids

There has been far less wor= k on this area than the others discussed below. This seems indicative of= the gap between statisticians and soft computing researchers. In my opi= nion however, this is the more fruitful area for research. I still look = at neural networks as a statistical technique, with all of the usual requ= irements, including the ability of the user to correctly use the techniqu= e.

Some of the interesting applications of neural-statistical hybri= ds that I've seem involve using statistical techniques as pre-processors = for the data. Often this isn't described as a hybrid system, but based o= n my definition it qualifies.

Related to this is the opportunity f= or using statistical techniques such as principal components to initialis= e a network's weights.

Neural Network-Fuzzy L= ogic Hybrids

This has become the most researched type of hybri= d systems with exponentially increasing numbers of publications appearing= =2E Most of these are based on using a neural network architecture to si= mulate a fuzzy system. This may allow for fuzzy rules to be inserted, an= d later extracted.

The most common alternative involves using a neu= ral network to adapt some parameters of the fuzzy system.

Neural Network-Genetic Algorithm Hybrids

Neural netw= orks lend themselves to be genetically optimised since it is fairly easy = to combine parts of networks together. Personally I don't think much of = this approach.

The other alternative that I've seen for combining n= eural networks and genetic algorithms is where the GA is used to optimise= some parameters for the neural network, such as training period, learnin= g rate (for supervised networks), etc. This has always seemed somewhat w= asteful in terms of computation to me.

Fuzzy = Logic-Genetic Algorithm Hybrids

As with neural networks above,= the use of genetic algorithms for fuzzy logic rulebases is fairly easy.<= /P>

Recommended Literature

General References

Neural Network-Statistical Hybrids

Neural Network-Fuzzy Logic Hybrids

<= br>

Neural Network-Genetic Algorithm Hybrids<= /h4>

Fuzzy Logic-Genetic Algorithm Hybrids

Relevant Web Sites

= General

Neural N= etwork-Statistical Hybrids

Neural Netwo= rk-Fuzzy Logic Hybrids

http://decsai.ugr.es/~herrera/fl-ga.html.
This pa= ge has links to a bibliography on neural network-fuzzy logic systems with= some genetic algorithm component. None of the papers are downloadable.<= /P>

http://divcom.otago.ac.nz:800/COM/INFOSCI/KEL/fuzzycop.htm.
= FuNN is a fuzzy-neural network simulator available separately and as part= of a system called FuzzyCOPE/2. This was developed by Nikola Kasabov at th= e University of Otago. FuNN is ba= sed on implementing a fuzzy-logic system within a four-layer MLP architec= ture. Modules are available for rule insertion and extraction. The soft= ware is free and can be downloaded from here. The software is described = as "under development" so a new version may be forthcoming.

http://www.cs.tut.fi/~tpo/grou= p.html.
Neuro-Fuzzy Systems Research Group at the Tampere Univers= ity of Technology. This includes Tommi Ojala's MSc Thesis (Neuro-fuzzy s= ystems in Control, 1995) which can be downloaded and a list of other rele= vant papers.

ttp://= sol.ibr.cs.tu-bs.de/~nauck/
Detlef Nauck. This page includes lin= ks to Detlef's papers and neuro-fuzzy bibliography (downloadable) as well= as software.

Neural Network-Genetic Algorithm H= ybrids

h= ttp://pages.prodigy.com/upso/biblio.htm .
This page provides a= list of neural network construction algorithms including many using gene= tic algorithms.

http://decsai.ugr.es/~herrera/fl-ga.html.
This page has links= to a bibliography on neural network-fuzzy logic systems with some geneti= c algorithm component. None of the papers are downloadable.

Fuzzy Logic-Genetic Algorithm Hybrids

http://decsai.ugr.es/~herrera/= fl-ga.html.
This page has links to several bibliographies on evo= lving fuzzy logic systems. None of the papers are downloadable

Acknowledgments
I would like to ack= nowledge the following people for their help in creating this FAQ (in alp= habetical order)

Jennifer Prattley, Department of Finance, Univers= ity of Otago
Associate Professor Nikola Kasabov, Departmant of Information Science= , University of Otago

This page is maintain= ed by Andrew Gray. Th= is is a University of Otago, Division of Commerce suppl= ied service.