OFAI

MetaL Publications Lists

88 references, last updated Tue Jan 14 21:47:25 2003

[Bensusan and Giraud-Carrier, 2000a]
H. Bensusan and C. Giraud-Carrier. Discovering task neighbourhoods through landmark learning performances. In D.A. Zighed, J. Komorowski, and J. Zytkow, editors, Proceedings of the Fourth European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD2000), pages 325-330. Springer, 2000.
Abstract: Meta-learning is concerned with the selection of a suitable learning tool for a given application. Landmarking is a novel approach to meta-leaning. It uses simple, quick and efficient learners to describe tasks and therefore to locate the problem in the space of expertise areas of the learners being considered. It relies on the performance of a set of selected learning algorithms to uncover the sort of learning tool that the task requires. The paper presents landmarking and reports its performance in experiments involving both artificial and real-world databases. The experiments are performed in a supervised learning scenario where a task is classified according to the most suitable learner from a pool. Meta-learning hypotheses are constructed from some tasks and tested on others. The experiments contrast the new technique with an information-theoretical approach to meta-learning. Results show that landmarking outperforms its competitor and satisfactory selects suitable learning tools in all cases examined.

[Bensusan and Giraud-Carrier, 2000b]
H. Bensusan and C. Giraud-Carrier. If you see la sagrada familia, you know where you are: Landmarking the learner space. Technical report, Department of Computer Science, University of Bristol, 2000.

[Bensusan and Giraud-Carrier, 2000c]
Hilan Bensusan and Christophe Giraud-Carrier. Casa Batló is in Passeig de Gràcia or landmarking the expertise space. In J. Keller and C. Giraud-Carrier, editors, Proceedings of the ECML-00 Workshop on Meta-Learning: Building Automatic Advice Strategies for Model Selection and Method Combination, pages 29-46, Barcelona, Spain, 2000.
Abstract: Task description is crucial not only to every meta-learning enterprise but also to related endeavours like transfer of learning. This paper evaluates the performance of a newly introduced method of task description, landmarking, in a supervised meta-learning scenario. The method relies on correlations between simple and more sophisticated learning algorithms to select the best learner for a task. The results compare favourably with an information-based method and suggest that landmarking holds promise.

[Bensusan and Giraud-Carrier, 2000d]
Hilan Bensusan and Christophe Giraud-Carrier. Harmonia loosely praestabilita: discovering adequate inductive strategies. In Proceedings of the 22nd Annual Meeting of the Cognitive Science Society, pages 609-614. Cognitive Science Society, August 2000.
Abstract: Landmarking is a novel approach to inductive model selection in Machine Learning. It uses simple, bare-bone inductive strategies to describe tasks and induce correlations between tasks and strategies. The paper presents the technique and reports experiments showing that landmarking performs well in a number of different scenarios. It also discusses the implications of landmarking to our understanding of inductive refinement.

[Bensusan and Kalousis, 2001]
H. Bensusan and A. Kalousis. Estimating the predictive accuracy of a classifier. In Proceedings of the 12th European Conference on Machine Learning (ECML-01). Springer Verlag, 2001.

[Bensusan et al., 2000a]
Hilan Bensusan, Christophe Giraud-Carrier, and Claire Kennedy. A higher-order approach to meta-learning. In Proceedings of the ECML'2000 workshop on Meta-Learning: Building Automatic Advice Strategies for Model Selection and Method Combination, pages 109-117. ECML'2000, June 2000.
Abstract: Meta-learning, as applied to model selection, consists of inducing mappings from tasks to learners. Traditionally, tasks are characterised by the values of pre-computed meta-attributes, such as statistical and information-theoretic measures, induced decision trees' characteristics and/or landmarkers' performances. In this position paper, we propose to (meta-)learn directly from induced decision trees, rather than rely on an ad hoc set of pre-computed characteristics. Such meta-learning is possible within the framework of the typed higher-order inductive learning framework we have developed.

[Bensusan et al., 2000b]
Hilan Bensusan, Christophe Giraud-Carrier, Bernhard Pfahringer, Carlos Soares, and Pavel Brazdil. What works well tells us what works better. In Proceedings of ICML'2000 workshop on What Works Well Where, pages 1-8. ICML'2000, June 2000.
Abstract: We have now a large number of learning algorithms available. What works well where? In order to find correlations between areas of expertise of learning algorithms and learning tasks, we can resort to meta-learning. Several meta-learning scenarios have been proposed. In one scenario, we are searching for the best learning algorithm for a problem. The decision can be made using different strategies. In any approach to meta-learning, it is crucial to choose relevant features to describe a task. Different strategies of task description have been proposed: some strategies based on statistical features of the dataset, some based on information-theoretic properties, others based on a learning algorithm's representation of the task. In this work we present a novel approach to task description, called landmarking.

[Bensusan, 1999]
Hilan Bensusan. Automatic bias learning: An inquiry into the inductive basis of induction. PhD thesis, School of Cognitive and Computing Sciences, University of Sussex, July 1999.
Abstract: This thesis combines an epistemological concern about induction with a computational exploration of inductive mechanisms. It aims to investigate how inductive performance could be improved by using induction to select appropriate generalisation procedures. The thesis revolves around a meta-learning system, called textsc The Entrencher, designed to investigate how inductive performances could be improved by using induction to select appropriate generalisation procedures. The performance of textsc The Entrencher is discussed against the background of epistemological issues concerning induction, such as the role of theoretical vocabularies and the value of simplicity. After an introduction about machine learning and epistemological concerns with induction, Part I looks at learning mechanisms. It reviews some concepts and issues in machine learning and presents textsc The Entrencher. The system is the first attempt to develop a learning system that induces over learning mechanisms through algorithmic representations of tasks. Part II deals with the need for theoretical terms in induction. Experiments where textsc The Entrencher selects between different strategies for representation change are reported. The system is compared to other methods and some conclusions are drawn concerning how best to use the system. Part III considers the connection between simplicity and inductive success. Arguments for Occam's razor are considered and experiments are reported where textsc The Entrencher is used to select when, how and how much a decision tree needs to be pruned. Part IV looks at some philosophical consequences of the picture of induction that emerges from the experiments with textsc The Entrencher and goes over the motivations for meta-learning. Based on the picture of induction that emerges in the thesis, a new position in the scientific realism debate, transcendental surrealism, is proposed and defended. The thesis closes with some considerations concerning induction, justification and epistemological naturalism.

[Berrer et al., 2000]
Helmut Berrer, Iain Paterson, and Joerg Keller. Evaluation of machine-learning algorithm ranking advisors. In Pavel Brazdil and Alipio Jorge, editors, Proceedings of the PKDD-00 Workshop on Data Mining, Decision Support,Meta-Learning and ILP: Forum for Practical Problem Presentation andProspective Solutions, Lyon, France, 2000.
Keywords: Meta-learning, Ranking, k-NN

[Blond, 2001]
P. Y. Blond. Model selection with inductive logic programming. Technical Report Unige-AI-01-02, University of Geneva, Computer Science Department, Geneva, Switzerland, 2001.

[Bohanec et al., 2001]
M. Bohanec, S. Moyle, D. Wettschereck, and P. Miksovsky. Software architecture for data pre-processing using data mining and decision support models. In Proceedings of the ECML/PKDD Workshop on Integrating Aspects of Data Mining, Decision Support and Meta-Learning (IDDM-2001), Freiburg, Germany, 2001.

[Brazdil and Soares, 1999a]
P. Brazdil and C. Soares. Exploiting past experience in ranking classifiers. In H. Bacelar-Nicolau, F. Costa Nicolau, and J. Janssen, editors, Applied Stochastic Models and Data Analysis, pages 299-304. Instituto Nacional de Estatística, 1999.
Comment: Presents three methods to aggregate performance information into a ranking of candidate algorithms; describes an evaluation methodology for ranking based on rank correlation; comparison of ranking methods using results of 6 algorithms on 16 datasets
Keywords: Ranking Methods, Ranking Evaluation, Rank Correlation, Meta-Learning

[Brazdil and Soares, 1999b]
P. Brazdil and C. Soares. Exploiting past experience in ranking classifiers: Comparison between different ranking methods. In C. Giraud-Carrier and B. Pfahringer, editors, Recent Advances in Meta-Learning and Future Work, pages 48-58. J. Stefan Institute, 1999.

[Brazdil and Soares, 2000a]
P. Brazdil and C. Soares. A comparison of ranking methods for classification algorithm selection. In R.L. de Mántaras and E. Plaza, editors, Machine Learning: Proceedings of the 11th European Conference on Machine Learning ECML2000, pages 63-74. Springer, 2000.
Comment: Presents three methods to aggregate performance information into a ranking of candidate algorithms; describes an evaluation methodology for ranking based on rank correlation; comparison of ranking methods using results of 6 algorithms on 16 datasets
Keywords: Ranking Methods, Ranking Evaluation, Rank Correlation, Meta-Learning

[Brazdil and Soares, 2000b]
P. Brazdil and C. Soares. Ranking classification algorithms based on relevant performance information. In J. Keller and C. Giraud-Carrier, editors, Meta-Learning: Building Automatic Advice Strategies for Model Selection and Method Combination, pages 61-71, 2000.
Comment: presents a IBL-based ranking method that uses general, statistical and information-theoretic measures to characterize datasets; presents a way of combining success rate and time in a performance measure; presents results based on meta-data for 6 algorithms on 16 datasets
Keywords: Meta-Learning, Ranking, Multicriterion Evaluation

[Brazdil and Soares, 2000c]
P. Brazdil and C. Soares. Zoomed ranking: Selection of classification algorithms based on relevant performance information. In D. Oblinger, I. Rish, J. Hellerstein, and R. Vilalta, editors, "What Works Well Where?" Workshop of ICML2000, 2000.
Comment: presents a IBL-based ranking method that uses general, statistical and information-theoretic measures to characterize datasets; presents a way of combining success rate and time in a performance measure
Keywords: Meta-Learning, Ranking, Multicriterion Evaluation

[Brazdil and Soares, 2001]
P. Brazdil and C. Soares. Reducing rankings of classifiers by eliminating redundant cases. In JOCLAD 2001: VII Jornadas de Classifica c c~ao e Análise de Dados, pages 76-79, 2001.
Comment: presents method to eliminate the algorithms in a ranking that are not expected to bring any improvement over the others; presents adequate evalution methodology capable of handling rankings of uneven length
Keywords: Ranking, Ranking Evaluation, Ranking Redundancy Elimination

[Brazdil et al., 2001a]
P. Brazdil, C. Soares, and R. Pereira. Reducing rankings of classifiers by eliminating redundant cases. In C. Giraud-Carrier, N. Lavrac, and S. Moyle, editors, Working Notes of the ECML/PKDD Worshop on Integrating Aspects of Data Mining, Decision Support and Meta-Learning, pages 37-44, 2001.
Comment: presents method to eliminate the algorithms in a ranking that are not expected to bring any improvement over the others; presents adequate evalution methodology capable of handling rankings of uneven length
Keywords: Ranking, Ranking Evaluation, Ranking Redundancy Elimination

[Brazdil et al., 2001b]
P. Brazdil, C. Soares, and R. Pereira. Reducing rankings of classifiers by eliminating redundant cases. In P. Brazdil and A. Jorge, editors, Proceedings of the 10th Portuguese Conference on Artificial Intelligence (EPIA 2001). Springer, 2001.
Comment: presents method to eliminate the algorithms in a ranking that are not expected to bring any improvement over the others; presents adequate evalution methodology capable of handling rankings of uneven length
Keywords: Ranking, Ranking Evaluation, Ranking Redundancy Elimination

[Brazdil et al., 2003]
P. Brazdil, C. Soares, and J. P. da Costa. Ranking learning algorithms: Using IBL and meta-learning on accuracy and time results. Machine Learning, 50(3):251-277, 2003.
Abstract: We present a meta-learning method to support selection of candidate learning algorithms. It uses a k-Nearest Neighbor algorithm to identify the datasets that are most similar to the one at hand. The distance between datasets is assessed using a relatively small set of data characteristics, which was selected to represent properties that affect algorithm performance. The performance of the candidate algorithms on those datasets is used to generate a recommendation to the user in the form of a ranking. The performance is assessed using a multicriteria evaluation measure that takes not only accuracy, but also time into account. As it is not common in Machine Learning to work with rankings, we had to identify and adapt existing statistical techniques to devise an appropriate evaluation methodology. Using that methodology, we show that the meta-learning method presented leads to significantly better rankings than the baseline ranking method. The evaluation methodology is general and can be adapted to other ranking problems. Although here we have concentrated on ranking classification algorithms, the meta-learning framework presented can provide assistance in the selection of combinations of methods or more complex problem solving strategies.
Comment: This paper presents a ranking methodology for algorithm recommendation.
Keywords: algorithm recommendation; meta-learning; data characterization; ranking

[Dzeroski and Zenko, 2002a]
S. Dzeroski and B. Zenko. Is combining classifiers better than selecting the best one? In C. Sammut and A. Hoffmann, editors, Proceedings of the Nineteenth International Conference on Machine Learning (ICML-2002), pages 123-130, San Francisco, CA, USA, 2002. Morgan Kaufmann.
Abstract: We empirically evaluate several state-of-the-art methods for constructing ensembles of heterogeneous classifiers with stacking and show that they perform (at best) comparably to selecting the best classifier from the ensemble by cross validation. We then propose a new method for stacking, that uses multi-response model trees at the meta-level, and show that it clearly outperforms existing stacking approaches and selecting the best classifier by cross validation.

[Dzeroski and Zenko, 2002b]
S. Dzeroski and B. Zenko. Stacking with multi-response model trees. In F. Roli and J. Kittler, editors, Proceedings of the Third International Workshop on Multiple Classifier Systems (MCS-2002), pages 201-211, 2002.
Abstract: We empirically evaluate several state-of-the-art methods for constructing ensembles of classifiers with stacking and show that they perform (at best) comparably to selecting the best classifier from the ensemble by cross validation. We then propose a new method for stacking, that uses multi-response model trees at the meta-level, and show that it outperforms existing stacking approaches, as well as selecting the best classifier from the ensemble by cross validation.

[Dzeroski and Zenko, 2004]
S. Dzeroski and B. Zenko. Is combining classifiers with stacking better than selecting the best one? Machine Learning, 2004. In press.
Abstract: We empirically evaluate several state-of-the-art methods for constructing ensembles of heterogeneous classifiers with stacking and show that they perform (at best) comparably to selecting the best classifier from the ensemble by cross validation. The best among state-of-the-art methods is stacking with probability distributions and multi-response linear regression. We propose two extensions of this method, one using an extended set of meta-level features and the other using multi-response model trees to learn at the meta-level. We show that the latter extension performs better than existing stacking approaches and selecting the best classifier by cross validation.

[Dzeroski et al., 2002]
S. Dzeroski, H. Blockeel, and L. Todorovski. Relational ranking with predictive clustering trees. In H. Motoda and T. Washio, editors, Proceedings of the ICDM-02 Workshop on Active Mining, pages 9-15, 2002.
Abstract: A novel class of applications of predictive clustering trees is addressed, namely relational ranking. Predictive clustering trees, as implemented in TILDE, allow for predicting multiple target variables from relational data. This approach makes sense especially if the target variables are not independent of each other. This is typically the case in ranking, where the (relative) performance of several approaches on the same task has to be predicted from a given description of the task. We propose to use predictive clustering trees for ranking. This allows us to use relational descriptions of the tasks. As compared to existing ranking approaches which are instance-based, our approach also allows for an explanation of the predicted rankings. We illustrate our approach on the task of ranking machine learning algorithms, where the (relative) performance of the algorithms on a given dataset has to be predicted from a given (relational) dataset description.

[Fürnkranz and Petrak, 2000]
Johannes Fürnkranz and Johann Petrak. Two remarks on landmarking. Unpublished Manuscript, 2000.
Abstract: Landmarking has recently been proposed as a novel technique for meta-learning. While conventional approaches typically describe a database with its statistical measurements and properties, landmarking proposes to enrich such a description with quick and easy-to-obtain performance measures of simple learning algorithms. In this working paper, we investigate two important aspects of landmarking. First, we suggest that the important information in landmarking is not the absolute value of the landmark, but the relation between the various landmarks on a dataset. Second, we propose to the use of sub-sampling estimates as a different way for efficiently obtaining landmarks.

[Fürnkranz and Petrak, 2001]
Johannes Fürnkranz and Johann Petrak. An evaluation of landmarking variants. In C. Giraud-Carrier, N. Lavrac, Steve Moyle, and B. Kavsek, editors, Proceedings of the ECML/PKDD Workshop on Integrating Aspects of Data Mining, Decision Support and Meta-Learning (IDDM-2001), pages 57-68, Freiburg, Germany, 2001.
Abstract: Landmarking is a novel technique for data characterization in meta-learning. While conventional approaches typically describe a database with its statistical measurements and properties, landmarking proposes to enrich such a description with quick and easy-to-obtain performance measures of simple learning algorithms. In this paper, we will discuss two novel aspects of landmarking. First, we investigate relative landmarking, which tries to exploit the relative order of the landmark measures instead of their absolute value. Second, we propose to the use of subsampling estimates as a different way for efficiently obtaining landmarks. In general, our results are mostly negative. The most interesting result is a surprisingly simple rule that predicts quite accurately when it is worth to boost decision trees.
Keywords: Meta-Learning, Landmarking, Subsampling

[Fürnkranz et al., 2002]
Johannes Fürnkranz, Johann Petrak, Pavel Brazdil, and Carlos Soares. On the use of fast subsampling estimates for algorithm recommendation. Technical Report OEFAI-TR-2002-36, Austrian Research Institute for Artificial Intelligence, Wien, Austria, 2002.
Abstract: The use of subsampling for scaling up the performance of learning algorithms has become fairly popular in the recent literature. In this paper, we investigate the use of performance estimates obtained on a subsample of the data for the task of recommending the best learning algorithm(s) for the problem. In particular, we examine the use of subsampling estimates as features for meta-learning, thereby generalizing previous work on landmarking and on direct algorithm recommendation via subsampling. The main goal of the paper is to investigate the influence of various parameter choices on the meta-learning performance, in particular the size of training and test sets and the number of subsamples.
Keywords: Meta-Learning, Landmarking, Subsampling, Ranking

[Gamberger et al., 2002]
D. Gamberger, N. Lavrac, and D. Wettschereck. Subgroup visualization: A method and application in population screening. In Proceedings of the ECAI-02 Workshop on Intelligent Data Analysis in Medicine and Pharmacology (IDAMAP-02), Lyon, France, 2002.

[Giraud-Carrier and Keller, 2002]
C. Giraud-Carrier and J. Keller. Meta-learning. In J. Meij, editor, Dealing with the Data Flood: Mining Data, Text and Multimedia, pages 832-844. The Hague, 2002.

[Giraud-Carrier and Povel, to appear]
C. Giraud-Carrier and O. Povel. Characterising data mining software. Intelligent Data Analysis, to appear.

[Hilario and Kalousis, 1999]
M. Hilario and A. Kalousis. Building algorithm profiles for prior model selection in knowledge discovery systems. In Proceedings of the IEEE SMC'99, International Conference on Systems, Man and Cybernetics. IEEE press, October 1999.
Abstract: We propose the use of learning algorithm profiles to address the model selection problem in knowledge discovery systems. These profiles consist of metalevel feature-value vectors which describe learning algorithms from the point of view of their representation and functionality, efficiency, robustness, and practicality. Values for these features are assigned on the basis of author specifications, expert consensus or previous empirical studies. We review past evaluations of the better known learning algorithms and suggest an experimental strategy for building algorithm profiles on more quantitative grounds. Preliminary experiments have disconfirmed expert judgments on certain algorithm features, thus showing the need to build and refine such profiles via controlled experiments.

[Hilario and Kalousis, 2000a]
M. Hilario and A. Kalousis. Building algorithm profiles for prior model selection in knowledge discovery systems. Engineering Intelligent Systems, 8(2), 2000.
Abstract: We propose the use of learning algorithm profiles to address the model selection problem in knowledge discovery systems. These profiles consist of metalevel feature- value vectors which describe learning algorithms from the point of view of their representation and functionality, efficiency, resilience, and practicality. Values for these features are assigned on the basis of author specifications, expert consensus or previous empirical studies. We review past evaluations of the better known learning algorithms and suggest an experimental strategy for building algorithm profiles on more quantitative grounds. Preliminary experiments have disconfirmed expert judgments on certain algorithm features, thus showing the need to build and refine such profiles via controlled experiments.

[Hilario and Kalousis, 2000b]
Melanie Hilario and Alexandros Kalousis. Quantifying the resilience of inductive algorithms for classification. In Proceedings of the 4th European Conference on Principles and Practice of Knowledge Discovery in Databases, pages 106-115. Springer, September 2000.
Abstract: Selecting the most appropriate learning algorithm for a given task has become a crucial research issue since the advent of multi-paradigm data mining tool suites. To address this issue, researchers have tried to extract dataset characteristics which might provide clues as to the most appropriate learning algorithm. We propose to extend this research by extracting inducer profiles, i.e., sets of metalevel features which characterize learning algorithms from the point of view of their representation and functionality, efficiency, practicality, and resilience. Values for these features can be determined on the basis of author specifications, expert consensus or previous case studies. However, there is a need to characterize learning algorithms in more quantitative terms on the basis of extensive, controlled experiments. This paper illustrates the proposed approach and reports empirical findings on one resilience-related characteristic of classifier inducers, namely their tolerance to irrelevant variables in training data.

[Hilario and Kalousis, 2002]
M. Hilario and A. Kalousis. Fusion of meta-knowledge and meta-data for case-based model selection. In Proceedings of the 5th European Conference on Principles of Data Mining and Knowledge Discovery (PKDD-01), Freiburg, Germany, 2002. Springer-Verlag.

[Hilario, 2002]
Melanie Hilario. Model complexity and algorithm selection in classification. In S. Lange, K. Satoh, and C. H. Smith, editors, Proceedings of the 5th International Conference on Discovery Science (DS-02), pages 113-126, Lübeck, Germany, 2002. Springer-Verlag.

[Kalousis and Hilario, 2000a]
A. Kalousis and M. Hilario. A comparison of inducer selection via instance-based and boosted decision-tree meta-learning. In Proceedings of the 5th International Workshop on Multistrategy Learning, Guimaraes, Portugal, June 2000.
Abstract: The selection of an appropriate inducer is crucial for performing effective classification. In previous work we presented a system called NOEMON which relied on a mapping between dataset characteristics and inducer performance to propose inducers for specific datasets. Instance-based learning was used to create that mapping. Here we extend and refine the set of data characteristics; we also use a wider range of base-level inducers and a much larger collection of datasets to create the meta-models. We compare the performance of meta-models produced by instance-based learners and boosted decision trees. The results show that boosted decision tree models enhance the performance of the system.

[Kalousis and Hilario, 2000b]
A. Kalousis and M. Hilario. Model selection via meta-learning: a comparative study. In Proceedings of the 12th International IEEE Conference on Tools with AI, Vancouver, November 2000. IEEE press.
Abstract: The selection of an appropriate inducer is crucial for performing effective classification. In previous work we presented a system called Noemon which relied on a mapping between dataset characteristics and inducer performance to propose inducers for specific datasets. Instance-based learning was used to create that mapping. Here we extend and refine the set of data characteristics; we also use a wider range of base-level inducers and a much larger collection of datasets to create the meta-models. We compare the performance of meta-models produced by instance-based learners, decision trees and boosted decision trees. The results show that decision trees and boosted decision trees models enhance the performance of the system.

[Kalousis and Hilario, 2000c]
A. Kalousis and M. Hilario. Supervised knowledge discovery from incomplete data. In Proceedings of the 2nd International Conference on Data Mining. WIT Press, June 2000.
Abstract: Incomplete data can raise more or less serious problems in knowledge discovery systems depending on the quantity and pattern of missing values as well as the generalization method used. For instance, some methods are inherently resilient to missing values while others have built-in methods for coping with them. Still others require that none of the values are missing; for such methods, preliminary imputation of missing values is indispensable. After a quick overview of current practice in the machine learning field, we explore the problem of missing values from a statistical perspective. In particular, we adopt the well-known distinction between three patterns of missing value (missing completely at random (MCAR), missing at random (MAR) and not missing at random (NMAR)) to focus a comparative study of eight learning algorithms from the point of view of their tolerance to incomplete data. Experiments on 47 datasets reveal a rough ranking from the most resilient (e.g., Naive Bayes) to the most sensitive (e.g., IB1 and surprisingly C50rules). More importantly, results show that for a given amount of missing values, their dispersion among the predictive variables is at least as important as the pattern of missingness

[Kalousis and Hilario, 2001a]
A. Kalousis and M. Hilario. Feature selection for meta-learning. In Proceedings of the 5th Pacific Asia Conference on Knowledge Discovery and Data Mining. Springer, April 2001.
Abstract: The selection of an appropriate inducer is crucial for performing effective classification. In previous work we presented a system called Noemon which relied on a mapping between dataset characteristics and inducer performance to propose inducers for specific datasets. Instance-based learning was applied to meta-learning problems, each one associated with a specific pair of inducers. The generated models were used to provide a ranking of inducers on new datasets. Instance-based learning assumes that all the attributes have the same importance. We discovered that the best set of discriminating attributes is different for every pair of inducers. We applied a feature selection method on the meta-learning problems, to get the best set of attributes for each problem. The performance of the system is significantly improved.

[Kalousis and Hilario, 2001b]
A. Kalousis and M. Hilario. Model selection via meta-learning: A comparative study. International Journal of Artificial Intelligence Tools, 10(4), 2001.

[Kalousis and Theoharis, 1999]
A. Kalousis and T. Theoharis. Noemon: Design, implementation and performance results of an intelligent assistant for classifier selection. Intelligent Data Analysis, 3(5):319-337, November 1999.
Abstract: The selection of an appropriate classification model and algorithm is crucial for effective knowledge discovery on a dataset. For large databases, common in data mining, such a selection is necessary, because the cost of invoking all alternative classifiers is prohibitive. This selection task is impeded by two factors: First, there are many performance criteria, and the behaviour of a classifier varies considerably with them. Second, a classifier's performance is strongly affected by the characteristics of the dataset. Classifier selection implies mastering a lot of background information on the dataset, the models and the algorithms in question. An intelligent assistant can reduce this effort by inducing helpful suggestions from background information. In this study, we present such an assistant, NOEMON. For each registered classifier, NOEMON measures its performance for a collection of datasets. Rules are induced from those measurements and accommodated in a knowledge base. The suggestion on the most appropriate classifier(s) for a dataset is then based on those rules. Results on the performance of an initial prototype are also given.

[Kalousis, 2002]
A. Kalousis. Algorithm Selection via Meta-Learning. Thesis number 3337, Department of Computer Science, University of Geneva, 2002.

[Keller and Nakhaeizadeh, 1999]
J. Keller and G. Nakhaeizadeh. Preprocessing and tranformation flow data indexes for concept design by knowledge discovery and data mining in an engine-intake-port database application. In C. Giraud-Carrier and B. Pfahringer, editors, Proceedings of the ICML-99 Workshop on Recent Advances in Meta-Learning and Future Work, Bled, Slovenia, 1999.

[Keller et al., 2000a]
Joerg Keller, Iain Paterson, and Helmut Berrer. An integrated concept for multi-criteria ranking of data mining algorithms. In Joerg Keller and Christophe Giraud-Carrier, editors, Proceedings of the ECML-00 Workshop on Meta-Learning: Building Automatic Advice Strategies for Model Selction and Method Combination, Barcelona, Spain, 2000.
Keywords: Multi-Criteria Ranking, DEA, Meta-Data, Meta-Learning

[Keller et al., 2000b]
Jörg Keller, Valerij Bauer, and Wojciech Kwedlo. Application of data-mining and knowledge discovery in automotive data engineering. In Proceedings of the Fourth European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD2000), pages 464-469. Springer, 2000.

[Köpf et al., 2000a]
Christian Köpf, Charles C. Taylor, and Joerg Keller. Meta-analysis: Data characterisation for classification and regression on a meta-level. In Antony Unwin, Adalbert Wilhelm, and Ulrike Hofmann, editors, Proceedings of the International Symposium on Data Mining and Statistics, Lyon, France, 2000.
Abstract: Meta-Analysis can serve as a base for Meta-Learning: To the user's support withautomated guidance in model selection and data transformation. The first applicationfield in METAL(Meta-Learning assistant, ESPRIT project 26.357) was classification wheredata characteristics, measures, and tests had to be evaluated and now in the phase ofregression learning, they had to be proved. We describe necessary statistics and datacharacteristics for regression learning. As a new approach, we present Meta-Regression:Regression learning performed on the meta level for Meta-Learning. This new directioncould ``sharpen'' the accuracy of Meta-Learning, in particular when compared to aclassification of error rates.
Keywords: Meta-Learning, Data Characterisation, Meta-Regression

[Köpf et al., 2000b]
Christian Köpf, Charles C. Taylor, and Joerg Keller. Meta-analysis: From data characterisation for meta-learning to meta-regression. In Pavel Brazdil and Alipio Jorge, editors, Proceedings of the PKDD-00 Workshop on Data Mining, Decision Support,Meta-Learning and ILP: Forum for Practical Problem Presentation andProspective Solutions, Lyon, France, 2000.
Abstract: An extended Meta-Analysis fertilizes a Meta-Learning, which is applied tosupport the user with an automated guidance in model selection and data-transformation.Two major application fields were selected in METAL(Meta-Learning assistant, ESPRITproject 26.357): classification and regression learning. In phase 1 of the project, the data characteristics, measures and tests have been evaluated for an automated use ofclassification algorithms. For regression learning, the statistics, informationtheoretical measures and tests had to be proved. This paper works out necessarystatistics and tests for regression learning. The new approach of this paper is touse a Meta-Regression: A regression learning on the meta level for Meta-Learning.In comparison to a classification of error rates, calculated for cross-validation tests,our new approach could improve the accuracy for Meta-Learning.
Keywords: Meta-Learning, Data Characterisation, Classification, Meta-Regression

[Köpf et al., 2001]
C. Köpf, C.C. Taylor, and J. Keller. Data characterization and multi-criteria performance measures as means of model selection in regression. In C. Giraud-Carrier, N. Lavrac, and S. Moyle, editors, Proceedings of the ECML/PKDD-01 Workshop on Integrating Aspects of Data Mining, Decision Support and Meta-learning. Positions, Developments and Future Directions (IDDM-01), 2001.

[Kuba et al., 2002]
P. Kuba, P. Brazdil, C. Soares, and Woznica A. Exploiting sampling and meta-learning for parameter setting for multilayer perceptron on regression tasks. Technical report, LIACC, University of Porto / Masaryk University, Brno., 2002.

[Leite and Brazdil, 2001]
R. Leite and P. Brazdil. Decision tree based feature selection via subsampling. In F. Moura-Pires, G. Guimar~aes, P. Brazdil, and A. Jorge, editors, Proceedings of Workshop on EKDB Associated with EPIA01, 2001.

[Leite and Brazdil, 2002]
R. Leite and P. Brazdil. A decision tree-based attributes selection via subsampling. In Proceedings of the Workshop WK1 on Learning and Data Mining, associated with Iberamia-02 Conference, Sevilla, Spain, November 2002.

[Lindner and Studer, 1999a]
Guido Lindner and Rudi Studer. AST: Support for algorithm selection with a CBR approach. In C. Giraud-Carrier and B. Pfahringer, editors, Proceedings of the ICML-99 Workshop on Recent Advances in Meta-Learning and Future Work, Bled, Slovenia, 1999.

[Lindner and Studer, 1999b]
Guido Lindner and Rudi Studer. AST: Support for algorithm selection with a CBR approach. In Jan Rauch and Jan Zytkow, editors, Proceedings of the 3rd International Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD-99), pages 418-423, Prague, Czech Republic, 1999. Springer, Berlin.

[Lindner and Studer, 1999c]
Guido Lindner and Rudi Studer. Forecasting the fault rate behavior of cars. In C. Giraud-Carrier and B. Pfahringer, editors, Proceedings of the ICML-99 Workshop on Recent Advances in Meta-Learning and Future Work, 1999.

[Nepil and Popelinsky, 2002]
M. Nepil and L. Popelinsky. Committee-based selective sampling with parameters set by meta-learning. In M. Bohanec, B. Kavsek, N. Lavrac, and Dunja Mladenic, editors, Proceedings of the ECML/PKDD-02 Workshop on Integration and Collaboration Aspects of Data Mining, Decision Support and Meta-Learning (IDDM-02), 2002.

[Peng et al., 2002a]
Y. Peng, P. Flach, P. Brazdil, and C. Soares. Decision tree-based data characterization for meta-learning. In M. Bohanec, B. Kavsek, N. Lavrac, and Dunja Mladenic, editors, Proceedings of the ECML/PKDD-02 Workshop on Integration and Collaboration Aspects of Data Mining, Decision Support and Meta-Learning (IDDM-2002), pages 111-122. Helsinki University Printing House, 2002.

[Peng et al., 2002b]
Y. Peng, P. Flach, C. Soares, and P. Brazdil. Improved data set characterisation for meta-learning. In S. Lange, K. Satoh, and C. H. Smith, editors, Proceedings of the 5th International Conference on Discovery Science (DS-2002), pages 141-152. Springer-Verlag, 2002.

[Petrak, 2000]
Johann Petrak. Fast subsampling performance estimates for classification algorithm selection. In J. Keller and C. Giraud-Carrier, editors, Proceedings of the ECML-00 Workshop on Meta-Learning: Building Automatic Advice Strategies for Model Selection and Method Combination, pages 3-14, Barcelona, Spain, 2000.
Abstract: The typical data mining process is characterized by the prospective and iterative application of a variety of different data mining algorithms from an algorithm toolbox. While it would be desirable to check many different algorithms and algorithm combinations for their performance on a database, it is often not feasible because of time and other resource constraints. This paper investigates the effectiveness of simple and fast subsampling strategies for algorithm selection. We show that even such simple strategies perform quite well in many cases and propose to use them as a base-line for comparison with meta-learning and other advanced algorithm selection strategies.

[Pfahringer et al., 2000]
Bernhard Pfahringer, Hilan Bensusan, and Christophe Giraud-Carrier. Meta-learning by landmarking various learning algorithms. In Pat Langley, editor, Proceedings of the 17th International Conference on Machine Learning (ICML-2000), pages 743-750, Stanford, CA, 2000.
Abstract: Landmarking is a novel approach to describing tasks in meta-learning. Previous approaches to meta-learning mostly considered only statistics-inspired measures of the data as a source for the definition of meta-attributes. Contrary to such approaches, landmarking tries to determine the location of a specific learning problem in the space of all learning problems by directly measuring the performance of some simple and efficient learning algorithms themselves. The experiments reported show how such a use of landmark values can help to distinguish between areas of the learning space favouring different learners. Experiments, both with artificial and real-world databases, show that landmarking selects, with moderate but reasonable level of success, the best performing of a set of learning algorithms.

[Popelínsky and Brazdil, 2000a]
L. Popelínsky and P. B. Brazdil. Combining the principal components method with decision tree learning. In R. S Michalski and P. B. Brazdil, editors, Proceedings of the Multistrategy Learning Workshop (MSL-00), pages 105-114, Guimar~aes, Portugal, June 2000. LIACC UP Porto.

[Popelínsky and Brazdil, 2000b]
L. Popelínsky and P. B. Brazdil. The principal components method as a pre-processing stage for decision tree learning. In P. Brazdil and A. Jorge, editors, Proceedings of the PKDD2000 Workshop on Data Mining, Decision Support, Meta-learning and ILP (DDMI 2000), Lyon, France, September 2000.

[Popelínsky, 2001]
L. Popelínsky. Combining the principal components method different learning algorithms. In C Giraud-Carier et al., editor, Proceedings of the ECML/PKDD2001 Workshop on Integrating Aspects of Data Mining, Decision Support and Meta-Learning (IDDM-01), pages 119-128, Freiburg, Germany, August 2001.

[Seewald and Fürnkranz, 2001]
Alexander K. Seewald and Johannes Fürnkranz. Grading classifiers. Technical Report OEFAI-TR-2001-01, Austrian Research Institute for Artificial Intelligence, Wien, Austria, 2001. Submitted for publication.
Abstract: In this paper, we introduce grading, a novel meta-classification scheme. While stacking uses the predictions of the base classifiers as meta-level attributes, we use ``graded'' predictions (i.e., predictions that have been marked as correct or incorrect) as meta-level classes. For each base classifier, one meta classifier is learned whose task is to predict when the base classifier will err. Hence, just like stacking may be viewed as a generalization of voting, grading may be viewed as a generalization of selection by cross-validation and therefore fills a conceptual gap in the space of meta-classification schemes. Grading may also be interpreted as a technique for turning the error-characterizing technique introduced by Bay and Pazzani (2000) into a powerful learning algorithm by resorting to an ensemble of meta-classifiers. Our experimental evaluation shows that this step results in a performance gain that is quite comparable to that achieved by stacking, while both, grading and stacking outperform their simpler counter-parts voting and selection by cross-validation.

[Seewald et al., 2001]
Alexander K. Seewald, Johann Petrak, and Gerhard Widmer. Hybrid decision tree learners with alternative leaf classifiers: An empirical study. In Proceedings of the 14th International FLAIRS Conference (FLAIRS-2001), Key West, Florida, 2001.
Abstract: There has been surprisingly little research so far that systematically investigated the possibility of constructing hybrid learning algorithms by simple local modifications to decision tree learners. In this paper we analyze three variants of a C4.5-style learner, introducing alternative leaf models (Naive Bayes, IB1, and multi-response linear regression, respectively) which can replace the original C4.5 leaf nodes during reduced error post-pruning. We empirically show that these simple modifications can improve upon the performance of the original decision tree algorithm and even upon both constituent algorithms. We see this as a step towards the construction of learners that locally optimize their bias for different regions of the instance space.

[Soares and Brazdil, 2000a]
C. Soares and P. Brazdil. Ranking classification algorithms with dataset selection: Using accuracy and time results. In R.S. Michalski and P.B. Brazdil, editors, Proceedings of the Fifth International Workshop on Multistrategy Learning (MSL 2000), pages 126-135, 2000.
Comment: presents a IBL-based ranking method that uses general, statistical and information-theoretic measures to characterize datasets; presents a way of combining success rate and time in a performance measure; presents results based on meta-data for 6 algorithms on 16 datasets
Keywords: Meta-Learning, Ranking, Multicriterion Evaluation

[Soares and Brazdil, 2000b]
C. Soares and P. Brazdil. Zoomed ranking: Selection of classification algorithms based on relevant performance information. In D.A. Zighed, J. Komorowski, and J. Zytkow, editors, Proceedings of the Fourth European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD2000), pages 126-135. Springer, 2000.
Comment: presents a IBL-based ranking method that uses general, statistical and information-theoretic measures to characterize datasets; presents a way of combining success rate and time in a performance measure; presents results based on meta-data for 6 algorithms on 16 datasets
Keywords: Meta-Learning, Ranking, Multicriterion Evaluation

[Soares and Brazdil, 2002a]
C. Soares and P. Brazdil. A comparative study of some issues concerning algorithm recommendation using ranking methods. In F.J. Garijo, J.C. Riquelme, and M. Toro, editors, Proceedings of the Eighth Ibero-American Conference on AI (IBERAMIA 2002), pages 80-89. Springer, 2002.

[Soares and Brazdil, 2002b]
C. Soares and P. Brazdil. Knowledge-based selection of data characteristics for algorithm recommendation using ranking methods. In M. Bohanec, B. Kavsek, N. Lavrac, and Dunja Mladenic, editors, Proceedings of the ECML/PKDD-02 Workshop on Integration and Collaboration Aspects of Data Mining, Decision Support and Meta-Learning (IDDM-2002), pages 129-134. Helsinki University Printing House, 2002.
Comment: This paper shows that manual selection of meta-features significantly improves the quality of recommended rankings, in terms of correlation. The results also show that the top-2 to top-5 strategies are good choices in terms of the trade-off between accuracy and time.

[Soares et al., 2000a]
C. Soares, P. Brazdil, and J. Costa. Measures to compare rankings of classification algorithms. In H.A.L. Kiers, J.-P. Rasson, P.J.F. Groenen, and M. Schader, editors, Data Analysis, Classification and Related Methods, Proceedings of the Seventh Conference of the International Federation of Classification Societies IFCS, pages 119-124. Springer, 2000.
Comment: discussion of three measures to evaluate rankings
Keywords: Ranking Evaluation, Rank Correlation

[Soares et al., 2000b]
C. Soares, J. Costa, and P. Brazdil. Distance to reference: A simple measure to evaluate rankings of supervised classification algorithms. In JOCLAD 2000: VI Jornadas de Classifica c c~ao e Análise de Dados, pages 61-66, 2000.
Comment: presents a multicriterion ranking evaluation measure
Keywords: Multicriterion Evaluation, Ranking Evaluation

[Soares et al., 2000c]
C. Soares, J. Costa, and P. Brazdil. A simple and intuitive measure for multicriteria evaluation of classification algorithms. In J. Keller and C. Giraud-Carrier, editors, ECML 2000 Workshop on Meta-Learning: Building Automatic Advice Strategies for Model Selection and Method Combination, pages 87-96, 2000.
Comment: presents a multicriterion ranking evaluation measure
Keywords: Multicriterion Evaluation, Ranking Evaluation

[Soares et al., 2001a]
C. Soares, J. Costa, and P. Brazdil. Improved statistical support for matchmaking: Rank correlation taking rank importance into account. In JOCLAD 2001: VII Jornadas de Classifica c c~ao e Análise de Dados, pages 72-75, 2001.
Comment: presents a weighted rank correlation coefficient
Keywords: Rank Correlation, Weighted Correlation

[Soares et al., 2001b]
C. Soares, J. Petrak, and P. Brazdil. Sampling-based relative landmarks: Systematically test-driving algorithms before choosing. In P. Brazdil and A. Jorge, editors, Proceedings of the 10th Portuguese Conference on Artificial Intelligence (EPIA 2001), pages 88-94. Springer, 2001.
Comment: Proposes a new data characterization approach, called Sample-Based Landmarks. The data characteristics are obtained by running algorithms on small samples of the data and further transformed to reflect relative algorithm performance.
Keywords: Landmarks, Sampling, Meta-Learning, Ranking, Algorithm Selection

[Soares et al., 2002]
C. Soares, P. Brazdil, and C. Pinto. Machine learning and statistics to detect errors in forms: Competition or cooperation? In P. Brito and D. Malerba, editors, Proceedings of the ECML/PKDD Workshop on Mining Official Data (MOD02), pages 129-135. Helsinki University Printing House, 2002.

[Soares, 1999]
C. Soares. Ranking classification algorithms on past performance. Master's thesis, Faculty of Economics, University of Porto, 1999.

[Soares, 2002]
C. Soares. Is the UCI repository useful for data mining? In N. Lavrac, H. Motoda, and T. Fawcett, editors, Proceedings of the ICML-2002 Workshop on Data Mining Lessons Learned, pages 69-75, 2002.

[Sykacek, 1999]
Peter Sykacek. Metalevel learning - is more than model selection necessary?. In C. Giraud-Carrier and B. Pfahringer, editors, Proceedings of the ICML-99 Workshop on Recent Advances in Meta-Learning and Future Work, pages 66-73, Ljubljana, Slovenia, 1999.

[Todorovski and Dzeroski, 1999]
L. Todorovski and S. Dzeroski. Experiments in meta-level learning with ilp. In J. M. Zytkow and J. Rauch, editors, Proceedings of the 3rd European Conference on Principles of Data Mining and Knowledge Discovery (PKDD-99), pages 98-106. Springer-Verlag, 1999.
Abstract: When considering new datasets for analysis with machine learning algorithms, we encounter the problem of choosing the algorithm which is best suited for the task at hand. The aim of meta-level learning is to relate the performance of different machine learning algorithms to the characteristics of the dataset. The relation is induced on the basis of empirical data about the performance of machine learning algorithms on the different datasets. In the paper, an Inductive Logic Programming (ILP) framework for meta-level learning is presented. The performance of three machine learning algorithms (the tree learning system C4.5, the rule learning system CN2 and the k-NN nearest neighbour classifier) were measured on twenty datasets from the UCI repository in order to obtain the dataset for meta-learning. The results of applying ILP on this meta-learning problem are presented and discussed.

[Todorovski and Dzeroski, 2000]
L. Todorovski and S Dzeroski. Combining classifiers with meta decision trees. In Proceedings of the 4th European Conference on Principles of Data Mining and Knowledge Discovery (PKDD-2000), pages 54-64. Springer-Verlag, 2000.

[Todorovski and Dzeroski, 2003]
L. Todorovski and S. Dzeroski. Combining classifiers with meta decision trees. Machine Learning, 50(3):223-249, March 2003.
Abstract: The paper introduces meta decision trees (MDTs), a novel method for combining multiple classifiers. Instead of giving a prediction, MDT leaves specify which classifier should be used to obtain a prediction. We present an algorithm for learning MDTs based on the C4.5 algorithm for learning ordinary decision trees (ODTs). An extensive experimental evaluation of the new algorithm is performed on twenty-one data sets, combining classifiers generated by five learning algorithms: two algorithms for learning decision trees, a rule learning algorithm, a nearest neighbor algorithm and a naive Bayes algorithm. In terms of performance, stacking with MDTs combines classifiers better than voting and stacking with ODTs. In addition, the MDTs are much more concise than the ODTs and are thus a step towards comprehensible combination of multiple classifiers. MDTs also perform better than several other approaches to stacking.
Keywords: ensembles of classifiers; meta-level learning; combining classifiers; stacking; decision trees

[Todorovski et al., 2000]
L. Todorovski, P. Brazdil, and C. Soares. Report on the experiments with feature selection in meta-level learning. In P. Brazdil and A. Jorge, editors, Proceedings of the Data Mining, Decision Support, Meta-Learning and ILP Workshop at PKDD2000, pages 27-39, 2000.
Comment: study on feature selection in meta-level learning using ranking and algorithm selection based on general, statistical and information-theoretic measures
Keywords: Meta-Learning, Feature Selection, Ranking, Algorithm Selection

[Todorovski et al., 2002a]
L. Todorovski, H. Blockeel, , and S. Dzeroski. Ranking with predictive clustering trees. In T. Elomaa, H. Mannila, and H. Toivonen, editors, Proceedings of the 13th European Conference on Machine Learning (ECML-2002), pages 444-455. Springer-Verlag, 2002.
Abstract: A novel class of applications of predictive clustering trees is addressed, namely ranking. Predictive clustering trees, as implemented in Clus, allow for predicting multiple target variables. This approach makes sense especially if the target variables are not independent of each other. This is typically the case in ranking, where the (relative) performance of several approaches on the same task has to be predicted from a given description of the task. We propose to use predictive clustering trees for ranking. As compared to existing ranking approaches which are instance-based, our approach also allows for an explanation of the predicted rankings. We illustrate our approach on the task of ranking machine learning algorithms, where the (relative) performance of the learning algorithms on a dataset has to be predicted from a given dataset description.

[Todorovski et al., 2002b]
L. Todorovski, P. Brazdil, and C. Soares. Experiments with feature selection in meta-learning. Technical Report IJS-DP 8640, Jozef Stefan Institute, Ljubljana, Slovenia, 2002.

[Wettschereck and Müller, 2001]
D. Wettschereck and S. Müller. Exchanging data mining models with the predictive modelling markup language. In C. Giraud-Carrier, N. Lavrac, and S. Moyle, editors, Proceedings of the ECML/PKDD-01 Workshop on Integrating Aspects of Data Mining, Decision Support and Meta-learning. Positions, Developments and Future Directions (IDDM-01), 2001.

[Wettschereck and Students, 2002]
D. Wettschereck and Students. Educational data preprocessing. In P. Berka, editor, Proceedings of the ECML-02 Discovery Challenge Workshop, pages 0-6, Helsinki, Finland, 2002.

[Wettschereck, 2002]
D. Wettschereck. A kddse-independent pmml visualizer. In M. Bohanec, B. Kavsek, N. Lavrac, and Dunja Mladenic, editors, Proceedings of the ECML/PKDD-02 Workshop on the Integration aspects of Data Mining, Decision Support and Meta-Learning (IDDM-02), Helsinki, Finland, 2002.

[Zenko and Dzeroski, 2002]
B. Zenko and S. Dzeroski. Stacking with an extended set of meta-level attributes and mlr. In T. Elomaa, H. Mannila, and H. Toivonen, editors, Proceedings of the 13th European Conference on Machine Learning (ECML-02), pages 493-504, Helsinki, Finland, 2002.
Abstract: We propose a new set of meta-level features to be used for learning how to combine classifier predictions with stacking. This set includes the probability distributions predicted by the base-level classifiers and a combination of these with the certainty of the predictions. We use these features in conjunction with multi-response linear regression (MLR) at the meta-level. We empirically evaluate the proposed approach in comparison to several state-of-the-art methods for constructing ensembles of heterogeneous classifiers with stacking. Our approach performs better than existing stacking approaches and also better than selecting the best classifier from the ensemble by cross validation (unlike existing stacking approaches, which at best perform comparably to it).

[Zenko et al., 2002]
B. Zenko, L. Todorovski, and S. Dzeroski. Experiments with heterogeneous meta decision trees. Technical Report IJS-DP 8638, Jozef Stefan Institute, Ljubljana, Slovenia, 2002.