STBenchmark

A fundamental problem in information integration is to precisely specify the relationships, called mappings, between schemas. Designing mappings is a time-consuming process. To alleviate this problem, many mapping systems have been developed to assist the design of mappings. However, there is not a clear method on how to evaluate and compare such systems. A benchmark for mapping systems is important for assessing their relative merits, which is in turn important to customers for making the right investment decisions, and to mapping developers for testing and features of their products, improving their capabilities, and comparing them to others.

The input to a mapping system is a mapping scenario. A mapping scenario is a triplet <S,T,P> where S is a source schema, T is a target schema and P is a transformation specification which given an instance I of the source schema S generates a target instance of the target schema T. The role of the mapping system is to assist the user in implementing the transformation specification P in some transformation language.

STBenchmark (Source-Target Benchmark) is a publicly available benchmark for testing and evaluating mapping systems. It consists of three basic components:

A basic suite of mapping scenarios that have been collected from the literature in information integration, data exchange, schema integration and schema evolution, as well as from real-life applications. We believe that these mapping scenarios represent the minimum set of transformations that should be readily supported by any mapping system.
A mapping scenario generator as well as an instance generator that can produce arbitrary complex mapping mapping scenarios and instances of varying sizes of the source schema.
A simple usability model that can be used as a first-cut measure on the use of a mapping system. The usability model measures in terms of human actions the effort requires by a mapping system user to implement a mapping scenario using the mapping system.

STBenchmark is the product of a collaborative effort between the University of California Santa-Cruz and the University of Trento.