News | about | Search Language: DE | EN | FR
 Recommendations on quality criteria

European Treasury Browser
ETB Handbook
Multilinguality and Thesaurus
Outputs and Documents
•  Controlled Vocabulary: Quality Selection Policy
•  Forum on Quality - Week I
•  Forum on Quality - Week I - Archive
•  Forum on Quality - Week II
•  Forum on Quality - Week II - Archive
•  Forum on Quality - Week III -Archive
•  Forum on Quality - Week III -Summary
•  Forum on Quality - Week III: Recommendation on Quality Assurance and Selection of resources for the ETB-network
•  Rationale behind the quality management
•  Recommendations on quality criteria
Search this area

Advanced Search

Print this page!
Tell a friend!
To get a reminder when this page is updated, please enter your email address here:

Other languages: [fr ]

"Guidelines": recommendations on quality criteria, content assessment, selection of resources to be included in ETB
by Maria Daniela Nenci, PDP
ETB partner: WP4 coordinator

In defining the selection criteria of educational resources to be included in ETB, the first question to ask oneself is "What do we intend by "resource quality". The interpretative key for the dual concept resource-quality cannot be anything else but the relationship between the user and the resource.

A resource can truly be defined as such if it produces real knowledge for a given type of user, if it transforms data in elements of knowledge, if it satisfy the information need of the user. DESIRE, borrowing from the commercial sector the definition of quality, affirms that it is possible to talk of quality, when the resource is related to "customer's" satisfaction (...)."In the context of a subject gateway, the quality of a resource will depend on the users of service, and the nature of the service, as well as the internal features of the resource itself."

The incidence of information on the knowledge of a given user group is the interpretative guideline followed by entities that produce and/or organise and/or disseminate information and documentation and, consequently, the first selection criteria applied to r resources. Data are, in fact, elements that produce new knowledge, both theoretical and practical. They can be processed, stored, circulated , but they are not bound to a limited and defined exploitation. Their function, as well as their value, changes according to different users, people with different interests and needs, and their placing in different frameworks of knowledge . It is only through this new placing that data become information.

Information is something that modifies knowledge, from a qualitative and a quantitative point of view it affects the knowledge framework of the users and modifies it: in this the dynamic characteristic of the datum can be seen. The value and significance of information depend on the relation between users and data which places itself in a framework of continuously changing knowledge.

Therefore, data are not lacking in significance, but their significance is highlighted through the dynamic relation with the user. The user appropriates data through this process and discovers their relations.

Documentation interprets data to communicate and diffuse them in function of the users and thus transforms data into information. Data acquire "information value" when they are analysed and interpreted for a specific target.

Every national documentation system has carried out documentation processing (source selection, documentary processing, information processing and diffusion) in relation to its target users whose needs have been analysed.
It is thus difficult to think that resources placed in the national repositories, i.e. selected from specific sources, processed and structured in relation to a specific user analysis, can automatically continue being resources, when different sources and users are considered.

Therefore, only documentation methodologies and means, criteria, procedures common to the system can make of ETB a European source for users at European level.
For this reason, this optimal European educational repository, the ETB, cannot originate, nor be developed through resources derived from information and documentary systems which are so different, as it has been highlighted by the WP2 survey. In fact, every national, regional and local system has already effected some choices relating to the sources selection, the target users, to the definition of the types of material functional to the users and to the subsequent documentary processing of the resources.ETB should instead represent a European documentation system, where the actors use documentary standards, selection criteria and quality controls mechanism which are common to the system and established in advance.

Only on the basis of this systemic structure ETB will be able to achieve its main objective that is to offer quality resources to the target users, to support teaching and learning but also research and the production of information and documentation.

Furthermore, a resource produces new knowledge in the identified users target groups, if the organisation, the structuring and the description of data are specifically addressed to the target user group. The chief selection criteria come from establishing the relationship between target user and the description of the resource.From this it follows that the structuring of information relating to a specific object or type of material (primary document) must be consistent, so that resources can be compared, assessed and selected by the user, according to his/her information needs.In addition, the semantic description of the resource must be analytical, accurate and complete, so that the user can extract all those elements that can help him to understand if he can use the resource effectively.
However, there remains the problem of finding parameters by which the effectiveness of a resource in relation to the user's information needs can be evaluated. Undoubtedly the planning of educational-teaching documentation services should address, through experimental methodologies, this area of research.

1. Types of resources

But which are the main types of resource and materials to select for ETB?
The variety of terms used by repositories to indicate the same type of resource as indicated by the survey carried out by WP2 may be due prevalently to the lack of normalised language, this variety may be also a consequence of different target groups, but it is also true that it is difficult to reach a general consensus on the concept of "resource".

Only those educational resources which meet the requirements for information of identified target group: teachers and pupils/students should be considered.

This approach in the processing of the results of WP2 survey has given as a result the following classification of the main types of resources:

· activity-based resources
-- resources on educational activity
-- resources on extra-curricular activity
· audio-visual materials
· best practice
· bibliographical material
· catalogues of resources
· calendar of educational/cultural events
· contact directories
· curriculum-based resources
· curriculum information
· discussion/forum -student corner
· distance education resources
· educational experiences
· educational/pedagogical research
· educational projects
· educational software
· exercises
· guidance and career information
· ICT information and support
· image set
· interactive resources
· (educational) legislation
· multimedia material
· online press
· partnership database
· self-study material
· software
· statistical data
· school magazine/journal
· sound
· teaching aids/guidelines
· teacher training resources
· youth information

The need to arrive at an agreed classification of resources is shared by different contexts and in relation to different initiatives, the co-operation between the Learning Technology Standards Committee Learning Objects Metadata (LTSC-LOM) Working Group of the IEEE (Institute of Electrical and Electronic Engineers) and the Dublin Core Metadata Initiative (DCMI) to develop interoperable metadata for learning, education and training has been recently announced . It would be auspicable to wait for the results of this co-operation in order to further define the ETB resource categories.

2. Scope of information covered

An other important selection criterion is provided by the definition of the scope of the information covered. This is another element that should originate from the analysis of the information needs of the target users. The following indications are very general and should be interpreted in a flexible way, either because it is difficult to identify the boundaries of a subject, or because the curriculum is continuously revised and changed.
Subject: the area "Education" is very broad but it is possible to identify the following:

· pre-school education
· primary education
· secondary education
· further education
· special education
· teacher education
· vocational training

Learning/teaching contents: resources should refer to the following contents:

Curriculum subjects

· applied science/technologies/techniques
· ethics/religion
· humanities (art, history, geography, language, literature)
· information/documentation/communication
· mathematics/ natural science/physics
· physical education/sport
· social and economic science

Cross-curricular/extra-curricular subjects

· Career education
· Consumer education
· Education for sustainable development
· Environmental education
· Equal opportunities
· European citizenship/
· European dimension
· Health/sex education
· Intercultural/multicultural education
· Media education
· Museum education
· Peace education
· Road safety education
· Technological education

The resources selection must be aimed at increasing the value of every single cultural identity, of the cultural heritage and folk culture of each country. An added value can be gained through the co-operation of European schools in planning and working together at common projects. The results of this kind of co-operation, as far as the process is concerned, would really represent the European dimension of resources.

With regard to geographical coverage, there should be no restriction relating to geographical coverage: the resources can be at national, international and local level. In all cases it is necessary to evaluate them in terms of reusability in different contexts.

About the language, the resources can be produced in the 11 official languages of the EU, in the national versions of the official languages and in the regional languages. Ethnical minorities needs should be taken into consideration. Problems related to multilinguism should be solved through WP6 proposed solutions

Because "Quality" is the key to the selection of resources, rigid time limits are not desirable or required (temporal coverage); all the same resources,for their scientific contents,should be as far as possible updated and up-to date.

A very important problem is the access, in the double meaning of technology and cost.
Resources based on the most friendly technologies for the users should be selected; resources which utilise advanced www technology should provide users clear indications on the type of technology, technical requirements, etc.
Special needs of users (blindness) should be taken into consideration as well.
Fee resources as well as free can be selected but in this instance it is necessary that information on the content (index) and the price of the resource should be clearly stated.

3. Relevant data content per type of materials and resources

It is essential for certain types of resource, such as experiences, projects, best practise examples, to intervene at the level of primary documents, to define the relevant content to provide homogenous thus comparable data, as well as at the level of secondary document (cataloguing) by proposing documentary standards that guarantee the retrieval of precise information which are functional to the users.
Therefore, only the integration of the above mentioned levels permits to document and describe every single "object" in a complete and exhaustive manner, as well as to allow their retrieval and elaboration in relation to the information needs of the users.

When one wishes to document experiences and examples of best practice the process which made the experience possible should highlighted, it is necessary that the primary document be organised and structured according to a defined "index". This is the only mean to make the documents homogeneous; only thus, data become comparable and then selectable by the target user of this type of material - the teacher.

In the primary document describing the best practice example, the relevant data contents must refer non only to the institution and/or the people who are the creator at the intellectual level of the experience, but above all to the relevant aspect of the process: the decisions taken at pedagogical, methodological and teaching level, which are at the base of the experiment, the general structural elements - the field of investigation, the specific contents, the curricular subjects involved, the individual subjects participating in the experience (teachers and pupils, in particular), costs and funding.

The most relevant data contents refer to the specific original features of a particular experience: such as its origin, the sequence of activities (time scale, actions, material, instruments), the mental processes, based on the reflections and on the emotions of subjects actively participating in the experience (students and teachers) which justify and explain decisions taken, knotty problems, and interpersonal dynamics.

The evaluation of the process, which represents the final, more "critical" stage, it is not meaningful unless elements which have been taken into consideration in the evaluation, are highlighted, such as the instruments, and the methods used to carry out the evaluation, the objectives that have been reached and the difficulties which have been encountered.

General considerations on the new knowledge produced by the experience (in terms of pedagogy, teaching, and organisation), the "ideas" which have developed in this experimental phase, are the new "resources" which must circulate, exactly because they represents the starting point for any successive development and proposal.

For the second level (the "catalogue card") the identification of the most significant elements must be achieved following certain parameters, as previously stated, which relate to the information need of the target user group both in terms of resources and material.

A complete and analytical description of a resource must include:

- general elements relating to the resource: title, description, subject, coverage, language, organisational structure, granularity, identifier;
- elements relating to the life cycle of the resource: creator, contributor, dates, publisher, number of edition, status of resource;
- technical elements: size, format, type of requirement, duration;
- elements relating to rights: copyright ,cost, price;
- elements relating to relationships: kind of relationship and source;
- elements relating to cataloguing: contributor, application used, date, scheme.

The elements relating to pedagogy and resource quality should be given prominence in the description of resources which are targeted to pupils/students and teachers.

The elements should relate in particular to:

- type of material
- interactivity type
- interactivity level
- semantic density
- users - target
- users - beneficiaries
- context of use (school level)
- age range of the pupils
- level of difficulty of the resource
- typical learning time
- description of use of the resource
- curricular standards
- teaching methods
- grouping
- assessment
- prerequisites of learning

Quality resource elements are:

- criteria
- category
- authority
- scale

These relevant data content are originated from the comparison of the descriptive elements proposed by DBS, IEEE/LOM, GEM, DCMI, VES, VEC, DBS, EdNA, EUN) and they represent the "maximum version" of all the elements required for the description.

4. Indicators for the selection

In short, the first criterion for resources selection is resource quality which must be evaluated in relation to the identified target user.
Resources must be selected on the basis of a clear definition of the target users and of the type of resource (see par. 1).

A possible definition of target users is the following:

- teachers in pre schools, primary, secondary schools, in institutes of further education and in vocational training
- pupils/students of primary secondary, institute of further education and in vocational training
- pupils/student with special educational needs
- pupils/students with learning difficulties
- trainers
- policy makers
- administrators
- families
- researchers
- providers
- all

Moreover resources must be selected on the basis of quality selection criteria listed below :

Content criteria

- Accuracy: the resource must be reliable, valid and produced by an authoritative source; information should be impartially presented; resource must not contain biases, mistakes or omissions.

- Appropriateness: the resource should contain information for the intended learners' level; the resource should use an appropriate and suitable vocabulary, language or concepts, avoid mistakes or stereotyping.

- Clarity: information should provide a clear tie between the purpose (goals, objectives) and the content and procedures suggested. Correlation should be comprehensive and obvious. Redundancy is usually unwelcome and isolated activities without a relationship are superfluous.

- Completeness: the resource should be complete, i.e., offer all essential information and elements, as well as inclusion of such components as self-contained activities, materials required, prerequisites, information for obtaining related resources, assessment criteria, links to quality indicators and standards. The resource should offer wide and in-depth information related to the topic.

- Motivation: the resource should achieve the active engagement of the learner and be interesting and appealing, build on prior knowledge and skills, and promote relevant action on the part of the learner.

- Organisation: the resource should be easy to use and logically sequenced, with each segment of the resource related to other segments. It should flow in an orderly manner, using organising tools (i.e. headings, a map, etc.) and avoiding use of unrelated elements that are potentially ineffective or overpowering; it should provide for references, bibliographies and other supporting materials available for the users.

All these aspects could be evaluated by the person in charge of approving the resource on the basis of an assessment mark scale to be defined:

Form criteria

The resources should have a documentary organisation based on the cataloguing integration and the access to the primary document; resources should be documented through the use of documentary standards common to the project.

The selected resources should be user friendly, easy to navigate and provided with an adequate user-support and make appropriate use of technology.
The resource should be easy to navigate and dead links should not be present; moreover, it should be easy to go back and forth through the links, etc.
Documentary supports should be available for the users for a proper use of the resource.

Process criteria

The resources are often volatile and they change with the passing of the time. ETB should aim at selecting resources which are actually accessible, stable and properly updated.

Information integrity
Information contained in the resources should be updated; the duration of the information should be indicated as well as their liability with the passing of the time, the intervals of their updating, on times and proceeding of the updating. The provider should be responsible for these indications.

Site integrity

The site integrity should permit the users to know if the site is current and accurate; in fact, the dates of the last updating should be indicated; there should be no dead links; the resource version number should be provided.
The updating frequency should be described. The organisation or the person should be responsible of keeping and managing the resource in a stable way.
The webmaster should be responsible for providing these indications.

System integrity

The selected resources should be technically acceptable, currently accessible, and should not present frequent overloads.
The system should be stable and adequate measures should be taken to keep the system integrity. The system administrator should be responsible for providing these indications.

The resources which should be included in ETB should be relevant to the ETB scope, to the target users and the identified materials and should be assessed according to agreed selection criteria.
The resources should be homogeneously organised and structured and should be described by means of common documentary standards and tools established by the system.

Author: Maria Daniela Nenci
Web Editor: Riina Vuorikari
Published: Monday, 29 Jan 2001
Last changed: Monday, 3 Sep 2001
Keywords: quality, recommendations, etb