News | about | Search Language: DE | EN | FR
 Forum on Quality - Week III -Archive

European Treasury Browser
ETB Handbook
Multilinguality and Thesaurus
Outputs and Documents
•  Controlled Vocabulary: Quality Selection Policy
•  Forum on Quality - Week I
•  Forum on Quality - Week I - Archive
•  Forum on Quality - Week II
•  Forum on Quality - Week II - Archive
•  Forum on Quality - Week III -Archive
•  Forum on Quality - Week III -Summary
•  Forum on Quality - Week III: Recommendation on Quality Assurance and Selection of resources for the ETB-network
•  Rationale behind the quality management
•  Recommendations on quality criteria
Search this area

Advanced Search

Print this page!
Tell a friend!
To get a reminder when this page is updated, please enter your email address here:

Find here all the comments that were given during the third week of the Virtual Forum on Quality.

Forum on Quality issues of ETB.
Week 3
Moderator: Riina Vuorikari

Vasiliki: I wish to ask the repository managers (if any)...

I wish to ask repository managers to share the experiences they have with teachers using their material. More specifically, I wish to ask what kind of information teachers consider to be valuable in order to use a resource.


Tommy: Re:I wish to ask the repository managers (if any)...

In our national Danish repository - - all of our subject editors are teachers, meaning that quality criteria are implemented by colleagues of the primary users of the service. We consider this important.

Based on usability testing and responses sent to us by mail, it seems that teachers (and especially students) are not looking for pages of description of individual resources. What they care about is a brief description, and the knowledge that somebody has actually had a closer look at this resource and found it valuable for educational purposes.

This means that in our listings of resources we presently only show: Title (as an active link to the resource), language of the resource, and (the first 255 characters of a) description. Below this information is a link giving you access to further information such as author, publisher, keywords etc. Around 20% of our user sessions (~ 10.000 per week) includes visits to such pages with additional info.
Responses from some teachers indicate that having the author/publisher information available is important/helpful for their own judgment of whether the material is from a trustworthy source or not.

Our notion is thus, that teachers tend to go quite quickly to the resources they find, and see for themselves whether they are useful for the specific educational purpose they (might) have in mind. In other words - they are happy that some of their colleagues have done most of the job, sorting out relevant hits from the often massive number of hits you get using major search engines, but the final evaluation of whether it is useful is of course done by themselves : )

An important difference between the user behavior of students and teachers is: Teachers often go directly to the part of our repository dealing specifically with the subject they are teaching, and navigate the links included there. Students tend to either search across all subjects, or choose our 'thematic' or 'Yahoo-style' browsing structure.

This is a few of our experiences - hope to hear from others :-)


abeckers: Re:Re:I wish to ask the repository managers (if any)...

Thanks for your experiences. If I understand it well from a quality perspective, there is 'less' need for a traffic agent who judges the usefulness of a resource than we expect.

Do I understand you correct that you opt for a quiet open structure?

Title:  Re:Re:Re:I wish to ask the repository managers (if any)...
Name:   tommy

Message: No - I don't opt for an open structure!
Actually I think it's essential to a repository to have a, not too big, well trained body of human subject editors in order to have a fair chance of implementing selecting criteria and a quality policy fairly homogenously across all subjects. At our repository we used to have close to a hundred subject editors, are now down to 40, but even that is a high number of persons to agree on what quality is :-) I'm not saying that the solution with the 'community as editors' doesn't work. I personally like very much, but stating that all 40.000+ editors share the same quality criteria would be an exaggeration IMHO : )
What I do argue though is that the _amount_of_metadata_ repositories add to a given resource should be limited. My experience is that average users are happy with a 'green light' (meaning that a quality policy has been implemented by some subject editor they trust, e.g. librarian, teacher etc.) and then just the 'bare necessities' of metadata, such as title, description, author/publisher and a browsing structure (e.g. implemented by a thesaurus or classification scheme) that makes sense to their particular universe. Another argument to limit the metadata is that it has to be reviewed frequently (say every three months) due to the dynamic nature of the web - and also because what was 'high quality' half a year ago might not fall into that category today...even though your quality policy hasn't changed in the meantime.

Hope I made myself clear ;^)

Title:  WEB sources and WEB documentary information: what teachers are asking for
Name:   Trigari

Message: As an operator in a national educational repository (, also responsible for managing an educational data-base, I would try to react to the discussion paper and some of the related messages on the basis of my experience; that without the pretension of covering a significant sample, because so far we have not conducted a study on the end-users of our repository in a scientific way.

The following variables being equal, i.e.:
-    teacher's familiarity with the computer;
-    teacher's access to Internet resources in his/her school environment;
what our teachers think to be valuable on the web in their teaching practice is -generally speaking -

1. The Internet in itself, a flexible space mobilising what Pierre Levy calls 'the collective intelligence" that creates/organises knowledge in a totally new way. In this case the medium is the first object of an analysis that can be carried out at different levels;
2. Paradoxically, all the materials that have not been conceived for teaching purposes, that is realia that the teacher might use in specific teaching/learning contexts (from literary full texts to newspapers; from statistical data to image databases; from the NASA site to the Louvre site; from the comic outcomes of automatic translators to any site in a foreign language);
3. All the facilities - from e-mail services to forums - allowing wide inter-connection at a world level;
4. Among specific student-targeted products, the ones that are:
4.1. Based on a problem-solving approach;
4.2. Highly interactive;
4.3. Simulation environments;
4.4. Stimulating students' creativity and personal contributions;
5. Among specific teacher-targeted products, the ones that are:
5.1. Authoritative selections of best sites (sites of sites) in different conceptual fields - here, again, the ones more focused on teaching subjects than on didactics, including software and  bibliographical sites. Professional/institutional information.
But also:
6. 'all-inclusive' educational packages - exercise-instructions-auto-evaluation - suitable to be downloaded, printed  in several copies and administered to the pupils;
7. reference materials (encyclopaedias, dictionaries, collected archival or historical sources for pupils, etc. etc.);

Points 2.,3.,5.,6.,7. refer both to teachers that are considered to be experienced and innovative and to those that are considered rather inexperienced and/or conservative, and at all educational levels. What obviously varies is how teachers use these resources.
As for the remaining points,  you can attempt to say that the approaches 1. and  4. are likely to be appreciated only by the more restricted brotherhood of  "good" teachers.

The most important issue: 'how effectively and ingeniously teachers use WEB resources' - in a certain way a variable independent of the quality of the used resources - cannot be solved - in my opinion - with short recommendations or teacher guides. As I wrote in the first round of this forum, this problem deals with the wider and extremely more complex issue of the teacher education nowadays.

What about internal quality, as for the above resources?
It is intuitive that for  resources not created for teaching purposes, quality - from an educational point of view - entirely lies in the way teachers use them. But our teachers will appreciate as a mark of quality an educational repository providing  plenty of links to interesting culture/information/debating sites and promoting a discussion on the medium in itself, without confining  itself to collecting teaching aids.

As for specific educational resources, at least in our educational setting, teachers keep trusting the opinion of eminent scholars/institutions, of the repository itself, if it has good reputation, of valued colleagues, as a quality mark much more valuable than the so-called 'objective' indicators - which - I fully agree with Margit - when they don't refer to technical features, are in fact highly subjective.
Good technical features and user friendliness (in great demand) are asked for; as for the contents, what's important for the teachers is that selectors and  selection criteria are made explicit, in order to - so to say - evaluate the evaluation.

As for teachers personally submitting their materials to an educational database, our experience (see ) suggests that this practice strongly needs both the foreseen EUN recommendations and a quality filter: good teaching doesn't necessarily imply expertise in creating both documentary units about 'best practice' in school and  educational resources suitable to be made available in a larger context . I don't think that the documentary and 'editorial' culture of the teachers is at the moment so advanced as to guarantee good results. A lot of totally insignificant stuff risk to enter the educational repositories.

What about documentary information quality, particularly related to Internet resources?

What our teachers seem to value the most in a catalogue of Internet resources is, in order of importance:
- the richest possible access to the documents by content (specific topic)/context (school/age level, discipline) descriptors;
- the possibility of  free searching including all the elements of  the record;
- presence of  an abstract;
- accurate but not redundant description.
Just to give you a piece of evidence for the above stated priority, searching by subject cover the 90% of the search statements in our data-bases.

Marisa Trigari  (INDIRE - Firenze)

Title: Dippe: Using web resources to improve the quality of teaching and learning

Message: In this short contribution I’ll start with giving the big picture of education and then gradually, via a real world example, focus on some findings and questions I feel are relevant for this discussion.

One of them is:
If we don’t know or know very little what happens in the class rooms of today, one might ask how we should approach quality issues when using IT and specifically using web based resources in education?

The European Treasury Browser project will put forth recommendations on quality assurance and selection of resources for the ETB network. Since the resources may be local, national or European wide, there’s an explicit need for a vital and broad discussion about quality issues.

My part within the ETB project is to write recommendations for the teachers and thereby hopefully add to the teaching and also learning quality when using web based resources.

In this short contribution I’ll start with giving the big picture of education and then gradually, via a real world example, focus on some findings and questions I feel are relevant for this discussion. It is purposely written for a broad category of readers.

Being a teacher today is very different compared with twenty years ago and our view of schools and especially education has changed dramatically during this short period of time. With the use of Internet based services and especially the web and e-mail, a broader audience has been introduced to concepts like i.e. life-long learning, flexible learning and on-line learning. Education is nowadays seen as an integrated part of our professional careers and not only as a part of the period in our lives before reaching adulthood.

I encourage the reader to communicate his/her own questions and also comment and criticize what I’ve written.
"The task is, not so much to see what no one has yet seen; but to think what nobody has yet thought, about that which everybody sees."
--Erwin Schrödinger

What is the goal of education?

Each country has it’s own goals but to find a description which summaries it into one “unifying goal” description, it may be useful to  cite UNESCO (1997), which states that
“The goal of education is to make people wiser, more knowledgeable, better informed, ethical, responsible, critical and capable of continuing to learn. Education also serves society by providing a critical reflection on the world, especially its failings and injustices, and by promoting greater consciousness and awareness, exploring new visions and concepts, and inventing new techniques and tools. Education is also the means for disseminating knowledge and developing skills, for bringing about desired changes in behaviours, values and lifestyles, and for promoting public support for the continuing and fundamental changes that will be required if humanity is to alter its course, leaving the familiar path that is leading towards growing difficulties, and starting the uphill climb towards sustainability. Education, in short, is humanity’s best hope and most effective means to the quest to achieve sustainable development” .

I will not write about the EU member countries different educational systems. This is described in detail on Eurybase,

My interest focuses rather on how these goals may be achieved in the local municipality and the importance of quality work for the outcome. It is my strong belief that we cannot discuss quality factors within the ETB project as separate entities but rather as part of a very complex system - the educational system.  This point of view will help us to better understand the local processes that are common throughout Europe and hopefully also give us a better understanding how our suggestions eventually will affect teachers and pupils in their processes of everyday teaching and learning using web based resources. I’ve been trying to find an example which by no means should be “the shining star” but rather what seems possible to achieve even with limited resources. Since my reading skills in foreign languages are limited to Norwegian, Danish, English, and German, I’m aware that there are certainly equally relevant examples in other European countries and also that I probably could have found better examples than the one presented here. You should be aware that Norwegian is not my mother tongue and it might therefore be some translation errors.

Improved quality in schools - a municipal example

Fet kommune is a small country side municipality with some 9000 inhabitants. It’s located 30 km northeast of Oslo, Norway. The municipality board decided in 1998 that the goals for the student’s study results and social development should be stated in a quality plan. The quality plan was achieved by starting a project which should involve politicians, parents, teachers, and students of the municipality. The decisions made by the politicians are based to a great extent on the outcome of the project and should be valid until 2010.

Three main questions were asked initially.

*What do we mean by quality in schools in our municipality?
*What should we aim at to improve the quality?
*What do we need to do to succeed in improving the quality?

The following goals shall be reached according to the project plan.

*More engagement from students, parents and teachers in the discussion “What characterizes a good school?”.
*High quality goals on all levels in the municipality including politicians, the director of education, principals, teachers, students and their parents.
*Better insight into the present conditions in the schools and how they compare to the goals set.
*Make it visible how resources are prioritized.
*Continuous evaluation how we are achieving compared to the goals.

The consequences of a successful project are also described and are thus described:

From a municipal point of view.
*Clear goals for the schools.
*A continous evaluation of the results in the schools compared to the quality goals of the municipality and thereby giving the politicians better insight how to prioritize the resources.

From a pupil’s point of view:
*A school where it is pleasant to be.
*Given appropriate challenges in their education.
*Achieve good academic results.
*Be in an environment which makes them socially apt and where they learn how to solve conflicts.
*Be proud of their school.
*Be “positively recognized” when studying at an upper secondary school.

From a parent’s point of view.
*A feeling that they are working together with the school to achieve the same goals.
*Knowing how the school is achieving compared to the goals.
*Have the same expectations as the school regards the individual pupil.
*Truly can say that they have a dialog with the school how the goals may be reached.

From a teacher’s point of view:
*Have clear goals for what results are expected.
*Have a greater knowledge of how they are achieving in comparison to the goals.
*Greater confidence that all involved will contribute with their respective part of the responsability to achieve the overall goals.
*Greater confidence that the goals and the resources set aside for achieving the goals are in harmony.

The consequence description also includes the municipality administration and the politicians but I will not include them here.

There is a shorter description about IT in the schools in Fet municipality.

*The use of IT must be a natural part of the work during the school day.
*The employees must have good competence using IT in education.
*It must be good supply of computers and software in the schools
*Evaluations must regualarily be done to ensure goal achievement.

How is then these very broad goals transformed into concrete goals in the school?
I refer to my own investigation which took place about a year ago where especially one secondary school stood out by involving all staff and also the pupils in the process of how IT should be used in education. The project group consisted of a teacher and a librarian who extensively used the different teacher groups as active partners when forming the local goals. On a regular basis other staff groups and pupils were also involved in the work. The principal initiated the project and also acted as a supervisor for the project group.

And what happens in the classroom?
The sad fact is that we know surpricingly little about this. A recent report on quality work in schools and municipalities from the National Agency for Education in Sweden stated that the municipalities don’t report changes concerning the learning process and what has been done to stimulate these changes. Resources and organizational changes are instead the means to improve the test results (National Agency for Eduation, V4-Uppdraget, 2000). The Swedish IT commission reported similar findings regards IT in 1998 by stating that “we know very little about the pedagogical use of IT in schools”.

I cannot at this moment for sure tell whether these findings apply to other EU countries but due to tight budgets around in Europe I’m inclined to believe that more attention is given to administrative and organizational issues rather than pedagogical ditto.

If we don’t know or know very little what happens in the class rooms of today, one might ask how we should approach quality issues when using IT and specifically using web based resources in education?
One might also ask how useful discussions regarding different pedagogic theories and their impact on education are, if we don’t know whether they actually have any impact at all on what takes place in the class room?

When we suggest teacher quality work in schools within this project, we should be aware that we most of the time presume that we know what is going on rather than that we actually know what is going on.
The quality work we suggest might fit into one teacher’s work but it might as well not fit.
Maybe a set of guidelines for using IT and more specifically web based resources are more appropriate and useful for the teachers work?

Basically I view teacher’s work consisting of three definite phases, although very simplified, as preparation for an educational acitivty, conducting this activity and evaluating the activity afterwards. These should be seen in a very broad sense and also seen from the point of somebody who’s responsible for a web based repository. This person or persons might have or might not have any knowledge of the educational sector other than as a pupil and later as a student. This could also be the key to how to approach quality issues when presenting them to the teachers from this projects point of view.

For those interested in different learning theories and who would like easy to read introductions I recommend visiting

Günther Dippe
Department of Pedagogics and Didactics
Göteborg University

Title: Re:Dippe: Using web resources to improve the quality of teaching and learning
Name: abeckers

Message: Your contribution is not as short as you intended. I like to think about the angle you propose to tackle the subject.

I am just stuck how we can retrieve the classroom information as you call it without increasing the administrative part. We often hear the admin. is already a burden and that the time used can better be spent to the pupils. Do you think vast contributions of teachers will help to understand what is going on in the classroom?

How can we reach and motivate the people?

Title: Re:Dippe: Using web resources to improve the quality of teaching and learning
Name: Riina

Message: It is truly pity that when we speak about the quality-issues here, very seldom there are any teachers balancing the conversation with their real life experiences.

One way to try to get the hint of what's going on in a classroom is, for example, to look into the projects where schools are directly involved. One such is the eSchola event that was co-ordinated by EUN in May 2001. It is still available at

There were more than 1000 schools participating in this European evet by submitting their own projects and resources to the different categories of the event, among other to eLearning Awards. A short listing of 100 projects is on the website. There is also interesting database called "Leading Edge" of proven examples and leading practices that will help you to learn from others including success stories, lesson plans and multimedia teaching material. Looking into this material gives a good clue how teachers actually deal with ICT in their everyday practices.

In my opinion there is a possibility to give a "framework-tool" to teachers. For sure it won't work for everyone of them, but be more like a recommendational guideline against which they can check their own criteria and maybe adapt them to better fit in their practices.

Title:  Re:Dippe: Using web resources to improve the quality of teaching and learning
Name:   tommy

Message: This is of relevance to those who speak Danish (Günther), sorry : )

I agree that we do have very limited knowledge of what is actually going on in the class rooms. But as you did mention Danish as one of the languages you master, I can't resist the opportunity to direct you to a recent survey concerning the use of IT in Danish class rooms - with a special focus on whether the national initiative of supplementary training (pedagogical ICT driving license) has any influence to the use of ICT.

See the report here:
kr, Tommy

Thread: ETB Documentation System as a Part of Quality
Asked for opinions of the Discussion Paper

          Margit: Re:ETB Documentation System as a Part of Quality
I have been reading your concept for Recommendation on Quality Assurance and Selection of resources for educational resources.

One part was especially interesting:
the idea to combine Trust Source, Attractiveness, Usefulness and Satisfaction for Quality Insurance is very useful, but I was wondering wether some factors might not exclude others. a.e.: I made the experience that very attractive and therefore motivating resources are mostly not really useful in the sense that one can not adapt it in classroom.

Are there any plans to implement a scoring system for the evaluators, based on the 4 factors?

Another remark would be concerning the Information Quality: some information is not to be judged as correct or not correct, rather reflect already an opinion. How would an evaluator be able to judge objectively?

kr Margit:

          Vasiliki : Re:Re:ETB Documentation System as a Part of Quality
Hello Margit, first of all thanks for bringing up the issue of "attractiveness". I will try to explain to you our idea of attractiveness or at least to explain to you how I perceive this notion.

I think that the idea is to evaluate a resource taking into consideration all the factors and sub-factors mentioned in the discussion paper.

Attractiveness, as pointed out in the aforementioned paper, is one of the factors. It might not be the most important but it should not be neglected. Attractiveness refers not only to an appealing interface (especially important for pupils, since it is the first thing they actually "come in contact") but also to the way that the material is presented in innovative and enjoyable way.

An attractive site is not necessarily a useless site. It depends on the other factors. In any case we don't evaluate only attractiveness. And I have seen several examples of resources that combine all elements.

I hope I gave you an idea...


          abeckers : Re:Re:Re:ETB Documentation System as a Part of Quality
would you mind sharing one or two resources that are for you an example?

Title:  Some examples
Name:   Vasiliki

Message: I could easily suggest:

Digital Dante (Institute for Learning Technologies, Teachers' College, Columbia University) at

Natural Park at

And several of the adventures presented in Educational Web Adventures site at

Title:  Re:Some examples
Name:   tommy

Message: Good examples, Vasiliki - I even found one to submit to our Danish repository, thanx ;-)

Concerning the question of whether information is objective and from a trustworthy source, we've had a few discussions at our local repository. Some (or should I say many Danish) teachers find that information that is _not_ objective usually are very good for e.g. animated discussions in class rooms. This would mean that in our Danish repository you could (in principle) find material from e.g political parties, different types of interest organisations and NGOs that are not exactly objective. But as educational resources for e.g. critical analysis of the information provided, they might be excellent. On the other hand, our description and metadata will be objective....and state that this resource gives one angle to the topic in question - others might be needed to get the full picture.

So what do you say, could a fascist web site prove to be a valuable educational resource or not (OK, this might be pushing the limit too far? :-)

kr, Tommy

Title:  Objective and trustworthy resources: are they valuable?
Name:   Vasiliki

Message: I totally agree with you,Tommy!
We (ETB wp for quality) actually had a discussion exactly one year ago, and we were arguing if we can use the terms "bad" and "good" resources. What we agreed upon is that it is very simplistic to characterize resources in such a way. Because, as you pointed out, a resource marked as "bad" (due for instance of lack of objectivity) may be excellent "tool" for a teacher for a specific purpose.
A single absolute truth cannot be found and moreover what is truth to one person maybe nonsense to another one.
And, I do think that the quality of a resource cannot be seen separately from its use in specific educational contexts.
And yes, within a specific context and for one teacher who knows how to deal with such a resource, even a fascist web site could be a valuable educational resource.

Riina: Re:Re:ETB Documentation System as a Part of Quality

About your comment on the scoring system within ETB:
From the beginning on it has been agreed that ETB doesn't want to make quality assurance a rating system that ranks the best quality material by giving points or start based on our recommendation. What is aimed within the quality recommendations, is the transparency of the system by indicating the factors that are favorable, and encouraging people to use those recommendations also in a wider context of their work.

Secondly, ETB tries to provide enough information for the user, so that s/he can make the decision by judging the meta-informaiton available. This is what we aimed for by the use of the metadata on the resource level and on the whole collection. Specially the "Quality Selection Policy" should be informative for users, they can find information about the local Quality Policy and its review processes that a piece of resource has gone through before being submitted to the ETB-network.

Author: Riina Vuorikari
Web Editor: Riina Vuorikari
Published: Tuesday, 9 Oct 2001
Last changed: Wednesday, 10 Oct 2001
Keywords: etb, quality-issues, workshop, archives