1) First of all the research question does not convince me, because of the choice of variables which seem too vague or inappropriate.

Participation in online activity cannot only be measured by counting posts ; quality, content, frequency are essential elements too. None of this is mentioned in the literary review . Some of this is added in the conclusions, but does not result from the analysis.  Also what kind of activities were run in the fora? Were fora moderated? were activities directed to stimulate discussion and deep learning?

Also I have a problem about using the term ‘learning ‘without defining it. Is it knowledge of facts or skills? these students may have acquired skills that course designers had never considered and that assessment is not measuring. Crook and Dymott (2005) say that technology doesn’t really change a property of the individual, but the manner they take part in learning activities. As a result of the online activities, students may have learned something that we are not assessing and that may still be relevant for their studies. I would be more interested to find out what the students have learned.

As a result I don’t believe that learning can always be accurately measured by assessment. If for instance assessment consists of multiple choice questions or requires students to regurgitate facts and data then interaction would not help that much. The analysis should have defined the kind of learning measured by assessment and then it should have tried to determine if that kind of learning had taken olace aand as aresult of which kind of activity

1. When we talk about the validity of research, we are often referring to the many conclusions we reach about the quality of different parts of our research methodology.
2. lots of different inferences or conclusions made while conducting research are influenced by the h methodology used, but are we measuring what we intend to measure and to what extent are our observations influenced by the circumstances in which they are made? (“Any research can be affected by different kinds of factors which, while extraneous to the concerns of the research, can invalidate the findings” (Seliger & Shohamy 1989, 95). Controlling all possible factors that threaten the research’s validity is a primary responsibility of every good researcher.)
3. Internal validity:

“Findings can be said to be internally invalid because they may have been affected by factors other than those thought to have caused them, or because the interpretation of the data by the researcher is not clearly supportable” (Seliger & Shohamy 1989, 95).

Internal validity is affected by:

• Subject variability
• Size of subject population
• Time given for the data collection or experimental treatment
• History
• Attrition
• Maturation
• Instrument/task sensitivity

4.   External validityIt is the extent to which you can generalize your findings to a larger group or other contexts. If your research lacks external validity, the findings cannot be applied to contexts other than the one in which you carried out your research. For example, if the subjects are all males from one ethnic group, your findings might not apply to females or other ethnic groups. Or, if you conducted your research in a highly controlled laboratory environment, your findings may not faithfully represent what might happen in the real world.

• What can affect external validity:

• Population characteristics (subjects)
• Interaction of subject selection and research
• Descriptive explicitness of the independent variable
• The effect of the research environment
• Researcher or experimenter effects
• Data collection methodology
• The effect of time

One must consider those issues at the time of the design of the research, i.e. at the earliest stages of the research process, because if you consider them only at the end, it will be too late to gather data on a research question that is of any relevance and quality at all.

From:

http://linguistics.byu.edu/faculty/henrichsenl/ResearchMethods/RM_2_18.html

http://www.socialresearchmethods.net/kb/introval.php

http://www.qmu.ac.uk/psych/Rtrek/study_notes/web/sn5.htm#5

Links that I want to explore

http://www.rogerwimmer.com/wrvalidity.htm

http://www.nova.edu/ssss/QR/QR8-4/golafshani.pdf

http://www.jisc.ac.uk/media/documents/programmes/jtap/jtap-041.pdf

A version od CF that helps describe collaborative environments (wikis,fora):

http://www.bton.ac.uk/cscl/jtap/paper1.htm

A critique of the two and an application to an ESP blended course:

http://www.zeitschrift-schreiben.eu/Beitraege/tench_SupportingWriting.pdf

1. hierarchical structure of activity: actions (conscious) into operations(unconscious)
2. object orientedness
3. internationaization/externalization . Internal processes develop into external actions and vice versa
4. mediation. Tools mediate human activity but they are also created and transformed during activities and are characterized by by features of their development. They influence external actions but also internal
5. development

Research has mainly concentrated on the effectiveness of educational technology. The term improve may imply that learning technologies are seen as an add-on to what is already done.

It also suggests that, given the number of variables that play a role and possibly define the learning, there is no unique recipe to promote learning, a combination of methods is used and we can only hope that each can improve the learning .

It also implies that learning is generated from the outside and not from within the individual (this would require support).

So learning technologies are seen and evaluated as tools used by teachers to cause learning to happen; the learners seem to be rather passive in all this.

When studying classroom talk, would the incorporation of computer based methods offer a good way of combining the strengths of qualitative and quantitative methods and reduce their respective weaknesses?

Setting

Primary schools. Childrens’ talk in collaborative work.

Concepts

Qualitative (context sensitive) versus quantitative (objective, systematic)analysis. Solution: combine context sensitive qualitative analysis with systematic treatment of data.

Use of concordancers to search for key words in context: systematic comparisons of lang use that keeps context into consideration. Possibility of combining different levels of abstraction. Researcher can abstract items from transcript to be systematically analyzed but at the same time they can click on any item and study it in context.

Method

Target group was given 8 lessons on exploratory talk. Children aged 9 and 10 years.

Target group and control group sat a pre and a post intervention reasoning test (to be done collaboratively) which were then compared.

Interaction was videoed and transcribed to allow for detaied analysis of context if necessary/ to evaluate the quality of the change.

First scores where compared then qualitative analysis highlighted key language items associated to one group solution of problem, then Kwic identified occurrence of these key words across all transcripts

Limitations

There is still subjectivity due to the choices made by the researchers : choosing groups, choosing activities.

However by integrating two techniques a wider view was achieved.

Also what did the control group do during the term? Were their characteristics similar to the control group: not much info is given.

Transcripts were used for qualitative analysis: it would have been useful to also to use video to include context to which the children were  referring

Findings

Target group : up 32%

Control group : up 15%

Difference between the two groups was significant.

Quality of talk in transcripts was anlyzed: students had used strategies taught during lessons.

Reflections

Pros:

1. possibility to handle large corpora of data
2. use of explicit criteria for categorizing data
3. coding and sorting can provide data ready for making systematic comparisons about communic. behaviour
4. insights can be used for further comparisons with other data (e.g. comm. behavior with measures of outcomes of collaborative tasks)
5. objective

Cons:

1. coding human behavior is arbitrary and difficult
2. utterances often have multiple and simultaneous functions
3. phenomena can be spread through several utterances; examining single utt. doesn’t help
4. negotiation of meaning throughout the conversation makes single utterances irrelevant or non repersentative
5. does not capture development of shared knowledge over time (atemporal inventories of utterances)
6. not really objective as coding and sorting is based on researcher’s own way of interpreting utterances and selecting criteria.

Coding and sorting may not be valid as it tries to classify utterances thus reducing the complexity that they contain.

The analysis is decontextualized.

Also quantitative analysis identifies links but not exactly which relationships exist and how they these are achieved.

Qualitative: collection, observation, interpretation and commentary of transcribed talk; insights obtained from discussions with teachers and students. Researcher is part of the research and not detached or objective.

Pros:

1. detailed analysis highlights interesting aspects

Cons:

1. case studies give only illusion of proof: no real evidence,
2. impossible to generalize and compare as only short texts are studied
3. can generate theories but not test them

Solution: combine context sensitive qualitative analysis with systematic treatment of data.

One could probably argue that if students used synchronous as well as asynchronous communication they would do better as they would have more opportunities for collaborative work.
However my experience with my online language course suggests that progress is not only related to the number of tools that are available but also to affective factors. Some students simply refuse to use the technology and do the bare minimum using only one of the tools; in some cases they don’t believe in technology and are not motivated to use it, in other cases they are not motivated to learn the language and the technology makes things even more difficult. Also the constructivist approach embedded in online tasks requires a level and a quality of participation that is simply too much for the students. In other words these tools require the students to be able to take charge of their own learning as well as have some idea about how to exploit all the opportunities that are offered and how to make the most of them. The more the tools the more acute the problem becomes.

Since the course I tutor is designed in such a way that most of the learning results through communication with peers and the tutor, those who don’t use the technology or are half hearted miss out a lot. They can only rely on a textbook which is little more than a grammar book. These students are not making much progress and, if they do well in the TMAs, it’s because they are displaying knowledge and skills acquired in other settings.

So I feel that more tools and opportunities would benefit those who are already making progress and further discourage those who are struggling. Autonomy and motivation seem to be the key factors here, so rather than adding more opportunities, design and support could concentrate on the scaffolding necessary to influence these two factors.
I would really like to find out if I’m right. The next step is to formulate a couple of questions and figure out how to conduct the research…

Ideas developed after reading the transcript of Podcast 3

The key issue is: what is the impact of information technology on education? Or better: on the way in which students learn?

I am particularly interested in the impact on the ability of students to take responsibility for their own learning (autonomy) as technology can both enhance an already existing ability and develop a new ability . If they already have an autonomous approach they can take advantage of the tools and opportunities and so there should be a direct relationship between the technology (tools and the opportunities they offer), the way in which learning takes place and the the outcomes. If they are not autonomous technology can guide them to develop a new approach which can help them take advantage of what has been offered and ultimately achieve deeper learning, motivation and good outcomes.

The podcast notes that there are various ways of using the technology. Both schools and HE often tend to adapt the technology to their current practice: for some, computers are for containing resources (placed there by their teachers) and for doing gap-fill exercises (again created by teachers) . If this happens than there is very little shift, no change in educational vision , just a change of tools which could actually prevent some from taking advantage from the resources. When students see this they often complain that ICT is wasting their time and often refuse to use it in future courses even if they offer new opportunities. On the other hand a course could be structured and run in a way that guides students to play a central part, to make decisions about their learning (strategies, resources, relationships with tutors and fellow students…) tools used.

What is my online course doing?

- Is it adapting technology to old practice and thus offering a lower quality product which simply forces the students to learn to use new tools? At its best this approach widens opportunities to access resources, but on the other hand it may just distract the ss from the subject (learn about computers and not how to communicate in Italian) and stress them by giving them resources that they have no time/wish to learn how to use.

or

- is it offering them a chance to be at the centre of the learning process? Is it offering a framework, activities, opportunities for communication, reflection on own learning, chances to develop/learn new language learning strategies, the support, opportunities to receive and give support…In other words is the course only transmitting knowledge or offering chances for knowledge construction and learner training (developing learners in terms of autonomy). Are both the framework and the tutors doing this? Are ss aware that this is done? Do they understand how it works?

- is the course a mixture of old an new? If so is the framework adequate to the level of the of most students. How autonomous do we assume they are? Can they cope with the number of resources and opportunities? Are the resources too may? Are we guiding the ss to make use of them?

So I would like to focus on the educational vision. Here are more questions:

1) What is the course offering and what is it  trying to achieve? In particular is it trying to help ss to become (more) autonomous? In what ways?

2) Are the students aware of this? Are the course aims clear to them?

and what about the tutors?

3) What kinds of outcomes result from the use of technology ? In particular are ss more autonomous ? Looking at this issue would mean looking at the cognitive framework : focusing on the impact of the online element on the way thinking and learning takes place

Too many questions!

For my project some kind of analysis of the course would be a good starting point, but question 3 is what I am really interested in. Looking at the whole framework is probably too much. Looking at blogs and/or  fora (to check if one or more aspects of an autonomous behaviour are achieved)  would be more feasible.

Problems/ issues:

How do I measure/ assess autonomy ? I need to read more to decide what I am going to look at and how.

• Is the virtual classroom a viable option for educational delivery? Are the outcomes at least as good as traditional f2f courses?
• What variables are associated with especially good and especially poor outcomes?

Settings

Post secondary education
Concepts
CMC: discussion fora for activities to be performed by ss: a question a quiz, seminar style presentations
Tutor’s role: ask questions, stimulate ss activity rather than giving all the answers.
Environment: a learning community, trust , ss comfortable with the medium
5 kinds of students : absentees,read-only participants, those who make few comments, those who make scattered contributions, those who contribute regularly, even daily
Level of autonomy : from cognitively immature ss that expect to receive the right answer, those who accept the possibility of many different views/solutions and those who are cognitively mature and look at materials/sources critically. If none is cognitively mature and have writing skills the process would be difficult to implement
Interaction and feedback available daily. Possibility to use pen names. More space and opportunities for less advanced ss .
Active participation: students are forced to think and respond independently;.
Collaborative learning: active engagementwithin a learning community knowledge is not obtained from outside but emerges from the interaction within problem solving activity

Methods

Pre and post course questionnaires
Quantitative data: grades, SAT results, amounts of online activity
Qualitative data: observations, interviews,reports
200 variables, 500 hypothesis, 5 courses 86/87

Hypothesis :

1) No significant difference in scores between VC and TC
2) Ss perceive VC to be superior

Limitations:

detailed methodological procedures and statistical analyses not included
Findings:
• VC gives more convenient access to professor, educational experience
• VC provides increased interest and participation
• VC does more communication and cooperation, but a greater increase is seen in mexed mode courses
• VC is seen by ss to provide better quality but it depends on the course

Implications:
• Ss are to be provided with the necessary equipment
• Teachers must be trained
• Match course objectives with available tools. Use VC for courses that cover a limited number of topics, not for intense survey like courses
• Embed study skills or limit access to ss with high SAT scores

This article as been written in 1989, yet there are still examples of courses run by untrained tutors, aimed at students who are not cognitively mature and who may not have the necessary equipment. Who reads research?

Reflective task

What counts as evidence?

• Observation of VC: students’ patterns of participation , common practice, how CMC is used, quality of work (both content and presentation)
• Analysis of subjective assessment: interest, access, participation, ability to communicate, quality of learning…
• Objective outcomes : results measuring mastery of subject matter
Overall a great number of variables being assessed. Yet VC can produce additional and different outcomes when compared with a traditional environment and these are not mentioned.For instance: how were motivation and autonomy affected? Did ss develop more effective study skills, did they learn how to learn through collaborative work? It ‘s a bit of an unfair comparison: however this research was done in the early days of CMC.
Also an eclectic approach was used and lots of data was gathered. As a result it was difficult to collate all the results to give a complete picture of the situation.

How do the two explicit research questions relate to the design of the research?
The questions were a bit vague, so it is difficult to follow the argument. Q1 was very general. What is a viable option? What are the variables involved? What counts as an outcome? Q2 what is good and what is bad? What are we measuring?
So the questions were too broad and the choice of variables to be studied should have been indicated and justified (referring to the literature) at the beginning. This would have helped to produce a more rigorous discussion at the end.

In what ways is the wider literature used in the paper?

It introduces some background issues but these are linked to the questions only in general terms. Narrower questions could have been selected and the literature review could have discussed issues related to those specific questions highlighting what is already known and what needs to be studied and why. Also it would have been a good idea to link the findings and conclusion to the literature.
Finally I missed some suggestions for further research, especially because this study offers a very broad investigation of the VC.

Views on education.

The main theory underpinning the research is constructivism. Learning is created by the students as they work collaboratively through tasks. According to the authors only cognitively mature learners- self-motivated, autonomous and with good writing skills – seem to succeed although CMC and appropriate support may help. As Ruslan notes in his blog, the idea of area of proximal development and the scaffolding that helps students to operate within it is implied but not developed. The paper seems to accept that unless very skilled tutors are hired to facilitate the learning process the use of a VC would not be beneficial to cognitively immature students.