CHAPTER IV
Introducing my research project
On the basis of literature review in the previous chapter, I found there are other mediating factors in relation to teachers’ instructional decisions (e.g. washback models by Alderson & Wall, Alderson & Hamp-Lyons, Bailey, Wall, Watanabe, and others). Their inconclusive results indicated washback effects have exerted differently to teachers and schools. Cheng (2004), therefore, asserts, “in order to further our understanding, we need to look at the phenomenon in a specific education setting by investigating in depth different aspects of teaching and learning.” As a result, my emphasis is on learning the most about both the individual case of EFL teacher and the phenomenon, especially the latter if the special circumstances may yield unusual insight into EFL testing programs.
The purpose of this study was to look at other variables (e.g., beliefs and contexts) together with washback which might influence EFL teacher decision-making. To profile thoroughly about EFL teacher behaviours mediated by factors such as the rooted beliefs and the circumstances of examination-based education, the present study focused on the language testing programs EFL teachers taught, e.g. ECL and TOEFL. I was distinguishing 2 types of EFL test which ECL is a traditionally linguistic-based test and TOEFL iBT is an innovatively communicative-based test. Under the contexts with potentially different types of washback, I compared and contrasted the differences between traditional oriented and communicative oriented EFL teachers. In their work for the departments of foreign language teaching, participating EFL teachers from air force colleges of
Meanwhile, the washback researchers (e.g. Alderson and Wall, 1994) strongly suggested a methodological basis for empirical evidence from direct classroom observations and other methods of data collection. Thus, this chapter has dealt with the methodological issues to find out teachers’ beliefs and their practices listed in research questions, and to further outline research methods to collect and generate empirical components. The chapter will start the aims of the research from my intents and remind the readers my research questions before describing the design of the study, the justifications of the chosen research methods and instruments in the light of the discussion of the previous chapter. In particular, I refined the observation schedule with many coding terms for the study, describing the ‘data generation’ and ‘data analysis’ procedures used in the study, and operationally summarized ethical issues.
Research aims
I hope that the present could understand the different types of washback from TOEFL and ECL while taking other mediating factors such as context, beliefs, teacher background into account. The present study aims to:
l To understand the extent to which washback effects exert on EFL teachers at tertiary level (military colleges) in Taiwan, and how they respond to the standardized language tests, such as TOEFL iBT and ECL.
l To understand the extent to which teachers’ attitude and beliefs have influenced teaching approaches, methods, and contents.
l To understand the extent to which contextual factors have influenced on teachers’ pedagogical decisions.
l To refine the washback model by understanding different cases to make test writers feel easier to lever the change in education.
With these aims I needed to stay contextually embedded in selected schools, but not attempted to generalize washback effects for all schools but my claims shall have truly reflected what happened in classrooms. Discovered beliefs and practice from an in-depth and holistic study will support the findings to be transferable to understand the processes of teacher instructional decisions. Thus, it was necessary to investigate how Taiwanese college teachers from foreign language teaching centers to respond to innovation. With good spirit I conducted teacher interviews and classroom observations for empirical data. My commitments included to bring expert knowledge to bear upon the phenomena studied, to round up all the relevant data, to examine rival interpretations and to ponder and probe the degree to which the findings have implications elsewhere. Thus, I inevitably offered local and practical reasons on the basis of culture and schooling system for my actions for choice of research focus, methodology, data analysis and interpretation. In this case I was able to debate the nature of washback, the principles of teachers’ teaching practice, and social and cultural context where teachers struggled with.
Research questions
Arriving with an idea of what to look at that I bring into the field, I am able to mould my theories according to facts and to see facts in their bearing upon theory. According to Stake (1998), the foreshadowed problems are the main endowment of a scientific thinker, and these problems are first revealed to the observer by his theoretical studies. Thus, my research addressed the following questions:
1. What attitudes to and beliefs about English language teaching and learning do experienced EFL teachers have at the air force military colleges in Taiwan?
2. What will washback effects of testing affect classroom practices at the air force military colleges in
3. What other contextual conditions affect classroom practices?
4. What is the relative weight of teacher beliefs, washback and other contextual factors in influencing classroom practices?
The design of the study
In my mind I had no preference to any teaching approach. My role in the research was as a participant observer that I explained later in section of data sources in this chapter. In the process of the research, I identified myself as a participant to explore the issues of curriculum change with a given group of individuals and their practices through an intensive involvement with these teachers in their natural environment. Admittedly, friendship with participating teachers had been built during the life of the study, but I honestly recorded and wrote what I had heard and observed, but not to give any judgment. This was to cultivate the climate of mutual trust between the participants and to make sure the readiness of participants to disclose their concerns (Weir & Roberts, 1994). Explicit and agreed criteria and processes were employed to have the impartiality of the evaluator. Meanwhile, I myself as an ethnographer who immersed in the field over an extended period of time was problem-oriented to examine how certain things get done and motivated by a commitment to the data complement to understand the interpretive context-bound teacher belief-practice interaction was most likely to achieve success.
My design of such research through participant observations involves a range of methods: 3 sets of semi-structured interviews, classroom observations, and field notes a data source. Using multiple ways for me is to address the situated work of accounts rather than as in Denzin’s to do ironies. Silverman (1993, p. 158; 2005, p. 121) argues triangulation is problematic as a test of validity, but the value is to reveal the existence of public and private accounts of an agency’s work. The study was undertaken over a long period of time from spring 2009 to winter 2010. Two years of observations supported me to obtain more detailed and accurate information about EFL teachers I was studying. Observable details (including daily time allotment, classroom practices, etc.) and more hidden details (including behaviours responding to the reform) are more easily observed and understandable over time. I could discover discrepancies between what these EFL teachers believe and what actually does happen, when they cope with 2 types of EFL test named TOEFL iBT and ECL with potentially different types of washback. Discordant voices have never been neglected and I had a great determination to use different methods for data collection. Research methods were explained in more details later in terms of justification. I discussed case study, sampling, open-ended interview, COLT classroom observation, data analysis and interpretation in my discussion of chosen qualitative inquiry research methodology to fit my research questions. Most importantly, the question of trustworthiness in doing research in a multi-case study was given special concern in the whole process of research.
Rationale of using a qualitative multi-case study
The rationale for the use of qualitative paradigm therefore was because this study tried to find out “how the participants interpret the phenomena, e.g. the influence of testing on teaching and learning occurring in a context and how it influences their actions.” In an educational context where several testing programs were used to evaluate students’ language proficiency, EFL teachers as the practitioners in teaching might yield different viewpoints and attitudes which consequentially influence their practices.
It was believed by Patton (1990) that a multi-case study was more appropriate to understand particular cases – unusual successes, unusual failures, or dropouts in educational programs. Thus, case studies in this research were used to understand some critical cases through collecting each teacher’s in-depth perceptions in relation to correspondent classroom behaviours due to testing programs implemented. Stake (2000) and Stark & Torrance (2005) point out that the strength of case study is to take an example of a subject or an activity and use multiple methods and data sources to explore it and interrogate it. Thus, it can achieve a rich description of a phenomenon in order to represent it from the participants’ perspectives. Merriam (1998) defines that the interest of a case study is “in process rather than outcomes” as a major characteristics. Unlike traditional evaluation methodologies which focus on outcomes, the case-study approach describes the participant, the situation, and the resulting interaction. Its primary focus is the participant experiencing the situation. Cohen, Manion & Morrison (2007, p.253) remark that case studies opt for analytic rather than statistical generalization, that they develop a theory which can help researchers to understand other similar cases, phenomena or situations.
Thus, naturalistic multi-case studies were employed to explore EFL teachers’ beliefs towards English language teaching and learning, their beliefs about testing programs, and their actual teaching practices within two military colleges in
Overview of the research
The procedures of the research consisted of 3 distinct phases as follows:
1. The first phase was to explore teacher attitudes and beliefs in general towards language teaching and learning and administrated language tests, and military school and language centre culture.
2. The second phase was to observe all classes taught by participating teachers and explore the issues that arose in classroom.
3. The third phase was follow-up site visits to clarify teachers’ interpretations on the issues and confirm the distinct process of the teaching.
For this study I decided to adopt multiple methods comprising basic quantitative and qualitative analyses of classroom observations, qualitative analysis of semi-structured interviews and field notes, and other data from the case study schools. The basic quantitative analysis in classroom observations was to weight components of teaching tasks and activities. It was complementary to count the interactions in classroom to give me to understand what EFL teachers put into the practice. The qualitative analysis would provide the detailed, deep, and holistic views of discourses between the teacher and students to give me an account of structure and meaning from within that perspective. The mixed approach could be best viewed to make better sense of the other in the context-boundedness of accounts, enabling me to collect useful rich data.
The following methods were used to generate data.
1. The Semi-structured Interview of Teacher Backgrounds and Teacher Beliefs about Teaching and Learning. These methods enabled me to discover teachers’ prior educational experience, and meanings that teachers give to language teaching and learning. In order to interpret the real world beliefs, the semi-structured interviews were adopted because "certain information is desired from all the respondents" (Merriam, 1988, p. 74). This format also allowed the researcher to respond to the situation at hand, to the emerging view of the respondent, and to new ideas on the topic (Merriam, 1988, p. 74). Furthermore, a semi-structured interview allowed “interviewees’ maximum freedom, ample and perhaps unexpected information might emerge” (Ferman, 2004, p. 195). In other words, with the themes I set interview questions pertaining to how the testing program might be affecting teaching, but to allow the respondents having freedom of expression, and to avoid leading the respondents with focused questions. I engaged in respondents’ dialogue, instead of a question and answer session. Therefore, I employed semi-structured interviews to gain perspectives and insights that would be of assistance to me in exploring teachers' feelings, attitudes, perceptions and beliefs.
2. The Semi-structured Interview of Teachers' Beliefs of Washback of Language Testing Programs. To enrich research data, second form of semi-structured interview was designed to collect teachers' statements about testing programs, classroom practice, and teaching context. The inquiries focused on beliefs how the traditional ECL testing program and the communicative TOEFL iBT testing program has affected respectively on English teachers’ beliefs of teaching and assessment. The interview questions were adapted from the questionnaires originally written by Abrams, Pedulla and Madaus (2003) as well as Choi’s (2008) study concerning the impact of EFL testing on education. In interview inquires, I collected data pertaining to teachers’ perspectives on how testing programs have influenced their teaching. The major themes were: a. types of skills are mainly taught; b. types of materials are mainly used; and c. kinds of activities are mainly used.
3. COLT – Classroom Observations. This method allowed me to build a richer picture of the processes of teachers’ decisions in pedagogy. It was to give a holistic interpretation working together with the interview data to obtain EFL teachers’ behaviours in English teaching for developing students’ communicative competence, or for developing lexical and grammatical ability, or other approaches for developing skills in EFL Classrooms. In addition, I closely examined the interactions between students and teachers. In order to capture potential pedagogical differences that subjects receive in their EFL classes, an observation scheme focusd on the communicative orientation of L2 teaching. Thus, it was adequate to use the Communicative Orientation of Language Teaching (COLT) (FrÖhlich, Spada, & Allen, 1985) as an instrument for probing pedagogical activities. The COLT observation scheme has been used primarily in classroom research to investigate relationships among instructional input, interaction, and second language acquisition. The COLT observation scheme consists of two parts: Part A, which describes classroom events at the level of activity, include the categories of activity type, participant organization, content, student modality, and materials; and Part B analyses the communicative features of verbal interaction between teachers and students within each activity. Furthermore, I saw this as opportunity to obtain more knowledge of each context, enhancing my understanding of culture and management of each case study. Most importantly, this process helped me to build relationships or even trustful friendships with teachers in each department, facilitating open discussions later in follow-up interviews.
4. Follow-up Interview. To find out why the teacher did the activities or made the choices recorded during the observations, a follow-up interview was held with the teacher. It was also an important way to check the accuracy of – further to validate or to disprove - the interpretations the researcher has gained through observation (Fraenkel & Wallen, 2003, p. 455) by checking contradictory statements and to explore the relevance between teachers’ beliefs and practices. It was important in conjunction with baseline interview data gathered to triangulate with classroom observations about practice done by EFL teachers and some follow-up accounts from EFL teachers of why they perceived and reacted differently. The follow-ups were to clarify any doubt or inconsistency I had found in classroom practices and from their previously addressed beliefs.
5. Analysis of Relevant Documentations. I jotted down some notes about classroom facilities, learning resources, bulletin boards, authority supervision, department meetings, school policies, student learning activities, and interactions in the classroom, etc. Documents such as teacher development and training were analyzed from school to school. They could truly reflect contextual factors around teachers and were encountered daily.
Hence, interviews and observations conducted in present study would find out how teacher beliefs systems and the washback effects interact to affect classroom practices. As required by research questions, the research processes were firstly to understand teachers’ experiences in teaching and learning under the context influenced by high-stakes tests, and secondly were to explore teachers’ attitudes and beliefs towards teaching and learning, and thirdly were to discover teachers’ practices. The in-depth interviews and observations would help me to compare and contrast the differences between traditional oriented EFL teachers and communicative oriented EFL teachers. It is interesting to see if teachers hereby to the standards of TOEFL/ECL, why and why not. Finally, the differences between what teachers believe and how they react under the washback effects could be interpreted on the basis of the sufficient data and qualitative analysis. It stressed the important factors of individual background, experience, motivation and attitude influencing teachers’ beliefs in dealing with teaching when mediated by contextual pressure of testing innovations. Therefore, the identified differences would greatly improve my knowledge and understanding of how and in what areas a change of testing program could actually influence English teachers’ practices and beliefs at the tertiary level.
The data were collected to investigate teachers’ beliefs on English teaching and learning, and their interactions with washback effects, and to explore how EFL teachers have responded to communicative principles of teaching and testing in the classrooms.
The research sites were the
Justifications of settings selected included:
• pragmatic
• permitted
• typical
• informative
Stake (1998, p. 105) argues “By comprehensively describing the program case, the researcher should help the reader draw naturalistic generalizations.” The design of the study was heavily influenced by the serious problems of communication and information gathering in
Understanding the critical phenomena depends on choosing the case well (Patton, 1990). Merriam (1988) also asserts the sampling is used in purpose to discover, understand, and capture the phenomena completely as the foundations of a new theory. A case study is to understand ‘the case’ through long-term immersion in the field rather than to generalize to a population at large. My goal was to select cases in order for gaining a deeper understanding of the processes of decisions, and I believed these cases allowed me to access different perspectives and beliefs. It allowed me to understand how these cases to construct their point of view to represent their own reality that they have experienced. Thus, their ‘thick descriptions’ facilitated the development of concepts relevant to the relationship between teacher beliefs and pedagogical actions for my research. This was consistent with the views of other researchers who developed a theory throughout the research process. For instance, social phenomena are always complex and require sensitive and dense theory to account for as much variation in the data as possible (Glaser & Strauss, 1987). David Silverman (2000, 2001) also argues that qualitative research follows a theoretical, rather than a statistical. Transferability is not the major concern in qualitative inquires. The selected teachers were no means to represent all types of EFL teachers in
The main selection criteria I required were:
1. The teachers of English taught test preparation courses, as the key characteristics.
2. It was essential that teachers have various teaching approaches based on their educational backgrounds and teacher training.
3. Teachers had widely disparate language levels. Emphatically, it was to highlight the ability to teach using communicative teaching principles that require more confidence and higher ability of the teacher in the target language.
4. Teachers’ attitudes to curriculum reform were either positive or negative.
5. Teachers who could be conveniently accessed by the researcher were willing to participate in the study.
6. There should be equivalent stability in the number of participating teachers to provide rich information in both Academy and Institute.
The first criterion involved in selecting teachers to participate in the study was based on the language testing programs they taught, e.g. ECL and TOEFL. Offered various types of courses, teachers' beliefs could be found to see teachers decide how they are going to carry out their teaching according to the syllabus. Moreover, the selected teachers were compared with equivalent teaching jobs, because my main concern in this study was to see if there was any difference in terms of teaching methods to test preparation courses. They were easily accessed in the field by the researcher and volunteered to participate in the study. Consequently, the participants in this study were two English teachers from the Air Force Institute of Technology and two English teachers from the Air Force Academy in the academic year of 2009-2010.
In their work for the Department of Foreign Language Teaching, they coped with the issues brought by the design and content change of current standardized language proficiency assessments. I identified the four participating teachers as Tina, Mary, Andy and Lee. Among four teachers, two (Tina and Andy) teachers are responsible for ECL test and two (Mary and Lee) teachers are responsible for TOEFL iBT test. All teachers participated both in the interview and observation procedures. A brief autobiography for participating EFL teachers is as follow:
• Tina: She was a Taiwanese at the age of 60s and has taught English classes for more than 20 years. Tina had 4 sophomore lower level classes and each class contained 25 students at the Academy. Owing to the lack of ability in using target language, her interest was of ECL test preparation courses and her specialty was the teaching of linguistic knowledge. Tina has a master degree in English literature. With a negative attitude to communicative approach, her teaching was more traditional linguistic oriented in classes.
• Mary: She was born and educated in
• Andy: At the age of 30s, Andy had the fewest teaching years among all participating EFL teachers in this project. He worked for the Air Force Institute of Technology as an English instructor for five years. Andy had 28 lower students in a fundamental English class for ECL test preparation courses. He has a master degree in Translation and he was enthusiastic about the approach of communicative grammar teaching through interacting with his students. Apparently, Andy liked communicative teaching because of his better ability in using the target language, but he chose to teach grammar because of his lack of teaching experience. Andy’s practices were changed often between communicative oriented and linguistic oriented. Somehow, he has a positive attitude to curriculum reform.
• Lee: Lee was male at 40s and was an experienced EFL teacher whose teaching years were more than ten years. At the Air Force Institute of Technology, Lee taught 30 intermediate students in an advanced English class for TOEFL test preparation courses. Lee has a master degree in ELT and his specialty was to integrate language skills. He managed to operate almost exclusively in the target language. Indeed, his attitude to curriculum reform was positive and his class was communicative oriented.
Table 4- 1 Setting and participants
| | Academy | Institute |
| |||||
| | Mary | Tina | Andy | Lee |
| |||
| | ECL Test Preparation | ECL/TOEFL Test Preparation | ECL Test Preparation | ECL/TOEFL Test Preparation |
| |||
| Age | 60s | 50s | 30s | 40s | ||||
| Gender | Female | Female | Male | Male | ||||
| Degree | MA | MA | MA | MA | ||||
| Years of Teaching | 20 ↑ | 20 ↑ | 5 ↑ | 10 ↑ | ||||
| Teaching-oriented | Traditional | Traditional | Communicative | Communicative | ||||
| Student Level | Lower | Intermediate | Lower | Intermediate | ||||
| Class size | 25 | 25 | 28 | 30 | ||||
| | | | | |
| |||
Data sources
To address a broader range of historical, attitudinal, and behavioural issues Yin (1994) suggests the approach to collect the evidence for case studies through the use of multiple sources of evidence. The most important advantage presented by using multiple sources of evidence is the development of converging lines of inquiry. As justified, the present study employed a range of data generation methods. Figure 4-1 shows how these methods relate to the research questions.
Figure 4-1. The design of the study
| Research question | Methods | Type of data |
| What attitudes to and beliefs about English language teaching and learning do experienced EFL teachers have at the air force military colleges in Taiwan? | l The Semi-structured Interview of Teacher Backgrounds and Teacher Beliefs about Teaching and Learning. | l EFL teachers |
| What will washback effects of testing affect classroom practices at the air force military colleges in | l The Semi-structured Interview of Teachers' Beliefs of Washback of Language Testing Programs. l COLT-Classroom Observations. l Follow-up Interviews. l Analysis of Documentations. | l EFL teachers l COLT coding sheet l Field notes |
| What other contextual (ecological) conditions affect classroom practices? | l The Semi-structured Interview of Teacher Backgrounds and Teacher Beliefs about Teaching and Learning. l Follow-up Interviews. l Analysis of Documentations | l EFL teachers l School and department documents l Field notes |
| What is the relative weight of teacher beliefs, washback and other contextual factors in influencing classroom practices? | l The Semi-structured Interview of Teacher Backgrounds and Teacher Beliefs about Teaching and Learning. l COLT-Classroom Observations. l Follow-up Interviews. l Analysis of Documentations | l EFL teachers l COLT coding sheet l School and department documents l Field notes |
The first stage in the research consisted of two interviews including Teachers' Background and Beliefs. Four EFL teachers were interviewed in 2009. Baseline interviews were conducted at the beginning of the data collection. Each teacher was given the interview questions days before the formal interview to see if they agree with this interview and if they have any questions about this interview. They were interviewed in their own office and the schedule will be on teachers’ convenience. The time spent on the baseline interview about 30 minutes to an hour to collect their educational background, teaching experience, and beliefs about teaching and learning.
Few weeks after conducting the baseline interviews, teachers’ beliefs about washback interviews were conducted on the same manners of the baseline interviews. Interview questions gave to the teachers before hand so they knew the questions being asked and to see if they agree to being interviewed. To have a quiet place, the interviews were conducted in teachers’ own office. Each interview took half hour or an hour to appropriately collect their perceptions about testing.
The baseline interview (see Appendix) conducted in April 2009 was adapted questionnaires from Johnson (1992) and Horwitz (1985) to include teacher background, teachers’ beliefs about teaching, and description of teaching context. The first part was about the subject’s individual English learning background, qualifications and experience in teaching English. The interview questions were concerning to years of teaching, degree and major, ways of being taught, courses, and motivation. The second category was to understand English teachers’ beliefs towards teaching and learning. Questions were asked about how they perceive EFL teaching and their responses were analyzed to find whether or not there are any differences in beliefs of EFL teaching between traditional teachers and those teachers on communicative lines.
Figure 4-2.
The themes in the second category included English teaching methods, designing an English course, the role of an EFL teacher, language skills taught, and teaching process. The third part asked the teachers about contextual factors such as social culture, political power and policy, school administration and management, resources and teacher professional development.
Likewise, second interview (see Appendix) to EFL teachers were conducted in May or June almost a month later from the baseline interview. 
Figure 4-3.
The purpose of second interview was to collect the data relevant to the influence of a test on teaching and learning. A list of pre-specified questions was used to lead the conversations with EFL teachers, and went into further discussions. The themes in second interviews were focused on language skills, materials, and activities in relation to language exams.
Phase two – Classroom observations
Phase two took place from the spring 2009 to the autumn 2010. Implementing school-based research, each classroom was observed for one hour one lesson a month each English teacher, making a total of 24 lesson observations (4teachers×3months×2semesters). For both school cases, I had an opportunity to stay on each campus at least for six months to know the whole school systems, joining in with academic activities and conversations. Teachers became familiar with the nature of my research. During the period of observation, friendship and trust were built to assist the implementation of research methods. In addition to general English courses in selected colleges, the teaching of EFL teachers in both semesters under ECL Test Preparation Courses and TOEFL Test Preparation Courses were observed. Three different sets of ECL test preparation courses were observed for traditional teachers and three units of TOEFL test preparation courses were observed for communicative teachers. It focused on the difference of teaching methods, activities and contents between TOEFL and ECL courses. All classroom observations were videotaped for the data analysis and future follow-up interviews. Portions of the transcribed data were translated into English to enable the use of direct quotations from the transcriptions. The transcriptions allowed me to obtain the pictures of the lessons observed, thus helping to explain in detail the classroom interactions and discourse. Four sub-questions that I observed were:
a. What types of skills are mainly taught?
b. What types of materials are mainly used?
c. What kinds of activities are mainly used?
d. Which communicative features are chiefly presented in those activities?
In order to answer these questions, I used classroom observations, the primary research method employed to address the four sub-questions pertaining to skills, materials, activities, and features employed in the classroom. This gave me a better understanding of how teachers actually teach in the classrooms. Robson (1993) points out that “As the actions and behaviour of people are a central aspect in virtually any enquiry, a natural and obvious technique is to watch what they do, to record this in some way and then to describe, analyse and interpret that we have observed (p. 190). The rationale of using observations includes: (1) allow the researcher to gather ‘live’ data from naturally happenings, (2) enable the researcher to have a reality check, (3) can offer an insight into everyday behaviours, relationships, teaching practices, interactions, physical construction of institutions, and contextual situations (Cohen, Manion & Morrison, 2000).
Furthermore, Alderson & Wall (1993) suggested that it is increasingly obvious that we need to look closely at classroom events in particular in order to see whether what teachers and learners say they do is reflected in their behaviour. Ellis (1994) stated classroom observation views the classroom as a place where interactions of various kinds take place, affording learners opportunities to acquire the second language. Allwright (1984) saw interaction as the fundamental fact of classroom pedagogy because everything that happens in the classroom happens through a process of live person-to-person interaction. Interactions in a language classroom include those between the teacher and the students, between textbooks and students, and between activities and students. Therefore, the type of my observations was mainly based on interactionism (Silverman, 1993) which is concerned with the creation and change of symbolic orders via social interaction. Denzin (1970) uses the term “participant observation” to describe the practice to interactionists. Such a method involves taking the viewpoint of those studied, understanding the situated character of interaction and viewing social processes over time, and can encourage attempts to develop formal theories grounded in first-hand data. Participant observation was suitable because of my ability to gain access to events or groups that are otherwise inaccessible to field investigation, my ability to perceive reality from the viewpoint of someone “inside” the case study rather than external to it, and my ability to manipulate interviews and observations (Yin, 1994).
My approach was a “methodology for listening” to centrally concern with “seeing the world from perspective of my subjects.” In this respect, I accepted assumptions about the “authenticity” of “experience” of my subjects as other interactionists do. I used methodological principles of participant observation to explore behaviours of EFL teachers in the presence of testing programs, to learn everyday conceptions of reality in their own environments, to gather data in naturally occurring situations, and to examine how pedagogical decisions vary during the period of time of research.
Since the potential contradictions between teacher beliefs and new teaching ideas (Lamb, 1995), the findings of the classroom observations were used as the recalls to stimulate more perceptions in teaching and learning. Silverman (1993) also concerns about how informants may distort social reality when interviewees are not fully moral or not intellectually up to scratch. Through the examination of video-taped lessons in classrooms, I was able to see where EFL teachers confronted their own routine practice and the value it is intended to serve.
Table 4-2 The detailed contents of practice observed
| Name | First Observation | Second Observation | Third Observation | Fourth Observation | Fifth Observation | Sixth Observation |
| Date | Date | Date | Date | Date | Date | |
| Content | Content | Content | Content | Content | Content | |
| Tina | March 2009 | April 2009 | Sept. 2009 | Oct. 2009 | Nov.2009 | Jan.2010 |
| ALCPT ECL | ALCPT ECL | | ALCPT ECL | | | |
| Mary | April 2009 | May 2009 | Sept. 2009 | Oct. 2009 | Nov.2009 | Dec. 2009 |
| TOEFL | TOEFL | TOEFL | | | | |
| Andy | March 2009 | April 2009 | May 2009 | Sep. 2009 | Oct. 2009 | Nov. 2009 |
| ALCPT | | ALCPT | ALCPT | ALCPT | | |
| Lee | Sept. 2009 | Oct. 2009 | Nov. 2009 | Jan. 2010 | Feb. 2010 | March 2010 |
| TOEFL | TOEFL | | TOEFL | TOEFL | |
The present study then was identical with those of the research studies that concentrated on teachers with the survey. I went to the classrooms to know how English teaching actually took place. The use of observation was to find out teachers’ instructional behaviours which were compared with teachers’ beliefs collected in previous interviews. It was to understand whether EFL teachers accommodate new ideas in TOEFL iBT test.
As noted in the Second Language Classrooms: Research on teaching and learning, Chaudron (1988, p. 180) claimed that units of observation were often unreported and led to the distorted results. Therefore, to have an appropriate measure, an episode, the basic unit of analysis in classroom observation, was operationally defined to count the frequency with which different kinds of behaviours were observed in the classroom. The ultimate objective was to explore correlations between different types of teacher and student behaviour. Constituent episodes occurred within each activity (e.g., drill, translation, discussion, game, and so on). Teacher introduced dialogue, teacher reads dialogue aloud, and students repeated dialogue parts after teacher (three episodes of one activity). This was achieved by a coding scheme based on the speech act in the COLT as the instrument aforementioned.
2. Taking Field-notes To take good notes, I prepared the questions I wanted to explore and went to class faithfully. I reviewed the notes from the previous class and I looked at parts I have marked. In the class, written records were done systematically. Ways to record notes as follows:
a. Keep a separate notebook for each teacher so I could review them easily.
b. Date each day’s notes.
c. Record whatever each teacher puts on the board. Record exams and quizzes that were announced as well as assignment that EFL teachers gave.
d. Use signals and listings. For instance, “
After the lecture and leaving the field, I read over the notes ad edited them. I clarified with EFL teachers anything I did not understand. In turn, I analyzed the documentary and added labels where needed.
Phase three Follow-up Interviews
As same as the procedures that employed for the teacher interviews, follow-up teacher interviews were conducted at teachers’ convenience within a few months of the first observation. The purpose of follow-up interviews were that questions about the differences between their beliefs told in earlier interviews and practices showed in later observations were asked so that each teacher had an opportunity to clarify their beliefs or behaviours. It was a very important process, because I could understand the reasons why they said were different from what they did in the classroom. The present study explored the differences not only between traditional English teachers and communicative English teachers, but also the differences between their beliefs and their instructional behaviours. What teachers said perhaps could be quite different with what they did. Therefore, it was important and necessary that the present study to validate teachers’ beliefs through actual classroom behaviours found in classroom observations.
Teachers were invited back for follow-up interviews which included more in-depth questions (see Appendix) to talk about the dominancy of traditional testing over communicative testing, teachers’ beliefs changed, teachers’ practices changed, tensions to align assessment and instruction, and EFL educational contexts. Follow-up interviews were given more attention to the relationship between teacher beliefs and learning experience, the relationship between the context and the decision-making, and looking for consistency across all interviews and observations. At this stage, it was to enrich the data needed and all follow-up interviews were recorded and transcribed.
Data analysis
In this qualitative case work, all my data analysis followed the interpretive analytic method addressed by Stake (1998) to emphasize the production of meaning and to learn the local meaning. I accessed to reality through shared meanings that people assign to them for the understanding of phenomena. Interpretive analysis that focuses on the full complexity of human sense making was employed to know the context of innovative reform and the process whereby new testing programs influence and is influenced by the context. With sensibility to data, I typified qualitative casework to code the data and continuously interpreted again and again. Through reflecting and revising meanings, my analysis sought to see what was natural in happenings, in settings, in expressions of value. In the analysis, I modified initial assumptions and theories because themes emerged from the data. The analysis was for intrinsic interests in the cases, rather than the contribution to generalization. Intrinsic case study would help me toward understanding of what is important about each EFL teacher within his or her own world, so that I could develop issues, contexts, and interpretations. I expected the readers to comprehend my interpretations but to arrive at their own. With the thick narrative, readers could vicariously experience these happenings and draw their own conclusions. The general analysis procedures have been implemented through
l Putting information into different arrays
l Making a matrix of categories
l Creating data displays
l Tabulating the frequency of different events
l Examining their relationships
These were important techniques, for instance, to count the various phenomena and to induct thick narratives. A complete case description has been developed for organizing each case study on later chapters (from chapter 5 to chapter 8) of the dissertation. These chapters cover a range of topics relevant to EFL teachers’ learning life, school context, teaching beliefs, attitudes to testing programs, and their actual teaching practices at colleges. A descriptive approach may help to identify a type of event and an overall pattern of complexity that ultimately was used in a causal sense to “interpret” why EFL teachers made their instructional decisions. However, such abundant data was difficult to proceed in the analysis without analytic strategies. The strategy of analysis is introduced in detail in the following sections.
Pattern-coding
Pattern-coding strategy is to identify and to relate to the different behaviours or to the variables. I applied this concept to the tracing of events when a testing program was intended to produce a certain outcome. Starting with the analysis of basic educational backgrounds and the analysis of teachers’ beliefs about teaching and learning allowed me to classify the typology with emerging phenomena. My attention was given to classify so-called pedagogical decisions by traditional and innovative teachers. Traditional cases were seen to use the traditional teaching methods such as translation and audiolingualism. Innovative cases were seen to integrate all language skills with interactive activities. New codes were applied to the data and revised codes after case studies were re-examined to their original transcripts to see if any additional light on the data. I ought to stress that this coding form was narrowed down what we were looking for. I became interested in how EFL teachers made their decisions. In analysis of interview data, I did not hear interview responses simply as true or false reports on reality. Instead, I treated such responses as displays of perspectives and moral forms. Analysis of interview data was mainly focused on the functions of EFL teachers’ accounts. I would show how interview accounts can be investigated in terms of the factor categorization devices (FCDs) employed by my respondents. It seemed likely that EFL teacher’s way of teaching was systematically related to test factors (like content), student factors (such as proficiency levels, motivation, etc.), micro-factors (school culture, school management, etc.) and macro-factors (Chinese culture, high-stakes, education policies, etc.). Table 4-3 indicates perceived important factors that imply common expectations in association to teaching and learning. Besides, according to Silverman (1993), tabulating many cases can avoid a danger of depending on short extracts to use them to support a preconceived argument rather than to test it. I overcame such a danger by giving a tabulating strategy as follow in Table 4-3.
Table 4-3 Factor categories for EFL teacher decision-making
| Categories | EFL teachers’ beliefs toward teaching |
| Teacher factors | |
| Student factors | |
| Test factors | |
| Micro factors | |
| Macro factors | |
Likewise, observational data were analyzed with initial coding in the COLT sheet. After re-examining videotaped data, new codes were added to allow me to generate theory. The emphasis throughout the data analysis phase was to grouping the factors and to extrapolate themes from the data of qualitative inquires. I normally generated emerging themes from the field visits. These themes were checked with categories I initial coded and categories my subjects used. The typical process of coding for the themes was selecting, focusing, simplifying, and abstracting “raw” data for reduction. I built inductive theories through conceptual development in order for EFL teacher behaviour and reasons behind it. I organized assembly of information that permits conclusions drawing to decide what things meant.
To some extent, however, any research must develop its own coding scheme, if it is to answer its own particular research questions. Thus, I made some adjustments in order to deal with the "washback" or testing impacts on teaching. I adopted four categories from the COLT scheme – into seven categories Using Items Released, Activity, Content, Material, Use of Materials, Student Participation, and Language Used. Table 4-4 implies common expectations about what sorts of activities EFL teachers employed in classroom.
Table 4-4 Analysis of EFL teacher behaviours and events in classroom
| Main categories of practice | ％ | |
| Using Items Released | | |
| Activity | | |
| Content | | |
| Student Participation | | |
| Material | | |
| Use of Materials | | |
| Language Used | | |
In “Using Items Released” category I added more sub-categories, e.g., “Mock Test practices,” “Answering test questions,” “Lecturing test-taking strategies,” “Assessing test-taking strategies,” “Tutoring tests and strategies,” “Assessing test contents,” “Student self-reviewing and practicing,” and “Test score reports and discussions.” related to testing. Additionally, in order to facilitate and supplement observations, I included some field notes for things that I noticed in the classroom but not in the categories of coding scheme. This step-by-step method of information-giving was to ensure to reflect the context of a solid body of original data. Complete coding form (in Appendix 4.1) allowed me to identify the essential elements of the EFL patterns.
Analyzing embedded units
Another analytic strategy was to analyze embedded units. The transcripts were dissected. For instance, by relating item on the language skills to the item teaching activities, I was able to see differences between traditional EFL teachers and others. This coding scheme was divided into the “episode” as the unit of analysis. For instance, a reading comprehension activity might consist of a 'warm-up' exercise (T <-> SS) followed by silent reading of the text (SS) followed by pairwork where the learners are working together on a set of reading comprehension questions (S <-> S), followed by a plenary session checking the answers to the questions (T <-> SS), and finishing with some language focus work (T <-> SS, or S <-> S). Each of these 'episodes' has a distinct and characteristic pattern of interaction between the participants. I therefore followed the COLT schemes that segment an event or activity into units, and each unit will need to be coded. I had video tapes and transcripts which offered endless opportunities to identify topics, ways of talks, themes, events, and categories. My goal in developing this complex cataloguing and retrieval system has been to retain good access to the world of the EFL teachers, without relying upon my personal memory.
Examining relationships and building explanations
After coding themes, I noted regularities, patterns, explanations, and propositions. Therefore, the best tool for analysis was my own mind. When I read the data, I linked themes with theories that previously discussed. This was what Silverman asserted “data-driven” analysis which closes to grounded theory. My theories came out slowly because I continuously asked myself “what” and “how” questions. It also took time to look at EFL teacher behaviours and at processes of change in the context. To test my emerging hypotheses, I divided my data into different sets and compared them against some theoretical propositions. I also provided sufficient “raw” data to allow the reader to separate data and analysis. Again, I revised the statement and compared more cases. Finally, I figured out the model of my washback theory with belief and contextual components. To evaluate my interpretive case study, I followed the fundamental principle of the hermeneutic circle, the principle of contextualization, the principle of interaction between the researcher and the subjects, and the principle of multiple interpretations. Thus, data analysis informed development of concepts, generation of theory, drawing of specific implications and the directions for the future study with new questions I found in the study.
Creditability and dependability
In terms of creditability and dependability (Guba, 1985), my strategies were 1.) To present the findings with multiple source of evidence, 2.) To confirm all the interpretations fitted their understanding of the experience under investigation, 3.) To use an auditable decision trail including memos, all data-reduction products, data interpretations, and all data reconstruction products. Thus, any finding was likely to be much more convincing and accurate if it was based on several different sources of information. A chain of evidence allowed the reader to follow the derivation of any evidence from initial research questions to ultimate case study conclusions. It could be traced the steps in either direction. Most importantly, my participants confirmed interpretations and reports which have made sufficient citation to the relevant portions of the case study database by citing narratives. In addition, my immersion in the field and the role of participant observer enabled a large enhancement in credibility for reflecting the reality of the life experiences of participants more accurately than do more contrived or laboratory settings.
Ethics
Since the present study was working with small case studies, all reasonable efforts and care were made to ensure the privacy, confidentiality, considerations of possible risk exposure and embarrassment, limits of accessibility, the right of withdrawal, agreements, oversight of protective system, rules for protection of human subjects, and code of ethics strict. During and after the research, I paid all my attention to minimize the risks. I especially strived for the protection of participants when the standard data collection techniques of interviewing and participant observation were employed in case studies. While conducting interviews, I gave complete information about the research and asked questions only related to the topic of the interview. Interview questions had been revised to reduce risks and embarrassment about teachers’ opinions. Meanwhile, participant observation was least susceptible to ethical violations since I looked at the public behaviour, but not to spy the privacy. In aspect of confidentiality, pseudonyms have been used in all data to protect teachers. They were always anonymous. All data could only be accessed by the researcher and was locked in a secured office to limit accessibility. All teachers had the right to withdraw from the study at any time. The value of the present study was not likely to outweigh injury to a person exposed. I especially highlighted ethical statements in the consent form to draw all participants’ attention. I confirmed that the research was undertaken in accordance with British Educational Research Association’s Revised Ethical Guidelines for Educational Research (2004) and Scottish Educational Research Association’s Ethical Guidelines (2005). I wrote the purpose of the study, methods of the study, agreements and ethics statement in the consent form. It was important all informants to receive summaries of how they were presented, quoted, or interpreted, and for me as the researcher to listen well for EFL teachers’ concerns. In addition to those rules of protection for human subjects, I also avoided low-priority probing of sensitive issues and invited department heads to oversee the protective system. Such steps enabled my participants safe and volunteered.
Summary of methodology
In summary, when designing and conducting this research I have reinforced the necessary engaged participation of the researcher in the setting being documented, not only for data gathering but for interpretation. My experience urges that the most meaningful evaluation contributes to the program’s goals and commitments have been carried out by the staff to improve the program as well as to the learning theory. To interpret the data I had gathered, I have reaffirmed and clarified my role in the participation in the research. My participation allowed me to examine and extend the interests and ideas I came with as well as observing and interviewing those EFL teachers brought. When I began documenting school’s testing programs, I had priorities based on theoretically semi-structured interviews, renowned observation tool and my teaching experience. I knew Taiwanese EFL teachers had difficulties to find various ways of integrating communication and other activities into their classrooms. In this process my engagement was deepened so the tasks of documenting have described and made various understandings, points of view and EFL practices visible. I accessed to the field and concentrated in interactions. A sense of the whole has been essential for me to observe and think over EFL teachers’ pedagogy. With thickened observation, I had seen all activities and patterns EFL teachers acted and thus became aware of new possibilities for relationships within the school. These steps ensured the collection of all relevant evidence for analysis. Interpretations of documentation were no longer personal opinions but corresponded with the context and with the phenomenon. Rival interpretations were also used for my findings. Seeking the most significant aspect of my case study, I came to a view of interpreting data collected as a process of seeking “reflecting” by reading between lines again and again. Looking openly at everything, I became more knowledgeable about EFL teachers’ instructional decisions mediated by other significant factors and more capable of bringing range and depth to the interpretation. Finally, essential themes and variations were identified to discover how specific lived experiences shape the phenomenon being studied. I have done everything to make sure that my analysis is of the highest quality.
