Validating a Research-Based Monograph for Teaching Post-Secondary EFL Reading Teachers the Meta-Cognitive Aspects of How to Teach Summarizing Strategies for Expository Text: Phase II of a Harvard Business School Type Case Study

This article reports the results of the second phase of a Harvard Business School type case study on the evaluation of a comprehensive research-based English language monograph for teaching Chinese EFL reading teachers the metacognitive aspects of how to teach summarizing strategies for English language expository texts to Chinese undergraduate students. This monograph could be used by native English speaking EFL teachers to teach native English speaking students the same skills, but the focus of this study was on the bilingual and bicultural aspects of such a monograph (text) and its development as a general model of such cross-language and cross-culture instructional materials development problems which are becoming increasingly more prevalent now and are a harbinger of the future of instructional materials. A cross-panel replicated expert reviewer (native Chinese EFL practitioners and academics) design was used to validate the monograph developed using the Carifio-Perla instructional materials development model as a guide. The expert reviewers used a 30-item previously validated structured responding protocol that reflected 7 evaluative criteria and 4 open-ended responding questions to review and rate the monograph chapter by chapter and then again for all 8 chapters. The reviewers unanimously agreed that the general Metacognitive Knowledge Framework, devised as a result of the literature reviews, analyses done, and numerous problems identified in Phase I of this study concerning views, definitions and strategies for analyzing and teaching summarizing strategies metacognitively, was appropriately constructed and effectively communicated and represented in The Monograph for the target audiences. The uniformly positive ratings by the two expert panels validated the high quality and consistency of the monograph in terms of the 7 evaluative criteria used. These results also showed aspects of skills, knowledge, understandings, and metacognitions both transcend and can be represented and communicated successfully across languages and cultures and to different professional audiences as well.


Introduction
When one teaches anyone something such as domain-based knowledge, skills, strategies, procedures, or sensibilities (i.e., values or attitudes), one should have a fairly complete and well-conceptualized schema and understanding of the domain in question, and one must analyzed the contexts and events of instruction and make decisions as to what to do and how to proceed, both before the fact and while the teaching (instruction) is occurring.More specifically, one must have metacognitive knowledge of 1) the domain to be taught, 2) the instructional pedagogies to be used or adapted to achieve the goals of instruction one has set, 3) the students' to be taught metacognitive skills, processes and knowledge, and 4) one's own personal metacognitive skills, processes, and knowledge.Teaching is not a simple and uncomplicated act and it is usually more complex and complicated than learning.Teaching someone to teach something expotentiates the complexity and complications even more, as does having systems, models and theories for each of the four frames just identified to track and keep clear these four frames and not "jumble them", as very often happens at both the individual and professional level, and in scholarly and instructional writings about these four frames, which in turn produces many problems and confusions.Further, it should be clearly noted that metacognition, metacognitive processes and metacognitive knowledge of various kinds are typically required and typically occur in each of the four frames identified above for various event and outcomes to occur successfully.
However, the concept of metacognition itself is a complex and fuzzy construct that varies from theorist to theorist and W. XU ET AL. field to field as well as from subject-matter area to subjectmatter area.So the discussion here will be brief and directed at establishing some key points and the reader will be referred to other sources for more detailed explication of the points made in this article.A concept as complex and fuzzy as metacognition demands that beyond such commonly acceptable definition as "thinking about thinking" or "cognition about cognition", researchers should elaborate the construct theoretically and practically with regard to different content domains, various situations, and specific processes to make the construct more useful, concrete and grounded relative to specific pedagogical contexts and situations.Flavell's (1979) seminal metacognition model, it should be noted, includes two kinds of metacognition.The first kind is metacognitive knowledge about self, task, and strategies and the second kind of metacognition is metacognitive experience or experiencing, which are "items of metacognitive knowledge that have entered consciousness" (p.908).Based on Flavell's metacognition model (1979), various studies have been carried out to explore the diverse functions metacognition serves in different content (subject matter) areas.More specifically, in the field of language arts and literacy, a research-based belief has been established that metacognition can be taught and learned (e.g.Flavell, 1979;Pressley & Woloshyn, 1995;Lovett, 2008;Lai, 2011) on the bases of such research evidence as the key role of metacognition in successful reading (e.g.Dagostino & Carifio, 1994a, 1994b;Brown et al., 1981;Brown, 1985;Baker & Brown, 1984), and the increasingly accumulated knowledge about what "the metacognitively sophisticated reader" is (Pressley, 2002).
With this understanding, reading teachers have been called upon to promote metacognition in students, or in Pressley's (2002) term, to be "the metacognitively sophisticated teacher".This view is in accordance with Hartman (2001) call "to teach for metacognition", referring to teachers' thinking about "how their instruction will activate and develop their students' metacognition, or think about their own thinking as learners" (p.149).This view and call is essentially focused on component three in the four component framework described at the beginning of this article.Teachers have also been expected to teach with metacognition; that is, teachers should think about their own thinking regarding their teaching before, during, and after conducting lessons to increase instructional effectiveness (Hartman, 2001).This view, definition, and focus on metacognition is essentially a focus on component four only in the four component framework previously-described.
Flavell also contends that his three kinds of metacognitive knowledge are the basis for particular metacognitive processes and experiences (e.g.Flavell, 1979;Garner, 1987).Thus for Flavell, metacognitive knowledge and metacognitive experiences are joined and dependent on each other with metacognitive knowledge being the gateway variable to specific metacognitive experiences and processes for both teachers and students.Therefore, it is important for teachers to acquire enough metacognitive knowledge if they are expected to teach for and/or with metacognition to have the metacognitive experiences and processes they need to have to achieve the desired goal for and/or with metacognition.
But there is another rub.All of the above four components of metacognitive knowledge are nevertheless complex in their own right and need to be understood both individually and relationally to teach or learn something well, as well as to teach someone how to teach other (pre or in-service) teachers the different elements of a given domain (i.e., knowledge, skills, strategies, procedures, or sensibilities), once they have sufficiently mastered them to do so to the targeted goals or levels set.This latter task involves components 2), 3), and 4) above, each of which involves or is heavily laden with various metacognitive processes, skills, and knowledge of one kind or another that are relevant to the tasks to be accomplished successfully in each component and across components by the teacher, the student or the teacher of teachers depending on the frame and component one is considering.It would seem logical, then, to both define and discuss metacognition and metacognitive knowledge, skills and processes, since they are so central to learning and teaching and teaching (pre or in-service) teachers about teaching some domain or one of its elements, or subelements such as strategies for summarizing expository text, the focus and subject of this article and study.
Unfortunately, only preliminary and formative work has been done along these lines in education and in teacher training and professional development for (pre or in-service) teachers.In addition, another problem is that Flavell and many of who built upon his work and model both jumble and confound the four components or frames associated with metacognition described at the beginning of this work, and the two types of metacognition Flavell identified (kinds of knowledge and experiences), and it is often more than just difficult to disentangle the four components and two types clearly in these works, which is a source of various confusions, misunderstandings, and miscommunications observed in this literature as it relates to both teacher and student metacognitions.It is for this reason that we use our "4 by 2" (components or frames by metacognition types) model to classify views and assertions to aid clarification and communication about this fuzzy construct and area.
The various models and theories used in our work, including the general one for guiding the development and production of a research-based monograph for teaching (pre or in-service) teachers how to effectively teach strategies for summarizing expository text (see Carifio & Perla, 2010;Xu, Carifio, & Dagostino, 2012, for details), are heuristic frameworks, aids and guides for understanding and conceptualizing the four components or frames of metacognition identified above, both individually and in relationship to each other, and for keeping them more distinguishable from each other for in terms of both explication and discussion.More will be said on this point below.

Prior Research
Limited research has been conducted to explore what metacognitive knowledge reading teachers should have so as to be able to teach for and/or with metacognition and to be able to judge when, how, and whether or not to teach metacognitively.Such explorations are especially rare for Chinese post-secondary in-service reading teachers in China.Most of the discussions on teacher knowledge in China have been conducted in terms of domain-based knowledge teachers should have, or component one only in the four component framework described at the beginning of this article.Next, as a pioneer in educational reform oriented studies on teacher knowledge, Shulman (1986Shulman ( , 1987) ) conceived that a knowledge base for teaching should be an amalgam of knowledge, skills, and dispositions that underlies the capacity to teach effectively, and he developed a typology to characterize teacher knowledge that other researchers have used, expanded, and refined later (e.g., Cochran et al., 1993;Grossman & Richert, 1988;Grossman, W. XU ET AL. 1990;Gudmundsdottir, 1991).Shulman's view is essentially component two in the four component framework described at the beginning of this article.Discussions of a knowledge base in TESOL, always nested in reviewing the general knowledge base of teacher education, have been fostered from two traditions: one is from the theoretical perspective, attempting to "identify and develop the knowledge teachers need to demonstrate," and the other, the practice and policy perspective, intending to "enhance the teacher preparation processes and teaching practices" (Fradd & Lee, 1998).In a recent study that examined the degree to which the requirements in teacher education programs in the United States reflected current theory and practice for teachers of English language learners (ELLs) in their coursework, each state's teacher preparation policies and requirements were still focused only on cognitive level of subject-matter knowledge, knowledge of pedagogy, knowledge of linguistics, and knowledge of cultural and linguistic diversity, without even a mention of metacognition or teaching metacognitively (see Lopez et al., 2013, for details). Furthermore, Fenstermacher (1994) tried to distinguish two major types of knowledge about teaching: formal knowledge and practical knowledge."Formal knowledge", described as knowledge for teachers, is the knowledge that is primarily known and produced by researchers."Practical knowledge", referred to as knowledge of teachers, is the knowledge that is principally known and produced by teachers themselves.In this article, the focus of attention is on what Fenstermacher noted, teachers' formal knowledge, but from the metacognitive perspective and in the area of reading strategies instruction.
At the metacognitive level, Pressley (2002) hypothesized that the metacognitively sophisticated reading teacher should know "what good readers know…" (p.305) and can teach comprehension strategies when needed (p.306).The question that logically and obviously follows from Pressley's assertion, then, is "What metacognitive knowledge and metacognitions do reading teachers need to acquire specifically in order to teach metacognitively?"Our work and study investigated this question in detail with regard to a particular group of reading teachers' (post-secondary in-service EFL teachers), specific teaching tasks (summarizing strategies for expository text) and a specific group of students (EFL undergraduates).
Another view of metacognition is that metacognitive knowledge and Flavell-like metacognitive experiences are used to control the cognitive processes used to achieve the goals of a specific cognitive task (Garner, 1987;Griffith & Ruan, 2005), while other kinds of knowledge, such as automatic functioning and automatic processing that occur in complex, well-learned and habit-driven cognitive activities like reading and studying a textbook chapter, do not have such "controlling influences" that can adjust or change these latter kinds of activities mid-process.In spite of this view and point, no qualitative difference exists between cognitive and metacognitive knowledge in Flavell's view and model (1985) of metacognition, which is consciousness driven with no acknowledgement of unconscious automatic processes and functions.This flaw and fuzziness in Flavell's and many other models of metacognition is a difficult hurdle to get by in exploring metacognitive knowledge and its functions in learning and teaching, as what at one time is conscious metacognitive experience can with learning, time and use become automatic unconscious processing and functions which creates many conceptual, theoretical and research difficulties and confusions.The key here, we believe, is that Flavell and many other metacognition theorist have a hidden assumption in their models and theories that the learner is a relatively novice learner in terms of the tasks to be learned, as compared to a more expert and experienced learners, and thus few if any automatic processes have been acquired relative to the task(s) to-be-learned, and most metacognitive experiences and processes will be conscious for the learner and thus metacognitive (self, tasks, and strategies) knowledge dependent.Flavell and other theorists, moreover, are not talking about the adult learner or even the young adult learner, which is another hidden assumption/restriction in their models and theories which make them only fuzzy or approximate fits to pre and in-service teachers and undergraduate students.Learners in Flavell-like models are fairly novice learners relative to the knowledge-domain and tasks to be learned, and are not adult learners in Knowles' sense and definition of the term (Knowles, Holton, & Swanson, 2005).Knowles' notion of adult learners corresponds to many aspects of the upper levels of "self-regulated" learners in Zimmerman's (2008) and Purdie & Hattie's (1996) conceptualization and models of this view and type of metacognition and controlling of cognitive processes, which, it should be clearly noted, can be both conscious and/or automatic and unconscious.Understanding such limitations of Flavell-like models of learners allows various aspects of Flavell's model and theory to be reasonably used to both model and teach a fairly simple and simplified model of metacognition to pre and in-service teachers and undergraduate students that is not overly complex or complicated, but also not grossly distorted or inadequate for the task.This approach is the approach we adopted in this work and some of the reasons why we adopted it.Further, according to our database searches, there is no prior work or studies like the ones we have done conceptualizing, operationalizing, and developing a research-based monograph for teaching the types of metacognition required by post-secondary in-service EFL reading teachers to teach summarizing strategies for expository texts and then validating that monograph using a detailed panel review of the monograph chapter by chapter by highly experienced practicing EFL teachers and of expert EFL professors and comparing the results and views of the monograph by these two different kind of expert panels (for further and more detailed elaboration of this point, see Xu, Carifio, & Dagostino, 2012).

Purpose
This article reports the second phase of the Harvard Business School like case study we did; namely, the validation of the monograph developed according to the formal monograph development model used (see below for details).Before a detailed report of this validation process is presented, a brief overview of the design of the entire study and the completion of Phase I of the study (the creation and production of the research-based monograph) is given to help the reader understand the process by which the monograph was developed and refined before the validation process (Phase II) of this study started (see Xu, Carifio, & Dagostino, 2012, for more details).The focus of this article, then, is on the method and findings of the monograph (product) validation and panel review processes done in Phase II.
W. XU ET AL.
service EFL reading teachers' understanding of the processes and metacognitive knowledge involved in teaching summarizing strategies for expository text to EFL undergraduates.An extensive and comprehensive literature review and content critical evaluation phase was required during the process of developing and validating the first draft of the monograph relative to various areas of research, such as metacognition, text comprehension, reading strategies instruction, TESOL, and so on.This process yielded certain theoretical frameworks, syntheses, ideas, and views, both anticipated and unanticipated beforehand that formed the core of this work and the monograph developed.To gauge and parameterize all expected and unexpected findings, the design of this study adhered to a formalized model of instructional materials development initially proposed by Carifio (1975Carifio ( , 1977) ) and further elaborated by Perla (2006) and Perla and Carifio (2011).This model is depicted in Figure 1 for the reader for the purpose of following the discussion here and the model is explained in detailed in Perla and Carifio (2011).
According to the Carifio-Perla model, the process of creating and developing academic materials involves three macro components, starting from developing a "Critical & High Quality Knowledge Base" (CHQKB) in a particular domain which identifies the key and critical features of the CHQKB and which is a form of meta-knowledge about the CHQKB as well.This "Critical and High Quality Knowledge Base" is then translated into "Appropriate Representations and Communications" (ARC's) in the form of instructional or academic materials.The "Appropriate Representations and Communications" (ARC's) then go through "Validation and Field Testing for Effectiveness" (VFTE).During the process of going from CHQKB to ARC's, and then from ARC's to VFTE, two types of unanticipated findings might emerge respectively, which the developer should be particularly sensitive to as they are research findings.The first type of unanticipated findings concerns inadequacies in the theories, frameworks, and claims encountered in the domain and contradictions between alternative views of all three found in the domain.The second type of unanticipated findings concerns inadequacies in the pedagogical theories, frameworks, and claims encountered in the domain and contradictions between alternative views of all three found in the domain.As both types of unanticipated findings must be reconciled to some degree, model driven and guided instructional materials development can help to refine and improve both the theory and the pedagogical knowledge in a given domain if done appropriately, which is not the commonly held view of the instructional materials development process today.
Phase I, then, was to utilize the Carifio-Perla model (Perla and Carifio, 2011) to create a research-based monograph (referred to as The Monograph hereafter).During Phase I, an unexpected finding was the many difficulties and contradictions that were encountered in constructing a general Metacognitive Knowledge Framework (MKF).The elaboration of this MKF was a process of exploring answers to the question "What exactly does a post-secondary in-service EFL reading teacher's metacognitive knowledge consist of as to teaching summarizing strategies with expository text?"The general MKF that was constructed was based on Flavell's (1979) tripartite metacognitive knowledge model and other researchers' conceptualization of metacognitive knowledge (e.g. Baker, 1989;Garner, 1987Garner, , 1990;;Glaser & Chi, 1988;Pressley, Borkowski, & Schneider, 1987;Reynolds, 1992;Schneider & Pressley, 1989).The MKF that was constructed from this process turned out to be a two-dimensional matrix as shown in Table 1 (for further and more detailed elaboration of this MKF, see Xu, Carifio, & Dagostino, 2012).
As can be seen from Table 1, Flavell's one dimensional view of metacognitive knowledge (i.e.person, task, and strategies variables) can be analyzed in terms of a second dimension as well; namely, metacognitive declarative, procedural, and conditional knowledge, and vice versa.It has to be pointed out again as it was done at the beginning of this article that the discussion of metacognitive knowledge on person variables is too complex, complicated, and jumbled (confounded) to be useful or to be included in this study.Thus, the focus of this study was only on exploring the metacognitive knowledge of Flavell's task and strategy variables.This is to say, post-secondary in-service EFL reading teachers' metacognitive knowledge of summarizing strategies instruction with expository text was investigated only from the six aspects numbered "1" to "6" in Table 1 and not in terms of those numbered 7 to 9 in Table 1.The monograph, therefore, elaborated the general MKF in terms of task and strategy variables for the dimension of metacognitive declarative, procedural, and conditional knowledge.The nature of this elaboration is given in detail in another article (Xu, Carifio, & Dagostino, 2012).
The process for establishing the finalized content outline and the content of The Monograph (i.e., the ARC's for this study) was an iterative, spiral and dynamic process.The finalized content for the version of The Monograph sent for review in phase II had eight chapters.The main idea of each chapter is listed as follow: 1) Chapter 1 outlined and summarized the focus of The Monograph.
2) Chapter 2 presented the construction of a general Metacognitive Knowledge Framework (MKF) via thorough analyses of different researchers' conceptualization of "metacognition" and "metacognitive knowledge".Copyright © 2013 SciRes.
W. XU ET AL.
3) Chapter 3 presented and elaborated a reading model to support the constituents of MKF, helping reading teachers better understand the nature and process of summarizing complex text and expository text comprehension at the post-secondary level.
4) Chapter 4 synthesized an instructional model, focusing on maximizing comprehension along with a range of comprehension improving strategies to further elaborate the theoretical MKF for reading teachers on teaching summarizing strategies in ESL/EFL instructional contexts.
5) Chapter 5 explored the constructs and teaching demands of summarizing strategies instruction with expository text, which comprises the metacognitive declarative knowledge of task variables in MKF.
6) Chapter 6 addressed and illustrated the specifics and details of teaching summarizing strategies with expository text (Category 2 in the framework) with a strong focus on the subteaching tasks of summarizing strategies instruction.
7) Chapter 7 continued to address the specifics and details of teaching summarizing strategies with expository text, concentrating on the selection of cognitive and metacognitive strategies with some accompanying conditional knowledge involved in teaching summarizing strategies.
8) Chapter 8 applied the previously discussed metacognitive knowledge and provided scenarios to show how a reading teacher's metacognitive knowledge can be translated into designing a lesson plan for teaching certain summarizing strategies with expository text to EFL undergraduates.
Because little literature was found in the various database searched (e.g.ProQuest, EBSCOhost, SAGE, JSTOR) that was directly related to the research topic and focus of this study, which surprised us, the meta-cognitive framework we constructed, as well as the pedagogical framework for teaching it, was very tentative and novel.We had apparently found a large "hole" in the literature and the various exhortations for teachers to teaching "metacognition" and to teach "metacognitively," repeatedly stated by many recent articles we encountered in this literature, as there was little that was concretely worked out and available in the literature to actually use to implement these numerous exhortations, and particularly so in the area of reading and expository text processing/summarization skills which are at the core of most academic instruction.Given these points and the numerous decisions and selections made in the development process, The Monograph developed needed to be further reviewed, evaluated, and cross-validated (or not) by independent and "expert" third-parties.Therefore, the completed monograph was sent to panels of expert reviewers for independent review, evaluation, and cross-validation.The methodology and results of the validation process; namely, Phase II of this study, are discussed in the following sections.

Validation of the Metacognitive Knowledge Exploration: Phase II Study
As previously stated, Phase II aimed to validate The Monograph.The Monograph was sent to two formal reviewer panels (seven EFL reading teacher expert practitioners and seven EFL academic reading professors of education) in China for external third party validation.This type of general research design and panel review methodology, which is an adaptation and implementation of Campell and Fiscke's (1959) convergent and discriminant validation design, was developed by Carifio (1990) and has been successfully used by others (See Dagostino & Carifio, 1994a;Flores, 2005;Erikson, 2006;Kwong, 2008).Perla (2006) summarized Kerlinger and Lee's (2000) explanation of the two concepts, convergent and discriminant, in his study as the following: Convergence here refers to different sources (observers or modes of observation) used to measure a construct or "traits" producing similar results, whereas discriminability refers to similar traits that can be empirically differentiated from one another or identifying traits that are not correlated or negatively correlated with each other or the construct (p. 168).
Therefore, the agreement and disagreement between data sources and reviewers provided triangulation evidences to assess and validate The Monograph created as well as the model and theory used to create The Monograph validated.Specifically, the validation process in Phase II was intended to address three questions: 1) How adequate and comprehensive was the general Metacognitive Knowledge Framework generated in the views of all reviewers?
2) How well and completely was the general Metacognitive Knowledge Framework elaborated in the views of all reviewers?
3) Were there important or key differences between the two panels of reviewers in evaluating The Monograph?

Methodology of the Validation
A cross-panel replicated expert reviewer (practitioners and academics) design was used to validate The Monograph.

Review Panels
The independent formal reviewers were grouped into two panels.Panel 1 was 7 post-secondary EFL teacher educators (the academic expert panel) who taught pre-service EFL teachers at a university in Zhejiang Province, China.Two of Panel 1 reviewers were senior teacher educators with around 30 years of experience teaching preparing EFL teachers.The other five panelists were more junior EFL teacher educators, who once were students of the two senior teacher educators during their undergraduate studies at the same university.One of the junior teacher educators also had a doctorate in Comparative Literature, while another one was also currently a doctoral candidate in Comparative Linguistics at the time The Monograph was reviewed.The other four Panel 1 reviewers also had master degrees in either English Language and Literature or English Language and Linguistics.The members of Panel 1 were welleducated as well as well experienced practitioners.
Panel 2 was made up of 7 post-secondary EFL professors/ teachers of reading from several universities in Shanghai, who taught EFL undergraduates in various majors.Most members of Panel 2 (the expert practitioner group) had master degrees in either English Language and Literature or English Language and Linguistics.All the EFL academics who were Panel 2 members also had many years of EFL teaching experience at the post-secondary level.Their professional opinions, therefore, should be more than just well-informed and "academic" relative to designing effective comprehension strategies instruction, and especially in terms of the summarizing strategies instruction discussed in The Monograph.
It should be pointed out that since it was very difficult to get qualified formal reviewers who were Chinese and living and W. XU ET AL.
teaching in China (one of the key criteria for being a member of these review panels), the formal reviewers for this study were chosen from the first author's former colleagues and classmates in China, based on their availability and interest of doing this kind of reviewing work.Two further points need to be made here however.
First as The Monograph was written in English for native Chinese professionals and students who would be learning and/ or teaching expository text summarizing skills in English to undergraduate Chinese EFL students, it was key to both the evaluation and validation of this Monograph that the review panel members were native Chinese experts who were bilingual and who taught English and the reading and understanding of English to undergraduate Chinese EFL students and not experts who were just English speaking EFL experts evaluating an EFL Monograph on the teaching of expository text summarizing skills.The latter groups would be highly flawed (and possibly biased) validation panels in several different ways, where the former groups (the panels we created and used) would not be, as they were the experts with the correct set of skills, training and experiences to more fully and comprehensively carry out the tasks, evaluation and validation of The Monograph.One feature of this study that is relatively unique and important, therefore, and unique and important relative to the research studies typically done in the EFL area is that it used bi-lingual and bi-cultural panelists to evaluate an instructional/academic monograph written in English and produced in the USA that was intended for EFL teachers, preparing teachers, and students in China; namely, in another country, culture and educational system contextually and implicitly regulated by different policies and values.Whether The Monograph would be judged to be of worth and a quality work of value in this context is not just external validation and an external validity question, but rather is ecological validation, and an ecological validity question is typically rarely addressed or done in most research works, making the views and reviews of The Monograph by the two bi-lingual and bi-cultural panels used important and unique in this larger and broader sense relative to multilingual and multicultural works and studies and international education.
The second point that needs to be addressed above is that members of the two review panels used were former colleagues of the senior author of this article.In Western cultures and contexts, the use of colleagues as reviewers and judges might be seen as a flaw and a biasing factor, but in China and China today, any biasing would be far less, as it would be culturally and personally as well as academically dishonest and unacceptable to render anything other than the frankest of evaluations and criticisms of a work and a work that was a social object that would be used in social contexts with social consequences even if it was the work of a known old colleague.Does this mean that there would not perhaps be unconscious biases operating in the review process?Not likely; only that they would most probably be comparative less due to differing cultural imperatives and effects as well as "averaged down" even further over the number of expert reviewers used in this study making this bias quite low for a panel review design.

Data Collection
The two panels of reviewers were provided a detailed review protocol to formally review and evaluate each of the eight chapters of The Monograph and The Monograph itself.This protocol was a modified version of an already field-tested and validated 30-item Likert responding format objective text protocol plus 7 open-ended written questions (See Carifio, 2003;Dagostino & Carifio, 1994a;Pellitier, 2004;Flores, 2005;Perla, 2006;Erikson, 2006).The protocol's purpose was to get both open and closed structured feedback about the appropriateness of the constructed MKF, and the effectiveness of the translation, representation and communication of the MKF in The Monograph.The protocol asked each formal reviewer to make specific judgments about each chapter as well as the whole work on a scale ranging from "NA (Not applicable)" to "RP (Rift with Problems)" to "E (Excellent)" in terms of eight categories of feedback.These categories were: 1) Accuracy, Saliency and Relevance of Content; 2) Thoroughness; 3) Quality of Supporting Theory, Research, and Scholarship; 4) Presence of Multiple and Alternative Views; 5) Tone; 6) Clarity of Writing Relative to Audiences; 7) Specificity and Concreteness of Key Points and Recommendations; and 8) Missing Elements (see Erikson, 2006 for details).These eight criteria were originally developed by Carifio (2003) and later effectively applied to research studies of this instructional materials development model by other researchers (e.g., Perla, 2006;Erickson, 2006;Kwong, 2008).All reviewers were also encouraged and allowed to write comments directly on The Monograph.
The whole monograph was sent to each individual reviewer chapter by chapter accompanied by the formal evaluation review protocol.A new chapter would not be sent until the previous chapter was finished by the reviewers so that halo effects were reduced and chapters were reviewed as independently from other chapters as possible.The same protocol was sent to each reviewer again to evaluate the whole document after they had finished reviewing all eight chapters separately.
Upon the completion of the formal review, a follow-up 15 to 20 minutes informal interview was also performed with reviewers on the phone if the interviewer (the first author of this article) felt the necessity of having the panelists further explain their responses or to help ensure no misunderstandings of their points and views were occurring.Thus, the discussed contents in each interview were different, and no formal interview guidelines were used.Notes were made both during and after these interviews and were referenced during the formal data analyses.The Likert data and other rating data in this study were analyzed as parametric data as this was the appropriate way in which these data should be analyzed (see Carifio & Perla, 2007, for details).

Data Analyses
Of the 14 external third party reviewers, only 6 (3 teacher educators and 3 in-service reading teachers) were able to complete the whole reviewing process.The other reviewers failed to finish the reviewing work due to either their heavy teaching load or personal issues.All data from the 6 reviewers were then analyzed both quantitatively and qualitatively as the core analyses in this study as they were a complete data set with no missing observations.For the 30-item objective text protocol, the responding keys were converted to numerical data from 0 to 6 and uploaded to SPSS for sequential statistical analyses except for Items 7 and 8 because of reviewers' inconsistent responses to these two items due to their misunderstandings and concerns in terms of the meaning and function of "NA (Not Applicable)" term and designation, which was a cross-cultural discovery for us.
First, mean and weighted mean combined responses by the W. XU ET AL.
teacher educators and the in-service reading teachers were calculated separately for each item for each chapter, across chapters, and all chapters as a whole, namely, The Monograph itself.Second, before conducting the next steps of data analyses, all items were aggregated into the first seven major evaluation criteria chosen for each chapter.The original eighth criterion "Missing Elements" was excluded because no reviewer pointed out any missing elements from The Monograph.
Third, mean responses as well as the overall mean responses for each criterion and each chapter were then calculated for the two review panels respectively.
Finally, a frequency matrix of mean ratings for each category was conducted respectively for Panel 1 reviewers, Panel 2 reviewers, so that they were directly comparable, and then reviewers of both panels combined.
A cut-off criterion of 3.5 on the 0 to 6-point scale was used as the minimally acceptable mean level of quality on each item, each category, and each chapter in the aggregate, as well as on the items and the categories across the chapters for the objective responding review protocol.
As to the open-ended written questions, the external reviewers' responses were analyzed using a thematic and simple tallies scoring system.The responses were coded using a simple 8-point Very Positive comment to Very Negative comment rating scale (see details in Erikson, 2006: p. 98).This rating scale had the following categories: (−4) Extremely Negative, (−3) Very Negative, (−2) Negative, (−1) Somewhat Negative, (0) Neutral/Unsure, (+1) Somewhat Positive, (+2) Positive, (+3) Very Positive, and (+4) Extremely Positive.The cut-off criterion of +1.0 was chosen as the minimally accepted level of quality on each chapter and the chapters in the aggregate.Simple tallies and means, both weighted and unweighted, were thus obtained to analyze the overall quality of the eight chapters relative to those positive and negative reviewer comments for both panels.Thus, the number and kind of responses to the questions could be reported, and the overall qualities of The Monograph could be assessed.Further, agreements and disagreements between Panel 1 and Panel 2 reviewers were also analyzed formally.With regard to the comments directly made on The Monograph, they were not coded using the +4 to −4 scoring rubric because not all reviewers commented directly on the text.However, these comments along with the results of the analyses reported below were used to revise The Monograph.

Results of the Validation
The following sections report the major findings of the validation of study done; namely, the formal review of The Monograph and the scoring and coding of these reviews as described above.

Structured Format Ratings
Table 2 is a summary table that presents the means and weighted combined means for both the structured format ratings and the rated open-ended responses by both the teacher educators and the reading teachers for the eight chapters as well as the overall averages for all chapters by the two panels of reviewers.
As can be seen from Table 2, the weighted combined means for all chapters for the structured format means ranged from 4.92 (highly acceptable) to 5.68 (good plus).This indicated that the ratings for all chapters were well above good ratings, Table 2. Summary of means and weighted combined means for the structured format ratings made by the academic teacher educators (n = 3) and the practitioner in-service reading teachers (n = 3) for the eight chapters of the monograph on a zero to six point response scale.and so were the averages for all chapters.The mean ratings by the three teacher educators (academic experts) were markedly higher than those by the three in-service reading teachers (practitioner experts) for seven chapters of the eight chapters of monograph, the exception being Chapter Five (Constructs and Teaching Demands).For Chapter Eight, and Chapter One, these mean rating differences were almost half a scale point or more for the academic experts, indicating obvious differences between the two panels of reviewers' viewpoints.

Structured
Relative to the findings presented in Table 2, the first three chapters of The Monograph are very theoretical and introduced the background of the conceptualization of metacognitive knowledge, while Chapter Five was about the constructs and teaching demands of summarizing strategies instruction.Thus, the teacher educators (the academic experts) seemed to buy more into those theoretically oriented chapters, but the reading teachers (the practitioner experts) were more interested in the chapters on the analyses of practical teaching issues.The academic teacher educators (professors) rated Chapter Eight (A Scenario) the highest, and in fact much higher than the practitioner reading teachers did, who also rated this chapter "good plus."This particular finding meant that though Chapter Eight was supposed to provide the readers a practical example, it could not be fully appreciated without the full understanding of its theoretical background elaborated in the previous chapters.Thus, overall the three teacher educators were more positive than the three in-service reading teachers on the 30 item structured rating scale.However, the difference in variability for the structured ratings between the two panels of reviewers was not that much.Actually, the range of the ratings for the structured W. XU ET AL. format means by both panels was 0.40 with the teacher educators' ratings basically 0.36 higher than those of the reading teachers.

Open-Ended Responses
In terms of the average rated comment means for the chapters, which were done on a −4.0 to +4.0 scale, Table 3 shows that all chapters except Chapter One (Introduction/Overview) more than met the cut-off criteria of +1.The low overall rating for Chapter One (+0.9) was caused by the low (negative) rating of this chapter by the teacher educators (+0.7).As previously stated, Chapter one was perhaps the most theoretical of the chapters.The recorded comments indicated that the reviewers expected more background description in Chapter One in terms of why the researcher chose this research topic and what specifically led to this research focus.
The rated comment means by the three academic teacher educators were markedly more positive than those by the three practitioner in-service reading teachers about the five chapters of the monograph: Chapter Two, Chapter Three, Chapter Four, Chapter Seven and Chapter Eight, two of which (Chapter Two and Seven) even higher than the double of the ratings by the reading teachers.The practitioner reading teachers tended to be somewhat more positive than the academic teacher educators about Chapter One (Introduction) and Chapter Five (Constructs and Teaching Demands), and significantly more positive about Chapter Six (The Sub-teaching Tasks).Thus, the means for all chapters by all reviewers except that for Chapter One by the academic teacher educators were above the acceptable rating of +1, and some were highly above +1 with the highest mean of +2.5 occurring for the academic teacher educators for Chapter Seven (The Selection of Strategies) and Chapter Two (A Model of Metacognitive Knowledge).The highest mean rating (positive comments) by the practitioner reading teachers was Chapter Six (The Sub-teaching Tasks).
These differences in making positive and negative comments on the chapters among the reviewers echoed the differences Table 3. Summary of means and weighted combined means for the ratings of comments made by the academic teacher educators (n = 3) and the practitioner in-service reading teachers (n = 3) for the eight chapters of the monograph on a one to eight (−4.0 to +4.0) point response scale.

Practitioner
Reading Teachers shown by their structured ratings.These differences once again revealed their differences in views while reading an academic monograph.The academic teacher educators seemed to buy more into those theoretically oriented chapters, such as Chapter Two (A Model of Metacognitive Knowledge), Chapter Three (A Reading Model), and Chapter Four (ESL/EFL Instructional Contexts), while the practitioner reading teachers were more interested in the chapters on the analyses of practical teaching issues, such as Chapter Five and Chapter Six.However, the real point of importance here is that both the academic teacher educators and the practitioner in-service reading teachers needed all 8 of the chapters in The Monograph really regardless of their preference and that point came out later in their overall evaluation.
All reviewers thought highly of Chapter Eight (A Scenario) as shown from their extremely positive comments (the average of 2.3 and 2.1 respectively by the two panels) according to the average all chapters and the total combined mean ratings.Thus, it can be concluded that all reviewers thought that Chapter Eight successfully applied theory into practice and effectively demonstrated how a reading teacher's metacognitive knowledge was involved in her decision making while designing a lesson plan for summarizing strategies instruction.
The average of all chapters by the teacher educators was 1.9, higher than that by the reading teachers (1.6).Thus, basically the three teacher educators were more positive than the three in-service reading teachers in their responses to the open-ended questions, but with a bit of more variability in their evaluations (range 1.8) as compared to the reading teachers who seemed less variable in their responses (range 1.4).This variability, it should be noted, makes it easier to understand the observed variability in the combined weighted means for the open-ended comments.
Table 4 presents the total combined means of the structured and comment ratings for the eight chapters of The Monograph made by the academic and practitioner expert reviewers in this study.As can be seen from Table 4, overall both the academic and practitioner reviewers were quite positive about the chapters of The Monograph in terms of the evaluation criteria used with the pattern of differences by chapter being as expected beforehand.Overall, the academic reviewers were slightly more positive than the practitioner reviewers which was also somewhat of an expected outcome.

Evaluation Criteria Analyses and Results
The 30-item structured survey also reflected seven major evaluation criteria for each chapter as mentioned earlier.Another and equally important way of depicting results as well as differences between the two review panels is in terms of each of the seven evaluation criteria used.Table 5 presents the mean responses by each of the seven evaluation criteria for the academic and practitioner review panels.The results depicted in Table 5 indicate that when The Monograph was reviewed chapter by chapter, an overall combined weighted mean for all criteria were either "good plus" or "excellent minus."These results further supported the high quality of content and consistency of The Monograph.Again, the academic teacher educators rated all seven criteria for each chapter higher than the practitioner reading teachers did.The largest variability was within the criterion of Professional Tone for which the mean rating by the practitioner reading teachers was 0.41 below that of the academic teacher educators.
W. XU ET AL.

Table 4.
Total combined means of structured and comment ratings made by the academic teacher educators (n = 3) and the practitioner in-service reading teachers (n = 3) for the eight chapters of the monograph.When The Monograph was evaluated as a whole according to the seven criteria, the mean rating for Professional Tone by the teacher educators was also much higher than that of the reading teachers (0.34 higher, see Table 6).This difference again showed that the academic teacher educators appreciated professional and scholarly discourse and presentation more than the practitioner reading teachers, but this result is relative as both were appreciative and satisfied with the tone of The Monograph.Again, this difference most probably can be attributed to the teacher educators' higher level of professional development from their doctoral degrees, which made them more Table 6.Mean and weighted mean combined responses made by the academic teacher educators (n = 3) and the practitioner in-service reading teachers (n = 3) on the seven major evaluation criteria for the monograph as a whole on a one to six point responses scale.

Academic
accustomed to communications in a professional academic tone, perhaps making them more "trilingual" than their bilingual practitioner counterparts.It was interesting to see that the mean and weighted mean combined responses made by both panels of reviewers for The Monograph as a whole (see Table 6) were higher than those for The Monograph reviewed chapter by chapter on almost all seven criteria except the criterion of Accuracy of Content.This result meant that after reading the whole work, all reviewers most likely came to see clearer the big picture of The Monograph and understood the contents of The Monograph better, so they gave The Monograph higher evaluation as a whole.And it may also be that because of this better and more global understanding that the reviewers' expectations or/and perceptions of the accuracy of content in turn became a bit higher.

Summary and Discussion of the Results
From the above results it is clear that all panel reviewers evaluated The Monograph positively and highly on all criteria and for all chapters respectively in terms of the structured format mean ratings and the average rated comment means for all chapters.In general, almost all reviewers' mean ratings were skewed towards the higher level of assessment.These very positive and uniform mean ratings indicated the high quality and consistency of The Monograph that was achieved by employing the guiding instructional materials development model which was presented at the beginning of this article; namely, the appropriate construction, and the effective communication and representation of the general MKF in The Monograph for the target audience.This result replicated the results of seven other recent uses of this instructional materials development model given at the beginning of this article but for somewhat more difficult, complex and fuzzy subject-matter in a bilingual and bi-cultural context which is both a new and very positive result and finding for this instructional materials development W. XU ET AL. model as it shows that aspects of skills, knowledge, understandings, and metacognitions both transcend and can be represented and communicated successfully across languages and cultures and to different professional audiences as well.This finding we believe is more than a hopeful finding and result.
We have also further synthesized our findings to answer the previously-mentioned three specific questions of importance related to the subject-matter of this study.These three questions are: a) how all reviewers viewed the construction of the general Metacognitive Knowledge Framework, b) how they viewed the elaboration of the general MKF, and c) what the differences between the two panels of reviewers were in evaluating The Monograph.
The construction of the general MKF was presented in Chapter Two of The Monograph.The means and weighted combined means for structured format ratings in Tables 2-4 by all expert reviewers for Chapter Two were all "good plus."The rated comments means by the academic teacher educators were more than positive, and those by the practitioner in-service reading teachers were somewhat positive.It can thus be reasonably concluded that all reviewers agreed that the general MKF was appropriately constructed for the purpose of this case study and its intended audiences even though a more simplified version of the MFK had to be presented for the reasons outlined at the beginning of this article than academic psychologist might prefer.Actually some reviewers made very positive comments about the MFK presented, saying that the researcher was quite successful in analyzing reading teachers' metacognitive knowledge and laying a very solid theoretical foundation for further discussions at this aspect.Further, all reviewers shared the opinion that reading teachers should acquire relevant metacognitive knowledge to conduct reading strategies instruction effectively.Therefore, one might conclude that for introductory instructional texts on meta-cognition (and similar content) the highly nuanced, complex and fully fuzzy representation of academic psychological theorists and researchers might not be the most successful approach and that simpler and less nuanced but accurate representations that may be built upon later will be more successful with certain audiences and that this is a point and criterion that should be kept in mind by instructional materials developers and publishers as well as those who select instructional materials for others.
Next, the results of data analyses also revealed that all reviewers thought that the general MKF was successfully elaborated in different chapters as to investigating the task and strategy variables of metacognitive knowledge in terms of their declarative, procedural and conditional metacognitive components.The key consensus specific reviewer findings concerning the elaboration of the general MKF presented were as follow: First, as to supporting the components of metacognitive knowledge that were identified with a reading model (Chapter Three), teacher educators thought that Kintsch's (1998) Construction-Integration (CI) Model was the best choice and appropriately presented for the purpose of exploring reading teachers' metacognitive knowledge concerning the process of summarizing and summarizing strategies instruction for expository texts.This CI model was the most useful and valid theory currently available to explain mature readers' skilled reading at the higher-level of information processing.Further, the use of macrostructural strategies illustrated in the CI model was crucial to summarizing and generating macropropositions to represent main statements or topic sentences in a text.Even reading teachers who rated this section of The Monograph somewhat lower than teacher educators in open-ended responses agreed that CI model was highly relevant to and appropriate for the investigation of reading teachers' metacognitive knowledge of reading strategies instruction.
Second, in terms of integrating the general MKF presented in The Monograph into the contexts of ESL/EFL reading strategies instruction (Chapter Four), reviewers thought that it was essential to reconstruct a more effective reading strategies instruction approach on the basis of current approaches to reading strategies instruction, such as those proposed for particular groups of students respectively by Pressley (1990), Almasi (2003), and Chamot and O'Malley (1994) rather than introduce a new model de novo and built everything around that model.The model of reading strategy instruction synthesized by the first author of this article includes:  The CALLA's framework of a five-stage strategy instruction procedure: preparation, presentation, practice, evaluation, and expansion (Chamot & O'Malley, 1996),  Seven cognitive instructional strategies respectively in CALLA's each stage of strategies instruction: 1) activating background knowledge for the preparation stage, 2) explaining and modeling for the presentation stage, 3) guiding practice, monitoring progress, and providing effective feedback for the practice stage, 4) evaluating for the evaluation stage, and 5) encouraging transfer for the expansion stage, and  Three general principles that teachers should follow: 1) Organize students' active involvement, 2) reduce processing demands when needed, and 3) follow the above key components of the five-stage instruction and principles in a recursive fashion to help create a "safe and risk-free environment that supports and facilitates motivated strategy use" (Almasi, 2003: p. 50).
The reviewers thought that this synthesized model of reading strategy instruction was very comprehensive, versatile, and instructive, and provided key components and principles for teaching reading strategies effectively.
Third, with regard to the more specific elaboration of the task and strategy variables of metacognitive knowledge for teaching summarizing strategies with expository text (Chapter Five to Seven), all reviewers were of the same opinion that to teach summarizing strategies effectively, reading teachers should have relevant metacognitive knowledge about the following: 1) What is summarizing?Both panels contended that the definition of summarizing synthesized by the first author of this article was very complete and coherent.Summarizing was thus considered the ability to construct from one's reading and understanding of the gist of a text an appropriate summarizing view that conveyed the important information for a particular reading purpose to different degrees ranging from disclosing the author's intention to fulfilling the reader's own goals and interests, to goals assigned to the reader by external sources or demands.
2) What does summarizing strategies instruction with an expository text entail?
With the above understanding of the concept of summarizing, reading teachers should know several important key constructs of summarizing strategies instruction such as teaching student how to identify important information and how to operate the three macrorules-deletion rule, generalization rule, and construction rule (van Dijk, 1980;Kintsch & van Dijk, 1978;van W. XU ET AL.Dijk & Kintsch, 1983)-at two levels (i.e., microstructure level and macrostructure level).Three major sub-teaching tasks should then be considered: a) teaching how to specify the type of summary to be composed, b) teaching how to identify important information in the text relative to the nature of the expository text being studied, and c) teaching how to generate the gist of a text via macrorules.Each major sub-teaching task can further be decomposed into certain minor sub-teaching tasks (discussed in detail in The Monograph).In addition to these key constructs, reading teachers should also consider such general teaching demands as the influence of students' background knowledge and comprehension proficiency, the different cognitive demands of summarizing strategies, students' awareness of expository text structures, the difficulties caused by different expository text structures for students of different comprehension proficiencies, different readers' concepts of importance, and their abilities and skills in identifying important information from a text.All these latter factors, it should be noted, involve the person variables of metacognitive knowledge, which, as stated previously, are too complex to be explored in this study.
3) What cognitive and metacognitive instructional strategies can be applied in summarizing strategies instruction?
An agreement was reached that reading teachers should make deliberate decisions on using the aforementioned seven cognitive instructional strategies (detailed in Chapter Two of The Monograph) to accomplish a particular teaching task, and adjust their teaching strategies in time to match students' development.Moreover, all reviewers agreed that the higher-level of metacognitive strategies at both the domain-specific and general level should be the part and parcel of effective teaching.General metacognitive instructional strategies, such as planning, monitoring, and evaluating, are ubiquitous in any metacognitive-like processes.However, it is difficult to propose a standard list of domain specific metacognitive strategies in a particular cognitive process, such as teaching summarizing strategies with expository text, because the adoption of any domainspecific metacognitive strategies depends on different teachers' cognition through years of teaching experience, their familiarity with different teaching tasks and different students' motivation and emotion as well as some other relevant factors.
Overall, the total combined mean ratings indicated that the teacher educators (professors) were much more positive about The Monograph than the reading teachers practitioners with the exception of Chapter Five (Constructs and Teaching Demands) and Chapter Six (The Sub-teaching Tasks).This result was not particularly surprising (but was validating) since Chapter Five and Chapter Six met the reading teachers' expectations for specific analyses of instructional issues related to their daily teaching practice.This particular difference indicates that the three teacher educators were able to see the big picture and understand the value of the theoretical background research to the creation of a scholarly educational document.This outcome may be due to the factors that all teacher educators have been trained in doctoral degree programs in either linguistics or comparative literature and all have comprehensive experience in pre-service teacher education.By contrast, the reading teacher practitioners tended to look more at the details of the usefulness and applicability of the elaboration to their day-byday teaching issues.They expressed such expectations to the senior author of this article from time to time in phone interviews.However, all practitioner and academic reviewers ex-pected to see more practical examples such as the one illustrated in Chapter Eight as to the application of teachers' metacognitive knowledge before, during, and after teaching and they were quite pleased about the many illustrative examples in the Monograph.
To sum up, the above comprehensive analyses in terms of structured format ratings and open-ended responses supports the conclusion that the general MKF was successfully generated and elaborated from the perspective of these two panels of reviewers though the two panels somewhat disagree with each other in their evaluation of The Monograph, both chapter by chapter and as a whole, due to their different teaching experience, academic expertise, and research interest.Such disagreements are not markedly different from their agreements.

Conclusion
One can conclude that this monograph validation study, which was Phase II of the instructional material development model used to guide the development of The Monograph, was a reasonable success.The various analyses done of the detailed structured format ratings and open-ended responses of the academic and practitioner expert panels strongly supported the adequacy, quality and validity of the monograph developed using the instructional materials development model used.The various kinds of feedback provided by the different kinds of reviewers both structured and open-ended also helped guide the revisions of The Monograph (Phase III) after the validation study was completed and analyzed.Both Phase I and Phase II of this work, as well as the revisions done to The Monograph as a results of the independent expert feedback obtained from the two panels in Phase II reported here, give new and very nuanced meanings and conceptual frameworks for the current much used phrase of "evidence-based practice," as The Monograph reflects and is evidence-based and evidence-driven from the primary research and scholarly literature to the rich and nuanced knowledge and evidence of various kinds and types of practitioners who also provide research evidence and even discoveries when they participate in formal reviewer panels such as those used in this study.
The panel validation process used in this study was a successful implementation of an adapted convergent and discriminant validation design for assessing the quality of developed instruction materials, which generated a variety of different kinds of information and results that were externally and ecologically validating, informative and useful in revisions.The results also generated a better understanding of the instructional materials development process and particularly for bi-lingual and bi-cultural instructional materials.The agreement among the bilingual and bicultural formal reviewers indicated that the two different groups of reviewers had "convergent" (similar) views and opinions about The Monograph, and this convergence of views provided strong evidence for the logical validity and theoretical or construct validity of The Monograph.The disagreement between the two review panels revealed how teaching experience, academic expertise, and research interest can influence reviewers' assessments of each chapter of a work (The Monograph) and the work (The Monograph) as a whole to some degree but a degree that does not deviant markedly from the consensus view.The reviews of reviewers, therefore, are not completely objective and absolute, or completely the reverse, and must be contextualized to some degree to be appro-W.XU ET AL.
priately interpreted, which is a metacognition if there was ever one which should not be forgotten!The pattern of findings in the present study, as previously stated, cross-validate those of several other recent studies that used the instructional materials development model used in this study (i.e., Flores, 2005;Erikson, 2006;Kwong, 2008) not only strengthening the findings of all of these studies and validating the instructional materials development model used further, but in the case of this study extending these finding to a bilingual and bi-cultural context and international audience.As previously stated, this finding is a very hopeful outcome and particularly so for cross-cultural scholarship, research, instructional materials development and instruction itself if all of the aforementioned as model and theory driven as in the case of this study and the similar studies on which it built.
The validation process used in this study, it should be noted, also provided the authors of this article and all the reviewers involved in this validation process and phase of The Monograph's development a workable platform to openly discuss, criticize, evaluate, and reflect on The Monograph and metacognition and the metacognitive knowledge needed for summarizing strategies instruction with expository text to EFL undergraduates.This type of interactional among communities of professionals is a form of professional development and professional education and a type of interdisciplinary scholarship of its own between academics and practitioners that needs to be encouraged and further developed for both better academic understanding and better pedagogy and pedagogical approaches and instructional materials.Both academics and practitioners have much to learn and gain from each other that is of value to their work and thus makes better overlapping halves of a whole.Such types of educational communication and communities are easier and more feasible today with the rapidly emerging social media technology available, together with models of the cognitive apprenticeship that are becoming increasing more popular approaches to instructional and professional development in all professional areas currently.Such types of educational communication and communities are still not widely employed, considering the teacher preparation/training programs going on in both the USA and China, which do not include metacognition and developing teachers' understanding of metacognition of their own or their students or how it works in instruction.The instructional materials and educational works development model presented in this article strongly "blends with" and enables these emerging approaches improving all those who participate in various ways.
One of the major findings of this research work, which was a surprising unanticipated findings to us, was how fuzzy, jumbled and contradictory the area and differing views of metacognition are today, and particularly as it applies to instructional and actual pedagogy and pedagogical strategies for implementing various exhortations from various academics and researchers to teach reading and language metacognitively and to develop the metacognitively sophisticated reader (Hartman, 2001;Pressley, 2002).Key aspects of the fuzzy, jumble and contradictory views that are present into the literature were summarized at the beginning of this article (and are presented in more detailed in The Monograph) but if there is one thing that the instructional materials developed model used to guide this work did and did extraordinarily well, it was to both identify and expose this problem of how fuzzy, jumble and contradictory the area of metacognition was relative to developing cogent and compre-hensive instructional materials about it.More subjective and narrative forms of instructional materials development, which are the more typical approach, would not have brought all of these problems to light as they have not for more than twenty years, and also would not have necessitated that these problems be reconciled and resolved and so in a manner that was considered to be adequate and acceptable to an independent panel of expert reviewers before the work developed to the point of completed instructional materials ready for independent expert review.Such a process and formal approach is not how instructional materials are typically developed and particularly so by academics and practitioners as opposed to publishing houses.This is the great benefits of the guiding model used.In fact, it was the MFK that was revised several times before the writing of The Monograph commenced and then several more times as the writing progressed.That was one of the most notable aspects and events of this study.This very same point held relative to the task of trying to analyze and unravel the different kinds of metacognitive knowledge that were needed to learn and teach summarizing strategies for expository texts, which also underwent numerous revisions both before and as the writing of The Monograph progressed.Therefore, every instruction and instructional materials developer needs a guiding model for carrying out such work as well as a community of peers and colleagues to help in such work and the production of such products.Every developer needs to both know and understand as well that they are engaged in a new form of scholarship that is not well-recognized or well developed yet but needs to be if we are to better understand how theory and research get translated into successful practice and successful instructional materials, particularly relative to teaching metacognitively and developing metacognitive readers.

Figure 1 .
Figure 1.A simplified version of Carifio-Perla Model (2011) for instructional materials development.CHQKB = Critical and high quality knowledge Base; ARC = Appropriate representations and communications; VFTE = Validated and field-tested for effectiveness.See original figures in Perla and Carifio (2011).

Table 1 .
A Two-dimensional Matrix of Metacognitive Knowledge Framework (MKF).

Table 5 .
Mean and weighted mean combined responses made by the academic teacher educators (n = 3) and the practitioner in-service reading teachers (n = 3) on the seven major evaluation criteria on a one to six point responses scale.