What in the Journal of the Learning Sciences : Evidence for Possible Biases

A content analysis of the Journal of the Learning Sciences (JLS) was conducted for a 5-year period. Article topics, methodology, the names of the contributing authors, their academic ranks, affiliations, and geographical locations were coded to reveal trends in publication patterns. A questionnaire was sent to all authors in the relevant period to find out how they selected journals for submitting manuscripts. It was found that the active involvement of the JLS’s editor in the journal’s orientation correlated with the great importance that the JLS’s contributors gave to editorial considerations when selecting journals. A possible geographical bias was identified, and possible solutions were discussed.


Introduction
The aim of the present study is to answer the question: What sorts of articles were accepted for publication in the Journal of the Learning Sciences?Two strategies may be followed in trying to answer a question of this kind.First, one can simply take a look at what articles actually made it to publication-What kind of articles were recently published in the JLS, in terms of methodology and topic?Who were the contributing authors, and to what institutions were they affiliated?Second, one may try learning from what the authors themselves have to say: How are they different from those of other journals with respect to publishing practices?What do they think are the important aspects of a journal?The former strategy can shed light on patterns discerned by following the latter strategy: as we shall see, for example, the extra importance that JLS's authors attached to the editor of a journal is partly explained by the active involvement of the JLS's editor in the journal's orientation.
Both of these strategies were adopted in the present study.First, a content analysis was conducted of the 5 volumes of the JLS from 2000 to 2004 to reveal trends and patterns of publication.Second, a questionnaire was sent to all first authors of the JLS in the period under study in order to obtain information about their practice in selecting journals for publication.Details of the implementing of these strategies are presented below.

Methodology Content Analysis
As part of a larger project, a retrospective review was conducted of all issues of the Journal of the Learning Sciences (JLS) and of another four leading journals in the field of education from 2000 to 2004.The data from the five journals were then combined and compared in a separate article to determine publication patterns in the broad field of education and among its specialised areas.The rationale for choosing these journals was that all of these are prestigious educational journals within the discipline of education in general and their specific specialty areas in particular, with high impact factors as calculated by the ISI Social Science Citation Index.The JLS was selected for in-depth study in the present paper.Only substantial articles were included; editors' introductions, notes to contributors, news, book reviews, and any other publications which were not peer-reviewed were excluded since these are generally not reflective of a journal's publishing trends and patterns.
The articles were coded according to the following criteria: the name of the first author, his or her nationality and institutional affiliation, the world rankings of his or her institution (according to the November 5, 2004 issue of the Times Higher Education Supplement), the country in which the research was conducted, article type/methodology, and the topic of research.Partly following English et al. (2005: p. 15), the types of article were divided into two main categories.The first category was theoretical, which was further divided into purely theoretical articles (including discussions about research methodology and theoretical models and frameworks) and review articles (including theoretical, methodological and historical reviews, among others, but excluding book reviews).The second major category was empirical or research-oriented, which was further divided into quantitative studies (involving mainly data derived from surveys or statistical analysis), qualitative studies (employing mainly qualitative or ethnographic methods, such as observation and interview), and those that combined both.
Because no Guideline for Contributors to the JLS could be identified on the Journal's website, specifying what topics it wished to include, the content of all the articles was used as the basis for establishing the categories of topics.This resulted in the following list of major topics: Science Learning, Mathematics Learning, Online Learning, Collaborative Learning, Scaffolding, Representations, Educational Standards, and Methodology.Some blurring of categorical boundaries occurred between Collaborative Learning and Online Learning (for example, an article by Mark Guzdial (2001) concerns the use of Website to promote collaborative learning).In general, how-ever, there were no substantial problems in subsuming an article under one of the foregoing categories.
Since none of the contributing authors was affiliated to an institution outside the USA, research articles were deemed to originate in countries where the research was carried out.With respect to theoretical articles, on the other hand, the countries of origin were considered to be those of the authors' institution.This special treatment of theoretical and review articles was due to the fact that the content of such articles was often marked by either a lack of, or a blurring of, national boundaries.

Survey
To determine whether and to what degree the patterns that were discovered in the content analysis reflected an author's reasoning as to his or her choice of journals, a questionnaire was developed that asked respondents to indicate the importance of a list of 24 factors in their selection of a publication outlet for a typical manuscript submission, employing a fivepoint scale with "1" representing "unimportant" and "5" representing "most important".The questionnaire was emailed in March 13, 2006 to each of the 60 authors who contributed to the JLS in the period concerned, with a covering letter that briefly explained the study and promised respondents anonymity.After approximately one month, a follow-up e-mail reminder was sent to non-respondents of the first wave; and on June 5, 2006, a third e-mail was sent to non-respondents of each wave.The three contacts yielded a total of 26 responses; a 43.3% response rate.

Results
During the five years under study, the JLS published a total of 66 articles (Vol.9-13).As shown in Table 1, except for the year 2003, there was a gradual increase in the number of articles per volume, with volume 9 containing 12 articles and volume 13 containing 19 articles.Except for the 2001 volume (where issue 1 and issue 2 are combined), the JLS was successful in consistently publishing four issues per volume.

Topics
Table 2 presents the frequency and percentage of the articles that are classified in the major categories for the years 2000 through 2004.As seen in Table 2, the largest group of articles (16.7%) deals with broad issues in science learning (specific issues for the different disciplines of science are not addressed).The categories of methodology and collaborative learning each contain the second largest group of articles (13.6%).Together these three categories account for 55.9% of the articles.Following these are the categories of mathematics learning (12.1%), online learning (10.6%), scaffolding (9.1%), the role of representations in learning (6.1%), and educational standards (6.1%).
No clear trends can be discerned in each category across the five years under study.This may be partly due to the fact that the JLS contains a relatively high proportion of thematic issues (including issue 4 of volume 9, issue 1/2 of volume 10, issue 1, 2, and 3 of volume 13), which is largely a matter of editorial arrangement and assemblage, and thus does not in general constitute a proper measure of current research interests.So, for example, articles addressing methodological issues were present only in issues 1 and 2 of volume 10 and issue 1 of volume 13.Similarly, articles on scaffolding appeared only in issue 3 of volume 13, while issue 2 of volume 13 was exclusively devoted to educational standards.Although the category of science learning, with at least one article published every year, reflects a fairly stable research interest, the large proportion of thematic issues together with the relatively short period covered by the present study make it difficult to identify clear topical trends in the JLS.
The editor's influence on the topics addressed is evident if we take a look at the editorial statements of the JLS.In a 2001 issue, editor Kolodner wrote: I have made it a point, as editor of JLS, to make methodology a central part of what we publish (Kolodner, 2001: p. 2).
Several paragraphs down, she anticipates themes to be covered in future volumes: [I] found [that] a whole host of other important issues have reached a new level of sophistication, and I hope we will be The journal is focusing on publishing three types of articles in addition to other regular sections.One type of articles related to the methodology of design studies and design experiments were published in Volume 13 of the journal.Another type of articles related to promotion of complex systems of learning has been published in this issue.Another type related to new ways of thinking about transfer will be published in Volume 15, Number 4 of the journal (Kolodner, 2006: p. 1).
Each of the themes mentioned in the forgoing paragraphs is covered by at least one subsequent special issue devoted to it.As mentioned, scaffolding is the theme of issue 3 of volume 13 of JLS; methodology in general is addressed in a special issue in 2001, while design research in particular is dealt with in a special issue in 2004.As is clear from the last quote above, other proposed themes have had their corresponding special issues as well.

Methodology
According to English et al. (2005: p. 15), approximately 60% of the research in most education journals is empirically based, which led them to declare that the high percentage (87.1%) of theoretical articles they had found in religious education literature was "somewhat problematic" (p.15).Related to this, the low percentage of articles in the literature on higher education that made explicit use of theory (25.6%) led Tight (2004: p. 409) to conclude that "there is a need for more theoretical engagement so that the field [of higher education] can develop further, and gain more credibility and respect." It may seem that, by English's standards, there were a disproportionately large number of empirical articles (83.3%) in the JLS, (see Table 3).However, it should be noted that, as editor Kolodner (2000: p. 2) notes in an editorial, "the [JLS] is distinguished from some of the other education and educational technology journals by the theoretical basis of the work it publishes"-namely, theories derived from psychology, cognitive sciences and computer sciences.Consequently, it is not sur-prising that the JLS should have a heavy empirical basis.

Authors
Table 4 lists and ranks the leading contributors to the JLS for the period under study.A total of 5 authors published more than once as first author in the JLS from 2000 to 2004.Interestingly, all five authors are associated with the JLS: one (Sasha Barab) is an associate editor of the journal, and the other four all belong to its editorial board.The majority of authors (n = 55, 83.3%) published only one article as first author in the relevant period, four of whom were second or third authors of other articles.

Academic Rank
As can be seen in Table 5, the vast majority of contributors to JLS are academics (n = 59, 98.3%).The different academic ranks are quite evenly distributed, with the percentage of full professors (33.3%) being the highest among the five journals under study (the overall percentage of full professor is 27.2%).
There was a clear increase in the proportion of higher-ranked authors over the period in question.In the year 2000, the proportion of full professors was zero, compared to that of assistant professors/PhDs, which was 60%, but by 2004, the proportion of full professors had risen to 36.8%, while that of assistant professors/PhDs had dropped to 26.3%.Assuming that a high correlation holds between tenure and academic rank (where full and associate professors are mostly tenured, whereas assistant professors and PhDs are mostly not tenured), the proportion of tenured authors rose from 40% in 2000 to 73.7% in 2004.These data point to a clear rise in the quality of the journal's authors, and also to the increasing difficulty for younger authors to publish in the JLS.
It may be asked whether there had been any academic promotions for authors during the five years under study, given that their employment positions were obtained through their personal websites around February, 2006, and that for most authors a considerable period in his or her career lifecycle had elapsed.Although this can be taken as a general caveat against assuming any inference from an authors' current academic rank to the general quality of an author's article for a journal over an extended period, it is curiously quite unimportant in the present case.The interesting fact is that none of the authors of the original 2000 volume of the JLS had been promoted to full professor even by the time of our study.It appears, then, that there is a genuine increase in the quality of the contributors to the JLS.

Institutional Contributions
Table 6 lists the institutions in order of their number of contributions to the JLS for each of the five years under study.A total of 12 institutions contributed at least twice to the JLS, 7 of whom contributed at least 3 times.In 2000, only four institutions met the 2-article criterion, accounting for 50 percent of total articles published in that year, while in 2004, 7 institutions met the criterion, representing 63.2 percent of all articles published in that year.Taken together, these 12 institutions represent 57.6 (n = 38) percent of the total number of articles published for the period from 2000 to 2004.Some of these institutions contributed to the JLS mostly in more recent years (e.g., the University of Michigan and Northwestern University), whereas others had virtually no publication except in the earlier period (e.g., Massachusetts Institute of Technology and Tufts University).
As seen in Table 7, a clear majority (74.2%, n = 49) of articles were contributed by top-200 institutions as ranked by Times Higher Education Supplement.Among these, 22 articles (n = 33.3%)were authored by those working for institutions in the top-50.Although no clear pattern over the period from 2000 to 2004 emerges from Table 7, the proportion of authors affiliated to institutions not in the top-200 was the lowest (15.8%) in 2004.

Geographical Locations
Not surprisingly, the vast majority (86.4%, n = 57) of articles were written by US authors and/or about research conducted in the US Contributions from Canada and UK included, the socalled "Anglosphere" accounts for 92.4% (n = 61) of all the articles.Although the more recent years saw some engagement by Israeli authors, the international representation in the JLS in 2004 (5.3%, n = 1) was much closer to the original position in 2000 (0.0%, n = 0) than to the intervening years of 2001 (16.7%, n = 2), 2002 (30.8%, n = 3), and 2003 (20%, n = 2).Is the low international representation much of a problem?It is worth noting that the JLS is published by the International Society of the Learning Sciences which "provides unprecedented opportunities for collegial interaction across national boundaries" (http://www.isls.org).Arguably, a journal's aspirations to international status cannot be fulfilled if over 85 percent of the articles in that journal are written by US authors and over 90% of the articles are written by authors from English-speaking countries.In the case of JLS during 2000 to 2004, authors from Asia, Africa, and Latin America were not represented at all; nor were authors from Northern Europe and Australia (both of which enjoy a high level of research capacity and a large number able researchers).Except for France, none of the countries on the European continent made contributions to the JLS in the relevant period.These facts suggest, it seems, that the JLS is not so international after all and is affected by a geographical/cultural bias (Table 8).

Criteria for Selecting Journals
Are the patterns that emerged in the foregoing content analysis of the JLS reflected in an author's actual practice when selecting journals for publication?To answer this question, it is crucial to look at what the authors themselves have to say.To be sure, there are two limitations to this strategy in the present case: to begin with, the ratings of journal selection considerations presented in Table 9 were based on a sample of 26 respondents out of a population of 60 (a response rate of 43.33%), which may involve some form of respondent bias.Moreover, respondents were asked what their routine practices were when selecting journals, and not why they had actually selected the JLS when trying to get one of their articles published.However, because, as we shall see, there actually are some significant differences between the journal selection practices of authors of the JLS and those of the other four journals surveyed, an interpretation which allows us to make more meaningful sense out of the original data would be quite in order.
As can be seen in Table 9, contributors to the JLS attached a significantly greater importance (p = 0.039) to how their articles coincided with the journal's foci when considering which journal to submit them to than did authors of the other four journals surveyed (In fact, as seen in Table 9, this item is rated as the most important consideration in JLS respondents' choice of journals).This agrees quite well with the fact, noted above in the section "Topics & Methodologies", that the editor of JLS is quite insistent on the journal's having clear foci and no less explicit about what those foci are-i.e., methodology (especially design research), scaffolding, complex systems, etc.The logic is quite obvious: if an editor has been vocal about his or her journal's foci, then one had better conform if one submits to the journal.
The importance of the JLS's editor also seems to correlate with other statistically significant differences.JLS authors rated the quality of editor (m = 3.58, p = 0.001) and personal knowledge of the editor (m = 3.04, p = 0.003) as significantly more important than did authors of the other journals.JLS authors also rated knowledge of an editor's intellectual interests (m = 2.88, p = 0.169) as more important than did the authors of other journals'.Although it is not easy to see how each of these differences can individually be explained in terms of an editor's actively orienting the journal, a general explanation is that when one is routinely presented with an active editor, other considerations about the editor-such as the editor's professional competence, intellectual interests, and personal qualities-become more salient.It remains unclear whether, for example, the fact that the authors of/contributors to the JLS placing more emphasis on personal knowledge of the editor was partly due to a lack of, or an abundance of, knowledge of the editor of the JLS.
Contributors to the JLS also paid significantly less attention than did authors of the other journals to the online accessibility

Table 4 .
Leading contributors to the JLS.

Table 5 .
Academic Ranks of contributors to the JLS.

Table 6 .
Leading institutional contributors to the JLS.

Table 7 .
Contributing institutions in order of overall rankings.

Table 8 .
The country of origin of articles.