What is “Good” Research? A Homeschool Researcher Responds to Harvard Professor’s Criticism: Research Methods 101

Brian D. Ray, Ph.D.*

The purpose of this paper is to place in context and evaluate the efforts of many of those who negatively criticize homeschooling, homeschool researchers, or particular research studies on parent-led home-based education. What is “good research” and how should one understand negative criticisms of homeschool research?

Elizabeth Bartholet’s recent criticism of homeschool research and researchers, which originally appeared in the Arizona Law Review, will serve as a point of contextualization for this article.[i ] Professor Bartholet called for a “presumptive ban” on homeschooling in the United States, such that parents would have to petition the civil government to get permission to home educate any child. Under her regime, the government would become the controller of all children’s educational lives, and parents would need government approval for their children to be allowed to have something different than an institutional state-run education. A significant portion of Professor Bartholet’s argument involved negative criticism of much homeschool research and the scholars who have conducted the studies. I will address Professor Bartholet’s claims in the second portion of this piece.

What is “Good” Research? Research Methods 101

Social science research (e.g., education research) almost never sees “the perfect study.” While teaching research methods at the university graduate and undergraduate levels across seven years, I explained to my students that many studies in education are exploratory, descriptive, and correlational in design.[ii] Some studies include designs that allow for more conclusions that address cause and effect. But researchers are not allowed (morally or legally) to execute a cause-and-effect design like this:randomly assigning one-third of 1,000 students in New Orleans to state or public schooling, one-third to private institutional schooling, and one-third to private homeschooling, letting things roll for 13 years to see how all the students “come out in the end” on academic achievement or amount of abuse experienced at the hands of adults and peers, then making a bunch of causal claims. Researchers are not allowed, by law or ethics, to force parents to put their children into these three types of schooling (i.e., educational treatment) and so may not conduct this type of study.

37 Years Ago

When I began doing research—about 37 years ago—on homeschool families, students, and groups, other scholars and I had a lot of questions. I wanted to know why parents chose homeschooling. What were their demographics? And how were the children performing academically?

When I first set out to measure homeschool educational attainment back in 1985, there were several challenges. How do I study a “hard-to-reach population?” What is a way to gain participation when there is no master list from which to randomly select families or students? How do I ethically engage their participation? How can I statistically control for background demographic traits so that reasonable comparisons can be made to public school student norms? How will I get test scores when there is no research budget for new testing? How can I do this on a shoestring of a budget and with very little manpower to assist me? (And especially when thousands of old-fashioned, multipage survey questionnaires would be going out and coming back via the US mail and all the data would be entered by hand, one question and click at a time, into a spreadsheet.)

As in any study, there were problems and challenges to executing my studies the ideal way. With hard-to-reach populations, there are typically even more obstacles. To learn something rather than nothing, researchers must come up with solutions and compromise between the ideal approach and the realistic or practical approach. I had to do that; just as most other social scientists have had to do. I had a lot of conversations and brainstorms with other scholars, leaders of homeschool organizations, academic achievement testing services, government officials, and homeschool families. It was and is complicated. It tries the mind. It pits the curious scientist against reality while trying to get questions answered. It was exhausting at times. It was thrilling at times. I thoroughly enjoyed it (and still do).

The Perfect Study

It is highly unlikely that an investigator will ever get to do a “random-assignment true experiment” regarding the “effects” of schooling type (e.g., public institutional school compared to private institutional school compared to homeschooling). Research opportunities and life are just not that way. Research scholars must work within budgetary, timeline, and ethical constraints to design and execute the best study that they can to meet their objectives. So, for example, since a scholar may not do a “true experiment,” the alternative might be some kind of “quasi-experimental” or non-experimental design, such as a matched-pair study (or a retrospective, explanatory study; or a cross-sectional, predictive study; or a cross-sectional, explanatory study).[iii]

Every study related to homeschooling is based on a particular research design, with an underlying theoretical framework; and each specific study has limitations and delimitations, based on its methods (which are often based on the aforementioned factors: budgetary, timeline, and ethical constraints), regarding what it can and cannot reveal and to what extent its findings can inform understanding and policy considerations. Every researcher should clearly state the study’s purpose and objectives; lay out a review of related literature; present a theoretical framework and explain methods; and, finally, present the findings and analysis. A researcher should also note the limitations of his study and provide a discussion of conclusions and interpretations. The conclusions should follow from and be coherent with the study’s methods and limitations.[iv]

The study is sound research if the researcher takes all these steps. If things are clearly stated so that they are conceivably reproducible and the conclusions that the scholar states reasonably follow from the methods, findings, and limitations, then it is sound, beneficial, or “good” research.

It is a bogus approach for any observer, evaluator, or critic to say that a research study is “bad” or “flawed” because it (a) does not meet all of the objectives that the critic wished it had included or met, (b) cannot lead to causal statements, or (c) led the investigator to suggest policies or ideas that the critic does not like. If a researcher is clear about the study’s theoretical framework, methods, conclusions, and limitations, and her conclusions and recommendations generally fit the findings and overall framework, then it is a decent study and people can learn from it. It can inform the field of inquiry.

Big Picture of Some Homeschool Research

Overall, research has found many positive things connected to homeschool students and families. There are many ways to do a review of research and, depending on the reviewer’s worldview and what message the reviewer wants to convey to the reader, the reviewer can easily slant things and affect the reader’s understanding of what the body of research says. There are several reviews of research on homeschooling, and a careful reading of each of them might reveal the author’s biases, whether slight or strong.

Reviews of research can be very helpful because they help us see the big picture. They can show us trends. They can indicate overall findings regarding a particular phenomenon. They can help us not get stuck on looking at an individual tree or two and miss the significance and beauty of the forest.

One unique review is the one that I did. It has been published in a peer-reviewed journal. Before I get to that, however, I would like to take a short side trail about the meaning of peer review.

Peer review

In academic publishing, the goal of peer review is to assess the quality of articles submitted for publication in a scholarly journal. Before an article is deemed appropriate to be published in a peer-reviewed journal, it must undergo the following process:

The author of the article must submit it to the journal editor, who forwards the article to experts in the field. Because the reviewers specialize in the same scholarly area as the author, they are considered the author’s peers (hence “peer review”).
Typically, the reviewers do not know the identity of the author and the author does not know the identity of the reviewers.
These impartial reviewers are charged with carefully evaluating the quality of the submitted manuscript.
The peer reviewers check the manuscript for accuracy and assess the validity of the research methodology and procedures.
If appropriate, they suggest revisions. If they find the article lacking in scholarly validity and rigor, they reject it.

Because a peer-reviewed journal will not publish articles that fail to meet the standards established for a given discipline, peer-reviewed articles that are accepted for publication generally exemplify the best research practices in a field.

The Arizona Law Review, in which Professor Bartholet published her criticism of homeschool research and researchers, is apparently not a peer-reviewed scholarly journal. “Founded in 1959, the Arizona Law Review is a general-interest academic legal journal. The Review is edited and published quarterly by students of the University of Arizona James E. Rogers College of Law.”[v] It is widely understood that law reviews are not considered peer-reviewed scholarship, and “law reviews are considered secondary scholarship because the articles are providing analyses and commentary on legal issues.”[vi] There is much debate about the integrity and value of law reviews.[vii]

My review in the peer-reviewed journal is a review of only and all peer-reviewed homeschool studies on the learner outcomes of academic achievement, social development, and success in adulthood. The article also gives the demographic characteristics of the US homeschooling population, reasons that parents choose to homeschool, and proposals for future research on parent-led home-based education. Here it is, open access:

Ray, Brian D. (2017b). A systematic review of the empirical research on selected aspects of homeschooling as a school choice. Journal of School Choice: International Research and Reform, 11(4), 604–621. https://doi.org/10.1080/15582159.2017.1395638

This review includes the peer-reviewed published research of roughly 45 scholars spanning about 30 years. Four of the studies were based on representative national samples in the United States. At least one involved a careful matched-pair design.[viii] Other studies control for various background variables while examining learner outcomes. The researchers live all over the United States and in other countries. Clearly, there is a breadth and depth of scholarly work and scholars investigating homeschooling.

What was uncovered in this peer-reviewed review of peer-reviewed research? In 11 of the 14 peer-reviewed studies (78%) on academic achievement, homeschool students performed significantly higher than institutional school students.

Out of 15 studies on social and emotional development, 13 of them (87%) showed clearly significant positive outcomes for the homeschooled students compared to those in conventional schools.

And 11 of the 16 studies (69%) on success into adulthood and college showed positive outcomes for the homeschooled compared to those in conventional schools.

Overall, 35 of the 45 peer-reviewed studies (78%) found that the homeschooled students or graduates of parent-led home-based education performed significantly better than their conventional or institutional school peers in terms of academic achievement, social and emotional development, and success into adulthood (including at college or university).

That is, 30 years of research by about 45 scholars in peer-reviewed journals has revealed significantly positive things related to homeschooling. There are very few pieces of empirical evidence from studies indicating that homeschool students or graduates of homeschooling are not doing as well as others; there is no body of research showing that they are being harmed at any higher rate than others. A helpful review of research for those who want to go deeper and that covers much more than only peer-reviewed publications is one that I published in a peer-reviewed journal, also in 2017.[ix]

Advocates and Advocacy

Some scholars will write or make claims about “advocacy research” or “advocates” of this or that. The first thing to keep in mind is that being an advocate of concepts, initiatives, or values is a core part of the human condition. Everyone is an advocate of something. One scholar might be an advocate of reducing child abuse. Another might be an advocate of Roman Catholic schooling. A third could be an advocate of seeing and understanding the world through the lens of critical theory, or of neo-Marxism. And a fourth might be an advocate of conventional government-run schools as the best educational format for US society.

People have asked me whether I am an advocate of homeschooling. My response is that because of my worldview and experience, I am an advocate of what helps children learn and enjoy learning so that they become adults who are literate and numerate, kind to their neighbors, and able and inclined to work hard and support themselves and their families and honor their Creator. I am a strong advocate of education in the classic sense.[x] I have been a state-certified teacher, have taught in private and public schools, and have taught teachers as a former professor of education. Because of all this, I am an advocate of much dialogue and interchange between adults and children. I advocate for customizing pedagogical approaches and curriculum for each child, according to their strengths, weaknesses, dreams, and desires. Mentoring and one-on-one instruction are great for children (and adults). Much opportunity for positive social interactions, cooperation, and age integration are good for children and adolescents. More parental involvement, rather than less, is good. Children having time to know and enjoy peers, but being adult-oriented rather than peer-dependent is good.

To the extent that any of the aforementioned conditions can or are likely to be met in homeschooling, government schooling, or private institutional schooling, I am an advocate for those types of schooling.

Professor Bartholet’s and Others’ Criticisms of Ray Are Criticisms of Many Scholars

Despite all of the aforementioned on what is sound or good research, some critics of homeschool research and some opponents of parent-led home-based education (or homeschooling, unschooling, or home education) in general are unfairly critical of specific studies or specific researchers. Dr. Elizabeth Bartholet (2020) is a fit example of such a critic.

It is easy to criticize a researcher or their study, on the other hand; it takes knowledge, skill, work, and time to thoughtfully evaluate what a study legitimately contributes to a body of inquiry. It takes time and skill to go to the original sources and accurately represent or evaluate them.

Professor Bartholet’s Criticisms

The main purpose of this paper is not to respond to one author’s criticism of me. However, a partial response to some of Professor Bartholet’s claims is in order, to set straight a few facts and to give an example of how certain criticisms are not based in sound methods of social-science research. Professor Bartholet is not the only scholar who has made unsound appraisals of homeschool researchers or their studies.

For the first example, Professor Bartholet writes, “Many of his [Ray’s] studies purport to show that homeschoolers’ academic performance is at least as strong as that of their public school peers”.[xi] My studies do not “purport to show,” they indeed find that homeschool students’ academic performance is typically stronger than the public-school average. Further, Professor Bartholet fails to mention that studies by many other scholars have also found that homeschool students’ academic performance is, on average, higher than that of their public-school peers .[xii] As do the other researchers who have found the same thing, I clearly explain my studies’ methods, limitations, and findings.[xiii] The other researchers’ and my studies, with their various nuances and limitations, add to scholars’, law professors’, and the public’s knowledge and understanding of homeschooling.

Second, Professor Bartholet cites another paper to claim that homeschool research is “generally ‘politically motivated’” and that one private organization, the Home School Legal Defense Association (HSLDA), predominantly funds the research.[xiv] The fact is that hundreds of studies, by over 100 scholars, have been done on homeschooling, its families, and its children. I am aware of no study that analyzes what percent of these scholars were “politically motivated.” I have personally met or interacted with many dozens of these researchers, and I am confident that most academics who investigate homeschooling would not fit Professor Bartholet’s claim of “politically motivated.” I am also confident that the hundreds of scholars who have conducted homeschool research would refute Professor Bartholet’s presentation about HSLDA and funding.

Perhaps the most obvious example of Professor Bartholet’s mishandling of social-science research is where she dives into the topic of child maltreatment. She spends many pages negatively critiquing research on homeschooling and implies that if a study does not include a known representative sample, then it is “debunked,” “not true social science,” or has “design flaws.” However, Professor Bartholet asserts “that homeschooling in its current unregulated form poses serious risks of abuse and neglect.”[xv] To support this claim, she cites one in-house report by a government agency that clearly does not use a representative sample. She then cites an “unpublished” study that is not based on a representative sample. The third report she cites is based on anecdotes and not a representative sample. Professor Bartholet finishes by citing a fourth study that has nothing to do with a representative sample or rates of maltreatment. She relies on studies that were not purposed to compare rates of abuse and maltreatment to argue that homeschool children are at risk. Professor Bartholet misapplies the purposes, methods, and findings of the reports she uses and clearly fails, according to her own standards about research, to support her claim—that homeschooling poses serious risks—with pertinent research evidence. Further, she cannot do so because such evidence does not exist.

A key focus of Professor Bartholet’s paper is to allude multiple times to the idea or claim that homeschooling, by definition, puts children at risk, more so than does parents placing their children in government public schools or institutional private schools.

Empirical scientist and distinguished University of Arkansas professor Patrick Wolf, , empirical scientist and University of Arkansas distinguished doctoral fellow Matthew Lee, , and empirical scientist and Johns Hopkins Institute for Education Policy senior research fellow Angela Watson, have studied and written about homeschooling along with other educational policy topics. They conclude the following about Professor Bartholet, and I agree:

Upon reviewing Professor Bartholet’s article, we conclude that it suffers from contradictions, factual errors, statements of stereotyping, and a failure seriously to consider that the alternative to homeschooling—public schooling—shares the problems that she attributes to home education.[xvi]

Professor Bartholet’s claims about abuse and homeschooling are not scientific, evidence-based claims. My experience as an expert witness in over 65 court cases across the United States and in Canada tells me that her approach would not pass muster in a court of law. This kind of claim without basis would not hold up in a scholarly peer-reviewed article. The limitations for which Professor Bartholet attempted to criticize other homeschool researchers’ and my studies are notably greater in the newspaper articles, blogs, surveys, and the one study that Professor Bartholet cites. “Professor Elizabeth Bartholet’s claims that homeschooling contributes significantly to the scourge of child abuse fail to survive scrutiny.”

The Real Point

Perhaps more important than what Professor Bartholet claims about me is that her negative criticisms of me are actually red herrings and amount to an overall negative criticism of all researchers who have found positive things related to homeschooling. As mentioned above, 45 scholars (as of about 2016) across the span of 30 years had published peer-reviewed research reports on three different homeschool learner outcomes. Dozens, if not hundreds, more professors, doctoral students, and think tank academics have conducted and published research that has found positive things related to homeschooling. Very few have found negative things. Professor Bartholet’s and other academics’ and commentators’ criticisms of me and my studies are really criticisms of nearly every scholar and study that has ever found something positive related to homeschooling.

Their charges against me are basically charges against all these scholars because most are working within the same genre of social science research. None of these studies on homeschooling that Professor Bartholet criticizes are “true experimental” in design, and relatively few involve guaranteed representative samples. However, the researchers do clearly lay out their purposes, methods, findings, limitations, and conclusions. They continue to add to the social-science body of research on homeschooling and inform and enrich our knowledge and understanding of parent-led home-based education.

Concluding Thoughts

A sound or respectable social science study and its report is one in which the investigator clearly states the study’s purpose, methods, and findings, and the interpretations and conclusions that the scholar makes reasonably follow from all that preceded them. A sound evaluation of a study considers all of this. Everyone who reads a study, or an evaluation of a study, should remember that every researcher and reviewer has a personal worldview that affects, biases, or slants what he or she does, writes, and says.

To date, the clear majority of research done on homeschooling finds positive things associated with the method of education. Consistently negative criticisms of various individual studies about homeschooling and homeschool parents and students or of a particular scholar who studies homeschooling are, in essence, a negative criticism of the large majority of homeschool researchers and their studies. The negative critics generally ignore or fail to recognize and explain what homeschool studies have legitimately contributed to the knowledge base and understanding of an important body of inquiry regarding a millennia-old and effective form of teaching and learning—homeschooling—in the United States and around the world.

References

American Psychological Association (2009). Publication manual of the American Psychological Association (6th ed.). Author.

Bartholet, Elizabeth. (2020). Homeschooling: Parent Rights Absolutism vs. Child Rights to Education & Protection, Arizona Law Review, 62(1). https://arizonalawreview.org/pdf/62-1/62arizlrev1.pdf

Gall, Meredith D., Gall, Joyce P., & Borg, Walter R. (2007). Educational research: An introduction (8th ed.). Pearson Education, Inc.

Johnson, Burke. (2001). Toward a new classification of nonexperimental quantitative research. Educational Researcher, 30(2), 3–13. https://doi.org/10.3102/0013189X030002003

Leedy, Paul D. (1993). Practical research: Planning and design (5th ed.). MacMillan Publishing Company.

National Education Association. (1990). The 1990–91 resolutions of the National Education Association. Author.

Posner, Richard A. (2004). Against the law reviews: Welcome to a world where inexperienced editors make articles about the wrong topics worse. Legal Affairs. https://www.legalaffairs.org/issues/November-December-2004/review_posner_novdec04.msp

Ray, Brian D. (1990). A nationwide study of home education: Family characteristics, legal matters, and student achievement. National Home Education Research Institute.

Ray, Brian D. (2000). Home schooling: The ameliorator of negative influences on learning? Peabody Journal of Education, 75(1 & 2), 71–106. https://www.tandfonline.com/doi/abs/10.1080/0161956X.2000.9681936

Ray, Brian D. (2010, February). Academic achievement and demographic traits of homeschool students: A nationwide study. Academic Leadership Journal, 8(1). https://scholars.fhsu.edu/alj/vol8/iss1/7/

Ray, Brian D. (2015). African American homeschool parents’ motivations for homeschooling and their Black children’s academic achievement. Journal of School Choice, 9(1), 71–96. http://www.tandfonline.com/doi/pdf/10.1080/15582159.2015.998966?needAccess=true

Ray, Brian D. (2017a). A review of research on Homeschooling and what might educators learn? Pro-Posições, 28(2). http://dx.doi.org/10.1590/1980-6248-2016-0009

Ray, Brian D. (2017b). A systematic review of the empirical research on selected aspects of homeschooling as a school choice. Journal of School Choice: International Research and Reform, 11(4), 604–621. https://doi.org/10.1080/15582159.2017.1395638

Ray, Brian D., & Eagleson, Bruce K. (2008, August). State regulation of homeschooling and homeschoolers’ SAT scores. Journal of Academic Leadership, 6(3). https://scholars.fhsu.edu/alj/vol6/iss3/17/

Webster, Daniel. (1828). Education. American Dictionary of the English Language. http://webstersdictionary1828.com/Dictionary/education

Wolf, Patrick J., Lee, Matthew H., & Watson, Angela R. (2020). Harvard law professor’s attack on homeschooling is a flawed failure. And terribly timed, too. EducationNext. https://www.educationnext.org/harvard-law-professors-attack-on-homeschooling-flawed-failure-terribly-timed/

* About the Author

Brian D. Ray, Ph.D., is president of the National Home Education Research Institute, and is internationally known for his research on homeschooling. Dr. Ray serves as an expert witness in courts and legislatures, and is a former classroom teacher in public and private schools and professor at the graduate university and undergraduate levels teaching courses such as research methods and science education. He holds a B.S. in biology from the University of Puget Sound, a M.S. in zoology from Ohio University, and a Ph.D. in science education from Oregon State University.

Endnotes:

[i] Bartholet, Elizabeth, 2020.

[ii] An exploratory study is a piece of “. . . preliminary research to clarify the exact nature of the problem to be solved” (www.oxfordreference.com) or one that is done “. . . to gain only the degree of familiarity with the properties of substances and procedures that is needed to manipulate them so as to achieve the desired effect or product” (https://www.researchgate.net/publication/309967516_Exploratory_research_in_the_social_sciences_what_is_exploration). Exploratory studies often precede ones that manipulate variables or purpose to determine causes and effects. Descriptive research simply finds and presents the facts about characteristics about some phenomenon. And correlational studies present patterns in data (e.g., as a person gets older, he is more likely to go bald, but we do not know the cause of the balding).

[iii] Within various quantitative research approaches, “. . . experiments provide the most rigorous test of causal hypotheses” (Gall, Gall, & Borg, 2007 p. 379). There are several experimental designs that attempt to control variables (e.g., by randomly assigning students to different purposive treatments) such that the investigator will test whether independent variable X (e.g., a new reading program) causes dependent variable Y (e.g., reading skills) to be changed. On the other hand, “[c]ausal-comparative and correlational methods . . . are similar [to one another] in that both are nonexperimental methods because they lack manipulation of an independent variable which is under the control of the experimenter and random assignment of participants is not possible” (Johnson, 2001, p. 5). Non-experimental designs often examine relationships among variables. It is generally agreed among research methodologists “. . . that the purpose of causal-comparative research is to examine [not establish] causality and the purpose of correlational research is to examine relationships and make predictions” (Johnson, 2001, p. 6).

[iv] Gall, Meredith D., Gall, Joyce P., & Borg, Walter R., 2007; Leedy, 1993; American Psychological Association, 2010.

[v] From https://arizonalawreview.org/about/, retrieved May 8, 2020.

[vi] From http://ncu.libanswers.com/faq/168512, retrieved May 8, 2020.

[vii] Posner, 2004.

[viii] Matching is used to equate two groups (e.g., homeschool students and public school students) on extraneous variables (e.g., sex, age, ethnicity) “. . . so that these extraneous variables do not confound the study of causal relationships involving the variables of primary interest . . .” (Gall, Gall, & Borg, 2007). An example of one of these “variables of primary interest” would be a student’s type of schooling.

[ix] Ray, Brian D. (2017a). A review of research on Homeschooling and what might educators learn? Pro-Posições, 28(2). http://dx.doi.org/10.1590/1980-6248-2016-0009

[x] Webster gave a holistic and multifaceted definition of education. It is the “. . . bringing up, as of a child, instruction; formation of manners. Education comprehends all that series of instruction and discipline which is intended to enlighten the understanding, correct the temper, and form the manners and habits of youth, and fit them for usefulness in their future stations [positions, occupations]. To give children a good education in manners, arts and science, is important; to give them a religious education is indispensable; and an immense responsibility rests on parents and guardians who neglect these duties [all italics original]” (from http://webstersdictionary1828.com/Dictionary/education, retrieved May 25, 2020).

[xi] Bartholet, Elizabeth, 2020, p. 24.

[xii] Ray, Brian D., 2017a, 2017b.

[xiii] See, for instance, Ray, Brian D., 1990, 2000, 2010, 2015; Ray, Brian D., & Eagleson, Bruce K., 2008.

[xiv] Bartholet, Elizabeth, 2020, p. 20.

[xv] Bartholet, Elizabeth, 2020, p. 15.

[xvi] Wolf, Patrick J., Lee, Matthew H., & Watson, Angela R., 2020.

What is “Good” Research? A Homeschool Researcher Responds to Harvard Professor’s Criticism: Research Methods 101