Advanced methodological_designs

Note: The German version of this entry can be found here: Advanced methodological designs (German)

Background[edit]

Designing a scientific study is rooted in experience and means standing on the shoulders of giants. Every scientific discipline that actively contributes to empirical knowledge creation has established pathways how knowledge can be created best. These norms are often specific to certain disciplines or fields, yet there are also co-evolutionary patterns that showcase that paths that produce optimific -meaning the maximum good consequences- are not only driven by the context of the research, but that instead there may be unified pathways that unite certain lines of knowledge creation.

There is no clear cutoff value or criteria that defines a research design to be advanced. The word "advanced" means modern and well developed, and this should allow us to derive a set of definitions that differentiate basic designs from advanced designs. Basic designs are long established yet did not age well. The knowledge that simple designs produce may be robust, but it is too simplistic concerning the research questions or hypothesis it allows us to answer. For example, simple statistical tests do still have some value for the initial testing of data, but much of the quantitative part of science has long been moved forward to more complex questions. Advanced designs are thus at the frontier of current knowledge. They create the knowledge that is currently the developmental edge in the respective field. Yet advanced designs differ from experimental designs by the latter being far out and not universally agreed upon by the majority of scientists. Hence we can form a succession from simple to advanced to experimental, where advanced designs represent a sweet spot between robust but anachronistic simple designs and experimental but not well tested and established designs.

This characterizes advanced designs with a certain complexity that demands some experience on the end of the researchers that utilizes them. In a sense of Kuhn advanced designs are the methods that stretch the fabric of normal science towards novel knowledge. Yet advanced designs operate already at scale, and do not represent extremely innovative outliers in knowledge creation that may be higher in their innovative potential, but also have a higher risk to fail. Advanced designs have a level of complexity that goes beyond mere textbook knowledge, but demand that researchers can adapt the design to the complexity of the knowledge production.

While intuitively the word "design" would be more closely associated with deductive research, inductive research can be equally designated to be designed. For example, interviews can even in their most open form still build on many normative steps, such as the identification of interviewees, initial communication with them, the setting of the interviews themselves and the diverse forms of analysis. All these give testimony of diverse approaches that can be considered to be more or less innovative and enable the creation of novel knowledge.

To this end much of what can be considered to be advanced is not necessarily purely driven by methodology. Instead it can be the research topics of the underlying theory that is the innovation that moves the research towards advanced designs. A prime example of such an innovation is the Ostrom framework, which in itself is focussing on the system dynamics of resource management. Yet it is the combination of Elinor Ostrom’s theoretical underpinnings with the investigation through a specific innovative lens that is the innovation itself. Advanced designs are the wider forefront of scientific innovation in empirical research.

What advanced designs do[edit]

Advanced designs approach knowledge that is novel and thus represents new terrain in the respective branch of science. Concerning the production of knowledge, advanced designs thus demand some innovation that goes beyond of what was already answered before. This is not restricted to the mere combination of a research object or topic, the underlying theory and the methods being utilized, but adds an element that is in and of itself novel. For example the mere reproduction of a previously conducted story is usually not an innovation, but instead nothing more but a confirmation, at least all things being considered equal. Advanced designs thus expand our knowledge through either a new research object, a modification of a theory, or a genuine new methodological approach. We should not restrict this line of thinking to one single study, but have to instead understand science more as a form of swarm intelligence, where a wider community of scientists shifts towards a new line of thinking or conduct.

A prominent example in statistics is the implementation of mixed effect models that allow for the creation and analysis of more complex data. In a nutshell mixed effect models allow for random information or more complicated nested designs in regression analysis. While this statistical approach has been known for decades before it started to gain traction, it was the implementation into openly available software that paved the road to its success. Without software solutions the wider implementation of mixed effect models would have been impossible, and the associated innovation in ecology, psychology, medicine and other branches of science would have been denied. This example illustrates that statistical research designs up to the point before the scaling of mixed effect models were widely resolved around few hypotheses, rather simple designs that tested for few interactions and were rather tamed concerning sample sizes and predictor variables. In other words, the widely ANOVA based designs that dominated advanced designs before the rise of mixed effect models were already tamed and aimed at simple, rather mechanistic and supposedly causal questions. These ANOVA study designs were once considered to be advanced designs, yet decades have passed since then. Established between the wars ANOVAS were the driver of statistical research designs up until the rise of computers allowed for the rise of open source software, which led to many methodological innovations. Mixed effect models thus allowed for the investigation of more complex research questions, opening new doors of knowledge previously unlocked, yet also creating other dead ends that needed to be explored first. For example the reproducibility crisis in psychology is an example of how all power corrupts, even statistical power. The discipline of psychology has ever since addressed this problem head on, and much has changed and improved since then. This crisis illustrates that designs can also become too complex or at least permit us to neglect information that may have been relevant to begin with. Any advanced designs are thus a complicated aspect of tinkering and tailoring that ultimately allows us to calibrate such designs best. Once such advanced designs become established and simple recipes are established many sciences reach a point of trivialization. What once was advanced is increasingly considered to be simple, and the knowledge that is created is hardly novel any more. The wheel of science keeps spinning. The developmental edge becomes normal science. Concerning mixed effect models, this effect can be well observed, as the diversification into novel approaches such as structural equation models, generalized additive models or time series analysis showcases that diversification and adaptation of known vectors of development towards novel approaches, and thus, new insight and knowledge.

Diversification is a notorious pattern in any methodology that went through a stage of being part of a revolution in the sense of Kuhn. Once the revolution is over, a smaller part of the formerly often buzzwordy community remains and becomes experts in the respective methods. Different schools of thought may emerge, sometimes reflecting different viewpoints or normative claims. All this is part of scientific dispute, and creates an ecosystem that can be lively at best and dogmatic at worst. Methods associated to formerly advanced designs sometimes shift, or may die out once their proclaimers retire or move on. This highlights that trends concerning methodological innovation have so far been usually trends that follow more a timeline of decades and not years. The AI burst of the mid 2020s is probably no exception, after all many of these models have been around for quite a while. The only difference now is that they are more widely available to the public, for better or worse. Advanced designs thus follow an overall timeline where they are proposed, implemented, adapted or diversified, and ultimately become widely replaced. Almost every generation of scientists had in every epistemological field their very own methodological revolution, yet this often only lasts some years and then lingers on for some more years, typically mounting 1-2 decades before a new revolution is proclaimed, following the generational shift in science. The advanced designs of the last generation are often the simple designs of the following generation. Methods may not age well, but at least they can diversify the canon of knowledge and create a pile of approaches that can be read like a sediment. Older layers are at the bottom, and novel layers start accumulating on top. It is usually the top research of their time and field that initiates a new layer on the highest level.

A way to engage with advanced designs is clearly in the application of methods on a repeated level. Building experience is rooted in continuous and wide-ranging application, where repetition will compound over time and can enable beginners to gain momentum and experience. Repetition is one key element, yet another important element to gain ground towards advanced designs is the application of methods in different contexts. No method can be utilized in all circumstances, but instead it depends on the context; understanding how different elements of a methodological application matter under different circumstances is essential to become experienced. Many beginners look for simple recipes in how to apply a method best, but this is not how any scientific method works. Instead one has to develop a wide array of variations, and these will compound over time. Interviews are a good example to this end. Conducting interviews is only one part of the whole method, and already getting interviews to represent a community can be a task in itself. Analysing interviews can be conducted also in different ways, hence one already has three moving parts, with preparing interviews and identifying interviewees, conducting interviews and documenting them and lastly analyzing the diversity of information being three independent components that form the overall outcome. Any given method thus has several fundamental elements that can all be explored in different variations, and the combination of variations makes all scientific methods like endless variations of a piece of music. Advanced designs are therefore an essential part of scientific reality, and provide the basis both of scientific progress as well as the reality that all knowledge forever changes. Whatever is considered the current basis for advanced designs thus provides the baseline for understanding the current complexity of the world. What is complex today may become simple in the future.

Strengths and challenges[edit]

The core strengths of advanced designs is that they represent the current best approach to generate novel knowledge. Safety to this end comes in numbers, since advanced designs are usually methodological approaches that are considered innovative and thus provide what most researchers within a branch of science will use to move the field forward. Yet since this means that methodologies are also evolving, there is often a lack of meta-experience in the community, and sometimes even the excitement towards the novel may get the better of a scientific community. For example, research on climate change did evolve for a long time mostly based on a perspective deeply rooted in modeling and natural science, and it took a shift in this community to include social scientists to enable a paradigm shift towards mitigation and adaptation. Hence a community that evolves science may particularly if this knowledge is discussed by society pose an almost unpredictable conservative twist, since being hyped by societal debate may ultimately translate into a standstill. Once a knowledge-bubble bursts, a usually smaller part of the community remains, and this will more often than not form a smaller yet more solid research community. In other words, a constant danger of advanced designs is to be corrupted into overplaying their hand concerning novel knowledge. This pattern is often increased by a certain lack of a critical perspective in empirical research, since many methodological innovations are driven by positivists to this day. These types of researchers may overplay their hand, or at least lack the critical capacity to identify weaknesses in the current methodological innovations.

Another challenge within advanced designs is the diversity of the different schools of thought. There is usually a gradient from pragmatists to dogmatists in any scientific community that evolves and the question is whether these two communities can reconsider their diverse approaches. Yet other schools of thought can also emerge, and some of these may as well be less valuable in the long run compared to others. Advanced designs as a testing ground for novel approaches are thus in their early stages prone to result in questionable if not wrong approaches, which is technically defined as a more experimental stage. Yet transitions are gradual, and it is in this gradual development of science that most challenges are located concerning questionable innovation. This process stands and falls with the boldness of the peer community that reviews and discusses the emerging approaches. Some communities are surprisingly dogmatic and conservative, almost actively suppressing progress. While this is counterintuitive to the original mission of science, it is still a problem that needs to be addressed.

This is associated with another problem that is associated to advanced designs. Such approaches are as we already concluded carried by a broader community, yet the fact that these can be thousands or even hundred of thousands of scientists does not make them more inclusive. Advanced designs can be a reason for deep dogmatic entrenchment, and many branches of sciences define neighboring branches by rejecting them altogether. This can lead to profound epistemological disagreement, where the instance of certain methodological approaches is driven by identity and not by knowledge itself. While this is difficult to track down, it is part of what defines many scientific disciplines. To this end, advanced designs can be a right means to a wrong end.

Another major problem in the establishment of advanced designs is the reality that by definition such designs cannot and should not be reduced to statistical textbooks. These can serve as a starting point and give valuable initial insight, but cannot replace a longer education that usually covers at least a decade if one considers a PhD to be included in it. This timeline creates another problem of temporality, and illustrates a point made before: Succession in science takes decades, and these are the same decades that students take to work their way into advanced designs. Science typically works at this timescale, where it takes at least 1-2 decades until methodological innovation happens that can be considered radical. Most scientists that use methods evolve these gradual, as paradigm shifts are exceptions and operate in communities and take time to be established.

One last strongpoint of advanced designs is the fact that they are typically informed or at least an indirect result of societal demand. If there are certain problems that demand an approximation of solutions, these problems are usually also complex for the societies that face them. The Manhattan project represents such an immediate and rather grim problem that the US faced during the second world war. Yet the preconditions formed already before, and it was the rush of the Manhattan project and the desperate resources associated with it that illustrate how advanced designs and a lot of tinkering can aid a scientific push. This example also illustrates that such solutions approximated by science may not necessarily create a better society, illustrating the potential moral danger that scientific innovation may encompass.

Normativity[edit]

A main point that emerges out of this mechanistic model of methodological innovation is the role of technology. We cannot and should not reduce methodological innovation to technical innovation, although this may seem more than tempting. The telescope contributed to the Copernican resolution, cameras and audio recorder revolutionized interview studies as well as media studies, microscopes revolutionized biology, genetics drove medical innovation, and the latest AI boom created severe ripples in science. All this may tempt people to consider methodological innovation to be a mere ratio of technology. This is not the case. Instead we have to see these developments as parts of an overall wider development. An example for such a process are the societal changes of the 1950s and 1960s after the second world war, when student protests and political movements were in tandem with an emerging critical perspective in academia. Hence there can be many co-evolutionary patterns where one could wonder whether one started the other, when ultimately science evolves as a complex system.

The individual students and researchers will have to decide whether they want to contribute to the established canon of science and contribute through incremental changes based on simple designs. There are a lot of jobs in industry, organizations, NGOs and wider society that scale science and its processes into a wider and more applied canon of knowledge. Yet scientists actively contributing to research will evolve and move forward, which is the very nature of science itself. By definition, we want to know more, and this includes not only the creation of knowledge, but also the responsibility that goes along with it, and the consequences that arise out of our gain in knowledge.

Outlook[edit]

Advanced designs are the engine of scientific empirical progress, for better or worse. The cycles of evolution followed so far are often widely similar to the timescale in which scientists utilized these methods. It remains to be seen if the great acceleration also creates ripples concerning scientific progress and its problems. The long shadow that AI currently casts on science can be seen as testimony of such an overarching and accelerating effect. Yet there is also the hope that scientific innovation can become more agile and pluralistic if overall science changes for the better. The identity of individual scientists could after all become more uncoupled from the evolutionary cycles of advanced designs. Time will tell whether scientific progress can evolve without further problems.