The opt out paradigm First steps towards a new experimental method that measures meta-linguistic awareness

A common assumption is that children learn a language implicitly and without conscious awareness of form and grammar, but this assumption has virtually never been tested experimentally. We propose a novel experimental method to examine if children’s ability to acquire linguistic regularities relates to awareness of these regularities. Traditional methods investigating awareness often rely on learners’ abilities to verbalize their awareness. For young children, such methods are not adequate because they often cannot reflect explicitly on their acquired knowledge, although they might be aware of it in a way they cannot verbalize. To test this, we adapted a method that is used to investigate awareness in animals, because it does not rely on verbalization for demonstrating awareness. Pilot results with 26 adults and 48 kindergartners show some important procedural prerequisites are met. In future research, this procedure could be used to investigate the development of meta-linguistic awareness in children.


Introduction
It is often assumed that children acquire a language implicitly without any awareness of form and grammar (Wijnen, 2013).Yet very little research has been conducted to investigate this assumption, as far as we are concerned.In contrast, the role of awareness in adult second language acquisition has been extensively investigated, and there is a wide range of theoretical perspectives and experimental methods that address the issue (Hulstijn, 2015;Andringa & Rebuschat, 2015;Rebuschat, 2015).Unfortunately, the work done in second language acquisition (SLA) does not yield suitable methodological tools for the study of meta-linguistic awareness in children.Here, we will try to bridge this methodological gap by applying a paradigm that has been used to investigate awareness in animals (Hampton, 2001(Hampton, , 2009) ) to a linguistic context.
Before we introduce this paradigm, we must clarify our conception of awareness.In this paper, we follow the idea that learners may either be phenomenally aware (P-awareness) or aware at the level of access (A-awareness; Cleeremans, 2008).P-awareness refers to the situation in which a person has a subjective experience of the object of awareness, whereas someone can be called A-aware when an experience is accessible to one's awareness, but this cannot be expressed subjectively.Several characteristics are typically associated with P-awareness; one should be able to verbalize and remember a particular experience s/he is aware of as a coherent whole.One can only be aware at this phenomenal level, if the mental representation of the experience is strong enough.This type of awareness bears resemblance to what is often called explicit knowledge.However, verbalization, coherence and memory might not be necessary requirements for awareness (Allport, 1988;James, 1890;Dennett, 1991).Less strong mental representations could be accessible to awareness, even though one cannot verbalize or memorize these representations coherently.This is perhaps the type of awareness that can be ascribed to animals who are clearly unable to verbalize their experiences, but nevertheless are shown to be aware of such experiences (see De Waal (2016) for an overview, although not in terms of A-and P-awareness).In this sense, Aawareness should be distinguished from implicit knowledge, which is knowledge someone has no awareness of whatsoever.Instead, A-awareness could be characterized as a state of awareness between explicit and implicit knowledge.
If we apply these forms of awareness to the study of language, someone can be said to be phenomenally aware, when s/he is able to explicitly describe certain linguistic patterns.On the other hand, one is aware at an access level if s/he perceives someone is speaking with an accent and can remark that 'something' is going on, without being able to determine which accent is used or that an accent is used at all.This A-awareness resembles what has been called noticing in the L2 literature (Schmidt, 1990).Furthermore, the object of meta-linguistic awareness could constitute of different linguistic phenomena.Someone could be aware that s/he is learning a language, but in this study we are not interested in this holistic type of meta-linguistic awareness.With meta-linguistic awareness we imply awareness of what one is learning in a particular language.
It is commonly assumed that language learners are only aware of linguistic patterns in the phenomenal sense if they can verbalize and remember them: meta-linguistic P-awareness.However, learners in general and children in particular might possess awareness of the patterns they acquire, without being able to verbalize those patterns: meta-linguistic A-awareness.Traditional methods of investigating meta-linguistic awareness, such as verbal reports or confidence judgement tasks, typically measure P-awareness (Timmermans & Cleeremans, 2015;Batterink, Reber, Neville, & Paller, 2015) and thus seem unsuitable to investigate awareness in children.Therefore, we adopt the 'opt out paradigm' , which has been used to investigate (A-)awareness in animals (Hampton, 2001(Hampton, , 2009)).The opt out paradigm assesses awareness, without requiring learners to verbalize their experiences.In this paradigm participants show strategic behavior that reveals their awareness.We believe this method can be used to assess whether children are aware of the linguistic patterns they are acquiring.
In this paper, we explore the possibility of using the opt out paradigm as a measure of meta-linguistic awareness.As this is a novel method with regard to children, the aim of the current study is twofold.First, we want to see whether the opt out experiment can reveal awareness of acquired knowledge in adults.In addition, we want to determine whether the procedure of our experiment is suitable for children.If both are the case, this would enable future research into the role of awareness in child language acquisition using an adaptation of this experiment.
In the remainder, we will elaborate briefly on the role of awareness in language acquisition and why it is worthwhile to develop a method for gauging awareness in child language learning.Subsequently, we will discuss in more detail how awareness is typically investigated and how our method can contribute to the range of existing methods.

Awareness in language acquisition
The role of meta-linguistic awareness in the process of acquiring a new language has been widely investigated (e.g.Hulstijn, 2015;Andringa & Rebuschat, 2015).Some authors have argued that awareness of the structures in a second language is necessary to acquire those structures (Schmidt, 1990;DeKeyser, 2003), whereas others have claimed awareness facilitates the acquisition process (Ellis, 2003).Others have suggested that the presence of awareness is just circumstantial and a consequence of how the language is learned, but has no influence on the implicit linguistic knowledge that is gained (Krashen, 1981).
Theories of child language acquisition are often less explicit about the role of meta-linguistic awareness in the acquisition process.Many theories assume that meta-linguistic awareness hardly plays any role.This seems to be the case in usage based theories (Tomasello, 2000), in generative approaches (Chomsky, 1986;Radford, 2004) and also in theories that assume language learning to be a statistical process (Erickson & Thiessen, 2015;Saffran, Johnson, & Aslin, 1996;Aslin, Saffran, & Newport, 1998;Endress & Bonatti, 2007).Whether children map forms onto functions, set grammatical parameters or track dependencies between linguistic elements, they have generally been assumed to be unaware they are carrying out these cognitive processes and they have also been assumed to be unaware of the regularities they acquire as a result of them.
The idea that primary language learning occurs without awareness also emerged in frameworks that seek a neurological foundation for language, such as the declarative/procedural memory model (Ullman, 2001(Ullman, , 2004)).In this model, procedural memory is responsible for pattern recognition in the auditory as well as the visual domain (Packard, 2009).Procedural learning is often equated with implicit learning, as people typically are assumed to be unaware they are acquiring grammatical regularities and to be unable to reflect on their acquired regularities (West, Vadillo, Shanks, & Hulme, 2017).On the other hand, declarative memory is thought to handle the storage of more arbitrary information, concrete facts and events.Declarative knowledge is said to be gained more explicitly, because we learn this kind of linguistic information consciously and we are able to retrieve this knowledge (Ullman, 2016).Many studies have shown that domain-general procedural memory is related to the acquisition of linguistic patterns and that child language acquisition in particular might rest on this memory system (Misyak & Christiansen, 2012;Misyak, Christiansen, & Tomblin, 2010;Evans, Saffran, & Robe-Torres, 2009;Hsu, Tomblin, & Christiansen, 2014;Ullman & Pierpont, 2005;Lum, Conti-Ramsden, Page, & Ullman, 2012).Because children are often portrayed as more skillful procedural learners, they are assumed to be unaware of the linguistic patterns they acquire (Wijnen, 2013).
However, whether children indeed acquire language completely implicitly is an issue that has hardly been put to test empirically.Importantly, mapping form unto function, setting parameters, tracking statistical regularities or proceduralizing knowledge might play a role in language acquisition, but these processes do not necessarily need to occur without any awareness involved.One reason why awareness during acquisition has not been investigated in children may be that people tend to equate awareness with P-awareness.Possibly, children do not have this type of awareness of the language they acquire, because they might be unable to verbalize the patterns of language.Yet, children could still possess A-awareness when acquiring a language: they are aware of the linguistic patterns they acquire though they are unable to verbalize this awareness.It is conceivable that some well described phenomena within language acquisition, such as overgeneralization (Barac & Bialystok, 2012), stem from children's A-awareness, but to our knowledge, links between such phenomena and awareness in children have not been as systematically investigated, as they are in the field of second language acquisition.If we want to make claims about the role of (A-)awareness in child language acquisition, the question to answer is thus how such awareness can be measured.

Measuring awareness
Typically, when researchers investigate the role of awareness, direct subjective methods are used (Timmermans & Cleeremans, 2015).These methods often consist of two parts.First, experimenters assess whether a participant learned or acquired a certain rule or regularity.Afterwards, the experimenters investigate whether a participant was aware of what s/he has learned or acquired.
To measure learning, experimenters can use grammaticality judgement tasks or Wug tests, obtain ERP signals, gather eye tracking data, or measure reaction times.Crucially, none of these measures reveal anything about the awareness that a learner has about his/her gained knowledge.If reaction times show a learner has acquired a certain regularity, the learner is not necessarily aware of this regularity.His or her behavior can be based entirely on implicit knowledge.For grammaticality judgment tasks and Wug tests (Berko, 1958), the same applies, although some scholars have argued they are a measure of meta-linguistic awareness (Bialystok, 1986;Barac & Bialystok, 2012).Perhaps some level of meta-linguistic awareness is needed to perform these tasks, as one has to recognize regularities in language or identify correct and incorrect sentences.The point is, however, that grammaticality judgment tasks and the Wug test do not reveal the status of the knowledge that was used to make choices in the tasks.In such tasks, learners may make choices because they are aware of what the regularities are, their behavior could also be the result of implicit knowledge of these regularities.
A similar rationale applies to the research showing that babies are able to distinguish between their native language and a foreign language (e.g.Bosch & Sebastián-Gallés, 2003) or that indicates that infants infer that non-native languages serve communicative purposes as well (e.g.Vouloumanos, 2018).In such studies infants could, for example by a preferential looking paradigm, show signs of awareness that language serves particular functions and that there is a difference between languages, but this paradigm does not reveal whether infants are aware of what these functions are or what the differences are.Of course, infants could possess such awareness, but these methods are not suitable to measure it.
Measures like grammaticality judgement tasks, Wug tests and preferential looking paradigms can thus chart learning, but they cannot gauge whether learners developed awareness of what the regularities in the acquired language are.To tap into this sort of awareness, learners typically have to report verbally on whether they were aware of the regularity (e.g.Batterink et al., 2015) or partake in a confidence judgement task, indicating how certain they are about their grammatical judgment (e.g.Bertels, Boursain, Destrebecqz, & Gaillard, 2015).These methods directly ask participants to reflect on the object of awareness in a way that relies on their subjective experiences, as they have to verbalize their knowledge or indicate themselves how confident they are.In other words, they tap into P-awareness.
If meta-linguistic awareness exists at the A-level, a different kind of measurement is required.To measure A-awareness, indirect objective methods are required (Timmermans & Cleeremans, 2015).In such methods, experimenters observe behavior that indirectly reflects some form of awareness, but does not require any subjective introspection, verbalization or reflection on knowledge.This makes these methods specifically suitable to tap into A-awareness and they may be a more fruitful way to investigate the role of awareness in child language learning.
An example of an indirect objective method that can potentially be adapted to a linguistic context is the opt out paradigm, which is normally used in studies with non-human animals (De Waal, 2016;Hampton, 2001Hampton, , 2009)).Unlike humans, non-human animals cannot verbally report on their awareness.This method is thus particularly interesting in this respect.Experiments reported in Hampton (2001) rely on the fact that monkeys like peanuts more than pellets.Rhesus monkeys were trained to remember a picture of a flower.After a certain time, the monkeys are shown an array of pictures of flowers and had to identify the flower they had seen before.If they identified the right flower they received a peanut, which they really like.If they answered incorrectly, they would not receive anything.After a training period, a new condition was introduced which gave the monkeys the opportunity to opt out of the test and for which they would receive a pellet.While the monkeys prefer peanuts over pellets, opting out would give them a guarantee of receiving at least some food.In the experiment, the option to opt out was given either shortly after seeing the flower they had to remember or after a longer period of time.
During the experiment the monkeys chose to opt out more often when this interval was longer and thus their memory of the target flower had decayed more.Hampton argued that this is indicative of at least some level of awareness on behalf of the monkeys.When the interval was longer, it would be a better strategy to opt out, because the chance of guessing the right picture was lower due to forgetting.Thus, the risk of not receiving any reward at all was relatively high.In order for the monkeys to make this type of decision, they should have been at some level aware of the fact that they had a certain memory of the picture of the flower that decays over time.Additionally, they could take this decaying memory into account when they had to make a decision that was based on the status of this memory.The monkeys thus did not have to engage in some subjective introspection with regard to their memory, but instead performed some objective observable behavior that was the indirect effect of their awareness.As such, this opt out paradigm is an indirect, objective measure that taps into A-awareness.
The main goal of this paper is to investigate whether the opt out paradigm can be adapted to a linguistic context successfully, such that it can be used in future research to investigate the role of awareness in child language acquisition.As this is only a first step towards implementing this paradigm, the concrete goals of the paper are twofold.To start off, we want to see whether the opt out experiment can reveal awareness of acquired knowledge in adults.
Next, we want to know whether the procedure of our opt out experiment is suitable for children.Several studies show that different populations react differently to rewards (Frederick, 2005;Eigsti et al., 2006), with for example children being much more likely to go for certain rewards than to seize the opportunity to obtain a bigger reward after a period of time or after the performance of a task (Mischel, Shoda, & Rodriguez, 1989).Given this observation, we also want to determine that the reward for opting out in our experiment is not too appealing, or that the task participants had to perform to earn a bigger reward is not too difficult.If either is the case, children in our experiment will be biased to opt out, regardless of their level of awareness.If we find that children choose to opt in and out, then (an adaptation of) our procedure might be suitable to probe into children's awareness in future research.

Participants
Twenty-six Dutch speaking adults (8 males, 18 females, M = 22;9, SD = 4;4) took part in the opt out experiment.No restrictions were imposed upon taking part in this pilot.A relatively large number of students of linguistics were part of the adult sample.Additionally, 48 Dutch speaking children (26 males, 22 females, M = 5;3 SD = 0;9) took part in the experiment.All children were in kindergarten and did not have any diagnosed language or communication disorders.Ethical approval for this study was obtained from the University of Amsterdam and before children participated, active consent was obtained from their parents or legal guardians.

Opt out experiment
The experiment consisted of a learning phase and an opt out phase.During the learning phase, participants had to discover a correspondence rule based on the Dutch determiner system.In Dutch, nouns can be of common or neuter gender, which each have their own definite article; de for common and het for neuter nouns.(cf.de koe ('the cow' , common) with het paard ('the horse' , neuter)).Participants were shown pictures that were accompanied by Dutch sentences in which a noun was replaced by a nonsense word.The sentences with a nonsense word would still refer to the concepts depicted in the illustration, but participants could arrive at the meaning of the nonsense words through the correspondence between the determiner that preceded the nonsense word and the determiner of the word that was illustrated in the picture.For example, the Dutch word koe 'cow' was replaced by the nonsense word lino which was always preceded by the determiner of the Dutch word for cow ('de').This resulted in the noun phrase de lino, which referred to a cow.In contrast, the Dutch word paard 'horse' was replaced by the nonsense word orbo which was always preceded by the determiner of the Dutch word for horse ('het').This resulted in the noun phrase het orbo, which referred to a horse.
To learn the correspondence rule, participants were presented with a spoken sentence and were shown two pictures.The sentence described only one of the two pictures and participants had to decide which picture matched the sentence they heard.The input during the learning phase consisted of 36 sentences of three different types; 12 sentences with a definite article and a Dutch noun (1a), 12 sentences with a definite article and a nonsense word (1b) and 12 sentences with an indefinite article and a nonsense word (1c).Figure 1 shows an example of the pictures that participants had to choose between.All test items can be found in Appendix A. The pictures always depicted two possible meanings of the sentence.One depicted a concept that in Dutch is described by a common noun and the other by a neuter noun, see Figure 1 for an example showing a cow and a horse.A correct decision in the picture matching task could be made on the basis of the final noun phrase, as the rest of the sentence could refer to both pictures.In the case of sentences with a Dutch noun in the final noun phrase, this decision could be made on the basis of the lexical meaning of that noun phrase.In the case of sentences that ended with a definite article and a nonsense word, this decision could be made on the basis of the earlier mentioned correspondence rule.
For the sentences ending with an indefinite article and a nonsense word, a correct decision could only be made if participants remembered the meaning of the nonsense word.In Dutch, gender is only marked on the definite article, but not on the indefinite article: both common and neuter nouns receive the article een 'a' .Therefore, the indefinite article introducing a nonsense words was not predictive of its meaning.Participants could give a correct answer in these cases, when they matched a nonsense word to its meaning when the correspondence rule could be used.They then had to remember this meaning and correctly apply it to the nonsense word when the correspondence rule could not be used to make the right decision.
Test items were presented semi-randomized, so that the Dutch nouns were always presented before the definite nonsense equivalent was presented.This way participants were presented with the Dutch noun and its article before the nonsense equivalent was presented, such that they had the opportunity to apply the correspondence rule to the right Dutch noun.Furthermore, a sentence ending with a definite article and a nonsense word had to be presented before its indefinite equivalent was presented, so (1c) could only be played when (1b) was presented.This was to assure that participants could give a correct answer to the indefinite cases, if they remembered the nonsense words and their matching concepts correctly.
Before this learning phase started, participants were presented with some practice items with both existing Dutch words and nonsense words.Importantly, after they had completed two-thirds of the learning phase, participants received a reward when they made a correct decision.At this point all sentences with an existing Dutch word had been presented, as well as half of the sentences with a definite article and a nonsense word and half of the sentences with an indefinite article and a nonsense word.For adults, the reward consisted of 2 euros, the currency being Monopoly® money.They were encouraged to obtain as much money as possible.Participants did not keep their reward after the experiment.Children helped a rabbit cross a river with 10 stepping stones to obtain a carrot.The rabbit would move 2 stones for every correct answer.Children received a sticker for every carrot the rabbit obtained.
In the opt out phase, participants heard the same sentences as before, up to and including the determiner, but without the last word, while seeing a blank screen.Participants would hear sentences as in (2).
(2) a.In het gras staat de/het … 'In the grass stands the …' b.In het gras staat een … 'In the grass stands a …' They were then given two options.They could hear the whole sentence including the last word.When hearing the full sentence, they would see the two pictures and had to make a decision between these pictures.As before, if they decided correctly they would earn 2 euros/2 steps and if they made a wrong decision, they would not obtain any reward.Participants could also choose to opt out.In this case, they decided not to hear the full sentence, but simply to move on to the next test item.When opting out, participants would receive 1 euro or the rabbit would move 1 stepping stone.Thus, in order to earn as much money or obtain as many carrots as possible, participants had to consider their chances of providing a correct answer.
If participants decided to hear the full sentence, the sentence would always end with a nonsense word that occurred in the learning phase.
The opt out phase consisted of only the sentences with nonsense words that were presented in the learning phase.There were 12 items that ended with a definite article that would end with a nonsense word if heard fully, and 12 items that ended with an indefinite article that would also end with a nonsense word if heard fully.In order to make sure that participants understood the consequences of opting for or against hearing the full sentence, they were presented two practice items before the test phase, to familiarize themselves with the procedure of the opt out phase.During the entire experiment, the target answer was counterbalanced between left and right.

Procedure
The experiment was administered to adult participants in a quiet room at the University of Amsterdam by one experimenter.The task was presented on a laptop and scores were recorded by hand using paper and pencil.Afterwards, there was an informal debriefing.Adults were not compensated for their participation.For children, the test was administered in a quiet room at their schools.The task was presented on a laptop using E-prime (Psychology Software Tools, 2016) and scores were registered automatically.Children were allowed to keep the stickers they earned during the experiment.Afterwards a debriefing took place in which children were asked whether they knew what the nonsense words meant, how they knew, why they decided to opt out and why to opt in.

Analysis
All analyses were carried out in R (R Core Team, 2015) using the lme4 package (Bates, Maechler, Bolker, & Walker, 2015) where needed.As mentioned before, the aims of this experiment were different for the two test groups.For adults, we wanted to know whether opting out can be used as a measure of meta-linguistic awareness, once a rule is learned.For children, we wanted to know whether they show sensitivity to a reward system based on linguistic input.We were thus not interested in group differences with regard to learning the correspondence rule.Therefore, separate analyses were carried out for the two groups.For each group, generalized logistic mixed effect models were used to investigate whether participants learned the rule in the learning phase of the experiment and whether they were aware of this in the opt out phase.Pearson's correlations were used to investigate possible relationships between the scores on both parts of this test.

Results
Results from the learning phase can be found in Table 1.To determine whether adults learned the correspondence rule, a generalized logistic model with mixed effects and orthogonal sum-to-zero contrast coding was carried out (see Baguley, 2012, pp. 590-621).This model took the target score as a dependent variable, article type as a within-participants fixed effect, participant as a between-participants random effect and item as a within-participants random effect.Results showed a significant main-effect of article type (odds ratio = 2.20, 95% confidence interval = 1.21 … 4.32 higher odds, z = 2.340, p = 0.019).Adults gave more correct answers when the nonsense word was introduced by a definite article than when it was introduced by an indefinite article.This suggests that adults have picked up upon the correspondence rule.To determine whether children learned the correspondence rule, a generalized logistic model with mixed effects and orthogonal sum-to-zero coding was carried out too.This model took the target score as a dependent variable, article type as a within-participants fixed effect, participant as a between-participants random effect and item as a within-participants random effect.Results did not show a main effect of article type (odds ratio = 0.98, 95% confidence interval = 0.59 … 1.64 higher odds, z = −0.080,p = 0.936).Children did not give more correct answers when the nonsense word was introduced by a definite article than when it was introduced by an indefinite article.This does not provide any evidence that children have picked up the correspondence rule.
Results from the opt out phase are shown in Table 2. To determine whether adults became aware of the correspondence rule, a generalized logistic model with mixed effects and orthogonal sum-to-zero coding was carried out.This model took the opt out score as a dependent variable, article type as a withinparticipants fixed effect, participant as a between-participants random effect and item as a within-participants random effect.Results showed a significant maineffect of article (odds ratio = 11.07,95% confidence interval = 1.86 … 65.92 higher odds, z = 2.696, p = 0.007).Adults opted out more often when the sentenced ended with an indefinite article than when it ended with a definite article.This suggests that adults were aware of the correspondence rule.To determine whether children became aware of the correspondence rule, a generalized logistic model with mixed effects and orthogonal sum-to-zero coding was carried out.This model took the opt out score as a dependent variable, article type as a within-participants fixed effect, participant as a between-participants random effect and item as a within-participants random effect.Results do not show a main-effect of article (odds ratio = 1.00, 95% confidence interval = 0.73 … 1.38 higher odds, z = 0.025, p = 0.980).Children did not opt out more often when the sentence ended with an indefinite article than when it ended with a definite article.This does not provide any evidence that children were aware of the correspondence rule.Nevertheless, it is important to note here that each child that completed the experiment opted out and opted in at least once during this phase.This means each child at least tried out both possible options available during this phase, which seems to indicate that a possible bias towards either of the two options is not overwhelming and leads to ceiling effects.

Correlation
To determine whether a correlation existed between awareness in the opt out phase and learning in the learning phase for the adult participants, a difference score was calculated for both phases.For the learning phase, this score was calculated by subtracting the number of target answers on sentences with an indefinite article from the number of target answers on sentences with a definite article.A larger difference expresses a greater likelihood that the participant had learnt the correspondence rule.For the opt out phase, this score was calculated by subtracting the number of times a participant decided to continue when a sentence ended with an indefinite article from the number of times a participant decided to continue when a sentence ended with a definite article.If this difference score was higher, it was more likely that a participant was aware of the rule s/he has learnt.A Pearson's correlation test showed a correlation between learning and awareness for the adult participants (r = 0.477, p = 0.014).

Discussion
We have seen that, during the learning phase, adults gave more correct answers when sentences ended with a definite article than when sentences ended with an indefinite article.This indicates that adults, as a group, most likely detected the correspondence rule that determined that the gender of the Dutch nouns corresponds to the gender of their nonsense equivalents.Only in the case of sentences that ended with a definite article, this rule could be applied and it was therefore expected that they would score better in this condition.
In the opt out phase, adults opted out more often in the case of sentences with an indefinite article than during sentences with a definite article.When participants were either A-or P-aware of the correspondence rule it would be more rewarding to opt out in the case of sentences ending in an indefinite article, because in such cases, participants could not give a correct answer on the basis of a rule.Therefore, it was advantageous to choose the guaranteed smaller reward.In contrast, when participants were aware of the correspondence rule, it would be more rewarding to hear the full sentence and try to give an answer when sentences ended with a definite article.In these cases, a correct decision could be made on the basis of the correspondence rule.The results thus indicate that participants as a group were probably aware of the correspondence rule.
Finally, a correlation was found between the degree to which the rule was learned and the degree to which participants opted out.This could indicate two things.Either, participants develop awareness of the correspondence rule before they can grasp it, or, awareness is a consequence of grasping the correspondence rule.Due to the nature of our experiment we cannot conclude anything about the causality within this relationship.
Our results do not provide evidence that children detected the correspondence rule and therefore, it is not surprising our results do not provide evidence that children were aware of the correspondence rule during the opt out phase.However, this particular experiment was not carried out to make any inferences about the development of meta-linguistic awareness in children.Instead, we wanted to know whether the procedure of our experiment was suitable for children.More concretely, we wanted to know whether the task was feasible for children and that the reward for opting out in our experiment was not too appealing.The results do not give reasons to assume the method is faulty.Although a proper linguistic motivation to choose one option over the other lacked in the current experiment, every individual child at least opted in and opted out once during the test.This indicates that our procedure did not lead to an overwhelming opt out bias.Future research should tell us whether or not children are able to opt out strategically on the basis of a regularity they can acquire.If they are able to do so, this could be indicative of A-awareness of the linguistic pattern they have acquired.
The main goal of our experiment was to investigate whether the procedure of our opt out experiment could be used in future research to measure metalinguistic awareness in children.For this to work, participants need to be sensitive to a reward system that is based on linguistic input, as this is a prerequisite for successfully adapting this paradigm originally developed by Hampton (2001Hampton ( , 2009)).The findings suggest that the opt out paradigm can be adapted to a linguistic context, because adults showed strategic behavior based on their rule knowledge and because our procedure seemed suitable for children, as there was no clear opt out bias in this population.Nevertheless, some issues should be taken into consideration before this method can be implemented in a child language learning study.
During this experiment, only a very limited amount of input was presented to participants.For a non-naïve population like our adult group, which contained a large number of students of linguistics, this input was sufficient to pick up the correspondence rule.For more naïve populations, like children, this input clearly did not suffice.Because the goal of this experiment was to investigate the workings of the reward system, the results on learning the particular correspondence rule are not of much importance for now.However, a future goal is to investigate whether children have the cognitive ability to develop any meta-linguistic awareness during language acquisition.Therefore, it is essential that we train children on a rule that they can learn.Also, it would be preferable if participants in an opt out experiment first undergo a training phase during which they do not have to perform any actions.Such a training phase would enable children to acquire a linguistic regularity properly, before they are actually tested on their awareness of what they have acquired.This could be achieved by first developing a miniature language that has to be learned during a training phase (see De Graaff, 1997;Lichtman, 2016or Andringa & Curcic, 2015 for examples of such a language).
When developing a miniature language for the opt out paradigm, there are some limitations with regard to the possible rule in that language that has to be learned.The paradigm requires a rule that allows for two types of linguistic elements: one linguistic element that predicts what comes next and another linguistic element that does not predict what comes next.Participants can only show strategic behavior if one linguistic element predicts what is coming next and it is appealing to opt in, while another element does not predict what comes next, giving the opportunity to opt out.Candidates for such a rule might be differential object marking (Aissen, 2003) or certain classifier systems (Passer, 2016).
As mentioned before, when using the opt out paradigm, it is extremely important to have a well-balanced reward system that rewards opting out and opting in differently, especially because children react differently to awards in comparison to adults (Frederick, 2005;Eigsti et al., 2006;Mischel, Shoda, & Rodriguez, 1989).If the reward a participant can earn for making a correct decision is insufficiently appealing, it will be attractive to opt out regardless of awareness of the target structure.Conversely, when the reward for opting out is insufficiently appealing, participants might be keen to take a risk regardless of their awareness of the linguistic regularity that was present in the input.Because all children both opted in and opted out in our experiment, our current rewarding procedure meets our desired criteria, and should thus be maintained as much as possible in future adaptations.

Conclusion
Despite the fact that further adaptations are required before the opt out paradigm can be used in future research, the results from this pilot are promising.The opt out experiment could be a valuable new tool to investigate meta-linguistic awareness.If the rule, training and rewarding system are controlled for carefully, this method can overcome some of the problems that direct subjective methods of investigating awareness have.Participants do not have to reflect on or verbalize their knowledge of the target structure.Therefore, this method can potentially tap into A-awareness, and provide new opportunities to make claims about the possible role of awareness in child language acquisition.Our next objective thus is to develop a miniature language that contains a rule that is suitable for this paradigm and learnable for children.

Figure 1 .
Figure 1.The two pictures between which a participant had to choose, when s/he heard the sentence In het gras staat de koe ('The cow is standing in the grass').The right picture was the target.
Appendix A. All test items and the pictures between which a participant had to choose, when hearing one of the six test items next to them during the experiment Test items Picture pair De jongen eet de appel De jongen eet de pato De jongen eet een pato De jongen eet het ei De jongen eet het domo De jongen eet een domo 'The boy eats the apple/egg'

Table 1 .
Scores from the first phase indicating the amount of target answers produced.Sentences with a definite article were predictable.Scores could range from 0-12

Table 2 .
Scores from the opt out phase indicating the amount of times a participant opted out.It should be more appealing to opt out in case of sentences with an indefinite article.Scores could range from 0-12

225 Test items Picture pair
In het gras staat de koe In het gras staat de lino In het gras staat een lino In het gras staat het paard In het gras staat het orboIn het gras staat een orbo 'The cow/horse is standing in the grass'Opting out as a measure of meta-linguistic awareness De vogel vliegt naast de boom De vogel vliegt naast de zumu De vogel vliegt naast een zumu De vogel vliegt naast het huis De vogel vliegt naast het teku De vogel vliegt naast een teku 'The bird is flying next to the tree/house' De man draagt de broek De man draagt de tigru De man draagt een tigru De man draagt het shirt De man draagt het wolgu De man draagt een wolgu 'The man is wearing the shirt/ pants' Het meisje zit op de stoel Het meisje zit op de herbi Het meisje zit op een herbi Het meisje zit op het bed Het meisje zit op het glavi Het meisje zit op een glavi 'The girl sits on the chair/bed' De vrouw leest de krant De vrouw leest de fundi De vrouw leest een fundi De vrouw leest het boek De vrouw leest het agri De vrouw leest een agri 'The women reads the paper/ book'