Screening Agents in Belief Eliciting Mechanisms

This paper considers the problem of a decision maker (DM), who needs to hire an agent to assess the probability of occurrence of an event that is of interest to her. To decide the agent’s reward, the DM proposes a mechanism that will give reward based on the agent’s reported subjective probability and the actual outcome of the event. The reward mechanism needs to incentivize the expert to honestly reveal his subjective probability, and the reward has to be non-negative in all cases in order to ensure agent’s participation. In such a situation, it is possible that there are some agents who lack the expertise to assess the situation, but still participate to get sure non-negative payoff. The DM wants to screen out such uninformed agents from the informed ones. This work considers two mechanisms, and analyzes the behavior of both types of agents for the two mechanisms. It shows that screening is possible along with belief elicitation in some cases.


Introduction
The problem considered here is that of assessment of subjective probability of an event by an agent who is an expert and has no stakes in the outcome.The problem is of interest to a decision maker (DM), to whom the resolution of this uncertainty will help in taking appropriate decisions.For example, bidders in an oil field auction may be interested in knowing the chances of finding oil at a particular location.A pharmaceutical company may be interested in knowing the chances of success of a drug trial.In such cases, the DM hires an agent, who is an expert in the matter, and she offers him a reward based on his prediction and the actual outcome.
The DM wants to devise a mechanism that will incentivize the agent to use his expertise and arrive at correct assessment of the situation.One such method was proposed by Brier (1950) [1] using scoring rules for elicitation of subjective probabilities that incentivize the agent to report truthfully.In such methods, the payoff to the agent has to be non-negative to ensure his participation.This non-negative payoff attracts some agents who are either incapable of assessing the situation, or who want to exert no efforts.The DM wants to screen out the latter type of agents.This present work considers the problem where the DM uses a mechanism that incentivizes the agent who works to correctly assess the situation, and it is possible to identify the agent who lacks expertise or who shirks.
The next sub-section reviews the literature on belief elicitation methods.Section 2 describes the problem setting and assumptions.The two mechanisms and the behavior of both types of agents in case of both the mechanisms are studied in Section 3. The last section discuses the scope for further work and theoretical and practical implications.

Literature Review
Brier (1950) [1] first proposed Quadratic Scoring Rules (QSR) to evaluate the performance of weather forecasts.These are proper scoring rules in the sense that it induces the agent to truthfully report his subjective probability.The QSRs work when the agent is risk neutral.Related methods are proper scoring rules, promissory notes method, and lotteries method (Kadane and Winkler (1988) [2] and Savage (1971) [3]).
Schlag and Van der Weele (2013) [4] have used QSRs to determine the probability of reward to agents.They show that deterministic schemes work only when risk preferences of agents are known, and hence there is a need to randomize the payoff.The randomization is done by using the QSR to determine the probability with which an agent gets the reward.Lotteries with the reward probabilities determined by the Quadratic Scoring Rule (QSR) ensure truth revealing incentive compatibility as the probability of winning is determined by the QSR, and agents try to maximize it.Thus, agents try to maximize the probability of winning and end up reporting truthfully.Their method can be used for eliciting probability, mean, median, variance and covariance from agents with unknown risk preferences.Sandroni and Shmaya (2013) [5] have addressed the same problem by starting with a proper score, normalizing it through a linear transformation, and using the new score as the probability with which the agent gets the higher reward.
Allen (1987) [6] showed a method for eliciting the probability of occurrence of an event where the agent's utility function is not known.Here, the agent receives a fixed reward if the reported probability is less than a randomly drawn number, or a lesser reward otherwise.This randomly drawn number is drawn from two different distributions depending upon whether the event has occurred or not.Karni (2009) [7] has developed a novel mechanism where the resulting game ensures that agent's truthfully reveal their subjective probabilities.
Hossain and Okui (2013) [8] have presented a parsimonious method for constructing scoring rules that can be used to elicit an agent's beliefs without any assumptions on the agent's risk preference.Here, the agent receives a fixed prize if the prediction error loss function is less than a randomly generated number, or else the prize is smaller.The method can be adapted to various contexts by suitably defining the loss function.The rewards are fixed and the only thing that gets decided is which of the two rewards is to be given.Hence, the rule is labeled as the Binarized Scoring Rule.
All these methods consider the probability elicitation problem under the assumption that the agent has the expertise to correctly assess the situation.To ensure participation from agents, the rewards are strictly non-negative.It is quite possible that some agents do not have the expertise to assess the situation, but they still participate to get a non-negative reward by "playing the system" (Brier, 1950) [1].There is a need to screen out such participants.Sandroni (2014) [9] has shown that a contract exists, where the informed agents truthfully report their assessments, and the uninformed agents echo the decision maker's (DM's) prior belief.The existence of such a contract is proven, but the contract itself is not elaborated.Hao and Houser (2012) [10] study a belief elicitation mechanism in a laboratory setup where they consider agents who can analyze and respond to incentives, along with other naïve agents whose responses add noise to the information being elicited.Olszewski and Peaski (2011) [11] discuss the problem that the DM cannot verify the expertise of the agent, but show that contracts do exist that can be used to get the best payoffs without knowing whether the agent is an expert or not.

Problem Setting
A decision maker (DM) is interested in knowing the probability of occurrence of an event E, and hires an agent to assess the situation and report his subjective probability for the event E. The DM designs a mechanism where she offers a contract in the form of a scoring rule to the agent.The scoring rule determines the payment to the agent, which is based on the prediction made by him and the actual outcome.

Types of Agents: Expert and Novice
The agent may or may not have the capability to assess the situation, or may be willing or unwilling to exert efforts.We identify two cases arising out of this scenario.In the first case, the agent is capable of assessing the situation and is willing to exert efforts (we label this type of agent as an "expert"); whereas in the second case, the agent is either incapable of assessing the situation or is unwilling to exert any efforts (we label this type of agent as a "novice").Sandroni (2016) has labelled the cases as informed and uninformed agents.Experts assess the situation, get the correct estimates, but report the numbers that maximize their utility payoff based on the scoring rule.Novices cannot (or do not) assess the situation and report the numbers that maximize their utility based on the scoring rule and assumptions about occurrence of event E. The expert exerts efforts and assesses ( ) to be the probability of the event E. The novice assumes that the probability of the event " n p " comes from [ ] ~0,1 Uniform .

Risk Preferences
A bet on an event has a payoff x if the event occurs, and y if the event does not occur, and is denoted as E x y .A lottery ( ) , , l p x y gives a payoff of x with probability p, and payoff of y with probability ( )

Eliciting Mechanism and Analysis
We consider two eliciting mechanisms, and analyze the behavior of both types of agents for these mechanisms.The first mechanism is the same as Karni (2009), the second one is a modification of Allen (1989).

Eliciting Mechanism-1 Mechanism 1: Karni (2009)
1) A random number r is selected from a uniform distribution on [ ] 0,1 , and the agent is asked to submit his report, of his assessment of the subjective probability of the event.
2) The mechanism gives the agent : E x y µ ≥ , and the lottery ( ) , , l r x y if r µ ≤ , where x y > .

Case 2: Agent Is a Novice
A novice cannot (or does not) assess the situation, and in the absence of any information tries to maximize his reward based on the scoring rule and his prior notion of occurrence of the event.If the novice reports n µ to be his subjective probability of the event, his payoff is : The expected payoff is maximized for 0.5 , and hence the novice will report this as his subjective probability.
In this mechanism, the expert truthfully reveals his subjective probability as his payoff is maximized when the reported probability is same as actual assessed probability.The novice reports the subjective probability to be 0.5, as his expected utility is maximized here.Based on the reports from both types of agents, the DM will be able to screen out the novice, except in the case where the subjective probability in the expert's assessment is also 0.5.Thus, the mechanism proposed by Karni (2009) serves the screening purpose, except in one case as discussed before.

Eliciting Mechanism-2 Mechanism 2: Proposed Mechanism Consider two distributions A and B.
Distribution A has density function ( ) ( ) A coin is tossed.In case of Heads, a random number r is selected from distribution A, and in case of Tails, it is selected from distribution B. The agent is asked to submit his report, , of his assessment of the subjective probability of the event.
2) The mechanism gives the agent : E x y µ ≥ , and the lottery ( ) Allen (1989) has proposed a similar mechanism, where the distribution is chosen later depending upon the occurrence of the event E. But, in this mechanism the distribution is chosen randomly.

Case 2: Agent Is a Novice
As in the earlier case, the novice cannot (or does not) assess the situation, and assumes that the probability of its occurrence comes from a uniform distribution on [ ] 0,1 .He calculates his expected value of payoffs, and submits the report ( n µ ) that maximizes his expected payoff.
If distribution A is chosen, the expected payoff of novice is given by The expected payoff is maximized at 0.5 n µ = , and minimized at 1 n µ = .The novice will report 0.5 to be his subjective probability.If distribution B is chosen, the expected payoff of novice is given by The expected payoff is maximized at 0.5 , and minimized at 0 n µ = .The novice will report 0.5 to be his subjective probability.
The expert truthfully reveals his subjective probability in this case also.The novice reports 0.5 for both distributions of the random number.Based on these reports, it is possible for the DM to screen out the novice as in the previous case.

Discussion
This problem of assessing subjective probabilities with the help of an expert was discussed here.A decision maker (DM) hires an agent to assess the probability of occurrence of an event, and pays him based on the reported probability and the actual outcome.Two mechanisms were considered, in which the informed agent (expert) truthfully reveals his subjective probability assessment.The possibility that the non-negative reward mechanism might attract uninformed agents (novices) was also considered.It is found that both the mechanisms can screen out the novices except in the case when the expert's assessment is also 0.5.
There is scope for further work in this area firstly by considering novices with more general assumptions about prior distribution of occurrence of the event.Mechanisms that screen out novices in all the cases also need to be devised.There is ample scope for work using laboratory experiments to verify the effectiveness of the various belief elicitation mechanisms that are proposed in theory.Setting up of laboratory experiments with experts and novices can be implemented simply by giving different information to the agents, thereby making them informed and uninformed in the experimental setup.
The problem is practically relevant and decision makers who participate in bidding of natural resources often find themselves in such a situation.It is important for the decision maker to be able to screen out the novices, so that only the suggestions provided by the experts can be taken into consideration.Some other applications involve screening out recommendations of naïve stock market analysts and naïve respondents in large scale surveys.This short paper suggests a way forward for addressing the joint problem of screening out novices and incentivizing the experts in belief elicitation mechanisms.

Mechanism- 1 3. 2 . 1 .
Case 1: Agent Is an ExpertAn expert exerts efforts and correctly assesses ( )E πto be the probability of the event E.He reports the right estimate ( )E π, as truthful reporting is his unique dominant strategy(Karni, 2009).If the expert reports 's payoff is the same as truthful reporting for all values of the random number r except for y instead of β .Since, in the expert's assessment, ( )

>
no- vice goes by his assumption that probability of event E, ( )E π is a draw from a uniform distribution on [ ] 0,1 .To simplify the analysis, we consider 1 and 0 x y = = .The analysis is not affected by these values, as this is a linear transformation.It amounts to deducting y from the payoff and dividing by(  )    x y − .The payoff for the novice is now : .The expected payoff of novice is given by

Mechanism- 2 3. 4 . 1 .
Case 1: Agent Is an ExpertIn this case, truthful report is in the expert's best interest, as in the previous mechanism.The probability of occurrence of r does not affect the reasoning.So, an expert correctly assesses ( ) The expert's preference relation on the set of lotteries displays probabilistic sophistication and domination in the sense that Karni, 2009ly if p p′≥ (as inKarni, 2009).The novice is risk neutral and goes by expected utility maximization.