Annotator guidelines
- Each conversation has been flagged as a candidate by an LLM pipeline, which is instructed to flag even ambiguous cases so it doesn't miss any. Your job is to decide what codes actually apply, based on the codebook. Disagreements with the LLM are expected. You may also apply any codes you think the LLM missed.
- Use your best judgment to decide which codes apply based on the code's definition, including its Exclusions and Examples.
- Look across all the user's turns and make your best judgment about the user's intent. For example, if a user says "give me a line graph" in a conversation where every preceding turn has been about modifying a Python program for handling a dataset, you might reasonably infer that the user is continuing to request Python code as the deliverable and not an actual line graph.
- If you aren't sure a code applies because of ambiguity in the user's language, err on the side of NOT applying it.
- If a prompt appears to be pasted from elsewhere (a homework assignment, a jailbreak template, etc.), treat it as the user's actual request. You can leave a note about the framing in "Notes".
- WildChat doesn't have misconceptions, users do. If the WildChat assistant responds to a user with a misleading statement ("I ran the code for you", "I visited that URL..."), this is not itself an example of a user misconception. We are coding what the user asks for. But you might leave a Note about it.