What should an emotion representation capture?

This is a research agenda and proposed experiment, not a report of completed results.

I’m interested in what an emotion representation should preserve when it becomes part of an AI system. A useful starting point is to separate the representation itself from the behavior we want it to support.

The question

What information should a representation carry so that it is useful for understanding context and generating an appropriate response?

I want to compare three possible starting points: discrete emotion labels, continuous dimensions such as valence and arousal, and learned representations. The goal is to understand the tradeoffs for a defined task, rather than assume that one representation fits every use.

A small experiment to design

Start with one task: predicting how listeners rate the emotional expression of a short speech sample. Compare representations using the same data splits and downstream evaluation wherever possible.

Before running the experiment, specify:

• The dataset, its permitted uses, and the source of the emotion annotations.
• Which aspects of emotion the labels are intended to describe.
• A speaker-disjoint test split, with no test examples used for model selection.
• Simple baselines, the training budget, and evaluation metrics.
• How differences between annotators and uncertainty will be reported.

What I would look for

Beyond an aggregate score, I want to inspect ambiguous examples, disagreement between listeners, and cases where the words and delivery suggest different interpretations. A label prediction alone would not demonstrate that a system understands a person’s internal state.

Reproduction plan

The eventual experiment should include the exact data preparation, split identifiers, model versions, configuration, environment, seeds, and commands. Any unavailable data or artifacts should be named explicitly, with their effect on reproducibility explained.

Next step

Define the task and assemble a focused literature review before selecting a representation or making empirical claims. Results and artifacts will be added after the experiment is run.