Last updated:Ā Aug, 2026
CV | Google Scholar | LinkedIn | Github
Embedded Files
Jihyun Lee
Hello! I am a Ph.D. candidate at POSTECH (Pohang University of Science and Technology), advised by three wonderful professors Prof. Gary Geunbae Lee, Prof. Yunsu Kim, and Prof. Hyounghun Kim in NLPlab@POSTECH.
My research aims to develop goal-oriented and socially aligned conversational intelligence. I am particularly interested in dialogue generation and evaluation, LLM alignment and personalization, and reinforcement learningābased simulation environments for improving real-world conversational agents, especially in counseling and mental health support contexts.
Ā Contact: jihyunlee [at] postech.ac.krĀ Ā
š° News
2026-06-15: I joined Amazon AWS AI in UK as an Applied Scientist Intern, working on Alexa.
2026-04-06: Our proactive, CBT-structured counseling framework, "PSY-STEP", is now available on arXiv.Ā
2025-10-27: Our keyword-based ASR error augmentation "Speak & Spell" has been accepted to AACL-IJCNLP 2025 Main. āØĀ
2025-08-21: Our LLM simulation based counselor model "Panic to Calm" has been accepted to EMNLP 2025 Main.āØ
2025-08-21: Our multimodal counseling dialogue "Mirror" has been accepted to EMNLP 2025 Main. āØ
2025-07-16: Our preference-based dialogue clustering has been accepted to the DSTC12@SIGDIAL 2025 (paper).
2025-01-23: Our multimodal personalized TOD dataset "PicPersona-TOD" has been accepted to NAACL 2025 Main.āØ
š Research Highlights
PSY-STEP: Structuring Therapeutic Targets and Action Sequences for Proactive Counseling Dialogue Systems (arXiv 2026, link)
Jihyun Lee, Yejin Min, Yejin Jeon, SungJun Yang, Hyounghun Kim, Gary Geunbae Lee
Can an AI counselor truly deliver meaningful therapeutic support beyond surface-level empathy?
This question motivates our work on PSY-STEP, which introduces a proactive and structured framework for CBT-based counseling. PSY-STEP explicitly distinguishes surface-level problems from underlying automatic thoughts by operationalizing therapeutic plans. Building on this dataset, we develop a counseling agent that proactively elicits automatic thoughts and executes strategy-consistent interventions. To further enhance therapeutic effectiveness and responsiveness, we refine the model via reinforcement learning in a simulated counseling environment. Together, this work demonstrates how LLM-based counselors can move beyond reactive responses toward cognitively grounded and clinically aligned support.
PanicToCalm: A Proactive Counseling Agent for Panic Attacks (EMNLP 2025, link)
Jihyun Lee, Yejin Min, San Kim, Yejin Jeon, SungJun Yang, Hyounghun Kim, Gary Lee
Can an AI counselor truly help someone in the midst of a panic attack?
This question motivated our work on PanicToCalm, which introduces PACE, a dataset of high-distress episodes constructed from first-person narratives, and PACER, a counseling model designed to deliver real-time Psychological First Aid (PFA). Trained within a simulation environment using the DPO algorithm, PACER demonstrates proactive and emotionally grounded intervention during real psychological emergencies. Together, this work provides a blueprint for developing LLM-based counseling agents capable of responsive and human-aligned crisis support.Ā
MIRROR: Multimodal Cognitive Reframing Therapy for Rolling with Resistance (EMNLP 2025, link)
Subin Kim*, Hoonrae Kim*, Jihyun Lee*, Yejin Jeon*, Gary Lee (* Equal contribution)
Can facial expressions influence how counselors understand and respond to client resistance ā and if so, how can we address the lack of multimodal data for such complex interactions?
To address this challenge, we present MIRROR, a synthesized multimodal counseling dataset that combines facial expressions, textual dialogue, and resistance-type annotations. Using this dataset, we train a visionālanguage counseling model that learns to interpret client resistance and adjust counselor responses accordingly.
Experimental results show that incorporating facial cues significantly enhances the modelās ability to detect resistance and generate adaptive, empathetic interventions.Ā
PicPersona-TOD: A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Persona (NAACL 2025, link)
Jihyun Lee, Yejin Jeon, Seungyeon Seo, Gary Geunbae Lee
How can we make task-oriented dialogue agents respond with personality and style, rather than generic, flat utterances?
To answer this, we introduce PicPersona-TOD, a š¼ļømultimodal dataset that enriches user personas with images and contextual attributes such as age or emotional state. Leveraging this, we train Pictor, a vision-language model that generates personalized responses aligned with visual persona cues.
Human evaluations show that incorporating image-based persona information significantly improves engagement and response appropriateness across previously unseen domains.Ā
š Publications
PSY-STEP: Structuring Therapeutic Targets and Action Sequences for Proactive Counseling Dialogue Systems
Jihyun Lee, Yejin Min, Yejin Jeon, SungJun Yang, Hyounghun Kim, Gary Geunbae Lee
arXiv, 2026
# Counseling # CBTĀ # Planning # Simulation-DPO
link
Speak & Spell: LLM-Driven Controllable Phonetic Error Augmentation for Robust Dialogue State Tracking
Jihyun Lee, Solee Im, Wonjun Lee, Gary Geunbae Lee
IJCNLP-AACL Main, 2025
# DST # ASR-errorĀ # Robustness
linkPanicToCalm: A Proactive Counseling Agent for Panic Attacks
Jihyun Lee, Yejin Min, San Kim, Yejin Jeon, SungJun Yang, Hyounghun Kim, Gary Lee
Empirical Methods in Natural Language Processing (EMNLP), Main, 2025
# Dialogue ModelĀ # Counseling # Dialogue Synthesis # Simulation-DPO
linkMIRROR: Multimodal Cognitive Reframing Therapy for Rolling with Resistance
Subin Kim*, Hoonrae Kim*, Jihyun Lee*, Yejin Jeon*, Gary Lee (* Equal contribution)
Empirical Methods in Natural Language Processing (EMNLP), Main, 2025
# Dialogue Model # Counseling # Dialogue Synthesis # Multi-Modal Ā
linkProgressive Facial Granularity Aggregation with Bilateral Attribute-based Enhancement for Face-to-Speech Synthesis
Yejin Jeon, Youngjae Kim, Jihyun Lee, Hyounghun Kim, Gary Lee
Empirical Methods in Natural Language Processing (EMNLP), Findings, 2025
# TTS #Face-to-Speech
linkThe Limits of Post-hoc Preference Adaptation: A Case Study on DSTC12 Clustering
Jihyun Lee,Ā Gary Lee
The Twelfth Dialog System Technology Challenge (DSTC12) Workshop @ SIGDIAL 2025
#Ā Dialogue Theme Clustering
linkPicPersona-TOD: A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Persona
Jihyun Lee, Yejin Jeon, Seungyeon Seo, Gary Geunbae Lee
Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), Main, 2025
# Task-oriented Dialogue # TOD Style-transfer # Dataset Synthesis # Multi-Modal
linkPrompt-Guided Selective Masking Loss for Context-Aware Emotive Text-to-Speech
Yejin Jeon, Youngjae Kim, Jihyun Lee, Gary Geunbae Lee
Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), Findings, 2025
# TTS # TTS with Dialogue
link
Inference is All You Need: Self Example Retriever for Cross-domain Dialogue State Tracking with ChatGPT
Jihyun Lee, Gary Geunbae Lee
arXiv, 2024
# Zero-shot, Dialogue State Tracking
linkExploring the Viability of Synthetic Audio Data for Audio-Based Dialogue State Tracking
Jihyun Lee*, Yejin Jeon*, Wonjun Lee, Yunsu Kim and Gary Geunbae Lee
IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2023
#Ā DSTĀ # ASRĀ # Synthesized Audio
linkTracking Must Go On : Dialogue State Tracking with Verified Self-Training
Jihyun Lee, Chaebin Lee, Yunsu Kim and Gary Geunbae Lee
Interspeech, 2023
# DST
linkDORIC : Domain Robust Fine-Tuning for Open Intent Clustering through Dependency Parsing
Jihyun Lee, Seungyeon Seo, Yunsu Kim, Gary Geunbae Lee
The Eleventh Dialog System Technology Challenge (DSTC11) Workshop at SIGDIAL & INLG 2023
# Dialogue ThemeĀ Detection
linkExploring Back Translation with Typo Noise for Enhanced Inquiry Understanding in Task-Oriented Dialogue
Jihyun Lee, Junseok Kim, Gary Geunbae Lee
The Eleventh Dialog System Technology Challenge (DSTC11) Workshop at SIGDIAL & INLG 2023
# Task-oriented Dialogue # TODĀ # Ā Augmentation
link
SF-DST: Few-Shot Self-Feeding Reading Comprehension Dialogue State Tracking with Auxiliary Task
Jihyun Lee, Gary Geunbae Lee
Interspeech, pp. 1233-1237, 2022
# DST
link
šAwards
Global Innovation Award, ICT Challenge 2025 (Minister of Science and ICT Award)
Jihyun Lee, Wonjun Lee, Sungjun Yang ā Multi-modal, Multi-session Counseling System
Ministry of Science and ICT, Republic of Korea (ź³¼ķźø°ģ ģ ė³“ķµģ ė¶ ģ„ź“ģ), 2025Ā Ā
š¼ļø Photos
Page updated
Google Sites
Report abuse