5 minute read

Meta info.

TL; DR

ํ–‰๋™ ์‹ ํ˜ธ(์„ ํ˜ธ ์Œ)์—์„œ ์‚ฌ์šฉ์ž ์„ ํ˜ธ๋ฅผ ์ž์—ฐ์–ด ์„œ์ˆ ๋กœ ๊ท€๋‚ฉ ์ถ”๋ก ํ•˜๋„๋ก 7B reasoning model์„ cold-start SFT + GRPO๋กœ ํ•™์Šตํ•˜๋ฉด, ์ด์ „ ์„œ์ˆ  + ์ƒˆ ์‹ ํ˜ธ๋งŒ ๋„ฃ๋Š” streaming ๊ฐฑ์‹ ๊นŒ์ง€ ๊ฐ€๋Šฅํ•ด์ง€๊ณ  ๊ฐ™์€ ํฌ๊ธฐ baseline๋ณด๋‹ค, ์ผ๋ถ€ ์„ค์ •์—์„œ๋Š” golden preference๋ณด๋‹ค๋„ ๋†’์€ judge ์ •ํ™•๋„๋ฅผ ๋‚ธ๋‹ค. (์„ ํ˜ธ ์ถ”๋ก ์„ "์„ค๋ช… ๊ฐ€๋Šฅํ•œ ์ƒํƒœ ๊ฐฑ์‹ "์œผ๋กœ ๋ฐ”๊พผ๋‹ค.)
  • ์™œ: ๊ธฐ์กด ๊ฐœ์ธํ™”๋Š” ํ–‰๋™ ์‹ ํ˜ธ๋ฅผ prompt, parameter, latent์— ๊ทธ๋Œ€๋กœ ๋„ฃ๋Š” direct mapping์ด๋ผ ์„ ํ˜ธ๊ฐ€ ๋ถˆํˆฌ๋ช…ํ•˜๊ณ , ์ƒˆ ์‹ ํ˜ธ๊ฐ€ ๋“ค์–ด์˜ฌ ๋•Œ๋งˆ๋‹ค ๋Š˜์–ด๋‚˜๋Š” ์ „์ฒด history๋ฅผ ๋‹ค์‹œ ์ฒ˜๋ฆฌํ•ด์•ผ ํ•จ
  • ์ œ์•ˆ: ์„ ํ˜ธ ์ถ”๋ก ์„ โ€œ์‹ ํ˜ธ ์ง‘ํ•ฉ + ์ด์ „ ์„ ํ˜ธ ์„œ์ˆ  โ†’ reasoning chain + ์ƒˆ ์„ ํ˜ธ ์„œ์ˆ โ€๋กœ ์ •์˜ํ•˜๊ณ , teacher ํ•ฉ์„ฑ ๋ฐ์ดํ„ฐ cold-start ํ›„ judge ๊ธฐ๋ฐ˜ reward๋กœ GRPO ํ•™์Šตํ•œ AlignXplore-7B
  • ๋ฐœ๊ฒฌ: streaming ์„ค์ •์—์„œ ALIGNX-test 71.47%๋กœ DeepSeek-R1-671B(70.47%)๋ณด๋‹ค ๋†’๊ณ , judge ๊ธฐ๋ฐ˜ reward๊ฐ€ ์ƒ์„ฑ ๊ธฐ๋ฐ˜ reward๋ณด๋‹ค ์•ˆ์ •์ ์ด๋ฉฐ, ์„ฑ๋Šฅ ํ–ฅ์ƒ์€ cold-start๋ณด๋‹ค RL์—์„œ ๋Œ€๋ถ€๋ถ„ ๋‚˜์˜ด

Overview (Fig 1) Two-stage training (Fig 2) Main results (Tab 1)

Online win rate (Tab 2) Reward function comparison (Tab 4)

Generalization and robustness (Tab 3)

Preference reversal and signal count (Fig 3) Streaming efficiency over rounds (Fig 6)

Reversing the first pair (Tab 6) Preference-description word clouds (Fig 4)

Background

  • ๊ฐœ์ธํ™” alignment๋Š” โ€œ๋ชจ๋‘์—๊ฒŒ ๊ฐ™์€ helpfulnessโ€๋ฅผ ๋„˜์–ด์„œ ๊ฐœ๋ณ„ ์„ ํ˜ธ์— ๋งž์ถ”๋Š” ๋ฌธ์ œ์ธ๋ฐ,
    ์„ ํ˜ธ๋Š” ๋Œ€๋ถ€๋ถ„ ๋ช…์‹œ๋˜์ง€ ์•Š๊ณ  ์„ ํƒ, ๊ฒŒ์‹œ๊ธ€, ์ธ๊ตฌํ†ต๊ณ„ ๊ฐ™์€ ํฉ์–ด์ง„ ์‹ ํ˜ธ์— ์ˆจ์–ด ์žˆ์Œ Fig 1
    • ์ˆ˜ํ•™, ์ฝ”๋”ฉ ๊ฐ™์€ deductive task์™€ ๋Œ€๋น„๋˜๋Š” inductive reasoning ๋ฌธ์ œ๋กœ ๋ด„
      : ์ •ํ•ด์ง„ ๊ทœ์น™ ์—†์ด noisyํ•œ ์ฆ๊ฑฐ์—์„œ ์ผ๋ฐ˜ ๊ทœ์น™์„ ๋Œ์–ด๋‚ด์•ผ ํ•จ
  • ๋™์‹œ์— extended reasoning(๊ธด chain of thought + RL)์€ deductive task์—์„œ๋งŒ ์ฃผ๋กœ ๊ฒ€์ฆ
  • Direct mapping ๊ณ„์—ด ๊ฐœ์ธํ™”: ํ–‰๋™ ์‹ ํ˜ธ๋ฅผ ๋ช…์‹œ์  ์„ ํ˜ธ ์—†์ด ๋ชจ๋ธ์— ๋ฐ”๋กœ ์—ฐ๊ฒฐ
    • prompt์— ๋„ฃ๊ธฐ: ํ–‰๋™ ์˜ˆ์‹œ๋ฅผ ๊ทธ๋Œ€๋กœ context๋กœ (Xu et al., 2022, Lee et al., 2024)
    • parameter์— ๋„ฃ๊ธฐ: ์‚ฌ์šฉ์ž๋ณ„ ํ•™์Šต (Kang et al., 2023, Tan et al., 2024)
    • latent์— ๋„ฃ๊ธฐ: VPL (Poddar et al., 2024)์ฒ˜๋Ÿผ ์‚ฌ์šฉ์ž ์ž ์žฌ ๋ณ€์ˆ˜๋กœ reward๋ฅผ ์กฐ๊ฑดํ™”
    • ํ•œ๊ณ„: ๋ฌด์—‡์„ ์„ ํ˜ธํ•œ๋‹ค๊ณ  ํŒ๋‹จํ–ˆ๋Š”์ง€ ๋ณด์ด์ง€ ์•Š๊ณ , ์ƒˆ ์‹ ํ˜ธ๊ฐ€ ์˜ค๋ฉด ๋ˆ„์  history ์ „์ฒด๋ฅผ ๋‹ค์‹œ ๋„ฃ์–ด์•ผ ํ•จ (scalability ๋ณ‘๋ชฉ)
  • ๋ช…์‹œ์  ์„ ํ˜ธ ์ถ”์ถœ ๊ณ„์—ด: history์—์„œ ์‚ฌ์šฉ์ž profile์ด๋‚˜ ์„ ํ˜ธ ์ ์ˆ˜๋ฅผ ๋จผ์ € ๋ฝ‘๊ธฐ
    • PBA (Li et al., 2025): ํ–‰๋™ ์˜ˆ์‹œ๋ฅผ ๋ฏธ๋ฆฌ ์ •ํ•œ ์„ ํ˜ธ ์ฐจ์›์˜ ์ ์ˆ˜๋กœ mappingํ•œ ๋’ค ์ž์—ฐ์–ด๋กœ ๋ณ€ํ™˜, (ALIGNX์˜ prior work)
    • LMInductReason (Qiu et al., 2024): LLM์˜ ๊ฐ€์„ค ์ƒ์„ฑ, ์ •์ œ๋ฅผ ๋ฐ˜๋ณตํ•˜๋Š” ๊ท€๋‚ฉ ์ถ”๋ก 
    • ํ•œ๊ณ„: ๋ฏธ๋ฆฌ ์ •ํ•œ ์ฐจ์›์— ๋ฌถ์ด๊ฑฐ๋‚˜(PBA), ์งง์€ ์ถ”๋ก ์œผ๋กœ out-of-domain์— ์•ฝํ•จ
  • ํฉ์–ด์ง„ ํ–‰๋™ ์‹ ํ˜ธ์—์„œ โ€œ์„ค๋ช… ๊ฐ€๋Šฅํ•œ ์„ ํ˜ธ ์„œ์ˆ โ€์„ ๊ท€๋‚ฉ์ ์œผ๋กœ ๋งŒ๋“ค๊ณ , ์ƒˆ ์‹ ํ˜ธ๊ฐ€ ์˜ค๋ฉด ๊ทธ ์„œ์ˆ ๋งŒ ๊ฐฑ์‹ ํ•˜๋Š” ํ•™์Šต๋œ ์ถ”๋ก ๊ธฐ๊ฐ€ ์—†๋”๋ผโ€ฆ

Problem States

  • ์„ ํ˜ธ ์ถ”๋ก ์˜ ์ถœ๋ ฅ์ด ๋‹ค์šด์ŠคํŠธ๋ฆผ ๋ชจ๋ธ๊ณผ ๋ฌด๊ด€ํ•˜๊ฒŒ ์žฌ์‚ฌ์šฉ ๊ฐ€๋Šฅํ•œ ์ž์—ฐ์–ด ์„œ์ˆ ์ด์–ด์•ผ ํ•จ
  • ์ƒˆ ์‹ ํ˜ธ๊ฐ€ ์˜ฌ ๋•Œ ์ „์ฒด history ์—†์ด ์ด์ „ ์„œ์ˆ  + ์ƒˆ ์‹ ํ˜ธ๋งŒ์œผ๋กœ ๊ฐฑ์‹ ํ•  ์ˆ˜ ์žˆ์–ด์•ผ ํ•จ
  • ์ •๋‹ต ์„ ํ˜ธ ์„œ์ˆ ์ด ์—†์œผ๋ฏ€๋กœ, ์„œ์ˆ ์˜ ์งˆ์„ โ€œ๊ทธ ์„œ์ˆ ๋กœ ๋‹ค์šด์ŠคํŠธ๋ฆผ ๋ชจ๋ธ์ด ์‚ฌ์šฉ์ž์˜ ์‹ค์ œ ์„ ํƒ์„ ๋งžํžˆ๋Š”๊ฐ€โ€๋กœ ๊ฐ„์ ‘ ์ธก์ •ํ•ด์•ผ ํ•จ
  1. [์ •์˜] streaming ์„ ํ˜ธ ์ถ”๋ก  ๊ณผ์ œ์™€ ๊ธฐ๋Šฅ์  ํ‰๊ฐ€(reward) ์„ค๊ณ„
  2. [ํ•™์Šต 1] teacher ํ•ฉ์„ฑ ๋ฐ์ดํ„ฐ๋กœ cold-start SFT
  3. [ํ•™์Šต 2] ๋‘ ๋ผ์šด๋“œ rollout์„ ์„ž์€ GRPO

Suggestions

[์ •์˜] Streaming Preference Inference์™€ ๊ธฐ๋Šฅ์  reward

  • ๊ณผ์ œ: ํ–‰๋™ ์‹ ํ˜ธ $\mathcal{E}=\lbrace e_1,\dots,e_T\rbrace$์™€ ์ด์ „ ์„ ํ˜ธ ์š”์•ฝ $\hat{d}$๋ฅผ ๋ฐ›์•„ reasoning chain $r$๊ณผ ์ƒˆ ์„ ํ˜ธ ์„œ์ˆ  $d$๋ฅผ ์ƒ์„ฑ

    \[(r, d) = \mathcal{M}(\mathcal{E}, \hat{d})\]
    • ์ฒซ ์ถ”๋ก ์—์„œ๋Š” $\hat{d}$๊ฐ€ ๋น„์–ด ์žˆ๊ณ , ์ดํ›„์—๋Š” ์ง์ „ ์„œ์ˆ ์ด ๊ทธ๋Œ€๋กœ ๋“ค์–ด๊ฐ โ†’ ์„œ์ˆ  ์ž์ฒด๊ฐ€ โ€œ์••์ถ•๋œ ์ƒํƒœโ€
    • ์ฃผ ์‹คํ—˜์˜ ์‹ ํ˜ธ๋Š” ๋น„๊ต ํŒ๋‹จ(์งˆ๋ฌธ + ์„ ํ˜ธ ์‘๋‹ต / ๋น„์„ ํ˜ธ ์‘๋‹ต), UGC(๋ฆฌ๋ทฐ, ๊ฒŒ์‹œ๊ธ€)๋„ ๊ฐ€๋Šฅ (ยง4.3)
  • ํ‰๊ฐ€์™€ reward: held-out ์„ ํ˜ธ ์Œ $(x, y_w, y_l)$์—์„œ $d$๋ฅผ ์กฐ๊ฑด์œผ๋กœ ์ค€ ๋‹ค์šด์ŠคํŠธ๋ฆผ ๋ชจ๋ธ์ด ์‚ฌ์šฉ์ž ์„ ํƒ์„ ๋งžํžˆ๋Š”๊ฐ€

    • R_jud: ์„ ํ˜ธ ํŒ์ • ๋ชจ๋ธ $S_{jud}$๊ฐ€ $d$๋ฅผ ๋ณด๊ณ  $y_w$๋ฅผ ๋” ๋†’๊ฒŒ ํŒ์ •ํ•˜๋ฉด 1 (ร— format ๊ฒ€์‚ฌ)
    • R_gen: ์ƒ์„ฑ ๋ชจ๋ธ $S_{gen}$์—์„œ $d$๋ฅผ ๋ถ™์˜€์„ ๋•Œ $y_w$์˜ log-prob ์ด๋“์ด $y_l$๋ณด๋‹ค ํฌ๋ฉด 1
    • ์ถ”๋ก  ๋ชจ๋ธ์€ held-out ์Œ์„ ๋ณด์ง€ ๋ชปํ•˜๊ณ , ๋‘ ์‘๋‹ต ์ˆœ์„œ๋Š” ๋ฌด์ž‘์œ„ โ†’ ํŠน์ • query์— ๋งž์ถฐ $d$๋ฅผ ๊พธ๋ฏธ๊ฑฐ๋‚˜ ์œ„์น˜ ํŽธํ–ฅ์„ ์“ฐ๋Š” shortcut์„ ๋ง‰๋Š”๋‹ค๋Š” ์ฃผ์žฅ
    • ํ•™์Šต, ํ‰๊ฐ€ ๋ชจ๋‘ R_jud ์œ„์ฃผ (model: frozen Qwen2.5-7B-Instruct)

[ํ•™์Šต 1] Cold-start SFT Fig 2

  • ๊ฐ ์‹ ํ˜ธ์—์„œ ๋จผ์ € ํ•ต์‹ฌ ์„ ํ˜ธ ์ฐจ์› $\phi$(์—ฐ๋ น๋Œ€, ์„ฑ๋ณ„, ํˆฌ์ž ์„ฑํ–ฅโ€ฆ)๋ฅผ ๋ฝ‘๊ณ , teacher(QwQ-32B)์— $\phi$์™€ ์‹ ํ˜ธ๋ฅผ ํ•จ๊ป˜ ์ค˜์„œ reasoning chain + ์„ ํ˜ธ ์„œ์ˆ ์„ G๊ฐœ ์ƒ์„ฑ
  • streaming ๋Œ€๋น„: ๊ฐ™์€ ์‚ฌ์šฉ์ž์˜ ์ƒˆ ์‹ ํ˜ธ 4๊ฐœ + ์ฒซ ๋ผ์šด๋“œ ์ถœ๋ ฅ ์ค‘ ๋ฌด์ž‘์œ„๋กœ ๊ณ ๋ฅธ $\hat{d}$(๋งž๋Š” ๊ฒƒ, ํ‹€๋ฆฐ ๊ฒƒ ๋ชจ๋‘ ๊ฐ€๋Šฅ)๋กœ 2๋ผ์šด๋“œ ๋ฐ์ดํ„ฐ๋ฅผ ๋งŒ๋“ฆ
  • reward = 1์ธ ์ƒ˜ํ”Œ๋งŒ ๋‚จ๊ฒจ SFT (7,000๊ฐœ ์ค‘ 1๋ผ์šด๋“œ 3,980๊ฐœ, 2๋ผ์šด๋“œ 5,278๊ฐœ ์ฑ„ํƒ)

[ํ•™์Šต 2] Two-round GRPO

  • GRPO, KL penalty ์ œ๊ฑฐ, G = 4 rollouts
  • ์‚ฌ์šฉ์ž๋งˆ๋‹ค (1) $\hat{d}$ ์—†์ด G๊ฐœ ์ƒ์„ฑ, (2) ๋‹ค๋ฅธ ์‹ ํ˜ธ ์ง‘ํ•ฉ $\mathcal{E}โ€™$ + 1๋ผ์šด๋“œ ์ถœ๋ ฅ์—์„œ ๋ฝ‘์€ $\hat{d}$๋กœ G๊ฐœ ์ƒ์„ฑ โ†’ ๋‘ ๋ผ์šด๋“œ 2G๊ฐœ๋ฅผ ํ•ฉ์ณ ์ตœ์ ํ™”
  • advantage๋Š” ๋ผ์šด๋“œ๋ณ„๋กœ ๋”ฐ๋กœ ์ •๊ทœํ™”

    \[A_i = \frac{R_i - \text{mean}(\lbrace R_j \rbrace_{j=1}^{G})}{\text{std}(\lbrace R_j \rbrace_{j=1}^{G})}\]
  • ํ•™์Šต์€ 2๋ผ์šด๋“œ๊นŒ์ง€๋งŒ, ํ‰๊ฐ€์—์„œ 4๋ผ์šด๋“œ๊นŒ์ง€ ํ™•์žฅ (App A.5)

Effects

  • Experimental setup
    • data: ALIGNX (Li et al., 2025)์—์„œ ์žฌ๊ตฌ์„ฑํ•œ 10,000๊ฐœ(ํ•™์Šต 7,000 / ํ…Œ์ŠคํŠธ 3,000), ์‚ฌ์šฉ์ž ๋‹จ์œ„ ๋ถ„๋ฆฌ
      • ๊ฐ target ์Œ๋งˆ๋‹ค ์ตœ์†Œ 5๊ฐœ ์„ ํ˜ธ ์ฐจ์›์—์„œ history ์ „์ฒด๊ฐ€ ์ผ๊ด€๋œ ๋ฐฉํ–ฅ์„ ๊ฐ–๋„๋ก ํ•„ํ„ฐ๋ง
      • ALIGNX-test: 90๊ฐœ ์„ ํ˜ธ ์ฐจ์›, 3,000๊ฐœ / out-of-domain P-SOUPS (Jang et al., 2023): expertise, informativeness, style ๊ฐ 600๊ฐœ
    • model: DeepSeek-R1-Distill-Qwen-7B backbone (๋ถ€๋ก์—์„œ Qwen3-8B)
    • setting: Base(์‹ ํ˜ธ 4๊ฐœ ๋˜๋Š” 8๊ฐœ ํ•œ ๋ฒˆ์—) vs Streaming(4๊ฐœ๋กœ $\hat{d}$ ๋งŒ๋“  ๋’ค ์ƒˆ 4๊ฐœ + $\hat{d}$)
    • baselines: Null, ์› ์‹ ํ˜ธ ๊ทธ๋Œ€๋กœ(E), Golden Preference / LMInductReason, VPL, EXPO, PBA / Qwen2.5-7B, DS-R1-Distill-7B, Qwen3-32B, QwQ-32B, GPT-4, DeepSeek-R1-671B๊ฐ€ ์ƒ์„ฑํ•œ ์„œ์ˆ 
    • evaluation: ACC_jud, ACC_gen, GPT-4 win rate(400๊ฑด), ์‚ฌ๋žŒ ํ‰๊ฐ€(300๊ฑด, 4๋ช…)
  • Results
    • ์„ ํ˜ธ ์ถ”๋ก  ์ž์ฒด๊ฐ€ ํ•„์š”ํ•˜๊ณ , ๊ธด ์ถ”๋ก ์ด ์œ ๋ฆฌํ•จ Tab 1
      • ์› ์‹ ํ˜ธ๋ฅผ ๊ทธ๋Œ€๋กœ ๋„ฃ๋Š” E(50.33%)๋Š” Null(51.37%)๊ณผ ์ฐจ์ด ์—†์Œ โ†’ ์‹ ํ˜ธ๋ฅผ โ€œ์ฝ๊ธฐโ€๋งŒ์œผ๋กœ๋Š” ์•ˆ ๋˜๊ณ  ์ถ”๋ก ์ด ํ•„์š”
      • ๊ฐ™์€ ๊ณ„์—ด์—์„œ thinking์ด non-thinking๋ณด๋‹ค ๋†’์Œ: Qwen3-32B 65.03 vs 57.60, DeepSeek-R1 70.47 vs GPT-4 66.10
      • VPL, EXPO ๊ฐ™์€ latent ๊ณ„์—ด๊ณผ LMInductReason์€ 50% ๊ทผ์ฒ˜, PBA๋Š” in-domain 62.77์ด์ง€๋งŒ P-SOUPS Style 31.33์œผ๋กœ ์ผ๋ฐ˜ํ™” ์‹คํŒจ
    • AlignXplore-7B๋Š” streaming์—์„œ ๊ฐ€์žฅ ๊ฐ•ํ•จ Tab 1
      • Base(4๊ฐœ) 65.33 โ†’ Streaming 71.47, ์‹ ํ˜ธ 8๊ฐœ๋ฅผ ํ•œ ๋ฒˆ์— ๋„ฃ์€ Base๋ณด๋‹ค ๋†’์Œ
      • backbone์€ ์‹ ํ˜ธ๋ฅผ ๋Š˜๋ฆฌ๊ฑฐ๋‚˜ streaming์œผ๋กœ ์ค˜๋„ ๊ฑฐ์˜ ์ด๋“ ์—†์Œ
      • ALIGNX-test์—์„œ Golden Preference๋ณด๋‹ค๋„ ๋†’์Œ
        • ์ €์ž๋Š” golden์„ ์ƒํ•œ์ด ์•„๋‹Œ ์˜๋ฏธ์  ์ฐธ์กฐ๋กœ ๋ด์•ผ ํ•œ๋‹ค๊ณ  ํ•ด์„
        • ๊ธฐ๋Šฅ์  ์ง€ํ‘œ๋Š” ํŒ์ • ๋ชจ๋ธ์ด ์“ฐ๊ธฐ ์‰ฌ์šด ์„œ์ˆ ์„ ๋†’๊ฒŒ ์นจ
      • ๋‹จ๊ณ„ ablation: w/o RL 61.80, w/o Cold-start 62.80 โ†’ RL ์ชฝ ๊ธฐ์—ฌ๊ฐ€ ํผ
    • Online, ์‚ฌ๋žŒ ํ‰๊ฐ€๋„ ๊ฐ™์€ ๋ฐฉํ–ฅ Tab 2
      • GPT-4 win rate: AlignXplore vs Qwen2.5-7B 63.00%(ALIGNX), 57.67%(P-SOUPS)
      • ์‚ฌ๋žŒ ํ‰๊ฐ€ overall 7.93 vs 7.11 (DS-R1-Distill streaming ๋Œ€๋น„)
        • specificity ์ฐจ์ด๋งŒ ์œ ์˜ํ•˜์ง€ ์•Š์Œ
    • ์ž…๋ ฅ ํ˜•์‹, ๋‹ค์šด์ŠคํŠธ๋ฆผ ๋ชจ๋ธ์ด ๋ฐ”๋€Œ์–ด๋„ ์œ ์ง€๋˜์ง€๋งŒ ๊ฒฉ์ฐจ๋Š” ์ค„์–ด๋“ฆ Tab 3
      • ์„ ํ˜ธ ์Œ ๋Œ€์‹  UGC๋กœ ์ถ”๋ก : 61.97%, DeepSeek-R1-671B์™€ ๋™๋ฅ 
      • ํŒ์ • ๋ชจ๋ธ์„ QwQ-32B, DeepSeek-R1์œผ๋กœ ๋ฐ”๊ฟ”๋„ 68.53, 67.59๋กœ 7B baseline๋ณด๋‹ค ๋†’์Œ
      • ์„ ํ˜ธ ์Œ์„ ์ „๋ถ€ ๋’ค์ง‘์–ด๋„ ALIGNX 62.13(โˆ’3.2)๋กœ ๋น„๊ต์  ์œ ์ง€
        • ๋ฐ์ดํ„ฐ์…‹ ํŽธํ–ฅ์„ ์™ธ์šด ๊ฒŒ ์•„๋‹ˆ๋ผ๋Š” ๊ทผ๊ฑฐ๋กœ ์ œ์‹œ
    • ์„ ํ˜ธ ๋ณ€ํ™”์™€ ์‹ ํ˜ธ ์ˆ˜ Fig 3, Tab 6
      • 8๊ฐœ ์‹ ํ˜ธ ์ค‘ ์•ž์ชฝ๋ถ€ํ„ฐ 0~3๊ฐœ๋ฅผ ๋’ค์ง‘์œผ๋ฉด Base๋Š” 64.3 โ†’ 50.67๋กœ ๋ฌด๋„ˆ์ง€๊ณ , Streaming์€ 71.47 โ†’ 56.73์œผ๋กœ ๊ฐ™์€ ์กฐ๊ฑด์—์„œ ํ•ญ์ƒ ๋” ๋†’์Œ
      • ์ฒซ ์Œ๋งŒ ๋’ค์ง‘๊ณ  ๋‚˜๋จธ์ง€์™€ test ์Œ์€ ์ตœ์ข… ์„ ํ˜ธ์— ๋งž์ถ˜ ๊ฒฝ์šฐ: Base(8๊ฐœ) 58.17 vs Streaming 66.60
      • ์‹ ํ˜ธ 1๊ฐœ๋งŒ์œผ๋กœ๋„ 61.23%(backbone 55.83%), backbone์€ 4๊ฐœ ์ดํ›„ ์˜คํžˆ๋ ค ๋–จ์–ด์ง
    • Streaming์€ ํšจ์œจ์ ์ด๊ณ  ๋ผ์šด๋“œ๊ฐ€ ๋Š˜์–ด๋„ ๊ฒฌ๋”ค Fig 6
      • 4๋ผ์šด๋“œ(์‹ ํ˜ธ 16๊ฐœ)์—์„œ Base๋Š” latency 396ms, ์ •ํ™•๋„ ๊ธ‰๋ฝ, Streaming์€ 157ms, 72.30% ์œ ์ง€
      • ํ•™์Šต์€ 2๋ผ์šด๋“œ๊นŒ์ง€๋งŒ ํ–ˆ๋Š”๋ฐ๋„ 4๋ผ์šด๋“œ๊นŒ์ง€ ์„ฑ๋Šฅ์ด ์œ ์ง€๋จ
    • (Finding 2) judge ๊ธฐ๋ฐ˜ reward๊ฐ€ ์ƒ์„ฑ ๊ธฐ๋ฐ˜๋ณด๋‹ค ๋‚ซ๋‹ค Tab 4
      • R_jud๋กœ ํ•™์Šตํ•œ ๋ชจ๋ธ์ด ACC_gen๊นŒ์ง€ ํฌํ•จํ•ด ๋Œ€๋ถ€๋ถ„ ์ง€ํ‘œ์—์„œ ์šฐ์„ธ, RL ๊ณก์„ ๋„ ๋” ์•ˆ์ •์  (๋ถ€๋ก A.6)
      • ์ €์ž ํ•ด์„: ์‘๋‹ต log-prob์—๋Š” ์œ ์ฐฝ์„ฑ, ๊ธธ์ด ๊ฐ™์€ ๊ต๋ž€ ์š”์ธ๊ณผ offline ์‘๋‹ต noise๊ฐ€ ์„ž์—ฌ ์žˆ์Œ
    • (Finding 1) ํ•™์Šต ๋‹จ๊ณ„์— ๋”ฐ๋ผ ์„œ์ˆ ์ด ์ผ๋ฐ˜๋ก  โ†’ ์ฐจ์› ๋‚˜์—ด โ†’ ์‹คํ–‰ ๊ฐ€๋Šฅํ•œ ์ง€์นจ์œผ๋กœ ๋ฐ”๋€œ Fig 4
      • backbone์€ โ€œhistoricalโ€, โ€œsituationโ€ ๊ฐ™์€ ์ผ๋ฐ˜์–ด, cold-start ํ›„ โ€œcommunication styleโ€ ๊ฐ™์€ ์ฐจ์›์ด ๋“ฑ์žฅ
        • ๋‘ ๋‹จ๊ณ„๋ฅผ ํ•ฉ์น˜๋ฉด โ€œavoidโ€, โ€œprioritizeโ€ ๊ฐ™์€ ํ–‰๋™ ์ง€์นจํ˜• ํ‘œํ˜„์ด ๋Š˜์–ด๋‚จ
      • ์ €์ž๋Š” ์ด๋ฅผ ์‚ฌ๋žŒ์˜ ๊ท€๋‚ฉ ์ถ”๋ก  ๊ณผ์ •(๊ธฐ๋ณธ ํŠน์ง• ํŒŒ์•… โ†’ ์‹คํ–‰ ๊ฐ€๋Šฅํ•œ ๊ฐ€์„ค)๊ณผ ๋‹ฎ์•˜๋‹ค๊ณ  ํ•ด์„
    • cold-start ๋ฐ์ดํ„ฐ ํ’ˆ์งˆ์— ๋‘”๊ฐ (App A.10)
      • ์•ฝํ•œ teacher(DS-R1-Distill-7B)๋กœ ๋งŒ๋“  ์ €ํ’ˆ์งˆ ๋ฐ์ดํ„ฐ๋กœ cold-startํ•˜๋ฉด RL ์ „์—๋Š” backbone๋ณด๋‹ค ๋‚ฎ์ง€๋งŒ(56.87), RL ํ›„ ํšŒ๋ณต๋˜๋”๋ผ

Personal note. ์„ ํ˜ธ๋ฅผ ์ž์—ฐ์–ด ์„œ์ˆ ๋กœ ๋ฝ‘๊ณ , ์ƒˆ ์„ธ์…˜์—์„œ๋Š” ์ด์ „ ์„œ์ˆ  + ์ƒˆ ์‹ ํ˜ธ๋งŒ์œผ๋กœ ๋‹ค์‹œ ์“ด๋‹ค๋Š” ๊ตฌ์กฐ๊ฐ€ PRefine๊ณผ ๊ฑฐ์˜ ๊ฐ™์€ ๋ฐฉ์‹์œผ๋กœ ์ ‘๊ทผํ–ˆ๋Š”๋ฐ, ๋‹ค๋งŒ ์—ฌ๊ธฐ์„œ๋Š” RLํ•™์Šตํ•ด์„œ ์ƒˆ๋กœ ์“ด๋‹ค๋Š” ๊ฒŒ ์ฐจ์ด๊ฐ€ ์žˆ๊ธด ํ•ฉ๋‹ˆ๋‹ค. ์ด ๊ณผ์ •์—์„œ ์ œ ๊ฐœ์ธ์ ์œผ๋กœ ๋А๋ผ๊ธฐ์—๋Š” ๊ฒ€์ฆ ๋‹จ๊ณ„์˜ ์œ ๋ฌด ์ •๋„์˜ ๋ฐฉ์–ด์  ์„œ์ˆ ์€ ๊ฐ€๋Šฅํ•  ๊ฒƒ ๊ฐ™์€๋ฐ,,
๋ณ„๊ฑด์ด์ง€๋งŒ ๊ฐ™์€ ํ•™ํšŒ์— AlignXplore+ ๋ผ๋Š” ๋™์ €์ž์˜ ํŽ˜์ดํผ๋„ main accept๋œ ๊ฒƒ ๊ฐ™์•„์„œ ๊ฐ™์ด ํ™•์ธํ•ด๋ณด๊ณ ์ž ํ•ฉ๋‹ˆ๋‹ค.