skip to content
← (projects)

gazeaide.jpg

GAZEAIDE

JetBrains IDE plugin that uses implicit eye-tracking gaze patterns to trigger AI code explanations during code review.

Context

GAZEAIDE (the plugin is published as "Implicit — Gaze-Triggered AI Assistance for Code Review") is a gaze-aware developer tool Heidi builds as a Research Engineering Intern in the HAX (Human-AI Experience) group, ML Research department, JetBrains Research Munich, supervised by Ilya Zakharov (Senior Researcher). The internship runs April–September 2026 under the project title "Enhancing Tech Creation through Multimodal Context".

The research question: can an IDE notice when a developer is struggling — purely from where and how they look at code — and offer AI help without the developer ever asking? Prior work such as GazeCopilot still needs an explicit keypress; GAZEAIDE's contribution is fully implicit triggering. Gaze is the first modality in a broader multimodal vision (gaze → voice → gesture).

What I did

  • Framed the research. Defined five scientifically grounded gaze triggers: T1 debugging fixation, T2 code-review clustering, T3 documentation shuttle, T4 cognitive-load collapse and T5 navigation anticipation. Key design insight: one gaze stream serves two purposes — cognitive-state detection (T1–T4 feed AI prompts) and context-switch prediction (T5 silently preloads files).
  • Wrote and defended the research proposal in LaTeX with TikZ/pgfgantt figures (system architecture, gaze-processing pipeline, 18-week Gantt chart). Every claim is grounded in literature: Bednarik & Tukiainen 2006, Uwano et al. 2006, Fritz & Begel et al. 2014, Rodeghero et al. 2014, Kevic et al. 2015, Sharif et al. 2012, Peitek et al. 2022, Serim & Jacucci (CHI 2019), CodeGRITS (ICSE 2024) and GazeCopilot.
  • Iterated with supervisor feedback, reframing GAZEAIDE as a general gaze-aware plugin with the triggers as a thin evaluation layer, and treating the LLM backend as model-agnostic.
  • Built and shipped the plugin (Java, IntelliJ Platform). It runs in IntelliJ IDEA and every other JetBrains IDE (PyCharm, WebStorm, GoLand, Rider, …):
    • Reading mode: a circle follows your reading position in the gutter.
    • Explaining mode: when gaze or mouse rests on a line for 3 seconds, the line is highlighted and an LLM explains it inline, using the surrounding function (extracted via PSI) as context.
    • Two interchangeable gaze sources: mouse dwell as a zero-hardware gaze proxy, or a real webcam eye tracker — UnitEye's browser pipeline (webcam → MediaPipe FaceLandmarker → EyeMU model via onnxruntime-web) running headlessly inside the IDE's embedded browser (JCEF). No Python or Unity install needed.
    • 9-point calibration with least-squares affine correction and exponential smoothing to damp jitter.
    • Session recording and export to CSV, JSON and a self-contained HTML heatmap + replay page, plus an in-IDE reading-coverage report.
    • Model-agnostic AI provider settings (OpenAI key or any local OpenAI-compatible endpoint; Groq/Llama 3.3 70B was used in the prototype).
  • Designed the VR/eye-tracker path. Architected the gaze-to-token pipeline: OpenXR gaze ray → ray-plane intersection with the virtual screen → UV normalisation → pixel mapping → IntelliJ PSI lookup (via CodeGRITS), with I-DT fixation filtering. Researched a device-agnostic GazeProvider adapter layer on OpenXR's XR_EXT_eye_gaze_interaction. This work became the separate open-source tool CodeGaze (see its own entry).
  • Hardware: started on the Pico 4 Enterprise; now building on the Bigscreen Beyond 2e headset.

Tech

Java · Kotlin · IntelliJ Platform SDK / PSI · Gradle · JCEF · MediaPipe · ONNX Runtime Web · OpenXR · Python (earlier gaze service with asyncio FSM + gRPC) · LLM APIs (OpenAI, Groq) · LaTeX/TikZ · pymovements + DuckDB for gaze analysis

Why this shows I can…

  • Run a research project end to end: from literature review and a defended proposal to a published, installable tool.
  • Combine eye tracking, IDE internals and LLMs in one working pipeline.
  • Design for real users: implicit, non-interruptive assistance rather than yet another chat panel.

Results

  • Plugin published on the JetBrains Marketplace (plugin ID 33763), version 0.3.0.
  • Working end-to-end prototype with both mouse and webcam gaze, recording/export and calibration.
  • Open-source spin-off CodeGaze for VR gaze recording.
  • Planned evaluation: within-subjects study comparing implicit gaze triggers vs. an explicit shortcut vs. no assistance, using code review of AI-generated code as the scenario. Candidate venues: UIST, VISSOFT, ICSME Visions, ASE NIER/tool demo, EMIP.

Vision

Reimagining coding environments in 3D and making programming more productive in 3D spaces — an IDE that understands what you're looking at, and eventually what you say and gesture.

Links