AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials
Uloženo v:
| Vydáno v: | arXiv.org (Dec 12, 2024), p. n/a |
|---|---|
| Hlavní autor: | |
| Další autoři: | , , , , , , |
| Vydáno: |
Cornell University Library, arXiv.org
|
| Témata: | |
| On-line přístup: | Citation/Abstract Full text outside of ProQuest |
| Tagy: |
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
MARC
| LEADER | 00000nab a2200000uu 4500 | ||
|---|---|---|---|
| 001 | 3144198202 | ||
| 003 | UK-CbPIL | ||
| 022 | |a 2331-8422 | ||
| 035 | |a 3144198202 | ||
| 045 | 0 | |b d20241212 | |
| 100 | 1 | |a Xu, Yiheng | |
| 245 | 1 | |a AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials | |
| 260 | |b Cornell University Library, arXiv.org |c Dec 12, 2024 | ||
| 513 | |a Working Paper | ||
| 520 | 3 | |a Graphical User Interface (GUI) agents hold great potential for automating complex tasks across diverse digital environments, from web applications to desktop software. However, the development of such agents is hindered by the lack of high-quality, multi-step trajectory data required for effective training. Existing approaches rely on expensive and labor-intensive human annotation, making them unsustainable at scale. To address this challenge, we propose AgentTrek, a scalable data synthesis pipeline that generates high-quality GUI agent trajectories by leveraging web tutorials. Our method automatically gathers tutorial-like texts from the internet, transforms them into task goals with step-by-step instructions, and employs a visual-language model agent to simulate their execution in a real digital environment. A VLM-based evaluator ensures the correctness of the generated trajectories. We demonstrate that training GUI agents with these synthesized trajectories significantly improves their grounding and planning performance over the current models. Moreover, our approach is more cost-efficient compared to traditional human annotation methods. This work underscores the potential of guided replay with web tutorials as a viable strategy for large-scale GUI agent training, paving the way for more capable and autonomous digital agents. | |
| 653 | |a User interface | ||
| 653 | |a Visual tasks | ||
| 653 | |a Graphical user interface | ||
| 653 | |a Annotations | ||
| 653 | |a Applications programs | ||
| 653 | |a Synthesis | ||
| 653 | |a Online tutorials | ||
| 653 | |a Trajectory planning | ||
| 653 | |a Task complexity | ||
| 653 | |a Training | ||
| 700 | 1 | |a Lu, Dunjie | |
| 700 | 1 | |a Shen, Zhennan | |
| 700 | 1 | |a Wang, Junli | |
| 700 | 1 | |a Wang, Zekun | |
| 700 | 1 | |a Mao, Yuchen | |
| 700 | 1 | |a Xiong, Caiming | |
| 700 | 1 | |a Yu, Tao | |
| 773 | 0 | |t arXiv.org |g (Dec 12, 2024), p. n/a | |
| 786 | 0 | |d ProQuest |t Engineering Database | |
| 856 | 4 | 1 | |3 Citation/Abstract |u https://www.proquest.com/docview/3144198202/abstract/embedded/6A8EOT78XXH2IG52?source=fedsrch |
| 856 | 4 | 0 | |3 Full text outside of ProQuest |u http://arxiv.org/abs/2412.09605 |