多模态与模块化AI厨师：从图像生成复杂食谱
The Multimodal And Modular Ai Chef: Complex Recipe Generation From Imagery
AI总结：
本文提出一种基于图像模型与LLM串行结合的轻量级模块化API方法，解决了从冰箱食材照片生成满足复杂约束食谱的问题，并证明其优于缺乏连贯记忆的单体多模态模型。
摘要：
AI社区已采用多感官或多模态方法来推动这一代AI模型，使其更接近预期的智能理解能力。结合语言和图像是针对图像描述或基于描述生成图像等特定任务的常用方法。本文将这些单体方法与一种轻量级且专业化的方法进行比较，后者基于使用图像模型标记物体，然后将生成的物体列表串行提交给大型语言模型（LLM）。这种使用多个应用程序编程接口（API）的方法在正确物体列表上实现了超过95%的平均精度均值，这些列表作为最新Open AI文本生成器（GPT-4）的输入。为了展示API作为模块化替代方案，我们解决了用户拍摄冰箱中可用食材照片，然后生成针对成本、准备时间、饮食限制、份量及多餐计划等复杂约束量身定制的新颖食谱卡的问题。该任务）。研究结论是，单体多模态模型目前缺乏维持上下文和格式所需的连贯记忆，而且直到最近，像GPT-2/3这样的语言模型在处理类似问题时，若不退化成重复或无意义的食材组合便无法格式化格式化（类似问题）。首次地（这），一位AI厨师或烹饪者似乎不仅成为可能，而且提供了以实用方式增强人类食谱库的增强能力。这项工作生成了一本100页的食谱书，使用超过2000张冰箱图像作为初始化列表，展示了三十种顶级食材。
英文摘要：
The AI community has embraced multi-sensory or multi-modal approaches to advance this generation of AI models to resemble expected intelligent understanding. Combining language and imagery represents a familiar method for specific tasks like image captioning or generation from descriptions. This paper compares these monolithic approaches to a lightweight and specialized method based on employing image models to label objects, then serially submitting this resulting object list to a large language model (LLM). This use of multiple Application Programming Interfaces (APIs) enables better than 95% mean average precision for correct object lists, which serve as input to the latest Open AI text generator (GPT-4). To demonstrate the API as a modular alternative, we solve the problem of a user taking a picture of ingredients available in a refrigerator, then generating novel recipe cards tailored to complex constraints on cost, preparation time, dietary restrictions, portion sizes, and multiple meal plans. The research concludes that monolithic multimodal models currently lack the coherent memory to maintain context and format for this task and that until recently, the language models like GPT-2/3 struggled to format similar problems without degenerating into repetitive or non-sensical combinations of ingredients. For the first time, an AI chef or cook seems not only possible but offers some enhanced capabilities to augment human recipe libraries in pragmatic ways. The work generates a 100-page recipe book featuring the thirty top ingredients using over 2000 refrigerator images as initializing lists.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
