When Should AI Ask About Your Meal?

Keeping a useful food diary takes effort. People need to remember what they ate, identify ingredients, and estimate portions. A meal photo makes logging quicker, but a picture cannot show everything that affects its nutritional value: what oil was used, how a dish was prepared, or what ingredients are hidden inside it. Asking a long list of follow-up questions can recover some of that detail, while making the process tiring.

For my MSc thesis at Wageningen University & Research, I explored how an AI food logging system could decide when a follow-up question is useful. I built Snap and Say, a multimodal app for older adults that combines a meal photo with optional voice input and asks for clarification only when its routing logic detects enough ambiguity to warrant it.

What the Existing Research Was Missing

The thesis began with a scoping review of AI systems for nutrition in older adults. Among the 25 studies that met the review criteria, 23 (92%) described systems still at the prototype or pilot stage, and 21 (84%) relied on private or otherwise unavailable datasets. The review also found recurring gaps in usability for older adults and in the connection between research prototypes and clinical workflows.

The review informed the design of Snap and Say. I wanted to test an approach that could reduce manual logging, handle uncertainty in a meal description, and make its decisions about follow-up questions inspectable.

A Meal Log That Can Ask for Context

Snap and Say lets someone photograph a meal and optionally describe it by voice. The system analyzes those inputs, estimates where the meal description is ambiguous, and combines that assessment with food-category rules in a transparent complexity score. A configurable threshold determines whether it proceeds with the estimate or starts a short clarification exchange inspired by the USDA Automated Multiple-Pass Method.

The exchange is bounded to at most two rounds. That limit keeps the interaction focused and prevents the system from turning a quick log into a lengthy interview. The routing is designed to ask more when the available information is unclear, and to leave straightforward entries alone.

What the Evaluation Found

I compared three modes on 500 meals from the Nutrition5k dataset: a single photo-based estimate, selective clarification through the agentic system, and an always-ask approach. Calorie mean absolute error ranged from 93.6 to 96.7 kcal across the three modes. The confidence intervals overlapped, and the analysis found no statistically significant difference in calorie error.

The clearest measured gain was in interaction effort. The selective system generated 56.7% fewer clarification questions than the always-ask mode and had 40.1% lower average latency (28.9 seconds compared with 48.3 seconds). It withheld questions for 74.6% of simple meals and routed 71.1% of complex meals to clarification.

This benchmark used an automated Oracle to provide ground-truth details whenever the system asked a question. It isolates the routing and estimation workflow under cooperative conditions; it does not show how much nutritional accuracy will improve when people answer questions in everyday use. In this test, clarification reduced interaction cost without a measurable change in calorie error.

Trying the App With Older Adults

In a separate usability evaluation, 19 adults aged 65 and older used the app to log meals. Most reported low task complexity and time demand, and participants generally did not find the clarification step cumbersome. Some did point to areas for improvement, including response speed and microphone or speech recognition issues.

This was a small, controlled session, so it cannot establish whether people will keep using the app over time or whether it improves health outcomes. The sample was also culturally homogeneous. Longer studies with more diverse participants and everyday meals are needed to evaluate sustained use and performance beyond the benchmark.

A Starting Point for Lower-Friction Food Tracking

Snap and Say shows how a food logging system can make follow-up questions conditional on the uncertainty it detects, rather than asking everyone the same questions. In this thesis, the routing architecture’s strongest demonstrated result was a reduction in questions and processing time while calorie estimates remained similar across test modes.

The full system is available in the Snap and Say GitHub repository. The next steps are to test the app over longer periods, improve voice interaction, and find out which kinds of clarification provide useful nutritional information in real-world use.