Applied Scientist · AWS Generative AI Innovation Center

Mengdie (Flora) Wang

Reliable agentic AI.
From research methods to working systems.

I am an applied scientist specializing in reliable agentic AI. I study how to evaluate reasoning, quantify uncertainty, and decide when an AI system should act, ask, or defer.

I pair this research with hands-on work on agent systems, evaluation pipelines, and model customization—taking ideas into enterprise applications.

Explore my research Google Scholar LinkedIn GitHub
Mengdie (Flora) Wang

What I work on

Agentic Systems

Retrieval, tool use, orchestration, and model customization for agents that work with real data and enterprise workflows.

Evaluation & Reliability

Calibration, uncertainty, and reasoning consistency: evidence for deciding when a system can act autonomously and when it should defer.

Research Engineering

Implementing fine-tuning and evaluation pipelines, designing experiments, and turning findings into reusable tools and implementation patterns.

I’m interested in connecting statistical reliability methods with the design and evaluation of agentic systems.

Methods & experiments

Research highlights

How can we measure an agent’s reliability—and use that evidence to decide what it should do next?

EMNLP 2026 · Main TrackSAC Highlight Award

From Debate to Decision

When agents agree, is their answer safe to act on?

A post-hoc safety layer turns multi-agent deliberation into calibrated act-or-escalate decisions. The evaluation intercepted 81.9% of wrong-consensus cases while preserving high autonomous-action accuracy.

Method: Conformal social choice for multi-agent reliability.

Read paperCode
ICLR 2026 · WorkshopBest Paper Award

Quantifying Consistency in LLM Logical Reasoning

Can inconsistent reasoning reveal unreliable answers?

Consistency-aware evaluation for black-box LLM reasoning uses ranking distributions, Bradley–Terry modeling, and PageRank-style aggregation to identify unreliable reasoning instances.

Method: Structural uncertainty and reasoning-consistency evaluation.

Read paper
Findings of EACL 2026

Reflect, Rewrite, Repeat

How far can a small model go with limited reasoning data?

Built the R3 pipeline using supervised fine-tuning and direct preference optimization. With 700 training samples, the pipeline improved AIME’25@10 from 20% to 30%, producing reasoning traces for annotation-light fine-tuning.

Implementation: Data generation, fine-tuning, and reasoning evaluation in the R3 pipeline.

Read paperCode

Implementation & evaluation

Selected engineering work

Architecture, evaluation, and rollout across enterprise GenAI engagements.

Agentic retrieval for pharmaceutical R&D

Problem. Support scientific question answering with retrieval and agent orchestration.

My contribution. Led a 12-week build using semantic chunking, Titan embeddings, custom retrieval tools, and reasoning-enhanced ReAct on Bedrock Knowledge Bases and Agents.

Outcome. The engagement secured a production phase.

Source-cited analytics through a semantic layer

Problem. Make a sprawling data warehouse usable for traceable, natural-language analytics.

My contribution. Distilled the warehouse into a governed semantic layer with a FastAPI and Claude retrieval service for source-cited answers.

Focus. Connecting data meaning, retrieval, and auditable responses.

Evaluation Systems & Research Tooling

Problem. Evaluate quality across varied GenAI tasks and outputs.

My contribution. Designed evaluation workflows for retrieval, Text2SQL, agents, model customization, and video systems, including structured outputs, temporal consistency checks, rubrics, and regression criteria.

Related work: Amazon Nova samples

Writing & Research

AWS Technical Blogs

Selected Publications

View the full publication list on Google Scholar.

Selected talks & community

Industry Talks & Workshops

  • LLMs Reflecting on Reasoning — AWS re:Invent 2025, distinguished-level chalk talk (INV508), with Jae Oh Woo.
  • Robo-Reviewer: Building AI Video Evaluators with Bedrock — AWS re:Invent 2025, hands-on workshop (SPS301).
  • From Hours to Minutes: Crafting Bite-Sized Video Stories with AWS — AWS re:Invent 2025, chalk talk (DEV334), with Haochen Xie.

Research community

  • Presented Reflect, Rewrite, Repeat at MathNLP (EMNLP 2025) and EACL 2026.
  • Presented From Debate to Decision at the ICML 2026 Statistical Frameworks for Uncertainty in Agentic Systems Workshop and EMNLP 2026.
  • Organizer, 2026 Amazon Machine Learning Conference — Test-Time Scaling Workshop.

Education

M.S. in Computer Science

University of Minnesota — Twin Cities

Fall 2020

First-Year CS & Engineering Graduate Fellowship Award

B.A. in Computer Science

Macalester College · Saint Paul, MN

May 2016

Best Capstone Presentation (top 1%) · Kofi Annan full merit scholarship

Get in touch