Work/AI/№01
Conversational AI
SDU
Designed, implemented, and stress-tested a specialized conversational agent to evaluate how system prompts manage safety-critical boundaries under conversational pressure.
Open-context Large Language Models (LLMs) deployed as automated health and lifestyle coaches suffer from a fundamental compliance bias. Because foundational models are trained to prioritize user helpfulness and conversational harmony, their built-in drive to satisfy the user systematically overrides static system constraints under conversational pressure.
In a healthcare or fitness context, this architectural vulnerability causes models to experience severe role decay, validate unscientific claims, ignore physical safety risks, and cross ethical boundaries into personalized medical prescription — creating significant compliance and user-safety hazards.
The strategy focused on engineering a user-centered system prompt framework across four structural iterations (V1–V4). The final system architecture decoupled the agent from a passive context-memory assistant into a strict, rules-based state machine.
01
Conducted a quantitative stakeholder survey (N=31) with a tech-literate cohort concentrated in IT and Business. Participants prioritized routine building, consistency, and functional mobility over isolated tracking metrics. Strict guardrails emerged against authoritarian lecturing, condescension, and monetization.
02
Ran controlled user trials (N=10) across six standardized scenario matrices using the Chatbot Usability Scale (BUS-15). Conversation analysis revealed an absolute usability-safety divergence: despite a 100% positive tone rating, the transformer architecture committed four systematic errors under pressure.
03
Engineered V4 to restructure system behavior without external wrappers. Added a mandatory baseline verification layer, preference locking to prevent style drift, and top-level domain restriction loops to block context-history contamination from out-of-scope requests.
Process
Achieved a 90% positive consensus for the agent's anti-shaming and setback management modules, proving system prompts can successfully regulate register and empathy parameters neutrally.
Secured absolute safety isolation during critical trauma and clinical tasks; the agent immediately terminated coaching loops, communicated its lack of diagnostic authority, and redirected users to certified human practitioners.
Empirically demonstrated that prompt engineering alone is mathematically insufficient for absolute domain containment under adversarial conditions, providing a technical blueprint for the integration of upstream programmatic intent classification layers in production environments.
90%
Positive consensus for anti-shaming and setback management tone
100%
High-risk medical containment during critical trauma tasks
4
Systematic error categories isolated under conversational pressure
V4
Final rules-based state machine with mandatory verification layer
Next