Overview
EyeCook is an award-winning augmented reality application designed to eliminate the daily friction of deciding what to cook. By simply taking a picture of available ingredients, users receive AI-generated recipes and hands-free, step-by-step cooking guidance directly through Snap Spectacles. This solves the common problem of using messy, inconvenient phones in the kitchen and empowers novice cooks, like students living alone for the first time, to create delicious meals with confidence.
Our solution integrates a powerful generative AI backend with three distinct frontend clients: a cutting-edge AR experience on Snap Spectacles, a native iOS application, and a responsive web app. This multi-platform approach ensures broad accessibility while showcasing a futuristic, hands-free cooking interface. The project was developed over a 24-hour hackathon and was recognized as the Winner of the Snap AR Challenge at HackPSU Fall 2024.
Key Features
-
AI-Powered Recipe Generation: Leverages Google's Gemini Pro 1.5 to analyze an image of ingredients, intelligently generate multiple recipe options, identify potential allergens, and create a descriptive dish image using the Flux Schnell model.
-
Hands-Free AR Cooking Guidance: Displays step-by-step instructions, ingredient lists, and interactive timers directly in the user's field of view on Snap Spectacles. This allows for a seamless, sanitary, and intuitive cooking process without needing to touch a screen.
-
Multi-Platform Experience: While the core innovation lies in the AR interface, the system is fully accessible via a native iOS app built with UIKit and a web application built with React, ensuring all users can browse, save, and view recipes.
-
Redundant & Scalable Cloud Architecture: The entire system is powered by a robust backend deployed on a multi-node Kubernetes cluster on Oracle Cloud, mirrored across US East and US West zones for high availability and fault tolerance.
Technologies & Implementation
We engineered a sophisticated, distributed system to deliver a real-time, AI-driven experience across multiple clients. The backend, built with Node.js and Express, serves as the core logic hub, orchestrating requests between the frontends and our AI service endpoints.
As the Lead Backend and Lead AR Developer, I was responsible for architecting and building the core systems that powered the EyeCook experience. I developed the robust Node.js backend, creating custom Gemini Pro JSON schemas to handle our AI features, including ingredient identification, recipe generation, and AI-powered image creation. I built the API that seamlessly integrated with Google's Gemini and the Flux image model, ensuring efficient data flow to all frontend clients. On the infrastructure side, I configured our Oracle Cloud environment, deploying the application on a region-redundant Kubernetes cluster for high availability, and handled the domain setup. Finally, I crafted the Snap AR interface, tying together the AI, cloud infrastructure, and augmented reality frontend to deliver a truly cutting-edge system.
graph TD
subgraph User Interfaces
A["Snap Spectacles (Lens Studio)"]
B["iOS App (UIKit)"]
C["Web App (React.js)"]
end
subgraph Backend on Oracle Cloud K8s
E["Encrypted REST API (Node.js/Express)"]
F{"Image Processing (Sharp)"}
G["AI Orchestration Service"]
end
subgraph AI Services
H["Google Gemini Pro 1.5 <br/>(Ingredient ID, Recipe Gen)"]
I["HuggingFace Spaces <br/>(Flux Schnell Image Gen)"]
end
subgraph Database
J["MySQL"]
end
A & B & C -->|Image Upload & Requests| E
E --> F
F --> G
G --> H
G --> I
G --> J
J --> E
E -->|Recipe Data & Instructions| A & B & C
Challenges & Solutions
The most significant challenge was developing for the brand-new Snap Spectacles 2024. As a bleeding-edge platform, documentation was sparse, and the development workflow was unfamiliar. We overcame this steep learning curve through direct collaboration with the Snap engineering team, whose guidance was invaluable. My role involved rapidly prototyping the API calls within the TypeScript environment for the Spectacles, translating our backend JSON responses into data structures that the AR Lens could render. This required iterative testing and creative problem-solving to create a stable connection.
A second challenge was maintaining feature parity and a consistent API contract across three separate frontend codebases developed in parallel. To prevent integration failures, we enforced strict adherence to our API specification and established a clear communication protocol. This ensured that as the backend evolved, all three client applications could adapt in unison without breaking, which was critical to our success in the compressed timeframe of a hackathon.
Results & Impact
The project culminated in a fully functional prototype that powerfully demonstrated the potential of combining AR and generative AI for practical, everyday applications.
-
Won the Snap AR Challenge: The project was selected by Snap as the best and most innovative use of their AR technology at HackPSU Fall 2024.
-
Rapid Prototyping: We successfully delivered a complex, multi-platform application—including a novel AR interface—from concept to working demo in just 24 hours.
-
Technical Showcase: The project served as a successful proof-of-concept for building highly available, AI-driven applications on a Kubernetes-based cloud architecture, handling image processing and multiple generative AI model integrations.
{TODO: Add screenshot of the Snap Spectacles AR interface showing a recipe step}