My Experience at the Hackathon: Building an AI Face-to-Face Assistant

Recently, I had the privilege of participating in an inspiring three-day hackathon organized by Digitaal Vlaanderen and Microsoft. The event brought together innovative minds to tackle real-world challenges using AI. My team—comprised of professionals from diverse backgrounds—focused on developing an AI-powered face-to-face (f2f) assistant aimed at improving communication during live interactions between users with different linguistic backgrounds.

Here, I’d like to share my experience of the event, our technical journey, the challenges we encountered, and how we overcame them.

The Challenge: Enhancing Communication with AI

The problem we addressed was clear: live interactions often face difficulties when participants don’t share the same language. While human translators are an option, they can be costly and logistically challenging. We envisioned a tool that could:

• Provide real-time transcription and translation with minimal delay.
• Offer live suggestions, such as prompts for scheduling follow-ups.
• Summarize conversations into actionable points, ensuring privacy compliance.
• Enhance control over AI outputs, allowing validation and refinement of key details.

This solution could make interactions more efficient and inclusive. However, developing it within three days was no small feat, and we faced some tricky hurdles along the way.

Day 1: Laying the Foundation

Our first day was all about the backend—the engine driving real-time transcription, translation, and summarization. Using Python, we built core functionalities by integrating Microsoft Azure Cognitive Services for speech recognition and translation. Here’s what we achieved:

• Real-Time Transcription: Azure Speech-to-Text allowed us to convert spoken words into text quickly and accurately, with minimal latency. Initially, we struggled with ensuring low delay, but tweaking configurations and processing priorities helped us hit our goal of under five seconds.
• Translation Service: Transcribed text was fed into Azure Translation services, enabling seamless communication in multiple languages. A key issue arose when handling less commonly spoken languages, where translation accuracy dipped. By dynamically switching between models and implementing fallback strategies, we improved consistency.
• Summarization and Action Points: With the help of OpenAI’s GPT-4, we generated concise summaries and extracted actionable points. Privacy was a major consideration, and prompt engineering ensured sensitive details were flagged or sanitized as needed. Early iterations often over-summarized, so we iteratively refined prompts to strike the right balance.

One unexpected issue during this phase was handling interruptions or overlapping speech—a common occurrence in live conversations. The transcription service occasionally jumbled the inputs, creating incoherent outputs. By implementing an audio separation algorithm and prioritizing primary speaker identification, we managed to mitigate these issues significantly.

The backend architecture orchestrated these processes efficiently, creating a robust foundation for subsequent development.

Day 2: Building an Intuitive Front-End

With the backend in place, we shifted our focus to creating a user-friendly interface using Angular. Our goal was to design a tool that users could rely on without needing extensive training. Key features included:

• Real-Time Updates: The interface displayed both original and translated text, ensuring accuracy and transparency.
• Interactive Summaries: Users could review, edit, and validate AI-generated summaries before saving them.
• Live Suggestions: The system provided real-time prompts, such as opening a calendar when scheduling was discussed, to streamline sessions.

During development, one challenge we faced was ensuring the UI updated seamlessly with backend changes. We encountered occasional delays in rendering real-time updates, but by implementing WebSockets and optimizing the front-end data pipeline, we ensured near-instantaneous feedback. We also tested various layouts to find the one that minimized cognitive load while maximizing usability.

An additional hurdle was managing user expectations. Some team members questioned whether the interface could intuitively highlight AI-generated suggestions without overwhelming users. We addressed this by introducing visual cues—such as icons and color coding—to draw attention to actionable items without being intrusive.

One significant technical snag was ensuring accessibility for users with varying levels of digital literacy. To address this, we conducted user simulations and implemented features like tooltips and contextual help, making the interface more approachable.

Day 3: Privacy and Final Touches

Privacy was a critical focus on the final day. We tailored our summarization process to highlight potentially sensitive information, giving users full control over what to retain or remove. This was reflected in the front-end interface, which allowed users to easily identify flagged content.

One unexpected issue we tackled was the over-flagging of benign information. For example, names of public figures were sometimes treated as sensitive. By fine-tuning GPT-4’s instructions and adding post-processing checks, we significantly improved accuracy. Ensuring compliance with privacy standards while maintaining the relevance and utility of generated summaries was a rewarding challenge to overcome.

We also encountered an unexpected bug where summaries occasionally excluded critical conversation points. This arose due to token limitations during processing. By breaking conversations into smaller segments and applying a context-preserving merging algorithm, we managed to resolve this issue without sacrificing accuracy.

An additional focus was data security during live processing. Ensuring secure data flows between the front-end, backend, and external APIs was paramount. By using encrypted channels and implementing access controls, we safeguarded sensitive information throughout the process.

Overcoming Complex Challenges

Several unexpected obstacles tested our creativity and resilience during the hackathon. One particularly challenging moment came when the transcription service struggled with dialectical variations. This caused inconsistencies in translations, potentially undermining the tool’s reliability. We addressed this by training a lightweight model to detect and adapt to specific accents or dialects, improving the system’s versatility.

Another hurdle was balancing speed and accuracy. Early tests showed that increasing the translation speed sometimes compromised the contextual integrity of translations. By refining our processing pipelines and parallelizing tasks where possible, we struck a balance that maintained both speed and contextual accuracy.

Finally, integrating live suggestions required a delicate balance between AI autonomy and user control. Feedback loops initially overwhelmed the system, suggesting irrelevant actions during conversations. By setting thresholds for suggestion triggers and continuously evaluating context, we fine-tuned this feature to align with user expectations.

A Modular and Scalable Architecture

The architecture of our solution was designed with scalability and flexibility in mind. It included:

1. Speech Analysis Layer: Azure’s Speech-to-Text processed audio input, converting it into text for further processing.
2. Translation Layer: Azure Translation services facilitated seamless real-time communication.
3. Text Analysis Layer: OpenAI’s GPT-4 handled summarization, action point extraction, and privacy-sensitive adjustments.
4. Front-End Integration: The user interface connected seamlessly with backend services, empowering users with actionable insights.

This modular design ensured each component could evolve independently, making it adaptable for future enhancements. That adaptability came in handy as we made on-the-fly adjustments to integrate feedback from initial tests during the hackathon itself.

The Mood and the Pitch: A Thrilling Experience

The atmosphere during the three days was electric—teams buzzing with ideas, tackling challenges together, and sharing breakthroughs. Each step forward brought a palpable sense of accomplishment, and the camaraderie among participants was infectious.

Despite the intense pace, there were lighter moments too—like joking about overly ambitious features or celebrating the first successful test of a critical function. The sense of shared purpose kept spirits high, even during late-night debugging sessions.

At the end of the hackathon, we presented our solution to a panel of judges. Demonstrating live transcription, translation, summarization, and privacy controls, we showcased how this assistant could revolutionize user interactions. The response was overwhelmingly positive, and we were thrilled to be named one of the top four finalists.

Later, at Digitaal Vlaanderen’s Trefdag in Ghent—a dynamic event celebrating digital innovation in Flanders—we had another opportunity to pitch. The audience at Trefdag was diverse, ranging from tech enthusiasts to policymakers. Delivering a concise, four-minute pitch in front of such a crowd was both nerve-wracking and exhilarating. Ultimately, our solution resonated with the audience, earning us first place with 36% of the vote.

Reflection and Future Potential

Participating in this hackathon was an incredible experience, both personally and professionally. It was inspiring to collaborate with talented peers and see the potential of AI in addressing real-world challenges. While our prototype was a significant step forward, I can’t help but think about the future possibilities, such as:

• Automated Form Filling: Streamlining administrative work with validated summaries.
• Enhanced Real-Time Suggestions: Offering even more context-aware prompts to support users.
• Language Inclusivity: Expanding capabilities to support more languages, including dialects, and improving cultural nuance in translations.
• Robust Offline Modes: Developing offline capabilities for areas with limited connectivity.

The hackathon reminded me of the power of collaboration and innovation in solving complex problems. Overcoming technical challenges under time pressure made the success even sweeter. I’m excited to see how this project evolves and contributes to making interactions more efficient and accessible.

Written by: