Signal
Signal is a retrieval-augmented question-answering system built during an internship. It grounds every answer in the company's real documentation and refuses to answer when the evidence is thin — a deliberate stance against confident fabrication.
- Role
- AI engineering intern
- Year
- 2025
- Status
- Completed
- Category
- Artificial Intelligence
- Tools & technologies
- PythonFastAPILangChainpgvectorClaude APIReact
The problem
The support team answered the same questions repeatedly, and the answers lived scattered across a wiki, a handbook, and people's heads. A naive chatbot would hallucinate confidently — worse than no tool at all.
Context
A twelve-week internship project. I was given the documentation corpus, a quality bar ('never invent policy'), and freedom over the approach.
Process
Grounding first
The whole system is organised around citations. Documents are chunked, embedded, and stored in pgvector; answers are generated only from retrieved passages and always link back to their sources.
I spent most of my time on retrieval quality and on the refusal behaviour — teaching the system to say 'I don't have a confident answer' when retrieval scores were weak.
- Semantic chunking that respects document structure
- Hybrid retrieval (dense + keyword) for acronym-heavy internal text
- An evaluation set of 120 real questions with graded answers
Research
- 01Built a 120-question evaluation set from real support tickets and scored answers on faithfulness, not just plausibility.
- 02Compared chunking strategies; structure-aware chunking beat fixed windows on faithfulness by a clear margin.
Key decisions
Faithfulness over coverage
I tuned the system to abstain rather than guess. A tool that is trustworthy 90% of the time and honest about the rest beat one that always answered.
Citations as the primary UI
Every claim is a link. Reviewers could verify in one click, which is what earned the team's trust.
Implementation
Serving
A FastAPI service handles retrieval and generation; the model layer is provider-agnostic behind a thin interface so the underlying LLM can be swapped. A small React panel surfaces answers with inline citations and a visible confidence state.
Gallery
Outcomes
- Answered 78% of a held-out question set faithfully, abstaining cleanly on most of the rest.
- Reduced repeat questions in the pilot channel noticeably over the internship.
- Handed off with documentation and an evaluation harness the team still runs.
Lessons learned
- In applied AI, the evaluation set is the product. Everything improved once I could measure faithfulness.
- A well-designed refusal is a feature, and users respect it more than a confident wrong answer.
Related work
Ledger
An AI-assisted personal finance tool that turns raw bank exports into a clean, categorised ledger you can actually reason about.
Atlas
A course-planning tool that helps Business Informatics students map degree requirements to a realistic multi-semester plan.