I built an agent that sits in a Microsoft Teams help channel. It does two things. It answers questions when the answer is already written down, and it offers to write the answer down when it isn't. The rest of the time, it says nothing.
The quiet part turned out to matter most.
What it does
- Answers from existing knowledge. If the wiki already covers a question, it replies with the answer and a link.
- Fills gaps. When a thread reaches a resolution that isn't documented, it drafts a short wiki entry in the thread and waits for approval.
- Otherwise stays quiet.
The "wiki" is a set of SharePoint pages. It isn't full documentation, just small facts people half-know, like "a SharePoint site needs at least two owners and no more than five."
How a gap becomes an entry
- The agent judges whether the thread has been resolved. If it isn't sure, it asks. It can also be called up manually.
- It checks the wiki. If the answer is already there, it does nothing.
- It posts a draft in the thread. Anyone in the group can approve it.
- Once approved, it updates the right wiki page and replies with a link to the exact section.
- The entry links back to the original conversation.
Nothing gets written without a human yes.
What I left out
I didn't build a semantic index. The usual advice is to embed everything and use vector search. Here the knowledge base was small and well structured, so an index would have added moving parts without better answers.
What went wrong first
It talked too much. It answered too often, and the knowledge it offered was accurate but didn't always match the question. An answer to a nearby question is still noise, and people learn to scroll past it.
The fix was a confidence threshold. If the agent isn't confident the existing knowledge matches what was asked, it says nothing. That one change did more for its usefulness than anything else.
What I didn't expect
People still ask the same questions, because getting an answer in the channel is easier than opening the wiki. I'm fine with that. The knowledge is captured and the answers are consistent. But repeat Q&A probably belongs in each person's own AI assistant, pointed at the same wiki. That would keep the shared channel for new problems worth capturing.
The takeaway
The pattern needs four things:
- A place where questions already get answered.
- A simple place for small facts.
- Clear rules for when to speak.
- A human approval step.
Get the rules for when to speak right first. Most of the value comes from restraint.
If your business knowledge lives mostly in chat threads and people's heads, I can help. Ask about a Process Friction Review →