What To Look For In An AI Note Taker

The AI note taker is a new form of software that automatically attends, records, and summarizes calls and meetings. It is designed to improve business processes by freeing up precious time for people to focus on the more creative and challenging aspects of their jobs.

The AI notetaker is an evolution of the call recording software that has been around for many years. Call recording has made a significant impact on customer service, allowing employees to quickly resolve issues by replaying the call in its entirety. However, by just recording calls, businesses are missing out on key insights that can be gleaned from the actual conversations taking place.

By using an AI notetaker, it is possible to go beyond call recording and capture key insights into how teams work together, how they communicate, and what they are concerned about.

The AI notetaker is a form of a virtual assistant, typically integrating with a person's calendar. The AI notetaker automatically joins calls and meetings on behalf of the user so that it can record them for transcription, summarization, and analysis.

To do this, the AI notetaker must first read the calendar to identify all upcoming meetings. It then chooses which one to attend based on various factors, such as who is attending the meeting, when it is, and whether it is a recurring meeting. This enables the AI notetaker to pick and choose what meetings it attends, and which ones it skips.

These systems typically appear on a call as an additional participant, named to indicate for whom the note taker is capturing the meeting. For instance, it may be named "Susan's Notetaker" or "SunCorp Inc's Notetaker".

A note-taker captures the audio and video of the meeting, making a recording that can be reviewed and replayed in the future.  This recording is also analyzed with a variety of tools to summarize and extract insights, including transcription, summarization, computer vision, and natural language processing.

The analysis is then often used to populate a knowledge base with all of the important information captured from the meeting. This enables people to quickly explore, search, or retrieve all relevant information that was captured during a meeting.

Beyond this, the analysis can be used to create alerts, rules, and suggestions. For instance, if someone mentions a specific type of product or service, it may generate an alert to the person responsible for that product or service. Or if it detects that customers are complaining about an important issue, it may generate an alert that notifies staff to take action.

The goal of AI notetakers is not only to remove the burden of taking notes but also to unlock the vast trove of knowledge trapped in meeting recordings. By applying natural language processing, machine learning, and other AI technologies, it is possible to find insights that would otherwise remain hidden.

Companies that deploy AI notetakers in their operations are seeing significant improvements in customer satisfaction, employee focus, cognitive load, and operational efficiency. The following are some of the reasons why an AI notetaker is a valuable tool.

Customer Satisfaction - Customers are more satisfied when they are provided with clear, easy-to-understand support. This is most easily done by immediately resolving their issue, but also by providing them with transparency into what is being done on their behalf. AI notetakers help achieve this by automatically creating a ground truth of all interactions that have occurred with a customer. This enables a support rep to better understand the issue at hand, what it is that the customer wants, and how to best solve their problem. By using an AI notetaker, support staff is able to better re-engage with customers that have talked to the company previously or interacted with another employee.

Employee Focus - Employees that are aligned around common goals and understanding are more productive. When people are attending meetings that are not critical, they are wasting time that could be better spent solving problems. Yet employees often attend unnecessary meetings due to FOMO (Fear of Missing Out) or because they don't want to make a decision without having all the facts. AI notetakers remove this pressure by automatically attending meetings, recording them, and making them available to employees. This means that people no longer have to attend a meeting to "just listen" or "stay informed". Instead, they can focus on working on important projects and solving important problems.

Cognitive Load - Taking notes is a mentally taxing activity. It requires active listening, is often hard to follow due to poor speaker/audience separation, and doesn't scale well as meetings get longer. This is why people often lose track of what was discussed or fail to fully process what was said. By using an AI notetaker, it is possible to significantly reduce the cognitive load of taking notes. This is especially important for people that are not good at taking notes or don't take enough of them.

Operational Efficiency - Meetings are often the most difficult to schedule, hardest to fit into someone's schedule, and most prone to cancellation. For employees that require information from a meeting, it can be frustrating when the meeting is canceled at the last minute, leading to wasted time and effort. AI notetakers help solve this problem by automatically attending meetings on someone's behalf and providing them with a copy of the meeting minutes, key events, and recording after the meeting. This enables people to more easily attend the meetings that are important to them.

The AI notetaker is the result of a convergence of many technologies, including speech transcription/voice AI, computer vision, natural language processing, and robotic process automation. These systems must be capable of connecting to a person's calendar, understanding the meaning of a meeting, joining the meeting, recording it, transcribing it, summarizing it, and analyzing it. 

Extracting value from a recorded call or meeting involves many steps, including transcribing any speech, understanding how many people are speaking, and who is speaking when, looking at the screen to understand any shared content (presentation slides, shared screens, etc). The transcribed text is not the end of the analysis process, but simply one important step in transforming recordings into actionable knowledge. A transcript must be analyzed to understand the topics being discussed, identify any key events, and summarize what happened in the meeting. This information is then presented to the user in a variety of ways, including a detailed report, an easily searchable knowledge base, or by providing recommended actions.

These systems typically employ many different machine learning modules to achieve these ends. Speech processing, language analysis, and computer vision can each involve dozens of AI components that must be seamlessly integrated into a unified system. Calls and meetings contain many challenging circumstances for an AI system, including multiple people talking rapidly, quick back-and-forth conversation flow, background noise, accents, bad audio conditions, acronyms, and other terminology, etc. Good AI note takers must employ the latest neural network approaches, custom-trained AI engines, and extremely large-scale training datasets and infrastructure to provide quality call analysis results.

There are many other important aspects to how a note-taker works, such as conversation indexing, enabling search and discovery, rules and alerting, anomaly detection, data visualization, and so on. These are outside the scope of this article but are equally important to ensuring that AI notetakers provide maximum value to organizations.

When choosing an AI notetaker, it is important to find one that offers features and capabilities suited for both your current and upcoming needs. A notetaker that doesn't offer features such as auto-attend meetings, flexible control over attendance, summarization vs transcription, advanced conversation search, and privacy features isn't going to be as useful as one that does. We will now dive into many of the important criteria that one should consider when choosing an AI notetaker.

A key feature of an AI notetaker is the ability to automatically attend meetings. These systems must be capable of reading a user's calendar and identifying upcoming meetings. It then must choose which ones to attend and which ones to skip. Companies today use a variety of calendar solutions, such as Google Calendar or Outlook Calendar, and a note-taker must be capable of interacting with these systems.

Employees have many different types of meetings, some of which may involve highly sensitive or confidential matters. For these meetings, it is important to have a note-taker that provides a high degree of flexibility. This involves allowing users to explicitly choose which meetings they attend, as well as which ones they do not. This can be enabled by offering a person the ability to mark specific meetings as ones to attend or skip, or by providing attendance options that analyze meeting attendees, titles, or descriptions to make this determination automatically. For instance, meetings with specific individuals (investors, law firms, etc) or about specific topics (sensitive HR matters, etc) could be marked as do-not-attend.

AI notetakers can either just capture and transcribe what is said in a meeting or it can perform a more advanced summarization of a meeting. A basic transcription just records all of the words that are spoken, and nothing more. It is then up to the user to review the transcript (which may encompass many thousands of words) and perform their own analysis of what was said.

Summarization systems go beyond this by transforming a transcribed recording into automatic meeting minutes and key event summaries. This enables users to quickly find the most important items, such as the decision that was made, key actions assigned, notable facts discussed, etc. This makes it easier to surface and consume all of the useful content in a meeting and reduces the need for users to manually review the entire transcript.

Some note-takers today are capable of capturing audio only, but it is increasingly important for them to also capture and understand video from a meeting. Video is the only way to truly capture what is happening at a meeting, including interactions around shared screens or content. While analyzing video is very challenging, AI systems are now capable of processing captured video from recorded calls and meetings, understanding what was being displayed, extracting readable text, identifying presentation slides or shared screens, and so on.

These video-based notetakers are able to support many additional features, such as indexing, summarization, and visualization of what was being displayed--not just what was being said. This is critical for ensuring that the meeting is captured in full, enabling people reviewing the meeting to gain a much better understanding of what happened.

A notetaker should be able to automatically identify the sentiment of any aspect of a meeting, such as whether it was positive, negative, or neutral. It should also be able to identify who is speaking and be able to identify positive vs negative sentiment from a speaker. This enables a variety of capabilities, such as generating alerts when a previously happy customer suddenly becomes unhappy.

Sentiment analysis is a challenging AI problem, but some notetakers are able to perform this analysis using a combination of natural language processing and machine learning. Conversations can involve sentiment being expressed in many different ways, such as "I'm excited to hear about your new product", "I'm was thrilled but now a little less sure", or "This is sick!". A good note-taker should be able to automatically and accurately identify and interpret these and other sentiment expressions.

AI note-takers should be able to automatically summarize calls and meetings, provide insights from the meeting, and alert staff to important things happening in the meeting. For this to happen, the notetaker must be able to perform natural language processing (NLP) on transcribed conversations, identifying key events, discussion topics, mentions of competitors or specific people, complaints, suggestions, or feature requests. Natural language processing systems typically involve many different modules for performing these various functions.

It is important for a notetaker to leverage NLP systems specifically built for conversations, meaning that they are designed for the specific structure of how people communicate verbally.  Traditional NLP solutions designed for text processing (eg, tweets or news articles) may work for written documents but won't work as well when used on human conversations. Conversations are very different than traditional text processing and require specialized systems specifically built for this task.

A notetaker should be able to enable people to search and retrieve specific content from a call or meeting. For instance, a user should be able to search for any mention of a specific topic, company, or person. These systems must also provide the ability to search for types of interactions, such as questions, opinions, complaints, or suggestions.

Conversations involve different people speaking, therefore the ability to search by the speaker is also very important. Did a customer ask a question or the sales rep? Who was discussing a specific topic? Who made a complaint? These are all important insights that need to be able to search for.

Traditional call recording repositories do not provide much use outside of reviewing individual recorded calls or meetings. AI notetakers should be capable of combining information across analyzed recordings to create a unified knowledge base of what is happening inside a company. A knowledge base should include information about specific people, companies, topics, products, and other aspects of a business--all extracted from analyzed calls and meetings. It should also be easily searchable and browsable, enabling users to find the information they're looking for, or easily discover new insights.

Knowledge graph technologies may be used to create a traversable knowledge base, capable of answering questions such as "what customers complained about a specific issue over the past 2 weeks", or even more complicated ones such as "what customers over $20,000 MRR who are managed by the federal sales team complained about a specific issue over the past 2 weeks". It is important that an AI notetaker transform analyzed calls into searchable and explorable knowledge, vs simply providing a collection of transcribed calls.

AI note-takers should incorporate the ability to customize the types of events they look for, and the behavior taken when such events are found, via rules and alerts. For instance, when analyzing a call, it may be important to know when customers complain about a specific issue. It may be even more important to know when customers complain about a specific issue and if the issue is not addressed in a timely manner.

Rules and alerts can help make this process efficient and automated. Users shouldn't have to wade through hours of calls to find when customers complain about an issue--the AI notetaker should be able to automatically alert the right people when important things happen.

Many organizations rely on group chat systems like Slack or Microsoft Teams for internal communication. Integrating with these systems enables users to bring the insights of their notetaker into existing processes. Notetakers should be capable of sending automatic meeting minutes or key event alerts to specific users via direct messaging (DMs) or to certain channels. For instance, if a support team needs to know about customer complaints, the notetaker should be able to send them an alert via Slack or Teams. With rules and alerts, this can be done automatically, without requiring users to manually search their transcribed calls and meetings.

A key aspect of using a note taker is that it is automatically recording your calls and meetings. Employees have many different types of meetings, some of which may contain more sensitive or confidential information than others. Some meetings may be ones that are not recorded at all, whereas others may be suitable for sharing only with specific people or groups within a company.

Quality privacy controls including private workspaces (similar to an email Inbox, but for transcribed conversations) and the ability to customize who has access to what is very important. It should be possible to customize which conversations are recorded and who has access to the resulting data. As an example, if a company has cordoned off teams it should be possible to limit who has access to what.

There are many cases where it is beneficial for multiple people to access the same call and meeting data. For instance, when there is a cross-functional team working on a shared project, it is important that they can easily collaborate and share information with each other. Shared workspaces can help facilitate this type of collaboration by enabling employees to share calls and meetings in a common space. These workspaces can be used to facilitate knowledge transfer or enable internal collaboration.

Workspaces should offer quality security and privacy capabilities, with role-based access controls. They should also support features like search, tagging, and notifications to make it easier for people to find and share information.

Quality notetakers should be capable of organizing and archiving call and meeting data into easy-to-access workspaces and folders. This enables people to quickly find the information they need, while also enabling data to be easily archived. This can be done using an intuitive interface, with drag-and-drop capabilities, and the ability to create folders, tags, labels, and filtered views of collections of conversations.

Good AI notetakers should offer a user interface that is intuitive, easy to use, and flexible enough to allow individuals to customize it to their liking. The system should be able to run in the background, and should not interfere with existing workflows and work patterns. The ability to use a note taker from any web browser is important as it enables people to access their call and meeting insights from anywhere.

It is also important that a notetaker provides a variety of ways to visualize and search call and meeting data. The system should be able to generate reports, dashboards, and visualizations. The system should provide advanced call and meeting playback and review capabilities, including video playback with subtitling, transcript viewing and search, automatic meeting minutes and summaries, and call analytics.

The future of AI notetakers is bright. These systems will continue to improve and expand, providing organizations with the ability to unlock all of the information trapped in recorded calls and meetings. The role served by notetakers will grow as well, from passive recording, transcription, and summarization to more active roles such as meeting facilitation and real-time information retrieval.

Notetakers will also become more integrated into existing business processes, leveraging AI-powered robotic process automation to automatically take action based on specific meeting insights. For instance, if an AI notetaker detects that a customer is upset about a particular issue, it may automatically create a trouble ticket and escalate the issue to a support team.

As AI notetakers continue to improve, it is important that organizations begin evaluating, testing, and implementing these systems. A good notetaker will remove the burden of taking notes, unlock the knowledge trapped in meeting recordings, provide useful insights, and generate actionable alerts. At Hyperia, we believe that AI notetakers will play an increasingly important role in many organizations.

If you are interested in leveraging AI notetakers in your business, check out the Hyperia AI Meeting Assistant. It works with Zoom, Google Meet, Microsoft Teams, and many other call and meeting solutions and SaaS apps.