Microsoft’s Copilot surpasses the capabilities of a mere chatbot. It represents a long-cherished goal for Microsoft, as it steadily evolves into an advanced AI assistant.
During my recent meeting with Microsoft to discuss Copilot, their new AI-powered tool amazed me by instantly summarizing our conversation and listing out the questions I had just asked moments before. I have seen Microsoft demo virtual assistant concepts for years, but Copilot is the closest to reality that I have ever witnessed.
According to Jon Friedman, corporate vice president of design and research at Microsoft, Copilot represents the future of computing and working with technology, and is the most adaptive technology they have developed yet. During our Teams call, Friedman activated Copilot mid-meeting, demonstrating its AI-powered abilities in real time across Office apps and Teams. Microsoft has a flashy marketing video showcasing Copilot’s potential, but witnessing its capabilities firsthand left me convinced that it will forever change how we interact with software, create documents, and ultimately, how we work.
While Copilot initially appears as a helpful AI chatbot on the sidebar of Office apps, it offers much more than that. It can seamlessly appear when you highlight an entire paragraph in the middle of a Word document, similar to how UI prompts highlight spelling errors. You can then use it to rewrite the paragraph, selecting from ten different suggestions for new text that you can freely edit. Alternatively, you can rely on Copilot to generate complete documents for you.
What sets Copilot apart from Microsoft just integrating ChatGPT into an Office sidebar is its adaptability. Copilot offers more than just a chatbot interface, as you can use it to command various Office apps, including Excel and PowerPoint. For instance, you can ask Copilot to change the color of every title on a slide deck from blue to orange instead of delving into PowerPoint features.
In Excel, Copilot can help you generate a PivotTable, create a graph, or understand the rows and columns of data before you. According to Friedman, “One of the ways we’re starting with Copilot is helping analyze and understand data.” With Copilot, you can ask about the data and obtain graphs based on trends it detects, which you can then insert into a spreadsheet. Excel also features a “show me” function that allows Copilot to teach you how it completed a command, allowing you to improve your knowledge of Office.
Microsoft appears to be slowly expanding on its previous visions for its Cortana assistant and Clippy from decades ago. According to Friedman, “I love that our heritage is full of products that try to adapt to people. Copilot shares some similarities with some things we’ve done in the past, but it’s far more capable, it’s humble, and it’s there to serve things up for you that help you save time.”
Microsoft has customized Copilot for every Office app, so it operates differently in each. Friedman showed me how Copilot can assist with writing emails in Outlook, presenting short or long message drafts with tone options. It even works in the mobile version of Outlook, which got me thinking about how it could speed up work while on the go.
Friedman stated that “Outlook mobile is the first place where we’re doing a big push,” with features like email summarization, draft generation, and a more streamlined inbox. Imagine creating entire Word documents from your phone without having to type on a tiny on-screen keyboard. “We’ll have more to talk about mobile in the coming months,” says Friedman.
As remarkable as Copilot is, we have seen various ways that large language models can fall short, such as inserting racial or gender biases into text and producing fake information. While such problems can be concerning in a search engine, they are even more worrisome when it comes to Excel or your email inbox, as it raises concerns about ethics, privacy, and data.
Friedman admits that Copilot “gets things right a lot of the time but not all of the time.” To address this, they have included features such as affordances that encourage users to review and edit content before sending it.
As Copilot is a system powered by large language models, Microsoft has incorporated several warnings into the tool. For instance, in PowerPoint, users will see a message stating, “Content is generated by AI and might contain inaccuracies for sensitive material. Be sure to verify information.” Similarly, prompts in other applications will remind users that “AI-generated content may be incorrect”. These warnings are aimed at reminding users that they are in control.
Microsoft has also provided tools to report incorrect information and prompts to help users write effective prompts. “Everything we’re doing in the user experience is to make it conversational and give you agency,” explains Friedman. However, as seen in Bing search engine, AI-powered chatbots can sometimes malfunction. Microsoft has learned from the experiences with Bing and is applying that knowledge to mitigate risks with Copilot. For now, Microsoft plans to roll out Copilot to just 20 businesses before expanding to more customers. It is also starting with enterprise customers first before introducing the tool to consumers.
Friedman states that Microsoft plans to iterate quickly and move fast to scale Copilot to more of the enterprise in a thoughtful and responsible manner to ensure an excellent experience. However, there are concerns about whether Microsoft is moving too fast in the AI race, especially since Google recently announced its own AI features for Gmail and Docs. Some experts are worried that tech giants are not fully considering the impact of these new tools.
Friedman explains that Microsoft is being both quick and thoughtful in its approach. The company is rolling out Copilot to just 20 customers initially and working closely with them. Despite reports of Microsoft downsizing its ethics and society team responsible for teaching employees how to develop AI tools responsibly, Friedman says that Microsoft is actually increasing its workforce in this area. “In terms of our investment in ethics and AI, we’ve grown more and more ethics and AI experts in all of the product teams working on this stuff,” he says. “We have to scale way bigger, so we’ve been investing more heavily, and it continues to grow year over year.”
Microsoft recognizes that Copilot is not perfect, and it will take time to improve. Although the system impressed the author during a Teams meeting, it could easily mistake someone’s voice for another’s if the microphone quality is poor, or Outlook could generate the wrong summary in an email thread. However, Microsoft is striving to make it easy for users to edit responses, correct sources, and provide feedback to enhance the system. Friedman acknowledges that AI can make mistakes and hallucinate confidently, and Microsoft is continuously working to minimize those errors while empowering users through the user experience.
Generating text is just the beginning for the Copilot system, as Microsoft plans to expand its capabilities to include image and video generation in the future when large language models become proficient enough to handle these features.
Microsoft has already incorporated OpenAI’s DALL-E model into its Designer app, which permits users to create images based on text, and it will assist PowerPoint in selecting the most appropriate images for AI-generated slides. Friedman confirms that Microsoft will integrate Designer further into Copilot, allowing users to modify things within Designer. “What you’ve seen today with Designer is just the beginning. I fully anticipate we’ll use Copilot to accomplish remarkable multimedia tasks,” he added.
When questioned about where else Copilot may be employed, Friedman stated that the company is exploring a variety of avenues and methods to expand Copilot. He believes that this is the next significant phase of computing and that it will revolutionize the way we work with all devices in the years to come.
The Copilot system’s future will extend beyond text-based generation to include image and video generation once large language models can competently handle these features. Microsoft has already integrated OpenAI’s DALL-E model into its Designer app, which allows users to generate images based on text. Designer will also help PowerPoint pick the most suitable imagery for AI-generated slides. Friedman claims that Copilot will be integrated further into Designer so that users can change things within Designer. The possibilities are endless, and Microsoft plans to use Copilot to perform incredible multimedia tasks.
Moreover, Copilot will include a multiplayer experience, as Loop components are available in Teams and Outlook. Loop components, which are blocks of collaborative text or content that can function independently and be freely copied, pasted, and shared, are the branding for Microsoft’s Fluid work. By copying a Loop component of text into an email, multiple people can edit it and interact with the Copilot. Friedman says that in the component while editing, the conversation is a clickable history of the content being generated that you can go back and forth through. This new mental model of working with Copilot and a group of people is revolutionary.
These Copilot features for Office and Microsoft 365 will significantly alter how people work and communicate, especially as these large language models continue to evolve. Microsoft’s decision to integrate AI deeply into its products could have a long-term impact on the job market. According to Friedman, every time there is a new technological advancement, there are both opportunities and things to consider. Microsoft believes that infusing this AI will create new job opportunities in the long run and increase job satisfaction in the short term. They expect that it will transform many jobs’ nature and create new jobs that did not exist before, which is why building a common design system and empowering people is critical to them.
