Join the movement and participate on Uniswap.
Participate
If you tried gemini live, you know that live agents are getting really advanced. With Gemini 3.1 you can now call with an agent as if you are calling with a human that’s actually really smart. More agents will come and they will get better and cheaper. The problem however is that they are incredibly generalized, they will help you generally but not specifically. For a lot of usecases you need to have context, think of recipes, working procedures, user context, memory etcetera. Language models will always need to find ways to get context to help you better and currently there is no universal way of doing this.
So you’re thinking okay, but what if I upload all my files to a google drive and give Gemini access. Yes that would work in some cases, when you need the agent once, like chatGPT helping you creating a presentation or something else. But you don’t want to manually upload all the instruction manuals of your car or CNC machine everytime you have a question. In fact there is a lot more to it like optimizing delivery and improving the context with each session. Google docs won’t do this for you. And besides that, if they would, would you want to outsource your company or your experience and sessions to a Tech company? You’ll lose your knowledge when there is no internet.
Mika proposes a system where you not only seperate the agent from your context, but also to keep the context locally on your premises, in your building. Each factory floor and each office building will have an invisible layer of insitutional context that grows and compounds over time.
He's building this for his agent “oneshot” that gives physical workers context they need for doing their job. The context delivery system and format should be something universal, I want to open-source the infrastructure. We will want to have one standard format so multiple agents, robots and platforms can benefit and grow together.
Technically this is how it works.
I have been building agents for physical work on smartglasses for the past few months. I started with Edge VLM’s that detect mistakes during a procedure, using the camera’s from smartglasses. This is still a very experimental field and it evolves fast, it also a hard problem to solve and there is no standard yet. To make my startup “Oneshot” a succes I decided that real time vision with my own pipeline is probably to hard and takes too much time to go to market. So I decided to work on the institutional knowledge first.
Oneshot pivoted to build a voice agent that helps during maintenance, it stores the machines information and procedures. The Oneshot agent harness that is bolted on top of a physical machine has an advanced data retrieval system. While you’re doing maintenance you can send voiced message to the machine and it responds with guidance.
With Gemini live 3.1 coming out I build an interface layer which is a mobile app connected to Gemini on smartglasses, and I created a QR code connected to my backend to fetch context. I used toolcalling from gemini to get more context if the agent demanded it. This new architecture is very promising and I expect many many agents to work like this in the near future. This is why I decided to build an maintain an open source context delivery architecture for physical agents, robots and copilots, named Shipmemory.
Auki is making the physical world accessible to AI by building the real world web: a way for robots and digital devices like smart glasses and phones to browse, navigate, and search physical locations.
70% of the world economy is still tied to physical locations and labor, so making the physical world accessible to AI represents a 3X increase in the TAM of AI in general. Auki's goal is to become the decentralized nervous system of AI in the physical world, providing collaborative spatial reasoning for the next 100bn devices on Earth and beyond.
X | Discord | LinkedIn | YouTube | Whitepaper | auki.com
