The context gap

If you tried gemini live, you know that live agents are getting really advanced. With Gemini 3.1 you can now call with an agent as if you are calling with a human that’s actually really smart. More agents will come and they will get better and cheaper. The problem however is that they are incredibly generalized, they will help you generally but not specifically. For a lot of usecases you need to have context, think of recipes, working procedures, user context, memory etcetera. Language models will always need to find ways to get context to help you better and currently there is no universal way of doing this.

So you’re thinking okay, but what if I upload all my files to a google drive and give Gemini access. Yes that would work in some cases, when you need the agent once, like chatGPT helping you creating a presentation or something else. But you don’t want to manually upload all the instruction manuals of your car or CNC machine everytime you have a question. In fact there is a lot more to it like optimizing delivery and improving the context with each session. Google docs won’t do this for you. And besides that, if they would, would you want to outsource your company or your experience and sessions to a Tech company? You’ll lose your knowledge when there is no internet.

Context on edge

Mika proposes a system where you not only seperate the agent from your context, but also to keep the context locally on your premises, in your building. Each factory floor and each office building will have an invisible layer of insitutional context that grows and compounds over time.

He's building this for his agent “oneshot” that gives physical workers context they need for doing their job. The context delivery system and format should be something universal, I want to open-source the infrastructure. We will want to have one standard format so multiple agents, robots and platforms can benefit and grow together.

Shipmemory

Technically this is how it works.

I have been building agents for physical work on smartglasses for the past few months. I started with Edge VLM’s that detect mistakes during a procedure, using the camera’s from smartglasses. This is still a very experimental field and it evolves fast, it also a hard problem to solve and there is no standard yet. To make my startup “Oneshot” a succes I decided that real time vision with my own pipeline is probably to hard and takes too much time to go to market. So I decided to work on the institutional knowledge first.

Oneshot pivoted to build a voice agent that helps during maintenance, it stores the machines information and procedures. The Oneshot agent harness that is bolted on top of a physical machine has an advanced data retrieval system. While you’re doing maintenance you can send voiced message to the machine and it responds with guidance.

With Gemini live 3.1 coming out I build an interface layer which is a mobile app connected to Gemini on smartglasses, and I created a QR code connected to my backend to fetch context. I used toolcalling from gemini to get more context if the agent demanded it. This new architecture is very promising and I expect many many agents to work like this in the near future. This is why I decided to build an maintain an open source context delivery architecture for physical agents, robots and copilots, named Shipmemory.

アウキ・ラボについて

Aukiはポーズメッシュという地球上、そしてその先の1000億の人々、デバイス、AIのための分散型機械認識ネットワークを構築しています。ポーズメッシュは、機械やAIが物理的世界を理解するために使用可能な、外部的かつ協調的な空間感覚です。

私たちの使命は、人々の相互認知能力、つまり私たちが互いに、そしてAIとともに考え、経験し、問題を解決する能力を向上させることです。人間の能力を拡大させる最も良い方法は、他者と協力することです。私たちは、意識を拡張するテクノロジーを構築し、コミュニケーションの摩擦を減らし、心の橋渡しをします。

ポーズメッシュについて

ポーズメッシュは、分散型で、ブロックチェーンベースの空間コンピューティングネットワークを動かすオープンソースのプロトコルです。

ポーズメッシュは、空間コンピューティングが協調的でプライバシーを保護する未来をもたらすよう設計されています。いかなる組織の監視能力も制限し、空間のプライベートな地図の自己所有権を奨励します。

分散化はまた、特に低レイテンシが重要な共同ARセッションにおいて、競争優位性を有します。ポスメッシュは分散化運動の次のステップであり、成長するテック大手のパワーに対抗するものです。

アウキ・ラボはポスメッシュにより、ポーズメッシュのソフトウェア・インフラの開発を託されました。