This looks like nonsense, but it's my nonsense.

My failed homage to Arrested Development's Larry Middleman

Jul 28, 2024

This is Episode 2 in the [Game On] series, where I attempt to design, code, and ship one new app every month. This is the tale of my FAILED attempt to launch Surrogate - a Vision Pro and iPhone app that acts like your eyes and ears using a smart dock.

Surrogate is available (kinda) for download on Testflight and for subscribers only.

Prologue

First of all, if you don’t know who Larry Middleman is, please watch this wonderful clip from 2006 presented in glorious 480p resolution. And if you’re already familiar with him, please enjoy one of my favorite moments of his from a show filled with incredible moments:

Anyways, back to the topic at hand: I decided to replicate Arrested Development’s faux Surrogate service with today’s technology by using an iPhone attached to a smart dock and a Vision Pro, and here is the final result:

“Wtf. Why?”

Yup — fair question.

Last month, we made Hacks - Golf Swing Journal (you can download it here). Hacks tested my determination and discipline by challenging me to take what I already knew how to do but to do it in a highly condensed timeframe. It did not necessarily test my technical abilities.

So for my second project, I wanted a technical challenge while also still being unabashedly unique, creative, and dare I say a little weird. But here’s the thing — I spent so much time writing for this Substack (I published 3 pieces in those 2 weeks, which is a lot for me!), that none of my time went into thinking about this upcoming project. And so I had nothing in the queue that fit what I was looking for. Crazier yet, I was on an even tighter timeline than last time because Ali and I were leaving for Croatia (vacation) in 15 days!

So I had 24 hours to come up with an app or else a newsletter proclaiming to ship one product per month would have stopped at exactly one 😅.

My only North Star was that I wanted to include the Belkin Stage Stand because it’s the only stand1 on the market that integrates with Apple’s new API DockKit. I bought the stand in January 2024 and, to be completely honest, it had been collecting dust since February. And it began collecting dust in February because that’s also when I got the ultimate ~~toy~~ work device — the Vision Pro. I purchased both of these things for “development purposes,” but had yet to develop with them. So what better way to develop with them than to panic smash it all together into one thought:

I want to move my dock-mounted iPhone using only my head!

Everything everywhere all at once

Turns out, giving yourself only fourteen days to work with relatively new technology without proper discovery is not a great way to start! In fact, I didn’t even know where to start! Every aspect was something I had (personally) never worked with:

Apple Vision Pro.
Less than a year on the market so everything on this platform is an experiment.
Apple’s DockKit API.
Most Apple engineers I talked to at WWDC24 didn’t even know what this was until I spoke to Brad Ford. Even then, he mentioned that the best way to get more information would be to request a 1-1 online session for later in the week.
Live-streaming.
Not new tech but foreign to me.
Real-time communication between devices.
Not new tech but foreign to me.

Unlike Owen Wilson in Armageddon, I was securely in the 98% scared camp as I tried to figure out which documentation to jump into first. I started with live-streaming but like a squirrel with an avoidant personality disorder, I was soon knee-deep in Apple forums looking for more information on DockKit and the Vision Pro. And this is how I spent my first two days on this project — in a vicious cycle of doom-scrolling dozens of newly opened browser tabs.

I think what I was trying to do was ensure each component of this complicated puzzle was realistic and doable. But in reality, I was afraid to mentally and emotionally commit to the task at hand. I had tricked myself into reading all this documentation to “not get started down the wrong path” so I could procrastinate just a little bit longer.

What ultimately woke me from this zombie state was recognizing and, more importantly, calling out the daily micro-actions that I was abandoning from my previous project:

Recording end-of-day demos to document progress.
Setting the next day’s goals the night before.
Utilizing every extra fifteen-minute block I had (i.e. between walking Frank and dinner) to implement non-core features like analytics, legalese, the onboarding experience, the launch screen, empty views and error states, and Google ads.

I could feel the timeline slipping away and I needed to regroup.

One foot in front of the other

The new strategy was really simple — to put one foot in front of the other and have faith that by solving this thing one piece at a time, it would all come together at the end like an episode of MacGyver (I’m more of an A-Team / Knight Rider guy but MacGyver seems like the right analogy for this specific project).

So with my priorities reset, here’s how it went down:

Day 3

Let’s, simply, just get the dock moving.

Forget about the Vision Pro. Forget about live streaming. Just mount the iPhone to the dock, write some code to get the phone and dock to talk to each other, and then tell it to move. Nothing crazy. Just. Get. It. Moving.

Up-up, down-down, left-right, left-right.
Great. Done. Moving on.

Day 4

Now, we need to get the Vision Pro to talk to the iPhone to execute those same basic movements. Again, nothing more and nothing less so today, find a way to get the Vision Pro and iPhone to communicate.

To do that, I use SharePlay. I have never used SharePlay before and I’m not entirely sure it’s the optimal choice but it’s the right choice for right now because it is a native Apple technology that gets me up and running the quickest.

However, it is not the most intuitive choice because it requires me to essentially FaceTime myself 100x a day, which is more cumbersome than it sounds. You have to create a FaceTime meeting link, join that from your iPhone, put the Vision Pro on, and then finally join that same FaceTime link “as a separate device”. Plus mute yourself while also turning off the cameras. It’s a bit of an ordeal but whatever, it gets the job done!

Day 5

Now that we’ve got the Vision Pro and the iPhone talking to each other, it’s time to answer our first true milestone question:

Can I use the Vision Pro’s physical orientation in space to move the dock?
And why yes, yes I can.

A simple Google search led me to this StackOverflow answer and introduced me to ARKit, World Tracking, and some property type called a simd_float4x4 🤷🏻‍♂️. I am still not very clear on what exactly a simd_float4x4 is but the only goal, at this present moment, was to see if I could move the dock with my head, and through some intense trial and error (pause for dramatic effect):

I have got the thing moving!!! Using my head!!! What the what!!!

A glimmer of hope enters my body as panic and excitement swap places. I’m able to let out a small smile alongside a big sigh from this incredible milestone but there’s still so much to do! So in the words of President Bartlett, “What’s next?”

Day 6 & 7

In the previous video, I was able to move the dock by wildly swinging my head in one direction or the other. It’s a good start but the real and final objective for this part of the project is to sync head movement with dock movement.

I know this is a newsletter dedicated to detailing more about the emotional process of building software than the technicals but to be honest, I kind of blacked out during these two days so I can’t tell you much. I do remember the key parts though:

confusion (what is a quaternion? Am I in the MCU with Dr. Strange?)
crying (whhhhyyyyyy am I so duuuummbbbbb)
cursing (at inanimate objects — sorry you had to hear all that, Frank)

But in the end, we got where we needed to be:

🥳🎊🎉
I know this video looks silly and I, myself, look incredibly silly, but like guys, come on. I’m now one step closer to building my own robot! With an iPhone! I just need to find a way to put some wheels on this thing and we’re only one billion or so steps behind Boston Dynamics.

So a moment of zen/respite for this unhinged adventure we are on right now, please.

Day 8

We are finally cooking with a little fire! Head movements are synced with the dock so all we need to do now is the part that should be substantially easier — live-streaming.

Foreshadowing: If you’ve ever been to therapy, you know that the word “should” always leads to trouble and a furrow in your therapist’s brow.

I spend some time exploring my available options but I stay mentally vigilant so as to not repeat my earlier mistake of wasting time on research. I choose GetStream because it appears to be the most user-friendly and I have something up and running on iOS in no time.

Here comes that trouble I mentioned …

However, I have veered so far in the direction of “no research, just get coding” that I have completely forgotten to check if these libraries also support VisionOS. And so of course, they do not nor do they plan on it. Womp womp.

I can read between the lines. Marked as [won’t do] in the JIRA and I understand. Developers are under-supported and over-committed.

Well F. Back to the drawing board. So what library does work with VisionOS?

GetStream? No.
Twilio? No.
Mux? No.
Agora? No.

Wait, hold that thought on Agora. A quick Google search of “Agora Vision Pro” leads to this VisionOS and Agora sample project which then leads to this branch (work-in-progress code). And it looks like it’s made by developers Rick Cheng and Max Cobb who appear to also work for Agora so … this might be the most official unofficial thing I can find and I can feel a small glimmer of hope return. But I’m going to have to dig into it tomorrow because I am completely spent from a full day of taking mostly L’s.

Day 9

I wake up anxious because I know that if this rogue branch doesn’t work, I’m kind of out of options. So I type a little slower and quieter as if that will somehow prevent any bug demons from waking or appearing in my code. It lets me linger a bit longer in purgatory, where as long as I haven’t tried it, it hasn’t failed yet.

I write my last line of code, build/run it, and wince painfully at the screen waiting for an error to fly back in my face. Except uhm, it works. Holy flipping madness, it works! It is the first time in this process that the entire concept of this chaotic journey is fully realized and wow. Just wow.

We have iOS to VisionOS streaming with synced movement. Unhinged.

Day 10

After that incredibly exhilarating day yesterday, it dawns on me today that I have given zero thought to UI and UX. Right now it’s just buttons haphazardly placed on the screen so I can run my experiments faster (or as fast as taking my Vision Pro on and off fifty times a day will allow for). But as I stop to give UI some actual thought, more technical questions arise:

Why is it malfunctioning when I try to look behind me?
Does it only work for 180 degrees?
SharePlay is a bit high-touch, how do I dial this down?
How do I make the syncing smoother?
Because I just gave myself motion sickness.

F@#$ me, why does every answered question have ten more behind it?

Day 11

There are now only 72 hours remaining and I am still frantically trying to solve technical issues in this Frankenstein I’ve built. It is a morale-killer being this late in the game and still being stuck in a stop-and-go workflow. I can’t keep my rhythm or momentum going and my brain is worn down and needs a reboot itself.

Admitting Defeat

Admitting defeat on Day 11 gave me the freedom to forego design polish and allowed me to focus on making the prototype available on Testflight for anyone to try out (if you have a DockKit-compatible device and a Vision Pro). It was the least I could do. Still, I felt disappointed. Two projects in and we’re already batting .500. But there are a few things I’m proud of in this failure:

I regrouped vs quit (or worse yet, pivoted to making some trivial to-do list app).
I learned so many new frameworks along the way.
I at least have a working prototype!
And I found a way to use my ~~toys~~, I mean development tools 😅.

I don’t want to shy away from the defeat so in keeping in line with my 480p throwback videos, here’s one of my favorite quotes from hall-of-fame coach Bill Parcells, “There are no free rides on the road to victory”.

This was not a victory but this was the furthest thing from a free ride.

What’s Next?

Croatia!
Project #3 and #4. I’m going to try not to make the same mistake I made here 😅
And if you want to collaborate on future projects with an app idea of your own, DM me!

[Bonus material] Frank says thank you for reading (from the comforts of his grandparent’s house) while his parents are off galavanting in Croatia.

This is not a paid promo, this stand was the only DockKit-enabled stand on the market when I was writing this. However, Insta360 did just release a stand of its own!