This is a great idea - I would dig into this.
I disagree on AI not winning here. Support crew mostly work on eliminating blindspots for operator. Yes you can eliminate blindspots with cameras and a 7inch TFT screen, but those are small, don't work great in direct sunlight. The current solution not being 10x better is why they still fallback to having those crew members. I genuinely believe we will change that.
I'd like to add that this was done in partnership with Frodobots.ai (https://frodobots.ai) and BitRobot.ai (https://bitrobot.ai). They are working on a great mission statement and were amazing partners!
I can relate to how you felt. Even though I developed this myself, when I saw it running remotely with no one sitting inside for the first time, it felt magical and weird. This is going to save lives and look magical and that's too much fun to put off and do something else!
The way I think about this - we should not have multi screens. Human field of vision is 60 degrees for central and about 120 degrees binocular. The bucket of the excavator is way narrower than this which means actual task doesn't require wide vision.
So if we are able to have really good autonomous safety layers to ensure safe movements, and dynamically resize remote teleop windows, you'd make the operator more efficient. So while we stream 360 degree view, we get creative in how we show it.
That's on the vision side. We also stream engine audio, and do haptic feedback.
Takeuchi are interesting! Rare ones to have blades even on bigger sizes - is that why you got one?
Just a suggestion from someone who's worked on industrial robots and autonomous vehicles, but I think you're underselling a lot of difficulties here.
Skilled humans have a tendency to fully engage all of their senses during a task. For example, human drivers use their entire field of vision at night even though headlights only illuminate tiny portions of their FoV. I've never operated an excavator, but I would be very surprised if skilled operators only used the portion of their vision immediately around the bucket and not the rest of it for situational awareness.
That said, UI design is a tradeoff. There's a paper that has a nice list of teleoperation design principles [0], which does talk about single windows as a positive. On the other hand, a common principle in older aviation HCI literature is the idea that nothing about the system UI should surprise the human. It's hard to maintain a good idea of the system state when you have it resizing windows automatically.
The hardest thing is going to be making really good autonomous safety layers. It's the most difficult part of building a fully autonomous system anyway. The main advantage of teleop is that you can [supposedly] sidestep having one.
I definitely agree with you - recreating the scene in teleop is challenging. In excavators, however, it does make it better. An excavator has huge blindspots on the right (due to arm), to the back and sometime near the bucket. Hence the workers who are hired to stand around (banksmen, spotters, signalmen) and signal to the operator.
It's like driving a Ford F150 without backup camera. You'd add the backup camera upfront, and not display the back view at the back window.
It's definitely challenging and we're far from something that's perfect. We're iterating towards something that's better everyday.
Well sure if you are just looking at where the bucket is digging but there is often a dump truck sitting on either your right or left flank waiting for what’s in your bucket (don’t forget the beep button lol). Having a monitor to either side duplicates what you are seeing out of your peripheral vision when operating the real thing. Would make transitioning from real to virtual much easier and imho safer.
Yes that is precisely why - makes for a much more versatile machine. TB180FR - it’s med-small, about 10 ton.
I think swinging (which is about 40% of dig and dump workflow by time spent) should not be manual. That's lowest levels of autonomy which requires roughly centering to the pit/truck which we have already achieved. Hence operator only has to look in front!
I disagree and here's a couple of reasons why I say that:
1. What am I going to do with the time between releasing control and regaining it from the autonomous control?
2. In that break of workflow my first thought is it will cause a break in my concentration.
3. When I am swinging back from the truck to the trench the bucket is naturally in my control. It seems that in autonomy mode the transition from autonomous to my control would be very unnatural and choppy. I suppose with time it would be okay but man seems to violate the whole smooth is fast concept.
> I think swinging (which is about 40% of dig and dump workflow by time spent) should not be manual.
It's been over a decade since I last operated an excavator, so grains of salt as usual - but I'd say it should be manual, or at least semi-automated. You need to take care where you unload the bucket on a truck, to avoid its weight distribution being off-center, or to keep various kinds of soil separated on the bed (e.g. we'd load the front with topsoil and fill the rear with the gravel or whatever else was below.
I agree - the dumping and digging itself (where you move boom, arm, bucket much more than swing/tracks) should be manual. But swinging to the truck and back to pit (pure swinging motion to center around these areas of interest) do not have to be manual. I agree with your and other comments that the transition has to be smooth and that's something we're working on.
Mining has had autonomous haulage (and to some extent excavation) for some years. They have been very tightly coupled to the individual machines, models and the quarry site.
Construction excavation workflows tend to be different and more varied (hence you would see more variations in the machines/attachments available for this industry). And so far there's no solution which just works or is plug and play. This is an attempt to solve that.
And we believe this is great time to do so due to market demands too. No way Gen Z is going to do these jobs when you can make money being delivery partner/influencer, etc.
The high-certified, high-skilled workforce is going to shockingly shrink in some years.
I think force feedback is key for small excavators, but not really true for 25+ tons excavators. Hence how easy it is for operators to accidentally kill someone with it.
That's indeed what we're trying to test to the extreme - to see how far we could go with just vision. We haven't done heavy excavation workflows yet, but we have some early success with some excavation workflows with just vision input and joystick action output (even without joint angle feedback!).
We're betting on having really huge data with compact observation input and experiment to see if it holds water. If not we can always dial it down and add more sensors/feedback.
