Thumbs up

I set up this “open test lane” singlehanded on a Saturday morning while a handful of single-engine piston airplanes flew overhead on finals for Runway 31. Briefed with papers and emails, Civil Air Patrol Maj. Leik Myrabo arrived to try his hand at flying a quadcopter 10 feet off the ground around the 100-foot course made with the cheapest two-by-fours money can buy.

Myrabo is a retired professor of physics at Rensselaer Polytechnic Institute currently serving as aerospace education officer for the CAP Vermont Wing. He is also president of the local Experimental Aircraft Association chapter, and vice president of the local Academy of Model Aeronautics club. He was interested in how CAP could use a method for testing aerial robots (and pilots who fly them) invented by a team of engineers at the National Institute of Standards and Technology (NIST).

The method they created is built around buckets, because the shape of a bucket or barrel obscures a full view of the bottom when observed from above at angles other than 90 degrees.

One bucket with a bull’s-eye at the bottom thus defines a line when the circles appear complete and unbroken; two buckets define a point in space; and what NIST engineers call an “open test lane” for aerial robots includes 20 buckets mounted on wooden stands that define by visual reference a series of waypoints for a remote pilot to navigate.

Pilot skill can be measured using this apparatus, although it was designed for a broader mission that includes testing and comparing the capabilities of cameras and thermal sensors. Setting a time limit and requiring that each target be photographed with enough precision to keep the outer circle unbroken demands skillful execution of planned, purposeful maneuvers.

This method is being embraced by the firefighters, police, and search-and-rescue specialists for whom it was designed. It has drawn international interest because it remains the only standardized tool in the public domain that measures the capabilities of both payload and pilot of any small, multirotor drone.

These methods can be applied to any discipline in the drone world. They can be used for training, and to objectively measure the skill of drone pilots who inspect cell towers or overhead electric transmission lines, and perform other jobs that require close approaches to trees and buildings—flights in tight spaces where an unexpected gust of wind can put expensive equipment, even people, at risk.

After about an hour of flying, explanation, and discussion of the nuances and fine points and flexibility, the scalability built into the design, Myrabo said he planned to get started building his own apparatus.

“You’ve opened my eyes,” Myrabo said, seeing it as a potent proficiency training tool as well as a test that is accessible for all personnel, from senior squadron members to cadets. “I think it’s going to be really useful for CAP.”

‘Wow, that works’

An aerial robot with a thermal imaging system is an expensive tool to put in the hands of a novice. While some firefighters and police officers are certificated to fly airplanes and helicopters, many more are newly minted drone pilots who have never flown anything before.

Flying traditional model aircraft remains arguably the best skill preparation for flying drones, other than drones themselves. Radio control aviation involves both the use of thumbsticks and the challenge that awaits all remote pilots the first time they turn an aircraft flying away from them around, and fly it back toward them. (Use caution the first time you try this, or you, too, might wind up on YouTube for the wrong reason.)

Flight automation built into many drones has reduced, to a degree, the demand for pilot skill, but many missions do require thumb skills, and muscle memory, since structure fires and highway crashes often happen at night, in marginal weather, and/or at locations dense with obstacles. Such flights cannot easily be automated.

This is why the NIST team led by robotics engineer Adam Jacoff spent the past few years refining their standardized test method built around buckets.

It was partly, Jacoff said, driven by a desire to give robotics novices a tool to measure system performance, and figure out what kind of camera (or other sensor) meets the needs of the mission.

“My whole goal in setting up this project was to bring robots that could do something reliably useful” into disaster response. The highly skilled specialists on regional teams organized by the Federal Emergency Management Agency, trained in urban search-and-rescue and ready to respond to disasters both natural and manmade, were among the first end users Jacoff worked with.

“I was helping them arm themselves against the marketing,” Jacoff said, by creating “basic tests to separate good systems versus bad systems.”

Jacoff’s team has for a few years toted their buckets and lumber to conferences and other events to give firefighters and police officers hands-on practice and show them how easy it is to build the apparatus.

Jacoff said the feedback has lately been overwhelmingly positive, on the order of 99-to-one approval.

“Kind of astonishing, isn’t it? I’m just amazed that no one has anything to say other than, ‘wow, that works,’” Jacoff said.

‘Everyone…on the measuring stick’

The “open test lane” consists of four omnidirectional stands made from 20-inch lengths of two-by-four cut on 45-degree angles attached to a four-by-four-by-six-inch center post to form an A-frame stand with four legs. The buckets are attached to the top and sides of each A-frame stand, and the entire apparatus can be built from scratch in a few hours with some basic carpentry skills and tools (a miter saw is the most important of these).

The gear can be set up in any open area, or even indoors in a school gymnasium, anywhere with high ceilings and open space. A smaller version can be made to fly in “confined” spaces, such as a shipping container, for those interested in testing their indoor maneuvering skills. (SWAT teams are keen to deploy aerial robots able to maneuver indoors.)

A printed target goes in the bottom of each bucket, although you could also draw the crucial circle around the edge with a marker, or paint. The important thing is to wind up with a circle around the outer edge of the bottom of each bucket that is fully visible only when the drone’s camera is aligned with the center. If the ring is eclipsed, the aircraft and camera need to be nudged into the correct alignment.

That, in a nutshell, is the genius of the system: “It gives you the feedback right away,” Jacoff said. “That’s what I like about it.”

Pilots of fixed-wing airplanes will recognize the “open test lane” as a version of ground reference maneuvers. An “orbit” with a drone is a variation on a turn around a point. Instead of carving S-turns over roads, the unmanned aircraft can be flown in patterns connecting one target to the next, with a pause at each to capture an image confirming proper alignment.

Jacoff said one of the design requirements was to make the patterns flyable by the most inexperienced novice, yet still able to challenge the most seasoned expert.

“Everyone needs to get on the measuring stick,” Jacoff said. If novices can’t fly the course and achieve a score, the test becomes much less useful, because “they can’t measure what ‘better’ looks like, or how much better they’re getting.”

Structured, purposeful practice

NIST has won many proponents for its aerial robot test methods, among them Los Angeles Firefighter David Danielson, who talked it up during a webinar in March hosted by the Association for Unmanned Vehicle Systems International. Danielson said flying the course well requires discipline:

“Every movement has a purpose.” Flying an open test lane, or a more challenging version that features obstacles, is not exactly the same kind of challenge that pilots encounter in the field; “what it does, though, is that it quickly hones some skill set that gives that person the confidence, the ability to perform those basic skills. From that point we can branch out…but we’re always falling back on that basic foundation.”

Jacoff said flying drones on the NIST courses also teaches another key aviation concept: crew resource management.

A pilot flying the test with a companion keeping score can work out between them a commonly understood phraseology they will use in the field as remote pilot and visual observer.

“It’s not just the flying,” Jacoff said. “All those little things are happening, all at once. People don’t even really realize it.”

The plans and patterns, any aspect of how the apparatus is used, can be adjusted to suit. Users are free to set their own standards. In fact, they are forced to, at least for now, because NIST has resisted posting scores online out of concern that doing so might run afoul of human ego, or make it seem more like a mandate.

“I don’t want people to shrink away from the test because they think this is official scoring,” Jacoff said.

Notching faster and faster “laps” through the open test lane patterns can bring out a competitive streak that lurks within some pilots. It is partly for this reason that Jacoff was disinclined to share performance data. If there’s a world record, Jacoff isn’t saying.

However, completing each pattern in less than three minutes with high accuracy is a pretty good indication of proficiency.

Alan Frazier, the UAS liaison for the Airborne Public Safety Association, is training trainers at online and live events, focused on a derivation of the NIST methods that he calls the Basic Proficiency Evaluation for Remote Pilots (BPERP). Frazier reports that seeing it, and flying it, has made a believer out of everyone he has trained so far.

“I [know] of no other sUAS pilot proficiency standard other [than those] developed and used internally by individual agencies (and those are rare),” Frazier wrote in an email, adding that he is also a member of the Civil Air Patrol and working with others at the national level to implement a BPERP for CAP pilots.

The National Fire Protection Association has incorporated the NIST test methods into NFPA 2400, the standard created to guide unmanned aircraft implementation in the fire service.

Jacoff said there has been international interest as well, and he has worked with public safety and government standards professionals in Australia, Japan, and elsewhere.

Brandon Karr, a police officer in Pearland, Texas, who is certificated to fly both manned and unmanned aircraft, said he is a former skeptic who came to appreciate the value of the NIST methods to validate human capabilities that the FAA does not require remote pilot certificate applicants to demonstrate.

“Let’s make sure everybody that’s coming to my scene, not only are they signed off on basic flight proficiency, but they’re also signed off on X, Y, Z scenario-based training, so that we all know what everybody’s doing, and how it’s supposed to be [done] in the interest of safety,” Karr said. “We are integrating in with manned aviation more often than not.”

Email [email protected]