HRC Aircraft Riveting Process

@ ZeMA, Germany

Assembly processes in aircraft production are difficult to automate due to technical risks. Examples of such technical challenges include small batch sizes and large product dimensions as well as limited work space for complex joining processes and organization of the assembly tasks.

A fully automated system can be expensive and requires a large amount of programming knowledge. For these reasons, ZeMA believes a semi-automated approach is the most effective means of success for optimizing aircraft production.

Many methods can be considered semi automation, one of which is Human-Robot-Collaboration. ZeMA will use the example of a riveting process to measure the advantages of Human-Robot-Collaboration systems in aircraft structure assembly.

In the assembly of the aircraft aft section the pressure bulkhead is mounted with a barrel section using hundreds of rivets. This assembly process is a non-ergonomic and burdensome task in which two humans must work cooperatively. The alleviation of such work can be achieved by using a collaborative scheme between operator and robot system. This approach uses dynamic task sharing between the operator and robot based on their skills and the process requirements. This is demonstrated by placing a robot inside section 19 on top of a lifting unit to position the counter holder while the human does the more complex task of inserting the rivet and operating the rivet tool.

Using dynamic task sharing and an intuitive control system, ZeMA aims to assist the operator with a collaborative robot and assistance system to improve the manufacturing quality as well as ergonomics for the human during the overhead riveting process. Interaction between human and machine is made possible by using smart devices such as a smart watch for remote control of the whole station or Mixed Reality glasses for visualization.

The results are part of the European Union’s Horizon 2020 research and innovation program at ZeMA and will present needs-based automation shown in the HRC riveting process.


Virtual presentation: Semi-automated HRC riveting station

A camera is installed at the end effector for quality control. Was the camera used during this process?

The camera is not used during the riveting process. In quality control, a differentiation is made between online quality control (force torque sensor, laser line sensor and human) and offline quality control (camera). Whereas the online quality control is performed during the riveting process, the offline quality control is only performed after the complete or paused riveting process. Afterwards, the evaluations of the online and offline quality control could be combined to increase accuracy.

How high was the pressure applied by the counter holder during counter holding?

The adjustment of the counter holder is neither force- nor pressure-controlled during the riveting process. The counter holder is adjusted by a spring and damper bearing of the mass. At the beginning of the riveting process, the counter holder is moved to the workpiece to generate a preload of the spring when the rivet gets inserted in the rivet hole. The force torque sensor is used to detect if the riveting process is performed.

ROS was used as operating system, how robust is that for an industrial approach/deployment? Which variant was used: ROS1, ROS2 or ROS Industrial?

The Universal Robots ROS(1) driver was used for communication between robot and computer, which is officially supported by Universal Robots. Robot manufacturers usually point out to use only their own certified hardware and software for warranty reasons. Theoretically, only the UR controller + PLC could be used for the riveting process. ROS brings some advantages for research purposes, which is why ROS was used in this case.

For security reasons and warranty reasons, I would refrain from a ROS-supported implementation at this point of time.

The size and dimension of the counter holder is currently very large. Can this be made smaller?

The counter holder is a prototype that was designed and fabricated under certain boundary conditions (large working space available, safe design, time). As a result, a relatively large counter holder was used. By redefining the boundary conditions and adjusting the spring, damper and mass values, a smaller counter holder can be designed. The only important thing is that only the rivet to be deformed is ever touched when several rivets are already placed.


Large process deviations can occur due to the kinematic structure, which means that a laser scanner is required for measuring and determining the rivet holes. Were / can markers also be used or are the workpiece contours used?

The measurement and detection methodology comes from the conditions of the project with the pressure bulk head. In the process, a flying calibration on the basis of significant workpiece structures was the most suitable method. However, other methods such as markers can also be used for calibration. The only condition is that the process accuracies are maintained so that the process can be performed.

In this case, we had to use a laser line sensor anyway due to the process accuracies and requirements, which made it equally suitable for a flying calibration.

How to calibrate without high additional effort?

On the one hand by using already existing sensors for the process. On the other hand, by using a calibration method that takes advantage of the conditions of the workplace and the component. In this case, this could be done via significant points on the component.

Integration of automation often failes because of size (End of arm tooling) and speed. This was not shown in this process, but is a relevant point for integration into production (especially the transition between the rivets).

At this point, the demonstrator is in a prototype research state. It is important that the processes operate cleanly without damage to the components as well as without injury to the research staff. Due to a lower speed, the intervention time for the worker is higher.

Of course, the demonstrator and the process can be run at increased speed. If there is no danger to the worker, the robot could be moved between rivets at increased speed without generating too much (over)vibration. Path and parameter optimization is usually one of the last steps in the robot development process.

What happens if the worker has chosen the wrong hole for riveting, did the robot recognize it?

The robot either follows a set sequence or is manually navigated between riveting points by the operator. At this stage, the robot is not able to independently detect if the robot is placed at the correct hole. However, with further logic and information, this could conceivably be implemented.

The sensors would be largely in existence for this purpose. A robot-side query via the force torque sensor and the robot position is no problem. Getting the counter information from the worker would have to be retrofitted.

What would it look like in real life because stations are not designed for automation? How flexible could the robot move between all the obstacles / interference contours?

The robot is mounted on a manually movable platform that can be moved between stations. The system can be commissioned at the corresponding system and workpiece by using a calibration concept. The robot itself has no collision detection. All the relevant information would have to be fed to the robot via appropriate teach-in or CAD data.

If there's a bad rivet, can you tell exactly where it is based on the data you've collected and go right back there?

When measuring on the workpiece, all rivet holes to be riveted are stored in a database. During riveting, all generated sensor data for the rivet are also stored in the database. Accordingly, an exact assignment can be made at a later time.

Afterwards, the worker can select the appropriate rivet and the robot moves to the corresponding position to carry out the process.

How do you handle the distance to the workpiece? Is the attached camera and laser line sensor used for this?

The laser line sensor is used to determine the position of the rivet holes. It can output point information in x, y, z. Accordingly, the distance (z-component) from the robot to the workpiece can also be measured.

What are the experiences with the different light effects with the sensor?

The laser line sensor uses a red laser beam and is relatively robust to light effects. Otherwise, depending on the environment, a sensor with a different wavelength could be used. With the camera, the light effects are more decisive. By selecting an appropriate lens, external lighting can be omitted at normal brightness in the hall. At this stage, we have not had any problems during the experiments due to light effects.

In the DEMO version, "next rivet" was always pressed. Would this also be the real process or can it be faster?

For demonstration purposes, manual mode was used, whereby the worker must independently command the robot to move to the next rivet. In the real process, automatic mode would be used, whereby the robot independently detects whether a riveting has been performed and then moves to the next position. This speeds up the process immensely and relieves the worker.

About sealing, did you also automatize surface preparation or only sealant beads disposition?

This is a human-robot collaboration in which skill-based tasks are divided between humans and robots. Accordingly, the robot only assists with sealant bead disposition. The surface preparation has to be done by the human. Nevertheless, tasks can be parallelized.

How do you train your AI classification? What is the size of your training DB?

As a training data set, 50 labeled images were used for camera-based quality control for image processing. A CNN algorithm was used.  As a result, a 97% accurate prediction could already be made.

A data set with over 1250 entries was used for the evaluation by the force-moment sensor for the online inspection. However, it was not possible to generate sufficient accuracies (82-86%) as with the camera-based evaluation. Accordingly, a much larger data set would be necessary. A NN method was used as well as an XGBoost method with different preprocessing approaches.

What is the "programmed" knowledge need by the robot on the product geometry?

For clear surfaces, the robot needs 3 coordinates to span a plane and measure it with the laser line sensor. Afterwards, it knows all the rivet holes that have to be approached (current use case pressure bulk head). For more complex structures, more information is needed to avoid a collision. CAD data would be best to detect interfering contours. Alternative would be to support the scan with a camera image. However, at this point nothing has been implemented to detect and avoid interfering contours.