publications

2026

T-RO

DDBot: Differentiable Physics-Based Digging Robot for Unknown Granular Materials

Xintong Yang, Minglun Wei, Yu-Kun Lai, and 1 more author

IEEE Transactions on Robotics, 2026

DOI
RCIM

Towards human-centric manufacturing: Task planning under uncertainties in human–robot collaborative assembly

Yingchao You, Ze Ji, and Changyun Wei

Robotics and Computer-Integrated Manufacturing, 2026

Abs DOI

Task planning plays a pivotal role in ensuring the smooth collaboration between humans and robots by efficiently allocating tasks among agents and scheduling available resources. Although some recently proposed task planners incorporate human factors into their frameworks, few explicitly account for human-related uncertainties, which can potentially lead to task failures. To address this gap, this study introduces a physical exertion–aware task planner that explicitly considers uncertainties in both human factors and task execution time. The uncertainties associated with physical exertion and execution time are modelled using the Single-Valued Triangular Neutrosophic (SVTN) Number method. Furthermore, a reinforcement learning-based approach is developed to learn adaptive task allocation policies and scheduling under these uncertainties. The experimental results indicate that the reinforcement learning-based approach effectively reduces performance variability compared with the benchmark methods.
TVT

CRAR: Coherent Risk-Aware Regulation for DRL-Based Mapless Robot Navigation

Shaojie Jian, Changyun Wei, Shunyu Tian, and 2 more authors

IEEE Transactions on Vehicular Technology, 2026

DOI

2025

T-ASE

Deep Reinforcement Learning With Multiple Unrelated Rewards for AGV Mapless Navigation

Boliang Cai, Changyun Wei, and Ze Ji

IEEE Transactions on Automation Science and Engineering, 2025

DOI
IJRR

Differentiable physics-based system identification for robotic manipulation of elastoplastic materials

Xintong Yang, Ze Ji, and Yu-Kun Lai

The International Journal of Robotics Research, 2025

Abs DOI

Robotic manipulation of volumetric elastoplastic deformable materials, from foods such as dough to construction materials like clay, is in its infancy, largely due to the difficulty of modelling and perception in a high-dimensional space. Simulating the dynamics of such materials is computationally expensive. It tends to suffer from inaccurately estimated physics parameters of the materials and the environment, impeding high-precision manipulation. Estimating such parameters from raw point clouds captured by optical cameras suffers further from heavy occlusions. To address this challenge, this work introduces a novel Differentiable Physics-based System Identification (DPSI) framework1 that enables a robot arm to infer the physics parameters of elastoplastic materials and the environment using simple manipulation motions and incomplete 3D point clouds, aligning the simulation with the real world. Extensive experiments show that with only a single real-world interaction, the estimated parameters, Young’s modulus, Poisson’s ratio, yield stress and friction coefficients, can accurately simulate visually and physically realistic deformation behaviours induced by unseen and long-horizon manipulation motions. Additionally, the DPSI framework inherently provides physically intuitive interpretations for the parameters in contrast to black-box approaches such as deep neural networks.
TNNLS

A Physics-Informed Demonstration-Guided Learning Framework for Granular Material Manipulation

Minglun Wei, Xintong Yang, Yu-Kun Lai, and 2 more authors

IEEE Transactions on Neural Networks and Learning Systems, 2025

DOI
TVT

Safe Learning for Multi-Robot Mapless Exploration

Wenxing Liu, Hanlin Niu, Kefan Wu, and 3 more authors

IEEE Transactions on Vehicular Technology, 2025

DOI
RCIM

Towards human-centric manufacturing: A reinforcement learning method for physical exertion alleviation in HRCA

Yingchao You and Ze Ji

Robotics and Computer-Integrated Manufacturing, 2025

Abs DOI

The advancement of the manufacturing system towards more human-centric, emphasising not only efficiency but also the well-being of workers. However, task planning in human–robot collaborative assembly (HRCA) remains challenging, when considering the physical exertion alleviation of workers, due to the complexities of physical exertion estimation and variations in human assembly operations. Different from conventional methods, this paper proposes a task planning method for physical exertion alleviation of workers in HRCA by leveraging the reinforcement learning (RL) method to train a policy. Initially, a musculoskeletal model-based method driven by human movement data to assess workers’ physical exertion is integrated into this work. Then, the policy is trained in a DuelingDQN-AM framework, utilising a carefully designed reward function informed by the estimated physical exertion of workers. The effectiveness of this approach has been validated through a simulation experiment and a proof-of-concept real assembly experiment. Simulation experiment results demonstrate the advantages of DuelingDQN-AM over other methods in terms of convergence speed and stability across multiple cycles and products of varying complexity. Additionally, real-world experiment results show that the RL strategy reduced physical exertion by 15.63% compared to the baseline method.
T-ASE

A Survey of Object Goal Navigation

Jingwen Sun, Jing Wu, Ze Ji, and 1 more author

IEEE Transactions on Automation Science and Engineering, 2025

DOI

2024

RAS

Mapless navigation via Hierarchical Reinforcement Learning with memory-decaying novelty

Yan Gao, Feiqiang Lin, Boliang Cai, and 4 more authors

Robotics and Autonomous Systems, 2024

Abs DOI

Hierarchical Reinforcement Learning (HRL) has shown superior performance for mapless navigation tasks. However, it remains limited in unstructured environments that might contain terrains like long corridors and dead corners, which can lead to local minima. This is because most HRL-based mapless navigation methods employ a simplified reward setting and exploration strategy. In this work, we propose a novel reward function for training the high-level (HL) policy, which contains two components: extrinsic reward and intrinsic reward. The extrinsic reward encourages the robot to move towards the target location, while the intrinsic reward is computed based on novelty, episode memory and memory decaying, making the agent capable of accomplishing spontaneous exploration. We also design a novel neural network structure that incorporates an LSTM network to augment the agent with memory and reasoning capabilities. We test our method in unknown environments and specific scenarios prone to the local minimum problem to evaluate the navigation performance and local minimum resolution ability. The results show that our method significantly increases the success rate when compared to advanced RL-based methods, achieving a maximum improvement of nearly 28%. Our method demonstrates effective improvement in addressing the local minimum issue, especially in cases where the baselines fail completely. Additionally, numerous ablation studies consistently confirm the effectiveness of our proposed reward function and neural network structure.
TVT

Integrated Learning-Based Framework for Autonomous Quadrotor UAV Landing on a Collaborative Moving UGV

Chang Wang, Jiaqing Wang, Zhaowei Ma, and 4 more authors

IEEE Transactions on Vehicular Technology, 2024

DOI
T-ASE

Efficient Hierarchical Reinforcement Learning for Mapless Navigation With Predictive Neighbouring Space Scoring

Yan Gao, Jing Wu, Xintong Yang, and 1 more author

IEEE Transactions on Automation Science and Engineering, 2024

DOI
TIM

Transformer-Based Multiscale Reconstruction Network for Defect Detection of Infrared Images

Changyun Wei, Hui Han, Zhichao Wu, and 2 more authors

IEEE Transactions on Instrumentation and Measurement, 2024

DOI
ICRA

VO-Safe Reinforcement Learning for Drone Navigation

Feiqiang Lin, Changyun Wei, Raphael Grech, and 1 more author

In 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024

DOI
RAL

GLSkeleton: A Geometric Laplacian-Based Skeletonisation Framework for Object Point Clouds

Qingmeng Wen, Seyed Amir Tafrishi, Ze Ji, and 1 more author

IEEE Robotics and Automation Letters, 2024

DOI
Neurocomputing

GAM: General affordance-based manipulation for contact-rich object disentangling tasks

Xintong Yang, Jing Wu, Yu-Kun Lai, and 1 more author

Neurocomputing, 2024

Abs DOI

Picking up an entangled object is a difficult manipulation task due to its rich contact dynamics. Most existing solutions fail to produce grasp poses to enable reliable manipulation due to the dependence on simplified assumptions for the motion policies. Grasps generated by these methods tend to drop objects or cause undesired movements of non-grasped objects. To improve such object-disentangling tasks, we propose to extend the concept of reinforcement learning (RL)-based affordance to include arbitrary action consequences and implement a general affordance-based manipulation (GAM) framework. In the GAM, we train an RL agent that uses more fine-grained actions and outperforms previous methods with a smaller chance of dropping objects and making contact with non-grasped hooks. Then, a manipulation affordance prediction (MAP) model is trained to estimate the performances of the RL agent. Finally, the manipulation affordance-based grasp filter (MAGF) selects grasp poses that afford the desired manipulation performances, showing substantial improvements in five challenging hook disentangling tasks in simulation. The experiments show (1) the limitation of TAG generators, (2) the effectiveness of filtering TAGs with predicted manipulation performances based on the general affordance theory, and (3) the importance of avoiding contact with non-grasped objects in contact-rich manipulation.
IET-CSR

Learning to bag with a simulation-free reinforcement learning framework for robots

Francisco Munguia-Galeano, Jihong Zhu, Juan David Hernández, and 1 more author

IET Cyber-Systems and Robotics, 2024

Abs DOI

Abstract Bagging is an essential skill that humans perform in their daily activities. However, deformable objects, such as bags, are complex for robots to manipulate. A learning-based framework that enables robots to learn bagging is presented. The novelty of this framework is its ability to learn and perform bagging without relying on simulations. The learning process is accomplished through a reinforcement learning (RL) algorithm introduced and designed to find the best grasping points of the bag based on a set of compact state representations. The framework utilises a set of primitive actions and represents the task in five states. In our experiments, the framework reached 60% and 80% success rates after around 3 h of training in the real world when starting the bagging task from folded and unfolded states, respectively. Finally, the authors test the trained RL model with eight more bags of different sizes to evaluate its generalisability.
NeurIPS

SCaR: refining skill chaining for long-horizon robotic manipulation via dual regularization

Zixuan Chen, Ze Ji, Jing Huo, and 1 more author

In Proceedings of the 38th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 2024

Abs

Long-horizon robotic manipulation tasks typically involve a series of interrelated sub-tasks spanning multiple execution stages. Skill chaining offers a feasible solution for these tasks by pre-training the skills for each sub-task and linking them sequentially. However, imperfections in skill learning or disturbances during execution can lead to the accumulation of errors in skill chaining process, resulting in execution failures. In this paper, we investigate how to achieve stable and smooth skill chaining for long-horizon robotic manipulation tasks. Specifically, we propose a novel skill chaining framework called Skill Chaining via Dual Regularization (SCaR). This framework applies dual regularization to sub-task skill pre-training and fine-tuning, which not only enhances the intra-skill dependencies within each sub-task skill but also reinforces the inter-skill dependencies between sequential sub-task skills, thus ensuring smooth skill chaining and stable long-horizon execution. We evaluate the SCaR framework on two representative long-horizon robotic manipulation simulation benchmarks: IKEA furniture assembly and kitchen organization. Additionally, we conduct a simple real-world validation in tabletop robot pick-and-place tasks. The experimental results show that, with the support of SCaR, the robot achieves a higher success rate in long-horizon tasks compared to relevant baselines and demonstrates greater robustness to perturbations.

2023

TNNLS

Deep Reinforcement Learning With Explicit Context Representation

Francisco Munguia-Galeano, Ah-Hwee Tan, and Ze Ji

IEEE Transactions on Neural Networks and Learning Systems, 2023

DOI
T-ASE

Efficient Hierarchical Reinforcement Learning for Mapless Navigation With Predictive Neighbouring Space Scoring

Yan Gao, Jing Wu, Xintong Yang, and 1 more author

IEEE Transactions on Automation Science and Engineering, 2023

DOI
TCDS

Recent Advances of Deep Robotic Affordance Learning: A Reinforcement Learning Perspective

Xintong Yang, Ze Ji, Jing Wu, and 1 more author

IEEE Transactions on Cognitive and Developmental Systems, 2023

DOI
IEEE Access

Affordance-Based Human–Robot Interaction With Reinforcement Learning

Francisco Munguia-Galeano, Satheeshkumar Veeramani, Juan David Hernández, and 2 more authors

IEEE Access, 2023

DOI
WACV

AFPSNet: Multi-Class Part Parsing based on Scaled Attention and Feature Fusion

Njuod Alsudays, Jing Wu, Yu-Kun Lai, and 1 more author

In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023

DOI

2022

ROBIO

Localisation-Safe Reinforcement Learning for Mapless Navigation

Feiqiang Lin, Ze Ji, Changyun Wei, and 1 more author

In 2022 IEEE International Conference on Robotics and Biomimetics (ROBIO), 2022

DOI
Mater. Des

Predicting laser powder bed fusion defects through in-process monitoring data and machine learning

Shuo Feng, Zhuoer Chen, Benjamin Bircher, and 3 more authors

Materials & Design, 2022

Abs DOI

Industry application of additive manufacturing demands strict in-process quality control procedures and high product quality. Feedback loop control is a reasonable solution and a necessary tool. This paper demonstrated our preliminary work on the laser powder-bed fusion feedback loop: predict local porosity through in-process monitoring images and machine learning. 3D models were rebuilt from in-situ optical tomography monitoring images and post-build X-ray CT images. They were registered to the original CAD. Dataset for machine learning was assembled from those registered 3D models. The trained machine learning model can precisely predict local porosity caused by lack of fusion and keyhole with multi-layer monitoring images. It also indicates the optimal processing window. It is impossible to be sure about the occurrence of defects in a layer based only on the abnormality of a single layer, and vice versa. Defects in a layer can be caused by improper parameters or anomalies in current layer or subsequent layers; defects in one layer can also be eliminated by proper parameters in the following layers. The work laid the basis for the next step feedback loop control of pore defect.
TASE

Anisotropic GPMP2: A Fast Continuous-Time Gaussian Processes Based Motion Planner for Unmanned Surface Vehicles in Environments With Ocean Currents

Jiawei Meng, Yuanchang Liu, Richard Bucknall, and 2 more authors

IEEE Transactions on Automation Science and Engineering, 2022

DOI
OE

Learning manipulation skills with demonstrations for the swing process control of dredgers

Changyun Wei, Haonan Bai, Yi Wei, and 1 more author

Ocean Engineering, 2022

DOI
AIM

Design and development of a robust vision-based tactile sensor

Prasad Rayamane, Ze Ji, and Michael Packianather

In 2022 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, Jul 2022
CASE

FRobs_RL: A flexible robotics reinforcement learning library

Jose Manuel Fajardo, Felipe González Roldan, Sebastian Realpe, and 3 more authors

In IEEE International Conference on Automation Science and Engineering (CASE), Aug 2022

2021

Int.J.Intell.Syst.

Safe distance prediction for braking control of bridge cranes considering anti-swing

Huili Chen, Guoliang Liu, Guohui Tian, and 2 more authors

International Journal of Intelligent Systems, 2021

DOI
Int.J.Intell.Syst.

Online human action recognition with spatial and temporal skeleton features using a distributed camera network

Guoliang Liu, Qinghui Zhang, Yichao Cao, and 2 more authors

International Journal of Intelligent Systems, 2021
TAROS

Reinforcement Learning-based Mapless Navigation with Fail-safe Localisation

F. Lin, Z. Ji, C. Wei, and 1 more author

In Towards Autonomous Robotic Systems Conference (TAROS), 2021
TAROS

Designing a Multi-Locomotion Modular Snake Robot

S. Djoudi, J.M. Frampton, A. Murphy, and 3 more authors

In Towards Autonomous Robotic Systems Conference (TAROS), 2021
Precis. Agric

RS-Net: robust segmentation of green overlapped apples

Weikuan Jia, Zhonghua Zhang, Wenjiang Shao, and 2 more authors

Precision Agriculture, 2021
Comput. Electron. Agric.

FoveaMask: A fast and accurate deep learning model for green fruit instance segmentation

Weikuan Jia, Zhonghua Zhang, Wenjing Shao, and 4 more authors

Computers and Electronics in Agriculture, 2021

Abs DOI

In the process of agricultural automation production, efficient and accurate segmentation of target fruit is the basis and guarantee for numerous applications including crop growth monitoring, yield prognosis, and machine picking. Green fruit is apt to affect by complicated scenes such as occlusions and overlaps, as well as the homo-chromatic background, which brings great challenges to the recognition and segmentation. In this paper, we propose a novel methodology named FoveaMask for improving the robustness and generalization of green fruit segmentation framework. Features of input images are firstly extracted by ResNet and fused by FPN. The classification and bounding-box regression of each spatial position on the feature maps are then carried out directly by the way of full convolution. RoI Align layer is applied to fix feature region of proposals to the same size yet preserve exact spatial locations. Finally, instance-level fruit segmentation is realized by pixel-level classification on each proposal using embedded mask branches. The whole network architecture of the new model does not involve the related design and operation of anchor, which greatly improves the generalization ability for different shape fruits, alleviates the computing and storage resources, and balances the contradiction between accuracy and efficiency simultaneously. Additionally, a position attention module(PAM) is introduced into the embedding mask branch to aggregate the effective information pixels, which can improve the robustness of the segmentation model in the actual complex environment of the orchard. We test the model on green apple and immature persimmon data sets and the experimental results show that FoveaMask performs best in terms of both recognition accuracy and model complexity compared with other 11 different types of detection and segmentation models.
IROS

ShorelineNet: An Efficient Deep Learning Approach for Shoreline Semantic Segmentation for Unmanned Surface Vehicles

L. Yao, K. Dimitrios, Z. Ji, and 1 more author

In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2021
TAROS

An Open-Source Multi-Goal Reinforcement Learning Environment for Robotic Manipulation with Pybullet

Xintong Yang, Ze Ji, Jing Wu, and 1 more author

In Towards Autonomous Robotic Systems Conference (TAROS), 2021
TNNLS

Hierarchical reinforcement learning with universal policies for multi-step robotic manipulation

Xintong. Yang, Z. Ji, J. Wu, and 4 more authors

IEEE Transaction on Neural Networks and Learning Systems, 2021
OE

Model predictive control for slurry pipeline transportation of a cutter suction dredger

Changyun Wei, Yi Wei, and Ze Ji

Ocean Engineering, 2021

Abs DOI

Cutter Suction Dredgers (CSDs) are a special type of ships designed for construction and maintenance projects of ocean and offshore engineering. During the dredging operation, CSDs can excavate nearly all kinds of soil on the sea bed, and then the dredged materials with coarse particles need to be sucked up by a slurry pump and transported to a disposal area through a long-distance pipeline. In order to avoid sedimentation of slurry in pipeline transportation, the flow rate must be maintained within a reasonable range. Otherwise, the pipeline can be blocked when the slurry density is too high. In this paper, we present a Model Predictive Control (MPC) approach to manipulate the flow rate of slurry in pipeline transportation for a CSD. To demonstrate the advantages of our proposed approach, we also implement three Proportional Integral Derivative (PID) controllers (i.e., conventional PID, Fuzzy-PID, and LQR-PID) to make a direct comparison. Moreover, in order to evaluate the effectiveness of our proposed approach in real scenarios, we have, in particular, built a slurry pipeline transportation platform. Both the simulation and experimental results show that our proposed MPC approach is more effective than other PID controllers in controlling the flow rate in the slurry pipeline transportation problem. The proposed approach can provide a guideline for the automated control of the slurry pump for a CSD.
Sens. Actuator

Self-validating high-g accelerometers through data-driven methods

Jingjing Wen, Houpu Yao, Ze Ji, and 2 more authors

Sensors and Actuators A: Physical, 2021

Abs DOI

In the aerospace industry, pyroshock testing is an indispensable step in designing space electronics. Yet, damages in high-g accelerometers, the core measuring instruments in pyroshock test systems, could result in various failures of pyroshock tests. To ensure the reliability of pyroschock tests for space electronics, a machine learning system is proposed to perform self-validation for high-g accelerometers. In this work, self-validation refers to the capability of identifying five key parameters, namely the validated shock signal, the validated uncertainty, the measurement status, the raw shock signal and the fault type, synchronously during measuring shock signals in pyroshock tests. To achieve the highest performance, we accomplish these tasks through combining an ensemble learning model and a deep neural network (DNN). The ensemble learning model, which integrates several k-nearest neighbors with different k values, is used to identify the sensorsâ health conditions from their measurements and diagnose their fault types synchronously if damaged. The DNN, a deep autoencoder-based neural network, is designed to correct corrupted measurements through constructing the mapping between faulty signals and their corresponding reference counterparts. Experimental results show that the proposed machine learning system is capable of not only accurately identifying the health conditions and fault types of the damaged high-g accelerometers from their measurements, but also recovering the corrupted shock signals to a large extent, and, meanwhile, outputting the five self-validation parameters.
SII

Accelerated Sim-to-Real Deep Reinforcement Learning: Learning Collision Avoidance from Human Player

H. Niu, Z. Ji, F. Arvin, and 2 more authors

In 2021 IEEE/SICE International Symposium on System Integration, 2021
SII

Model Checking for Decision Making System of Long Endurance Unmanned Surface Vehicle

H. Niu, Z. Ji, A. Savvaris, and 2 more authors

In 2021 IEEE/SICE International Symposium on System Integration, 2021
Automat Constr

Deep learning-based prediction of piled-up status and payload distribution of bulk material

ZongWei Yao, Qiuping Huang, Ze Ji, and 2 more authors

Automation in Construction, 2021

2020

JAMT

A novel image feature descriptor for SLM spattering pattern classification using a consumable camera

Ze Ji and Quanquan Han

Int J Adv Manuf Technol, 2020

DOI
OE

Energy efficient path planning for Unmanned Surface Vehicle in spatially-temporally variant environment

Hanlin Niu, Ze Ji, Al Savvaris, and 1 more author

Ocean Engineering, 2020

Abs DOI

Unmanned Surface Vehicles (USVs) are increasingly used for ocean missions, which typically require long duration of operations under strict energy constraints. Consequently, there is an increased interest in energy efficient path planning for USVs. This work proposes a novel energy efficient path planning algorithm to address the challenges with the presence of spatially-temporally variant sea current and complex geographic map data, by integrating the following algorithms, namely Voronoi roadmap, Dijkstra’s searching, coastline expanding and genetic algorithm (GA). The selection, crossover and mutation operators are employed as part of the GA algorithm. The dividing, smoothing and exchanging operators are proposed to improve the quality of the path and adapt to the Voronoi-Visibility roadmap. The Global Self-Consistent Hierarchical High-Resolution Shorelines dataset and historical sea current dataset are applied to demonstrate the flexibility and practicability of the proposed algorithm. To evaluate the performance, the Voronoi-GA energy efficient algorithm and Voronoi-Visibility energy efficient path re-planning algorithm are also implemented to provide the baseline for comparison. The proposed algorithm generates the most energy efficient paths in ten USV missions, while keeping a configurable clearance from the coastlines. The practicability and scalability of this algorithm is also demonstrated by analysing the computational time in these ten missions.
IEEE Sensors

On Fault Diagnosis for High-g Accelerometers via Data-Driven Models

J. Wen, H. Yao, Z. Ji, and 2 more authors

IEEE Sensors Journal, 2020
RA-L

Particle Swarm Optimization for Cooperative Multi-Robot Task Allocation: A Multi-Objective Approach

C. Wei, Z. Ji, and B. Cai

IEEE Robotics and Automation Letters, 2020
IEEE Sensors

Efficient Multi-View Multi-Target Tracking Using a Distributed Camera Network

L. He, G. Liu, G. Tian, and 2 more authors

IEEE Sensors Journal, 2020
IEEE Sensors

A Deep Learning Approach to Recover High-g Shock Signals From the Faulty Accelerometer

J. Wen, H. Yao, B. Wu, and 2 more authors

IEEE Sensors Journal, 2020
AIM

Online Collision Avoidance for Human-Robot Collaborative Interaction Concerning Safety and Efficiency

G. Liu, H. He, G. Tian, and 2 more authors

In 2020 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), 2020

2019

IEEE Robio

Visual SLAM Based on Dynamic Object Removal

H. Liu, G. Liu, G. Tian, and 2 more authors

In 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), 2019
IEEE Robio

Distributed Human 3D Pose Estimation and Action Recognition

G. Liu, T. Liu, G. Tian, and 1 more author

In 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), 2019
IEEE Robio

Simultaneous Monocular Visual Odometry and Depth Reconstruction with Scale Recovery

Y. Luo, G. Liu, H. Liu, and 3 more authors

In 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), 2019
taros

An Auto-Correction Teleoperation Method for a Mobile Manipulator Using Gaze Tracking and Hand Motion Detection

Junshen Chen, Ze Ji, Hanlin Niu, and 2 more authors

In Towards Autonomous Robotic Systems, 2019

Abs

Situational awareness in remote environments is crucial for human operators to teleoperate mobile manipulators confidently and reliably. Visual feedback is the most common way for environment perception, providing rich information to human operators. This paper proposes an intuitive teleoperation method by combining gaze tracking and hand motion detection to teleoperate a mobile manipulator. A camera is fixed on the end-effector of the mobile robot’s arm to provide visual feedback, acting as the eye of the teleoperator. Rather than direct remote control of the robot, an adaptive auto-correction mechanism is introduced for helping human operators to achieve better hand-eye coordination of the teleoperation experiences. The mobile manipulator can adjust its behaviours, such as speed, while gaze and hand movements of the operator are in different states. The experiments carried out demonstrated the effectiveness of the proposed algorithm and the survey evaluation verified the practical application value of the system.
Voronoi-Visibility Roadmap-based Path Planning Algorithm for Unmanned Surface Vehicles

Hanlin Niu, Al Savvaris, Antonios Tsourdos, and 1 more author

Journal of Navigation, 2019

DOI
ICASSP

Low-cost Measurement of Industrial Shock Signals via Deep Learning Calibration

H. Yao, J. Wen, Y. Ren, and 2 more authors

In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019
JNE

Feature extraction of four-class motor imagery EEG signals based on functional brain network

Qingsong Ai, Anqi Chen, Kun Chen, and 4 more authors

Journal of Neural Engineering, Feb 2019

Abs DOI

Objective. A motor-imagery-based brainâcomputer interface (MI-BCI) provides an alternative way for people to interface with the outside world. However, the classification accuracy of MI signals remains challenging, especially with an increased number of classes and the presence of high variations with data from multiple individual people. This work investigates electroencephalogram (EEG) signal processing techniques, aiming to enhance the classification performance of multiple MI tasks in terms of tackling the challenges caused by the vast variety of subjects. Approach. This work introduces a novel method to extract discriminative features by combining the features of functional brain networks with two other feature extraction algorithms: common spatial pattern (CSP) and local characteristic-scale decomposition (LCD). After functional brain networks are established from the MI EEG signals of the subjects, the measures of degree in the binary networks are extracted as additional features and fused with features in the frequency and spatial domains extracted by the CSP and LCD algorithms. A real-time BCI robot control system is designed and implemented with the proposed method. Subjects can control the movement of the robot through four classes of MI tasks. Both the BCI competition IV dataset 2a and real-time data acquired in our designed system are used to validate the performance of the proposed method. Main results. As for the offline data experiment results, the average classification accuracy of the proposed method reaches 79.7%, outperforming the majority of popular algorithms. Experimental results with real-time data also prove the proposed method to be highly promising in its real-time performance. Significance. The experimental results show that our proposed method is robust in extracting discriminative brain activity features when performing different MI tasks, hence improving the classification accuracy in four-class MI tasks. The high classification accuracy and low computational demand show a considerable practicality for real-time rehabilitation systems.

2018

Autonomous 3D scene understanding and exploration in cluttered workspaces using point cloud data

Z. Ji, R. Schruoffeneger, and R. Setchi

In 2018 IEEE 15th International Conference on Networking, Sensing and Control (ICNSC), 2018
taros

Seeing the Unseen: Locating Objects from Reflections

Jing Wu and Ze Ji

In Towards Autonomous Robotic Systems, 2018

Abs

Inspired by the ubiquitous use of reflections in human vision system, in this paper, we present our first step exploration of using reflections to extend the FOV of cameras in computer vision applications. We make use of a stereo camera, and establish mathematical models for locating objects from their mirror reflections. We also propose a pipeline to track and locate moving objects from their reflections captured in videos. Experimental results demonstrated the efficiency and effectiveness of the proposed method, and verified the potential use of reflections in locating non-line of sight (NLOS) objects.

2017

Autonomous boat dynamics: How far away is simulation from the high sea?

Y. Huang and Z. Ji

In OCEANS 2017 - Aberdeen, 2017