Abstract

In this paper, we address the following question: How can multi-robot self-organizing systems be designed so that they show the desired behavior and are able to perform tasks specified by the designers? Multi-robot self-organizing systems, e.g., swarm robots, have great potential for adapting when performing complex tasks in a changing environment. However, such systems are difficult to design due to the stochasticity of system performance and the non-linearity between the local actions/interaction and the desired global behavior. In order to address this, in this paper, we propose a framework for designing self-organizing systems using Multi-Agent Reinforcement Learning (MARL) and the compromise Decision-Support Problem (cDSP) construct. The proposed framework consists of two stages, namely, preliminary design followed by design improvement. In the preliminary design stage, MARL is used to help designers train the robots so that they show stable group behavior for performing the task. In the design improvement stage, the cDSP construct is used to explore the design space and identify satisfactory solutions considering several performance indicators. Surrogate models are used to map the relationship between local parameters and global performance indicators utilizing the data generated in the preliminary design. These surrogate models represent the goals of the cDSP. Our focus in this paper is to describe the framework. A multi-robot box-pushing problem is used as an example to test the framework’s efficacy. This framework is general and can be extended to design other multi-robot self-organizing systems.

Graphical Abstract Figure
Graphical Abstract Figure
Close modal

References

1.
Werfel
,
J.
,
Petersen
,
K.
, and
Nagpal
,
R.
,
2014
, “
Designing Collective Behavior in a Termite Inspired Robot Construction Team
,”
Science
,
343
(
6172
), pp.
754
758
.
2.
Reina
,
A.
,
2016
, “
Engineering Swarm Systems: A Design Pattern for the Best-of-n Decision Problem
,”
PhD dissertation
,
Ecole Polytechnique do Bruxelles
,
Paris, France
.
3.
Bras
,
B.
, and
Mistree
,
F.
,
1993
, “
Robust Design Using Compromise Decision Support Problems
,”
Eng. Optim.
,
21
(
3
), pp.
213
239
.
4.
Gershenson
,
C.
, and
Heylighen
,
F.
,
2003
, “
When Can We Call a System Self-Organizing?
,”
Proceedings of the European Conference on Artificial Life
,
Springer
, pp.
606
614
.
5.
Banzhaf
,
W.
,
2003
, “
Self-Organizing Systems
,”
Encycl. Compl. Syst. Sci.
,
14
, pp.
589
598
.
6.
Turing
,
A.
,
1990
, “
The Chemical Basis of Morphogenesis
,”
Bull. Math. Biol.
,
52
(
1–2
), pp.
153
197
.
7.
Howard
,
J.
, and
Fuller
,
B.
,
2010
, “
Self-Organization in Biology
,”
Res. Perspect.
, pp.
28
29
.
8.
Chertow
,
M.
, and
Ehrenfeld
,
J.
,
2012
, “
Organizing Self-Organizing Systems: Toward a Theory of Industrial Symbiosis
,”
J. Ind. Ecol.
,
16
(
1
), pp.
13
27
.
9.
Hamann
,
H.
,
Khaluf
,
Y.
,
Botev
,
J.
,
Divband Soorati
,
M.
,
Ferrante
,
E.
,
Kosak
,
O.
,
Montanier
,
J.-M.
, et al
,
2016
, “
Hybrid Societies: Challenges and Perspectives in the Design of Collective Behavior in Self-Organizing Systems
,”
Front. Rob. AI
,
3
, p.
14
.
10.
Cools
,
S.-B.
,
Gershenson
,
C.
, and
D’Hooghe
,
B.
,
2013
, “
Self-Organizing Traffic Lights: A Realistic Simulation
,”
Adv. Appl. Self-Org. Syst.
, pp.
45
55
.
11.
Zhang
,
J.
,
Yao
,
X.
,
Zhou
,
J.
,
Jiang
,
J.
, and
Chen
,
X.
,
2017
, “
Self-Organizing Manufacturing: Current Status and Prospect for Industry 4.0
,”
Proceedings of the 2017 5th International Conference on Enterprise Systems (ES)
,
Beijing, China
,
Sept. 22–24
, pp.
319
326
.
12.
Humann
,
N. K.
, and
Jin
,
Y.
,
2016
, “
Adaptability Tradeoffs in the Design of Self-Organizing Systems
,”
Proceedings of the International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, American Society of Mechanical Engineers, Paper No. DETC2016-60053
.
13.
Rahimi
,
M.
,
Gibb
,
S.
,
Shen
,
Y.
, and
La
,
H. M.
,
2019
, “
A Comparison of Various Approaches to Reinforcement Learning Algorithms for Multi-Robot Box Pushing
,”
Proceedings of the International Conference on Engineering Research and Applications
,
Thai Nguyen, Vietnam
,
May 30–31
, Springer, pp.
16
30
.
14.
Long
,
P.
,
Fan
,
T.
,
Liao
,
X.
,
Liu
,
W.
,
Zhang
,
H.
, and
Pan
,
J.
,
2018
, “
Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning
,”
Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA)
,
IEEE
, pp.
6252
6259
.
15.
Kakish
,
Z.
,
Elamvazhuthi
,
K.
, and
Berman
,
S.
,
2021
, “
Using Reinforcement Learning to Herd a Robotic Swarm to a Target Distribution
,”
Proceedings of the International Symposium Distributed Autonomous Robotic Systems
,
Virtual Online
, Springer, pp.
401
414
.
16.
Bae
,
H.
,
Kim
,
G.
,
Kim
,
J.
,
Qian
,
D.
, and
Lee
,
S.
,
2019
, “
Multi-Robot Path Planning Method Using Reinforcement Learning
,”
Appl. Sci.
,
9
(
15
), p.
3057
.
17.
Lowe
,
R.
,
Wu
,
Y. I.
,
Tamar
,
A.
,
Harb
,
J.
,
Pieter Abbeel
,
O.
, and
Mordatch
,
I.
,
2017
, “
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
,”
Adv. Neural Inf. Process. Syst.
,
30
.
18.
Ji
,
H.
, and
Jin
,
Y.
,
2019
, “
Designing Self-Organizing Systems With Deep Multi-Agent Reinforcement Learning
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, American Society of Mechanical Engineers, Paper No. DETC2019-98286
.
19.
Werfel
,
J.
,
2012
,
Collective Construction With Robot Swarms, Morphogenetic Engineering: Toward Programmable Complex Systems
,
Springer
,
Berlin/Heidelberg
, pp.
115
140
.
20.
Khani
,
N.
,
Humann
,
J.
, and
Jin
,
Y.
,
2016
, “
Effect of Social Structuring in Self-Organizing Systems
,”
ASME J. Mech. Des.
,
138
(
4
), p.
041101
.
21.
Ji
,
H.
, and
Jin
,
Y.
,
2021
, “
Evaluating the Learning and Performance Characteristics of Self-Organizing Systems With Different Task Features
,”
AI EDAM
,
35
(
4
), pp.
404
422
.
22.
Bar-Yam
,
Y.
,
2002
,
General Features of Complex Systems, Encyclopedia of Life Support Systems (EOLSS)
,
UNESCO, EOLSS Publishers
,
Oxford, UK 1
.
23.
Chen
,
C.
, and
Jin
,
Y.
,
2011
, “
A Behavior Based Approach to Cellular Self-Organizing Systems Design
,”
Proceedings of the International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Washington, DC
, pp.
95
107
.
24.
Trianni
,
V.
, and
Nolfi
,
S.
,
2011
, “
Engineering the Evolution of Self-Organizing Behaviors in Swarm Robotics: A Case Study
,”
Artif. Life
,
17
(
3
), pp.
183
202
.
25.
Jin
,
Y.
,
Meng
,
Y.
, and
Guo
,
H.
,
2010
, “
A Morphogenetic Self-Organization Algorithm for Swarm Robotic Systems Using Relative Position Information
,”
Proceedings of the 2010 UK Workshop on Computational Intelligence (UKCI)
,
Colchester, UK
,
Sept. 8–10
, pp.
1
6
.
26.
Wang
,
Z.
,
Li
,
M.
,
Dou
,
L.
,
Li
,
Y.
,
Zhao
,
Q.
, and
Li
,
J.
,
2015
, “
A Novel Multi-Objective Artificial Bee Colony Algorithm for Multi-Robot Path Planning
,”
In 2015 IEEE International Conference on Information and Automation
,
Yunnan, China
, IEEE, pp.
481
486
.
27.
Rubenstein
,
M.
, et al.
,
2013
, “
Collective Transport of Complex Objects by Simple Robots: Theory and Experiments
,”
Proceedings of the 2013 International Conference on Autonomous Agents and Multi-Agent Systems
,
St. Paul, MN
,
May 6–10
, pp.
47
54
.
28.
Kovac
,
K.
,
Zivkovic
,
I.
, and
Dalbelo Basic
,
B.
,
2004
, “
Simulation of Multi-Robot Reinforcement Learning for Box-Pushing Problem
,”
Proceedings of the 12th IEEE Mediterranean Electrotechnical Conference
,
IEEE
, Vol. 2, pp.
603
606
.
29.
Huang
,
B.
, and
Jin
,
Y.
,
2023
, “
Social Learning in Self-Organizing Systems for Complex Assembly Tasks
,”
Adv. Eng. Inform.
,
57
, pp.
1
15
.
30.
Huang
,
B.
, and
Jin
,
Y.
,
2022
, “
Reward Shaping in Multiagent Reinforcement Learning for Self-Organizing Systems in Assembly Tasks
,”
Adv. Eng. Inform.
,
54
, pp.
1
12
.
31.
Ferreira
,
S. C. C.
,
Bruns
,
R. E.
,
Ferreira
,
H. S.
,
Matos
,
G. D.
,
David
,
J. M.
,
Brandão
,
G. C.
,
da Silva
,
E. G. P.
, et al
,
2007
, “
Box-Behnken Design: An Alternative for the Optimization of Analytical Methods
,”
Anal. Chim. Acta
,
597
(
2
), pp.
179
186
.
32.
Mistree
,
F.
,
Hughes
,
O. F.
, and
Bras
,
B.
,
1993
, “Compromise Decision Support Problem and the Adaptive Linear Programming Algorithm,”
Structural Optimization: Status and Promise
,
M. P.
Kamat
, ed.,
AIAA
,
Washington, D.C
, pp.
247
286
.
33.
Nagar
,
D.
,
Pannerselvam
,
K.
, and
Ramu
,
P.
,
2022
, “
A Novel Data-Driven Visualization of n-Dimensional Feasible Region Using Interpretable Self-Organizing Maps (iSOM)
,”
Neural Networks
,
155
, pp.
398
412
.
You do not currently have access to this content.