⑤ Execute simulation of evaluation scenarios¶
A simulation is performed based on the evaluation scenario file that has been created. The simulation is performed using the traffic simulation software used by the customer.
Files used in the simulation¶
In Example 1: Make and download evaluation scenario files based on the evaluation specification "No4_Foreseeable", three OpenScenario files and one OpenDrive file were created. A total of three simulations will be performed, but the OpenDrive file will be used in common for each simulation.
In Example 2: Make and download evaluation scenario files based on the evaluation specification "No4_Preventable", 16 OpenScenario files and one OpenDrive file were created. A total of 16 simulations will be performed, and the OpenDrive file will be used in common for each simulation.
Preparation before execute a simulation¶
If you are using a simulation software that cannot read OpenScenario files or OpenDrive files, prepare the simulation environment by referring to those files.
If you are using simulation software ※1 that can read OpenScenario files or OpenDrive files, please set it up to read them. However, if necessary information is missing, the simulation may not run as expected. Please modify the OpenScenario file or OpenDrive file to match the requirements of the simulation software ※2.
※1 DIVP®: Driving Intelligence Validation Platform, CARLA, etc.
※2 In the scenario "No. 4" discussed here, the scene starts with the ego vehicle and other vehicles moving. If you want to drive the ego vehicle in autonomous driving mode using the simulation software CARLA, you need to start the ego vehicle from a stopped state. In other words, you need to add a scenario from the ego vehicle stopped state to the start scene of scenario "No. 4".
Execute the simulation and record the results¶
Use your own simulation software to execute each evaluation scenario. Then, record the simulation results in a json file using a text editor or similar.
We will explain using the evaluation specification "No4 Foreseeable" as an example. Execute the simulation for the following evaluation scenario file.
43e11c28_TRAFFIC-DISTURBANCE-No4-00000001.xosc
43e11c28_TRAFFIC-DISTURBANCE-No4-00000002.xosc
43e11c28_TRAFFIC-DISTURBANCE-No4-00000003.xosc
The results of each simulation should be reflected in a json file (43e11c28_sim_result.json). The json file has a section for specifying the variable "Value" for each evaluation scenario, and if a collision is avoided, enter "PASS", and if a collision occurs, enter "FAIL" ※ .
For example, if only scenario number 1 (evaluation scenario file: 43e11c28_TRAFFIC-DISTURBANCE-No4-00000001.xosc) collided and all others were avoided, it would be written as follows.
※ The initial value of "Value" is "FAIL", so please rewrite it if necessary.
The scenario information for this evaluation is as follows:
Evaluation scenario number |
1 |
2 |
3 |
\(V_{e0}\) [km/h] |
50 |
50 |
50 |
\(V_{o0}\) [km/h] |
30 |
30 |
30 |
\(d_{x0}\) [m] |
20 |
30 |
40 |
\(V_{y}\) [m/s] |
1.1 |
1.1 |
1.1 |
×Collision／〇Collision avoidance |
× |
〇 |
〇 |
For the evaluation specification "No4_Preventable", simulations are execute for the following 16 evaluation scenario files.
7a1532f9_TRAFFIC-DISTURBANCE-No4-00000001.xosc
7a1532f9_TRAFFIC-DISTURBANCE-No4-00000002.xosc
：
7a1532f9_TRAFFIC-DISTURBANCE-No4-00000016.xosc
Each simulation result should be reflected in the json file (7a1532f9_sim_result.json). For example, if a collision occurs when the scenario number is odd and collision is avoided when the scenario number is even, it should be written as follows.
{
"Version": "2023-10-25",
"Results": [
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000001_1",
"Observer": [
{
"Name": "Collision",
"Value": "FAIL"
}
]
},
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000002_1",
"Observer": [
{
"Name": "Collision",
"Value": "PASS"
}
]
},
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000003_1",
"Observer": [
{
"Name": "Collision",
"Value": "FAIL"
}
]
},
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000004_1",
"Observer": [
{
"Name": "Collision",
"Value": "PASS"
}
]
},
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000005_1",
"Observer": [
{
"Name": "Collision",
"Value": "FAIL"
}
]
},
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000006_1",
"Observer": [
{
"Name": "Collision",
"Value": "PASS"
}
]
},
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000007_1",
"Observer": [
{
"Name": "Collision",
"Value": "FAIL"
}
]
},
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000008_1",
"Observer": [
{
"Name": "Collision",
"Value": "PASS"
}
]
},
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000009_1",
"Observer": [
{
"Name": "Collision",
"Value": "FAIL"
}
]
},
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000010_1",
"Observer": [
{
"Name": "Collision",
"Value": "PASS"
}
]
},
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000011_1",
"Observer": [
{
"Name": "Collision",
"Value": "FAIL"
}
]
},
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000012_1",
"Observer": [
{
"Name": "Collision",
"Value": "PASS"
}
]
},
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000013_1",
"Observer": [
{
"Name": "Collision",
"Value": "FAIL"
}
]
},
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000014_1",
"Observer": [
{
"Name": "Collision",
"Value": "PASS"
}
]
},
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000015_1",
"Observer": [
{
"Name": "Collision",
"Value": "FAIL"
}
]
},
{
"Sid": "7a1532f9_TRAFFIC-DISTURBANCE-No4-00000016_1",
"Observer": [
{
"Name": "Collision",
"Value": "PASS"
}
]
}
]
}
For reference, we reprint the information for each evaluation scenario. We set the simulation results (collision avoidance/collision) of the 16 evaluation scenarios to match avoidable/unavoidable.
Evaluation scenario number |
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
13 |
14 |
15 |
16 |
\(V_{e0}\) [km/h] |
50 |
50 |
50 |
50 |
50 |
50 |
50 |
50 |
50 |
50 |
50 |
50 |
50 |
50 |
50 |
50 |
\(V_{o0}\) [km/h] |
30 |
30 |
30 |
30 |
30 |
30 |
30 |
30 |
30 |
30 |
30 |
30 |
30 |
30 |
30 |
30 |
\(d_{x0}\) [m] |
23 |
27 |
18 |
22 |
15 |
19 |
14 |
18 |
12 |
16 |
11 |
15 |
11 |
15 |
10 |
14 |
\(V_{y}\) [m/s] |
0.35 |
0.35 |
0.5 |
0.5 |
0.65 |
0.65 |
0.8 |
0.8 |
0.95 |
0.95 |
1.1 |
1.1 |
1.25 |
1.25 |
1.4 |
1.4 |
×Unavoidable／〇Avoidable |
× |
〇 |
× |
〇 |
× |
〇 |
× |
〇 |
× |
〇 |
× |
〇 |
× |
〇 |
× |
〇 |