How To Evaluate the Navigation of Autonomous Vehicles Around Pedestrians?

The navigation of autonomous vehicles around pedestrians is a key challenge when driving in urban environments. It is essential to test the proposed navigation system using simulation before moving to real-life implementation and testing. Evaluating the performance of the system requires the design of a diverse set of tests which spans the targeted working scenarios and conditions. These tests can then undergo a process of evaluation using a set of adapted performance metrics. This work addresses the problem of performance evaluation for an autonomous vehicle in a shared space with pedestrians. The methodology for designing the test simulations is discussed. Moreover, a group of performance metrics is proposed to evaluate the different aspects of the navigation: the motion safety, the quality of the generated trajectory and the comfort of the pedestrians surrounding the vehicle. Furthermore, the success/fail criterion for each metric is discussed. The implementation of the proposed evaluation method is illustrated by evaluating the performance of a pre-designed proactive navigation system using a shared space crowd simulator under Robot Operating System (ROS).


I. INTRODUCTION
A UTONOMOUS vehicles (AVs) have the ability to trans- form urban lifestyle and reduce the carbon footprint of cities by optimizing travel times, reducing congestion and the overall requirement for vehicle ownership [1], [2], [3].Furthermore, AVs are accessible equally to all demographics (families, elderly, infirm), and they can reduce human losses and injures caused by traffic accidents [4].However, these driverless vehicles cannot be fully integrated into our daily lives without the capability to navigate safely and efficiently around vulnerable road users.Developing a fully autonomous vehicle navigation system that can operate around pedestrians is an increasingly critical issue, especially with the growing appearance of the shared space concept in city planning [5].This growing interest is motivated by the premise of more green and pedestrian-friendly cities where vehicles and pedestrians share public spaces in a safe and efficient way [4], [6].However, building, testing and evaluating AVs navigation systems in shared spaces with pedestrians are highly challenging tasks, each task being a standalone multidisciplinary problem.These steps of the development process (Build → Test → Evaluate) are equally crucial before letting a potentially harmful system, such as an AV, operate in proximity with humans.This work discusses the methodology of evaluating the performance of a pre-designed AV navigation system in a simulated environment.The navigation system evaluation process translates into ensuring that the system is functional, efficient and above all safe in all possible working scenarios.However, the evaluation in shared spaces is much more challenging than in any other structured environment.In a more structured environment, such as a highway, the structure of the space itself and the strict driving rules limit the number of possible use cases.On the contrary, the free and open nature of shared spaces creates a wide range of possible working scenarios to test and evaluate.Moreover, the concepts of efficiency, natural and socially compliant behavior are much more challenging to define in such a space.
Evaluating the navigation system in shared spaces with pedestrians requires a set of adapted performance measures.The metrics traditionally used for autonomous robots to validate the navigation are necessary but not sufficient.The validation process should consider the safety and comfort of pedestrians around the navigating system, and of the passengers inside the vehicle.This problem can be summarized in selecting and/or designing a set of performance metrics that, firstly, evaluate the system's motion safety in proximity with pedestrians.Secondly, the proposed set of metrics should evaluate the quality of the resulting AV motion in terms of traveled distance, traveled time and even nature of the generated trajectory.Moreover, the analysis of the AV's trajectory should assess the comfort and safety of the vehicle's passengers.Finally, the proposed metrics should be able to evaluate the AV's social compliance and comfort of the pedestrians around the vehicle.This reduces the evaluation process in shared spaces into three main categories: motion safety, trajectory quality, and pedestrian comfort.In this work, a set of adapted metrics is proposed to evaluate the performance of an AV around pedestrians in a simulated environment using only the spatio-temporal information for both the pedestrian and the AV.The proposed evaluation process and the designed set of metrics are applied to test the performance of a proactive navigation system using a crowd-based simulator under ROS.
The structure of this paper is as follows: Section II gives a background on the methods used in the literature for performance evaluation in autonomous navigation applications.Section III discusses the process of designing the set of test simulations.After establishing the test set, Section IV proceeds to the performance metrics required to evaluate the different aspects of the AV's performance around pedestrians.Furthermore, this section presents how each metric can be implemented and its evaluation criterion.The navigation sys-tem under evaluation along with the simulation environment are presented in Section V. Finally, the implementation of the suggested evaluation process and the analysis of the main simulation results are discussed in Section VI.

II. RELATED WORK
The problem of AV navigation around pedestrians has only started getting researchers attention in recent years.Therefore, not many works in the literature arrived at the performance validation step of an AV system around pedestrians.However, works in robotics and transportation engineering have targeted the issue of evaluating navigation systems around humans.To conduct a thorough evaluation of the system, a combination of different metrics is needed, quantifying different aspects of the system.Therefore, we can classify the navigation performance metrics into three main categories: motion safety, trajectory quality and pedestrian comfort.
The Motion Safety of a navigation system is mostly evaluated by reporting the collision rate, i.e. how many pedestrians who interacted with the vehicle had an accident [7].The near missed rate may be used instead of the collision rate, by counting the occurrences of very close approaches between pedestrians and the vehicle [8].A more continuous metric like the minimum and mean distance between pedestrians and the AV can be used [9].Other research qualifies the motion safety by analyzing the severity of potential collisions.This can be measured by the distance or the temporal proximity to the collision when the evasive action starts.For example, the Time To Collision (TTC) before the vehicle starts reacting [10], [11], [12] or the Post Encroachment Time (PET) corresponding to the TTC at the minimal approach distance [11], [12] have been used.The speed of the vehicle before it starts reacting can also be used, as proposed by [11].
The Trajectory Quality can be defined by the smoothness of the trajectory produced by the vehicle along the navigation, as well as its efficiency.Moreover, the trajectory quality can also translate to measuring the level of comfort of any passengers along the AV's trajectory.
A smooth trajectory limits abrupt changes in direction and speed.Trajectory smoothness has been measured by the total AV acceleration along its trajectory in [8], by the number of AV accelerations and decelerations in [7], or by the intensity of acceleration and decceleration in [10].However, these metrics are insufficient to evaluate navigation in an open space, where the AV can adapt its steering in addition to its speed.In order to more thoroughly evaluate the trajectory smoothness, a trajectory stability metric based on the turning angle, the path's length and duration, as well as the speed of the AV was proposed by [13].A metric used for mobile robots, based on the bending energy and the smoothness of curvature of the robot's trajectory has also been reported in [9].
The most efficient, i.e. optimal, trajectory is a straight line from the starting point to the goal, of minimal length, covered in minimal time.To measure a trajectory's efficiency, the trajectory length or the travel time to reach the destination can be used [9], as in [7] and [8].[7] also used a success rate metric, reporting the percentage of simulations where the AV reaches its goal within an arbitrary time limit.However, in order to obtain comparable measures for different trajectories, it is preferable to use relative distances and times, as in [11].The AV's travelled trajectory can be compared to its optimal trajectory, in order to obtain the delay and the excess distance travelled.
Although passenger comfort is an important factor in the acceptance and adoption of AVs, only a few studies have addressed this point.To determine passenger comfort, studies usually collect data through a survey of the passengers in the vehicle.In simulation, only data from the vehicle trajectory can be used to estimate passenger comfort.A metric including the lateral acceleration of the AV can for example be used as an indicator of passenger comfort, as shown by [14].
In addition to safety, the Pedestrian Comfort around the vehicle is an important factor in the performance evaluation in shared spaces.The conflict duration has been used as a comfort metric by [11], based on the idea that the longer the conflict with the vehicle, the less comfortable the interaction is for the pedestrian.The minimal approach distance between the pedestrian and the vehicle can also be used, as in [10] and [11].A pedestrian discomfort metric has been proposed by [15].This metric reflects the average degree and frequency of pedestrian velocity changes, due to avoidance maneuvers.Similarly, variations in walking speed and in pedestrian trajectory have been used by [10], [11] and [12] to measure pedestrian comfort.These metrics assume that changes in walking behavior to avoid the car require effort from pedestrians, which may cause discomfort.Following a similar idea, [10] has proposed using the severity and complexity of the evasive action performed by the pedestrian.The severity is the degree of speed change and the complexity is related to whether the pedestrian performs a change in speed, direction or a combination of both.

III. TEST CASES DESIGN FOR PERFORMANCE
VALIDATION The first step of the performance evaluation is the process of designing the tests themselves.The interpretation of the test results is only valid in the context of the considered test cases.Therefore, the designed test set should be diverse, including all the possible user cases and scenarios in order to be sufficient to validate the performance.Firstly, we start by identifying the set of testing environments E = {e k } 1≤k≤K .Each test environment e k is defined by the static map of the testing space that includes the grids occupied by static obstacles, the free space and the semantics associated with the occupied space.These semantics (exit/entry points, walls, etc.) should drive the movement of the agents in the space.Therefore, they can be used to evaluate the trajectory and to estimate if it is socially acceptable and if it followed the rules associated with the specific semantics.Secondly, a set of test scenarios S = {s j } 1≤j≤J is defined within each environment.Each test scenario s j is defined by the configuration of the interaction between the agents in the space (pedestrians-vehicle and pedestrians-pedestrians).This is set by defining the start and goal point of each agent in the space.
In each test scenario, a set of variable test parameters should be identified.This set of parameters P = {p m } 1≤m≤M represents the different testing conditions that can occur in the identified use cases of the system.This can include the space density or the speed and acceleration limits for example.Each of these parameters is assigned a range of possible values F m identified based on experience and prior knowledge: Afterwards a discrete set of test values is selected for each parameter within its range of possible values.Let L m be the number of test values selected for the parameter p m , then the set of test parameters is written as: For example, if the parameter of interest is the crowd density in the space, the possible range of values can be assigned based on the observations in highly dense crowds, which can reach a limit of 9 pedestrian/m 2 in some dense gatherings [16].The discrete set of test values can be chosen closer to the range of the expected normal crowd density ([0, 0.5] pedestrian/m 2 ) [17].The set of test configurations C is the union of all possible combinations of the test parameters values, which will have a total of N C configurations: Finally, each test simulation with a parameter configuration c i ∈ C is repeated for i times to ensure the convergence of the performance.Therefore, the total number of simulations required to validate the performance in the context of the designed tests is equal to:

A. Motion Safety
The main indices used to validate the motion safety around pedestrians are derived from analyzing the collisions occurring in the simulations.A collision is detected in the simulation when an overlap occurs between the footprints of both the pedestrian and the vehicle.The footprint of a pedestrian in the 2D plane is considered circular with a radius R p whereas the footprint of the rectangular AV is approximated with the outer Löwner-John ellipse of the rectangle [18].
The Collision Rate is an important factor in estimating the safety of the navigation policy.A near-zero collision rate is required to validate the navigation.The collision rate in a set of simulations is calculated as: Number Of Simulations (5) However, when looking at the collision rate in a simulated world it is important to evaluate if this collision could actually occur in the real world.A number of non-cooperative purposeful pedestrian collisions can be detected in the simulator that resulted from a behavior on the pedestrian side and did not result from an error on the vehicle's side.Therefore, the rate of such collisions does not evaluate the vehicle's navigation scheme.An example is shown in Figure 1a where the vehicle is very slow and a pedestrian walks into the side of the vehicle.This case is detected as a collision in the simulator whereas in real-life this is highly unlikely to happen.Such a situation would mean, for example, that a distracted person bumped into a slow vehicle which is not considered as an error in the navigation algorithm.An example of a Realistic Collision is shown in Figure 1b.To assess if a collision is realistic or not, a Velocity Obstacle based test is performed.The Velocity Obstacle (VO) is a well-known method in robotics used for motion planning.The method calculates the set of robot velocities that can lead to a collision with another moving obstacle, or what is referred to as the collision cone (Figure 2b).When planning a trajectory, the velocity command is selected outside the collision cone for obstacle avoidance [19].The idea of the VO-based Collision Analysis is to examine the AV's velocity commands in a time period preceding the collision with a pedestrian.If the AV's velocity belongs to the pedestrian's collision cone, then the collision is considered "Realistic".Otherwise, the collision is considered "Non-cooperative Purposeful Pedestrian Collision" as the AV is moving away from the pedestrian and the pedestrian still collided with the AV.One main limitation of the VO method is that it assumes a circular shape in the 2D plane for both of the colliding objects.However, assuming a circular footprint for the rectangular AV would result in large errors.Therefore, the footprint of the vehicle is approximated with a minimum number of enclosing circles (Figure 2a) [20].
Algorithm 1 shows how the approximated footprint C f p of the AV with a pose (x o (t), y o (t), θ V (t)) is obtained in the local frame of the AV then in the global frame using the transformation matrix R G L : Once the set of footprint circles is obtained, the VO-based analysis is applied to each circle.The collision is realistic if any of the circles yields a positive result (Algorithm 2).

B. Trajectory Quality
Let T j = {(x j , y j , t j ) : j ∈ [0, .., M ]} be the overall trajectory of the AV during the simulation.The quality of the Input: AV's pose: 2 W ) : 0 ≤ j ≤ i} ▷ Transform to global frame AV's trajectory refers to its cost (smoothness), its efficiency, and to the comfort of the AV's passengers along this trajectory.

1) Cost:
The cost of the trajectory is estimated using two metrics: the Path Cost which is based on the maneuvers along a trajectory, and the Dynamic Cost which is a speed-based trajectory cost estimate.
The path cost measures the smoothness of the path by evaluating the average tangent of the path during the vehicle's maneuvers.This measure is defined using the formulas in [21], where we used a tangent variation cost rather than the angle variation cost defined in [21].On the other hand, the dynamic cost measures the average instantaneous variation of the speed from the preferred driving speed v pref .This preferred speed is equal to the driving speed in an obstacle-free space, which corresponds to the maximum allowed speed in a shared space v pref = v max .The dynamic cost is defined based on the energy consumption impact of this speed change along the trajectory, similar to the cost adapted in [22].
end while end for return 0 evaluated as: v(j) = ẋ2 (j) + ẏ2 (j) : When evaluated in the path's Frenet frame fixed at the AV's starting point, E P ∈ R + increases when a path contains more frequent and sharp maneuvers (E P = 0 for a straight path).Figure 3 shows some examples for the path cost corresponding to a U-turn path with a Quadratic function, and a lane change path with a Quintic function [23].The limit accepted value for E P is selected based on the working scenario.For example, in the case of uni-directional road driving where the vehicle is not supposed to do any 180 o turns, the limit value can be selected around 0.5 to ensure a low-cost lane change.On the other hand, this limit is set higher in scenarios where the vehicle is supposed to go on roundabouts or make u-turns.The dynamic cost E T ∈ [0, 1] increases when the trajectory consists of large instantaneous velocity variations and E T = 0 for an optimal constant maximum velocity trajectory (v = v pref ).Therefore, smaller values of the dynamic cost E T → 0 is desired from a trajectory quality point of view.However, it is unlikely to operate with the maximum velocity while interacting with the pedestrians.
2) Efficiency: The efficiency of the trajectory can be expressed in terms of distance or time [22].Firstly, the Relative Traveled Distance (C L ), also called sinuosity, measures the length of the traveled path relative to the shortest distance path Secondly, the Relative Traveled Time (T T G ) measures the traveled time relative to the time to reach the goal at the average speed: In efficient trajectories, the length of the traveled distance is close to the shortest path distance.This corresponds to the optimal path between the start point and the goal, in an obstacle free case C L → 1.Moreover, more efficient trajectories do not introduce large time delays, i.e.T T G ≤ 1.
3) Passengers' Comfort: The AV's trajectory can further entail information on the comfort of the AV passengers, which is an important part of the trajectory quality evaluation.A recent case study shows that passengers comfort is mainly related to their experience of the centripetal force [14].Passengers experienced more discomfort when passing small radius curves with high speed.Therefore, the average Centripetal Acceleration is used as a passengers' comfort measure: where κ(i) is the curvature of the path at point i.The results of the study in [14] recommends different a C values depending on the driving speed range.For the case of shared space (speed ≤ 20 km/h), a C ≤ 1 m/s 2 is recommended for a very comfortable drive and a C ≤ 1.75 m/s 2 for an acceptable comfort level.Finally, similar to measuring the passenger comfort by the centripetal acceleration, the longitudinal acceleration and even the longitudinal jerk can have a role in the passenger comfort as well.Which is a point worth inspecting in future studies to examine the dependency of the comfort index on both the longitudinal and the lateral dynamics.

C. Pedestrian Comfort
The first metric used to evaluate the pedestrian comfort during the interaction with the AV is the Discomfort Index Īucf suggested in [15].This index reflects the average degree and frequency of the linear velocity change a pedestrian experienced during the navigation.However, this index only considers variations in walking speed, and not variations in direction.The evaluation of pedestrian comfort was therefore completed with an additional Directional Discomfort Index Īucf θ .In a simulation defined at times {t j : j ∈ [0, .., M ]}, the two discomfort indices are evaluated as: with: where hi and li are the mean square velocity and the mean square orientation of agent i respectively.ȳi and zi are the average of the squares of the deviations from the mean speed and the average of the squares of the deviations from the mean orientation of agent i respectively.Finally, ḡi and fi are the average velocity and average orientation of agent i over the simulation period.When Īucf or Īucf θ is equal to 0%, this indicates that the pedestrian walks at his average speed or orientation throughout the simulation, which is very comfortable.On the contrary, higher values of one of these indices indicate higher level of discomfort.
The average Īucf and Īucf θ are computed for pedestrians who interacted with the AV and for pedestrians who did not interact with the AV.A pedestrian has interacted with the AV if the AV was perceived by the pedestrian during the simulation, i.e. the AV was within the perception zone of 220°up to 10 metres or 360°up to 3.3 metres around the pedestrian [24].This distinction helps to overcome differences in pedestrian discomfort due to the context (e.g.crowd density) and to identify discomfort specifically related the interactions with the AV.In more comfortable trajectories, the AV does not cause any change in pedestrians' walking speed or direction, i.e. the part of discomfort related to the AV is close to 0.

A. The proactive navigation system
The proactive navigation system under study is a local navigation system that works in combination with a higher level path planner by locally modifying the global path based on the perceived state of the dynamic space and the local interactions with the nearby pedestrians.The system relies on the concept of cooperation between the vehicle and the pedestrians.The concept is based on observed human behavior where individuals concurrently assist other individuals or robots and comply with their trajectories when navigating a shared space [25].Pedestrians can adopt similar cooperative behaviors around vehicles to ensure safety or to facilitate the movement of this bigger sized robot.The proactive navigation system starts by quantitatively estimating the pedestrian's cooperation during an interaction with the vehicle [26].In the proposed pedestrian behavioral model, the cooperation of a pedestrian with the vehicle is defined as the tendency of the pedestrian to modify and adapt its optimal trajectory (both direction and speed) to facilitate the movement of the vehicle in the space.Based on this cooperation estimation the system drives the vehicle proactively by invoking more pedestrian cooperation while ensuring the safety of the surrounding agents and maintaining a socially compliant vehicle trajectory.
Pedestrian cooperation is modeled with a time-varying parameter called the cooperation factor CF (t) ∈ [0, 1] (CF = 1 being the most cooperative).This factor varies for each agent based on the behavior of the vehicle and other surrounding pedestrians, as well as the agents own goal and state.This cooperation estimation is then used to predict a short term future trajectory for each pedestrian in the space, based on the model proposed in [26].The outputs of the behavioral model are used to locally modify a global path provided by an A* global planner based on ROS navigation stack as shown in Figure 4.
Firstly, the longitudinal velocity is controlled using the proactive controller proposed in [27].This proactive controller computes the longitudinal velocity using a cost function to maximize the pedestrians safety and cooperation factors.Secondly, the proactive dynamic channel method proposed in [28] is used to obtain the local steering control.The method finds natural steering in the unstructured and open shared space by imposing an imagined structure of the space and dividing it into a set of navigation channels.This new imagined structure resembles a road where each channel is similar to a lane.The optimal navigation channel is selected based on a fuzzy logic model and the transition between channels is performed similarly to a lane change on a highway, which ensures that a natural driving pattern is produced.The two parts of the proactive local navigation system overcome the shortcomings of reactive controllers and allow the AV to navigate pedestrian populated environments while avoiding the freezing robot problem in dense spaces.

B. SPACiSS Simulator
The SPACiSS simulator [29] is based on the pedestrian simulator Pedsim ros [30] that has been adapted in order to simulate pedestrians in shared spaces with an AV.The pedestrian model in SPACiSS is built on the Social Force Model [31], [32].This model uses physical forces to describe the internal motivations of a pedestrian to move towards its destination and to avoid other pedestrians and obstacles, and the sum of the forces drives the pedestrian motion in the simulation.
The simulator integrates social models of pedestrian behavior in shared spaces [33].The visual perception and Fig. 4: Overview of the proactive navigation system and the simulator under ROS attention of pedestrians are modelled, as well as the concept of personal space.It integrates social groups of pedestrians, such as friends, couples, coworkers and families [34], as well as interactions with a car in shared spaces.In shared spaces, pedestrian-AV interactions are diverse.In a front or rear interaction with a car, pedestrians slow down and deviate from their trajectory to avoid the car's path, while in a lateral interaction, pedestrians may accelerate to cross in front of the AV, slow down or stop to let it pass, or hesitate to cross and step back.Pedestrians in social groups avoid the car together, unless a collision with the AV is imminent.Pedestrian-vehicle interactions are modelled in SPACiSS by combining the SFM with a newly defined decision model that integrates these empirical observations [24].In this model, the pedestrian's decision depends on the time to conflict with the AV, on their interaction angle and on their expected crossing order at the conflict point.Depending on these parameters, simulated pedestrians decide to run, stop, step back or turn.The pedestrian's decision thus depends on his individual properties and on the vehicle's properties (position, speed, orientation and size).
The pedestrian model parameters have been calibrated on the CITR dataset [35] that contains 16 scenarios of realworld pedestrians-car interaction.The scenarios have been reproduced in simulation and the pedestrian model parameters have been calibrated using a 3-block cross-validation and an optimization algorithm that minimize the distance between simulated and real trajectories [24].The model has then been validated by comparing the simulated trajectories with ground truth trajectories from several datasets [24].As the simulated pedestrians have validated behaviors, we assume that the simulator is suitable for preliminary tests of navigation systems.
A Renault Zoé car was used in the simulator using a realistic car chassis and footprint along with the kinematic bicycle model.This allows integrating an external navigation system under ROS to control the car, as shown in Figure 4.The AV model perceives in all directions, i.e. 360 degrees, for a distance of 10 meters.The simulator provides pedestrians' information in the AV perception zone as point clouds, as well as vehicle odometry information, to simulate the AV sensors.As a result, the navigation system can use the simulated pedestrian information in its local planning, and the simulated pedestrians dynamically react to the AV movements.
In order to test the navigation system, SPACiSS can be used to build realistic shared space scenes and simulate a large variety of pedestrian-vehicle interactions.The simulation model is stochastic, so each scenario can be simulated multiple times, to assess the robustness of the navigation system in a given environment.To perform such tests efficiently, the simulator is designed so that several simulations can be executed in parallel.Finally, all simulation data are recorded and used for real-time visualization of the scene.Once the simulation is complete, agents' trajectories can be analyzed to evaluate the performance of the navigation system.

A. Designed Test Cases
To test and validate the proactive navigation around pedestrians, we selected one test environment E = {e 1 } (K = 1) that represents a shared space with no specifically assigned pedestrian or vehicle areas.This means that the space does not contain any roads, sidewalks, pedestrian crossings, traffic signals, etc.The space is bounded by two side barriers which can represent a parking lot or a shared business area like the shared space in Exhibition Road in London for example.
The navigation is tested in environment e 1 in J = 7 interaction scenarios: Frontal Crossing (F), Back Crossing (B), Frontal-Back Crossing (F-B), Lateral Crossing (Lat), Bi-Lateral Crossing (Lat-Bi), 45 o Uni-Directional Crossing (45 o ), 45 o Bi-Directional Crossing (45 o -Bi).The AV's goal in all scenarios is to cross the shared space safely and efficiently, while ensuring pedestrians' and passengers' comfort.Examples of four interaction scenarios are shown in Figure 5.The variable test parameter set includes the space density and the space sparsity (i.e. the distribution of the pedestrians in the space).The density of the space D is measured as the average number of pedestrians per m 2 .The space sparsity is estimated with a Gini Index that measures the distribution of pedestrians among the interaction space partitions [36].
Let a simulation be defined at times {t j : j ∈ [0, .., M ]}, A the area of the shared space in m 2 and N (j) be the number of pedestrians in the space at time t j .Then, the average pedestrian density and sparsity are measured as follows: where the interaction space is first divided into a grid of N (j) equal cells.Then, N i is the number of pedestrians in a cell i of the grid.For example, if the distribution of the pedestrians in the space is homogeneous, then there is one pedestrian in each of the equal grid cells, yielding GI = 0%.The rest of the parameters that are required to define the scenario are set to a constant value.The parameter values and the range of values for the variable test parameters are as follows:

B. Motion Safety
The collision rate and the analysis of the plausibility of each collision is performed in the test simulations set.The realistic collisions caused by the navigation system were analyzed.This was done after discarding the non-cooperative purposeful pedestrian collisions detected in the test simulations set.Table I shows the detected collision rate in each interaction scenario.A higher rate of realistic collisions was detected in the interactions including back crossing with pedestrians (B and F-B).This can be a result of not accounting for the pedestrian perception range in the navigation system [27].The behavior of the pedestrians in a back crossing scenario was assumed to be similar to their behavior in frontal crossing.A higher collision rate was also detected in the 45 o crossing scenarios (both uni-directional and bi-directional).This can be a limitation in the pedestrian behavior prediction model used in the system and which did not include such an interaction scenario [26].Overall, in the interaction scenarios an average of 0.2 realistic collisions per simulation was detected.
As a result of the motion safety test, the system requires further development to avoid the percentage of detected collisions.It is worth noting that the lowest collision rate was detected in the case of frontal and lateral crossing, which are the scenarios used to establish the pedestrian behavioral model used in the system under evaluation [26].Therefore, the performance might be significantly improved by adapting the pedestrian death cooperation model to account for all the different interaction scenarios.Moreover, the previous system might be implemented in combination with an emergency braking system to take over in critical situations.

C. Trajectory Quality and Efficiency
The vehicle trajectory in each simulation was qualified by analyzing its path and dynamic costs, the efficiency in terms of the traveled distance and the traveled time, and the passengers comfort using the centripetal acceleration.Table II shows the corresponding performance metric values to the set of testing simulations across all scenarios, density and sparsity levels.Firstly, the navigation system resulted in low cost both in terms of the manoeuvring (path cost) and the linear velocity.The values of the dynamic cost E T are all within the desired low limit to maintain a smooth linear velocity variation.Although the maximum value of the path cost E P is a little higher than the desired limit, most of the values fall within a very low acceptable cost range.Secondly, the resulting trajectories are efficient in terms of the traveled distance.We can notice that C L has values close to 1, which means that the vehicle did not deviate very far from its optimal path.For the efficiency in terms of the traveled time, we can see that most of the test simulations did not have large time delays.Only a small portion had larger time delays with T T G reaching over double the max desired value.Finally, the values of the centripetal acceleration of the vehicle along its traveled trajectory was overall within the desired range to maintain a comfortable ride for any potential passengers.

D. Pedestrian Comfort and Social Compliance
Figure 7 shows the average discomfort index Īucf for pedestrians in each scenario.Mean values and standard deviations are presented for pedestrians who interacted and for those who did not interact with the AV.The pedestrians who did not interact with the AV have an average Īucf ∈ [1.9; 6.1]% depending on the scenario, as shown in blue in Figure 7. Pedestrians who did interact with the AV have an average Īucf ∈ [1.6; 7.6]% depending on the scenario.The average discomfort in case of interaction with the AV is very close to the average discomfort without interaction with the AV.
Figure 8 shows the average directional discomfort index Īucf θ for pedestrians in each scenario, for all parameter combinations.Pedestrians who did not interact with the AV have an average Īucf θ ∈ [5.4 ; 89.6]% depending on the scenario, while pedestrians who interacted with the AV have an Īucf θ ∈ [8.6 ; 86.3]%.Again, the average discomfort in case of interaction with the AV is very close to the average discomfort without interaction with the AV in all scenarios.With both metrics, pedestrian comfort varies according to the interaction scenario.In frontal and back interactions, pedestrians have a low Īucf , while in 45°and lateral interactions, the average Īucf is higher.The Īucf index only considers pedestrians' speed adaptations and not direction adaptations.Pedestrians mainly adapt their direction in frontal and back interactions with vehicles, and their speed in lateral interactions.A higher Īucf in lateral interactions is therefore expected.Accordingly, in frontal, back and 45°interactions, pedestrians have a higher Īucf θ than in lateral interactions.Lateral interactions are more comfortable for pedestrians in terms of variations in their walking direction as these interactions require little directional adaptation from the pedestrian.These results confirm the importance of using both metrics to properly assess pedestrian comfort in various scenarios.
The statistical results for the discomfort related to the AV (i.e. the difference of discomfort between interacting pedestrians and non-interacting pedestrians) are shown in Table III.
Note that in some scenarios, especially with Īucf θ , pedestrians who interacted with the AV have even slightly lower discomfort than those who did not interact.This is due to the fact that sometimes the pedestrians stayed interacting with the AV for a long time, e.g.waiting for the AV to pass.They therefore had less discomfort due to interactions with other pedestrians than those who did not interact with the AV.Overall, the extra discomfort caused by the AV is very low, with values close to 0. The AV trajectory is then considered as comfortable for pedestrians.

VII. CONCLUSION
AV navigation in shared spaces introduces close interactions with pedestrians.Therefore, it is essential to test and evaluate the navigation system using simulation before allowing the AV to operate around pedestrians.This leads to an imperative need for an evaluation methodology.In this paper we formalized the evaluation process for an AV navigation system in a simulated shared space.The process starts with the design of the test set to include different targeted scenarios and working conditions.After running a sufficient number of simulated tests, the evaluation of the performance is done based on the spatio-temporal information for the recorded pedestrians and AV trajectories.We divided performance evaluation into three main categories: the AV's motion safety, the quality of the generated AV's trajectory and the comfort of the pedestrians around the AV during an interaction.A set of adapted metrics along with the evaluation criteria was introduced for each category.
Firstly, evaluating the motion safety using the detected collision rate allowed us to identify the challenging scenarios for the navigation system.Secondly, the AV's trajectory was evaluated using the suggested metrics based on the traveled distance, traveled time and the generated path and trajectory costs.While it is convenient to use the trajectory information to evaluate its quality, it is less straightforward to evaluate the comfort of the pedestrians or the passengers in simulation.We suggested evaluating the comfort by using the concept of interacting and non-interacting pedestrians with the AV.The discomfort levels for each type of pedestrians is evaluated.Then, the system was evaluated as comfortable if the interacting and not-interacting pedestrians had similar average levels of discomfort during the simulation.Meaning that the navigation system maintained the comfort of the pedestrians while interacting with them.The passengers' comfort, on the other hand, was evaluated based on the centripetal acceleration as part of the trajectory quality evaluation.
The suggested evaluation process was used to study the performance of a pre-designed proactive navigation system using a crowd simulator under ROS.We found that the proactive system produced good-quality trajectories that maintained an acceptable level of comfort for both pedestrians and passengers.The system's safety can be improved by including more diverse interaction scenarios in the pedestrian behavioral model used in the navigation framework, or by integrating an emergency braking system in the framework.
Furthermore, the proposed metrics are relevant for the evaluation of a navigation system in normal scenes.However, the simulator and evaluation metrics would need to be fitted if the case study is a mass-panic, emergency evacuation, or stampede situation.In such scenes, pedestrians may have very different behaviour (trying to escape, acting irrationally, ignoring their environment, or being unable to move) and the crowd may reach a very high density.
Moreover, the work was focused on evaluating the performance of the AV around pedestrians and without the presence of other vehicles on the scene.In future works, the evaluation method can be generalized to a multi-vehicle scene.This requires a re-calibration of the metrics and a re-formulation to isolate the effect of the ego AV from the effect of other vehicles on pedestrians.
Finally, future works can also consider using the evaluation method for different simulated scenarios using multiple available simulators (such as [37] and [38]).The results of the evaluation method in simulation can also be compared with the results of real-life scenarios evaluation.

Fig. 1 :
Fig. 1: Two collisions detected at time t C and the state of the AV and the pedestrian in the previous t CH period: (a) Non-cooperative Purposeful Pedestrian Collision, (b) Realistic Collision

Fig. 2 :
Fig. 2: (a) Circular approximation of the AV footprint (Algorithm 1), (b) VO method applied to one part of the AV footprint with radius R o , and a pedestrian with circular footprint of a center P and radius R P

Fig. 3 :
Fig. 3: Examples of path cost for U-turn and lane change

•
Constants: AV max speed 5.5 m/s, Pedestrian max speed 6.5 m/s, • Variables: Pedestrian density D ∈ [0.003, 0.56] ped/m 2 , Pedestrian distribution (Sparsity) GI ∈ [6, 34]%.Every simulation is run 20 times with 8 different values for the pair (D, GI) resulting in 2240 test simulations.Figure6shows examples of the resulting trajectories in the XY-Plane for the pedestrians and the AV in some of the testing scenarios.

TABLE I :
Collision rate values across the testing scenarios

TABLE II :
Trajectory quality statistical results

TABLE III :
Pedestrian comfort statistical results