EWSN 2020 Dependability Competition - Call for Competitors

Following the success of the past four editions (2016, 2017, 2018, and 2019), the International Conference on Embedded Wireless Systems and Networks will host also this year a dependability competition to benchmark the performance of low-power wireless communication protocols in harsh RF environments.

Background

Over the last decade, a large number of low-power wireless communication protocols has been proposed by both academia and industry. Their performance, however, has rarely been benchmarked under the exact same settings. Furthermore, little is known about the dependability of low-power wireless protocols in the presence of RF interference, which is known to significantly increase packet loss, end-to-end latencies, and energy consumption. Since 2016, the EWSN dependability competition aims to fill this gap by quantitatively comparing the dependability of state-of-the-art low-power wireless protocols under the same settings and in harsh RF environments.

Scenario and Categories

This year’s competition scenario will emulate the operation of a wireless sensor network where several co-existing Wi-Fi devices are crowding the RF spectrum. In particular, the messages generated by several source nodes need to be collected at a single destination (sink) in a reliable, timely, and energy-efficient way despite the presence of RF interference.

Data transmissions. Each source node is connected via I2C to an EEPROM that contains messages (of variable length) to be transmitted to a given destination. The benchmarking infrastructure will generate messages in the EEPROM of each source node asynchronously and signal their availability by toggling a pre-defined GPIO pin. These messages should be forwarded to the intended destination as efficiently as possible within a maximum per-message delay bound ∂, i.e., the end-to-end delay of every message from its generation to its reception at the sink should be lower than ∂. The latter is fixed and applies to all messages exchanged during a test run. Contestants can expect the values of ∂ to be set in the order of hundreds of milliseconds / a few seconds. As soon as the destination node receives a new message, it needs to write the corresponding data to the EEPROM and raise a pre-defined GPIO pin accordingly. The benchmarking infrastructure will then verify the correctness of the information written in the EEPROM and use the GPIO rising edge time-stamp to compute the end-to-end latency. If a message has been received with an end-to-end delay greater than ∂, it is considered to be lost.

Testbed infrastructure. As in the previous editions, the D-Cube testbed infrastructure will be used to benchmark the competing solutions. D-Cube allows to unobtrusively profile in hardware the energy consumption of each node in the network and to accurately measure the reliability and end-to-end latency of individual transmissions. Furthermore, thanks to its binary patching capabilities, D-Cube allows to benchmark the competing solutions using different input parameters, such as the address of source and destination nodes and the length of the packets being transmitted. These input parameters will not change during a test run and will be directly injected by D-Cube into the firmware under test by applying patches to the provided ihex binary, as explained in this document. This allows to make sure that results are not setup-specific. An example of how to integrate the necessary code for the binary patching will be provided beforehand.

Type of nodes. In contrast to the previous editions, this year’s competition will make use of nRF52840-DK wireless nodes deployed in an office building over several floors in an area of approximately 1440 m2. The nRF52840-DK board supports multiple technologies and radio modes (e.g., IEEE 802.15.4, BLE 1Mbps, BLE 2Mbps, etc.): contestants are free to use any of the supported technologies and modes without any limitation.

RF interference. Competing solutions will be tested both in the absence and in the presence of RF interference. The latter will be generated across the whole 2.4 GHz ISM band in a repeatable fashion by running JamLab-NG across the testbed infrastructure using D-Cube's observer nodes. The generated RF interference will resemble the patterns of common Wi-Fi appliances and will vary over time. The contestants cannot assume that some of the frequencies in the ISM band will be constantly interference-free.

Categories. Based on this scenario, two different categories are planned for this year’s edition:

Test runs using different configurations (i.e., a different set of input parameters) will be performed for each category. The raw sensor values will be generated either periodically or aperiodically during a test run. The period (or the lower/upper bound for aperiodic traffic), the delay bound ∂, the length of the packets to be transmitted, as well as the identity of the source and destination nodes will be injected to the firmware under test using D-Cube’s binary pathing features, and will apply for the whole duration of a test run. Additional information can be found in the “Frequently Asked Questions” section. More details will be provided as part of the logistics information before the beginning of the preparation phase.

Binary firmware. For each category, contestants are asked to provide a single binary in hex format (Intel) for all nodes. This ihex binary will be uploaded to all nodes in the network using the onboard JLink-OB debug adapter. The provided binary firmware should support different input parameters as described previously.

Evaluation Procedure

The competing solutions will be evaluated based on (i) the reliability of communications, i.e., the number of messages correctly reported to the destination node within a specified end-to-end delay ∂, and (ii) the energy-efficiency of the solution. The latter will be measured in hardware as the average energy consumed by each node in the network using D-Cube's observer nodes. Contestants can expect that the number of configurations using periodic and aperiodic traffic will be the same during the evaluation, i.e., periodic and aperiodic traffic are weighted equally.

Winner selection. The team that performs best across these evaluation metrics on the entire set of input parameters will be selected as winner in each category. Note that relative differences between teams will be considered and that achieving a reliable communication will be given more importance than achieving a high energy-efficiency. Competing teams can expect honorable mentions in case their solution performs exceptionally well for a specific set of input parameters.

Format

The EWSN 2020 dependability competition will take place remotely and the final results will be announced during the main EWSN conference in Lyon.

Entry abstract. Contestants willing to participate to the competition must submit a short entry abstract describing their solution by the competition entry deadline. The entry abstracts will be used to pre-select the teams participating to the competition.

Remote preparation phase. Accepted contestants that have successfully registered to the conference will obtain the credentials to remotely access the competition infrastructure. During the official preparation phase, all competing teams will have the possibility to test and optimize their solution for this year's competition scenario. In particular, contestants will be able to program all nodes in the competition infrastructure and download their serial output, as well as the evaluation results. The latter will be published on a leaderboard that will be visible to all other competing teams.

Final evaluation phase. At the end of the preparation phase, each team needs to provide one single ihex binary file for each competition category. During the evaluation phase, this binary file will be used to extensively benchmark the performance of the competing solution.

Competition abstracts. Each team will have the possibility to publish a competition abstract as part of the EWSN proceedings that will appear in the ACM Digital Library. In case the competing teams wish to publish a competition abstract, they should follow the same camera-ready deadline and formatting rules as the EWSN poster/demo abstracts. The competition abstracts are expected to include a description of the competing solution, as well as preliminary results and lessons learned from the preparation phase and will undergo a light review process.

Poster session. A poster session dedicated to the dependability competition will take place during the main EWSN conference. All competing teams must present their solution in the poster session and will have the possibility to engage in lively discussions with the other conference attendees.

Awards and winners' presentations. The winners of each competition's category will be awarded during a dedicated plenary session of the main conference. After receiving the award, the winning teams must hold a plenary short presentation about their solution, followed by a discussion session.

Eligibility

Both academia and industry submissions are encouraged. Any low-power wireless protocol is eligible and there is no restriction on the operating system used to program the sensor nodes, nor on the wireless technology used. There is also no requirement on the originality of the competing solutions, which can be based on well-known protocols or previously-published material. The same team can participate in multiple categories. The EWSN 2020 dependability competition will take place if at least six teams competing in the same category respond to this preliminary call for competitors.

Submission Guidelines

Contestants must submit a short entry abstract describing their approach by the competition entry deadline. This entry abstract should prominently include the name of all members of the team, as well as their affiliation and contact information. Furthermore, the entry abstract should clearly state the categories for which the solution is intended. For teams that intend to participate using a previously-published solution, a shorter entry abstract is sufficient, provided that it clearly embeds a reference to the prior work and that it points out which changes or extensions are planned compared to the published version. All submissions will be treated as confidential until the camera-ready deadline.

Formatting requirements. Entry abstracts must be written in English and can have a maximum length of 1 page including figures, tables, and references. Pages must have 8.5" x 11" (letter) two-column format, using 10-point type on 11-point leading, with a maximum text block of 7" wide x 9" deep with an inter-column spacing of 0.25". Authors may use the LaTeX templates provided here.

Submission. To submit your competition entry, please email your entry abstract to cboano(at)tugraz.at and markus.schuss(at)tugraz.at with the following subject line: "EWSN 2020 Competition - Submission".

Tentative Dates

Organizers

Contact

As for the 2019 edition, a blog and a Slack group will be made available after the notification date for a quicker communication between organizers and contestants. The competition’s testbed can be accessed at: https://iti-testbed.tugraz.at/. For further questions and clarifications about the competition scenario, please send an e-mail to cboano(at)tugraz.at, markus.schuss(at)tugraz.at, or nathalie.mitton(at)inria.fr.

Supporters

Frequently Asked Questions

Can input parameters change at runtime?
No, the input parameters remain the same for the whole duration of a run. It is foreseen that a single firmware should support different periods, end-to-end latencies ∂, or payload lengths: those values will be patched by the competition infrastructure during the uploading phase and will not change at runtime (i.e., they remain constant during a run). The same set of input parameters will be used for preparation and evaluation phase.

Is there any correlation between consecutive packets?
No: the data to be transmitted by source nodes is unique and random, i.e., there is no correlation or trend between consecutive sensor readings generated by the same source node.

Which frequencies are allowed?
The use of frequencies below 2400 and above 2485 MHz are strictly forbidden, whilst there is no limitation about the usage of frequencies between 2400 and 2485 MHz. Note that the frequency usage will be monitored during the evaluation phase and any detected violation will lead to a disqualification.

How can I get access to the competition’s testbed infrastructure?
In order to get access to the competition's testbed infrastructure, at least one member of each accepted team needs to register to the EWSN conference. After this step is completed, each team needs to accept the terms and conditions for the use of the competition's testbed facility (further information will be sent after the notification date). Once these steps are completed, the credential information to access the competition's testbed facility will be provided to all team members.