Heltec Wireless Stick v3 and LoRaWAN network – part 2

Posted on 2024/11/162024/11/23 by Nettigo Team

In the previous part we have created simple app sending some data to LoRaWAN. We have used OTAA (Over The Air Activation) and that means that while joining between node and gateway communication has to be bidirectional. Every time you power on device (and it is joining network) you can’t be on the maximum range for your node/gateway – device need to get response from gateway. And on the maximum range usual device can send packet which is being received by gateway, but end device has usually not enough sensitive antenna/radio to receive response.

This is a series post. Click to see the whole list:

Using Heltec Wireless Stick to send data to The Things Network (TTN)
Persisting data after OTAA and avoid unnecessary joins to TTN (this post)s
Mapping LoRaWan coverage

And more – TTN says that you, as developer should avoid unnecessary joins. TTN has own, good reasons for that so how we can achieve that?

OTAA vs ABP

Over the air activation means that node send request for join and TTN responds with generated device address, network and application session keys (so you see why bidirectional communication is required for join). ABP (activation by personalization) means that keys are generated in TTN console an then put on node (via code upload). In ABP mode keys are fixed and stored in node (usually in code you upload to the board).

To use OTAA join only once – after it succeeds just extract required keys and store in some persistent way. So we have arrived to ESP32 preferences. ESP32 has small part of its flash memory reserved to store some persistent information. Usually it is 16 kB, however this area is being used by ESP32 – WiFi related information is being stored there, so you can not count that you have so much space available.

How ESP32 preferences work?

Preferences is key/value storage wrapped into namespace. What does that mean?

#include <Preferences.h>
Preferences appConfig;

void setup(){
  appConfig.begin("lorawan");
}

You need to include Preferences.h to have access to needed functions. Then You create object of Preferences type. Before anything You write or read You configure namespace via begin("namespace"). By default namespace is opened in read/write mode, You can provide second argument true to have it read only mode.

In two different namespaces You can have two the same keys with different values stored. For example You can have two different presets/configs selected at boot depending on some conditions and Your app can behave differently.

Regular code upload does not overwrite values in preferences, so you can store some settings for different code when you switch back and forth between two projects on Your bench. Just an idea.

So, we have namespace open how to write data? Library has many different functions storing different type of data. Documentation for library has table with all available types. For example to store unsigned integer value: appConfig.putUInt("cycles-no",24); This will create key cycles-no in current namespace and store value 24. Function return size (in bytes) stored data. Remember to check that – if returned value is 0 then variable has not been stored!

To retrieve value store under that key: unsigned int cycles = appConfig.getUInt("cycels-no",10) 10 is value returned if key is not present in namespace (optional, default is 0). You have to use function matching type of stored data or you get corrupted result. You have stored UInt – use getUInt to retrieve. In case You can have different data type stored under given key You can use appConfig.getType("keyname") to check what type of data is stored.

With data types like int, float we have constant data size. What about strings and arrays of bytes? To store bytes Preferences provide putBytes function. Lets assume we have array uint8_t NetworkSessionKey[16]. To store it in prefs use appConfig.putBytes("ntwsk", NetworkSessionKey, 16). First is key used to store value (each key used in Preferences can have up to 15 characters), then array and finally size of array. Function will return number of stored bytes. Check that value, since otherwise Your code may not notice that saving has failed. You can use following code:

    if ( !appConfig.putBytes("ntwsk", NetworkSessionKey, 16)){
        appConfig.remove("ntwsk");
        return false;
    };

Remember that ! negates boolean values? Nonzero is treated as true, so nonzero value returned by putBytes will be converted by ! to false – code in if won’t be executed. On the other hand, failed putBytes will return 0. ! will convert to true and code will be executed.

I have assumed that if storing value have failed I need to remove old one if present, this seems to be suitable in most cases, but Your code can behave differently if needed.

Retrieve bytes using getBytes – to read array from previous example appConfig.getBytes("ntwsk", NetworkSessionKey, 16) – first You give key where data should be stored, then pointer to array/buffer where You want data to be stored and size of Your buffer. Function will return number of bytes stored in buffer or 0 in case of error.

One of reason for error is too small buffer – when stored bytes blob is to big to fit into created buffer. To get information of how many bytes are saved in prefs under given key use appConfig.getBytesLength("keyname").

OK, to complete topic there are few helper functions more. To check if key named keyname is present in namespace use appConfig.isKey("keyname"). To remove key and stored value use appConfig.remove("keyname"). You can also remove all keys in namespace via appConfig.clear(). Read more in Preferences API description.

And one more thing – clearing whole Preferences. As You probably noted if You use device with different namespaces and store a lot of data then You can run out of place in Preferences. Can happen on device used in lab to test many different projects. Over time Preferences can accumulate some old, unused data. How to get rid of it?

In such case You can clear and format that 16 kB of flash used by Preferences library. Run simple sketch:

#include <nvs_flash.h>

void setup()
{
    Serial.begin(115200);
    Serial.print("Clearing NVS flash...");
    nvs_flash_erase();      // erase the NVS partition and...
    nvs_flash_init();       // initialize the NVS partition.
    Serial.println("\n\nDone. Upload regular code.");
    while(true){delay(1);}
}
void loop(){
}

This code uses not Arduino library but calls ESP32 API directly and deletes all data in area used by prefs (it is called NVS flash). Run this code only once and upload “normal” code.

Other option is to erase whole flash using tool like esptool.py. On Ubuntu, when Wireless Stick is connected to USB0 command will look: esptool.py --port /dev/ttyUSB0 erase_flash. That clears all data on flash, including NVS part.

So, we know how to store data in Preferences, but what data?

As You should remember from previous part, AppEUI and nodeAppKey are common for all devices who can join to our TTN application. After successful join there are three parameters used to communicate between device and LoRaWAN TTN network. These are network session key, application session key and node device address. If You have tried to run some code doing ABP instead of OTAA those names should be familiar and that’s correct.

After successful join via OTAA Your node can connect to network in ABP mode (no join request) using keys and device address from OTAA join. So, we need to store those two keys and address in prefs. Keys are 16 byte arrays and address is 32 bit number. How to retrieve them? After successful join (after lorawan_has_joined_handler):

    lmh_getAppSkey(AppSessionKey);
    lmh_getNwSkey(NetworkSessionKey);

and device address is being returned by lmh_getDevAddr(). We will use this flow:

open prefs namespace
try to read device address, network session and application session keys
if read was successful – work in ABP mode using these values. Otherwise run OTAA join request.
in case OTAA join is successful – store keys for next boot

That way Your device will do OTAA each time until it succeeds, later will be in ABP mode.

Final code review

Ok, so we will talk about changes from code used in part 1.

Preferences – we use appConfig global variable to manage persistent storage. In setup we open prefs namespace, I have chosen lorawan name. Then we try to read keys from previous runs using get_stored_keys function:

static bool get_stored_keys(void){
    if (!appConfig.isKey("ntwsk") || !appConfig.isKey("appsk") || !appConfig.isKey("devaddr"))
        return false;
    if (!appConfig.getBytes("ntwsk", NetworkSessionKey, 16))
        return false;
    if (!appConfig.getBytes("appsk", AppSessionKey, 16))
        return false;
    nodeDevAddr = appConfig.getUInt("devaddr");
    return true;
}

First – it returns false if any required parameter is missing – we don’t even want to try sending data when keys are not complete… Next we try to read each stored data and if we have any problem with any session key then returns false – we need to try join again.

I have small problem with device address, since it is harder to get info if reading was successful. In case of problems with that value (for example storage in NVS has ended and we have session keys saved but not device address) getUInt will return 0 as default value. I have read a bit TTN forum and found no trace that device address can not be 0, since 0 is valid 32bit integer… So, there is some very low chance that code will read two keys and since device address was not saved it will have 0. To get proper communication with TTN all 3 values are required – device wont be able to send data.

I think that is so unlikely that we can ignore that – if Your node does use Preferences to store only few values and there is no other namespaces eating up space we are safe. In case You wonder how to mitigate that risk I can suggest two approaches – add fourth variable stored in preferences, saved after all 3 parameters were saved properly. And get_stored_keys would use that 4 variable. In case that variable did not save due to lack of place then our code will think it doesn’t have correct keys and it will try new join. As result there is a chance it will be sending data.

Second option it would be to access device address in NVS using ESP32 API directly, nvs_get_u32 returns error code if can not read value. Preferences library is from ESP32 Arduino core and does not expose that info. Read more in NVS flash API interface.

OK, let get back to setup:

 if (get_stored_keys()){
        OTAA_request = false;
        Serial.println("Keys restored from storage");
        lmh_setAppSKey(AppSessionKey);
        lmh_setNwkSKey(NetworkSessionKey);
        lmh_setDevAddr(nodeDevAddr);
        print_keys();
    }

We know here that session params were retrieved from Preferences. So, we use lmh_setXXXX functions to set session keys and device address. Now we are ready to work in ABP, therefore we set global flag OTAA_request to false. This flag (set to true by default) we use in setup when configuring library:

    err_code = lmh_init(&lora_callbacks, lora_param_init, OTAA_request, CLASS_A, LORAMAC_REGION_EU868);

So when no session data were restored from flash, then we will use OTAA mode otherwise ABP.

On first run after upload we don’t have any data stored in Preferences, so OTAA_request will be true and OTAA join procedure will be run. If we have JoinEUI, DevEUI and AppKey set correctly and we are in gateway range then after successful join lorawan_has_joined_handler will be called (like in code in part 1). So, in this function we add code to store data:

    lmh_getAppSkey(AppSessionKey);
    lmh_getNwSkey(NetworkSessionKey);
    print_keys();
    store_keys();

First we retrieve session keys and store them in variables, then try to store them in store_keys function:

tatic boolean store_keys() {
    if ( !appConfig.putBytes("ntwsk", NetworkSessionKey, 16)){
        appConfig.remove("ntwsk");
        return false;
    };
    if (!appConfig.putBytes("appsk", AppSessionKey, 16)) {
        appConfig.remove("appsk");
        return false;
    };
    if (appConfig.putUInt("devaddr",lmh_getDevAddr()) == 0) {
        appConfig.remove("devaddr");
        return false;
    }
    return true;
}

Next time our device boots it will use saved keys and no join request will be sent. It can be seen on TTN console – after You reset device, no join request.

OK, this is end for now. You can download code from here. In next part we will add GPS and will send location data. With TTN Mapper integration enabled we will be able to see real coverage for our device.

2 thoughts on “Heltec Wireless Stick v3 and LoRaWAN network – part 2”

Pingback: Heltec Wireless Stick Lite – sending data to LoRaWAN – part 1 – Starter Kit
Pingback: Heltec Wireless Stick v3 and LoRaWAN network – part 3 – Starter Kit

OTAA vs ABP

How ESP32 preferences work?

Final code review

2 thoughts on “Heltec Wireless Stick v3 and LoRaWAN network – part 2”

Leave a Reply Cancel reply