feat(esp_modem): Add mqtt example in AT-only mode #156

david-cermak · 2022-10-07T18:31:53Z

Similar to the pppos-client, but without PPP mode.
This uses standard mqtt client and a loopback service that forwards the TCP traffic from localhost to the modem and vice-versa. (next step is to create a dedicated tcp-transport layer for modem)

I (525) sock_dce: Got IP +IPADDR: x.x.x.x
I (525) sock_dce: Socket created
I (525) sock_dce: Socket bound, port 1883
I (525) system_api: Base MAC address is not set
I (535) system_api: read default base MAC address from EFUSE
I (535) modem_client: MQTT other event id: 7
I (965) sock_dce: Connected!
I (1015) modem_client: MQTT_EVENT_CONNECTED
I (1025) modem_client: sent subscribe successful, msg_id=17361
I (1095) modem_client: MQTT_EVENT_SUBSCRIBED, msg_id=17361
I (1095) modem_client: sent publish successful, msg_id=0
I (1155) modem_client: MQTT_EVENT_DATA
TOPIC=/topic/esp-pppos
DATA=esp32-pppos
E (121345) mqtt_client: No PING_RESP, disconnected
I (121345) modem_client: MQTT_EVENT_DISCONNECTED
E (122345) sock_dce: Failed to get idle
E (122345) modem_client: Loop exit.. retrying
I (126885) sock_dce: Got IP +IPADDR: y.y.y.y
I (127115) sock_dce: Connected!
I (136345) modem_client: MQTT other event id: 7
I (136415) modem_client: MQTT_EVENT_CONNECTED
I (136415) modem_client: sent subscribe successful, msg_id=1196
I (136485) modem_client: MQTT_EVENT_SUBSCRIBED, msg_id=1196
I (136495) modem_client: sent publish successful, msg_id=0
I (136575) modem_client: MQTT_EVENT_DATA
TOPIC=/topic/esp-pppos
DATA=esp32-pppos

jonathandreyer · 2022-10-10T22:07:30Z

Hi @david-cermak, very interesting PR.
I had planned to implement the same idea with the MQTT AT command (available for Quectel BG96 & SIMCom SIM7000 modules). If you need some help to implement and/or test this PR, maybe I can help you. Don't hesitate to ping me.

david-cermak · 2022-10-13T06:45:19Z

@jonathandreyer Thanks for the interest. Yes, I also like this idea of the localhost listener, since we're able to use standard IDF libraries (the same way as we do for WiFi/Ethernet) and even using the IDF TLS stack (with mbedTLS)!

The disadvantage is that these socket commands are (very) different for different devices. My plan is to look at other devices and try to add some abstraction layer. It would be helpful if you could test it then, once we add support for BG96.

Another point we'd like to improve is the composition of commands for specific devices, so we could add custom commands as well as include certain type of commands (like addons), e.g. for socket commands, GNSS commands or as you mentioned the MQTT commands. It's party addressed in this PR, but I wanted to make the modem factory more general, so we're able to build/create a DCE including commands from these groups (general V.25TER, specific control commands, network commands,...). Once this is supported, it would be very helpful if contributors implement these command groups for their devices and their use-cases.

jonathandreyer · 2022-10-16T16:23:10Z

@david-cermak Thanks for your reply. I fully agree about the abstraction and the customization of commands.
If I can add my two cents, another point will be parsing of URC’s message. Not all but some URC seems to me important, like when SMS received, change in networks, etc. In the previous example (pppos_client in esp-idf repository), I have tried to implement this idea, but it is not easy to detect when data are received by ESP32.

diplfranzhoepfinger · 2022-10-19T07:32:45Z

Hi, agree that URC is important.
especially when a pppos Session exit (crossing Border e.g. from Germany to Czech) to be informed about this.

david-cermak · 2022-10-21T05:05:18Z

This PR actually addresses these asynchronous messages as the notifications about the received messages come as URC. At this point only the Rx messages, but need to watch for socket close and others...

esp-protocols/components/esp_modem/examples/modem_tcp_client/main/sock_dce.cpp

Lines 276 to 285 in a299192

    
           void DCE::check_async_replies(std::string_view &response) const 
        
           { 
        
               ESP_LOGD(TAG, "response %.*s", static_cast<int>(response.size()), response.data()); 
        
               if (response.find("+CIPRXGET: 1") != std::string::npos) { 
        
                   uint64_t data_ready = 1; 
        
                   write(data_ready_fd, &data_ready, sizeof(data_ready)); 
        
                   ESP_LOGD(TAG, "Got data on modem!"); 
        
               } 
        
           }

diplfranzhoepfinger · 2022-11-24T08:28:40Z

i would suggest to have a more general URC Handling, independend of this PR.

i will look into it.

jonathandreyer · 2023-01-13T12:20:23Z

Hello @david-cermak, I have tried to use this WIP PR, but I wasn't able to build sources. Into the build log, it seems that a file is missing ("socket_commands.inc"). Is this possible or is it a problem on my side?

david-cermak · 2023-01-13T12:57:13Z

Hello @david-cermak, I have tried to use this WIP PR, but I wasn't able to build sources. Into the build log, it seems that a file is missing ("socket_commands.inc"). Is this possible or is it a problem on my side?

Oh, sorry, forgot to add this file into my initial commit. Have just added.

jonathandreyer · 2023-01-15T16:13:32Z

Hello @david-cermak, thanks for the reply and the fix. I have created a rebase branch (jonathandreyer/esp-protocols:feature/modem_example_at_mqtt) with few fix to be able to build it with CI and also be able to run it with v4.4.
I will test the feature in the next few days.

jonathandreyer · 2023-01-22T22:13:19Z

Hello @david-cermak, I have updated the branch with an initial for the BG96 modem. Don't hesitate if you have any comment?
With this modification, I was able to send messages to the default MQTT server and also another server with authentication (username & password).
In this version, there are two simplifications with the document, one for ack validation of data sent and another for validation for the length of data received.

In this initial commit, I didn't modify functions tcp_send() and tcp_recv(), because in the current version it is not used.

david-cermak · 2023-01-23T08:47:25Z

Hello @jonathandreyer

Thank you for your help with promoting this PR from just an idea to the real implementation, and supporting BG96 modem!

I think I'd clean this up and make it ready for review (here I would borrow your ci-fix commit). Then it should be easier for you to raise a new PR with the BG96 support.

david-cermak · 2023-01-24T08:07:12Z

Hello @jonathandreyer

Thank you for your help with promoting this PR from just an idea to the real implementation, and supporting BG96 modem!

I think I'd clean this up and make it ready for review (here I would borrow your ci-fix commit). Then it should be easier for you to raise a new PR with the BG96 support.

Note the last commit that support fragmented received data. This allows receiving/sending bigger chunks which are typically send during TLS handshake. (MQTT over TLS didn't work before this fix).

As for supporting other devices, it's quite simple to build an abstraction on these commands:

esp-protocols/components/esp_modem/examples/modem_tcp_client/main/sock_commands.cpp

Lines 36 to 40 in 588a73c

    
           command_result net_close(CommandableIf *t) 
        
           { 
        
               ESP_LOGV(TAG, "%s", __func__ ); 
        
               return dce_commands::generic_command(t, "AT+NETCLOSE\r", "+NETCLOSE:", "ERROR", 30000); 
        
           }

But, not so easy for the Tx/Rx state machine, such as:

esp-protocols/components/esp_modem/examples/modem_tcp_client/main/sock_dce.cpp

Lines 113 to 132 in 588a73c

    
           if (state == status::SENDING) { 
        
               if (memchr(data, '>', len) == NULL) { 
        
                   ESP_LOGE(TAG, "Missed >"); 
        
                   state = status::SENDING_FAILED; 
        
                   signal.set(IDLE); 
        
                   return; 
        
               } 
        
               auto written = dte->write(&buffer[0], data_to_send); 
        
               if (written != data_to_send) { 
        
                   ESP_LOGE(TAG, "written %d (%d)...", written, len); 
        
                   state = status::SENDING_FAILED; 
        
                   signal.set(IDLE); 
        
                   return; 
        
               } 
        
               data_to_send = 0; 
        
               uint8_t ctrl_z = '\x1A'; 
        
               dte->write(&ctrl_z, 1); 
        
               state = status::SENDING_1; 
        
               return; 
        
           } else if (state == status::RECEIVING || state == status::RECEIVING_1 ) {

Will think about it more, but if you have an idea or suggestion, please let me know

components/esp_modem/include/cxx_include/esp_modem_command_library.hpp

euripedesrocha · 2023-02-06T14:13:50Z

components/esp_modem/include/cxx_include/esp_modem_dce_factory.hpp

+    explicit Creator(std::shared_ptr<DTE> dte): dte(std::move(dte)), device(nullptr), netif(nullptr)
+    {
+    }
+
    ~Creator()


Unrelated: By declaring this destructor move operations are not declared by the compiler. Could we achieve this error LOG by other means?

The meaning of this error message is that some part, added to the factory was left behind, unused for constructing. There's no easy fix I know about. The only thing is to make the factory state-less, but that would need complete refactor.

components/esp_modem/include/cxx_include/esp_modem_dte.hpp

components/esp_modem/src/esp_modem_dte.cpp

euripedesrocha · 2023-02-07T09:05:50Z

components/esp_modem/examples/modem_tcp_client/main/sock_dce.cpp

+        char *recv_data = (char *)data;
+        if (data_to_recv == 0) {
+            const std::string_view head = "+CIPRXGET: 2,0,";
+            auto head_pos = (char *)std::search(data, data + len, head.begin(), head.end());


Have you considered making a string_view from data as an alternative to the usage of memchr and pointer manipulation?

I have, but since the replay contains not only string delimiters, but also and mainly data (and that could be very long data), I'd prefer using memchr and treating this as raw binary, rather than string_view.
As you can see, some portions of those data are parsed using string_view, but that doesn't universally work for the entire data we receive.

euripedesrocha · 2023-02-07T09:28:17Z

components/esp_modem/examples/modem_tcp_client/main/sock_commands.cpp

+
+using namespace esp_modem;
+
+command_result net_open(CommandableIf *t)


Could t have a better name?

Have renamed to term (this could be any class, that implements commands and as such, we call it "commandable", but there are in most cases just more or less abstracted terminals

components/esp_modem/examples/modem_tcp_client/main/sock_commands.cpp

components/esp_modem/examples/modem_tcp_client/main/CMakeLists.txt

components/esp_modem/examples/modem_console/main/my_module_dce.cpp

components/esp_modem/examples/modem_console/main/CMakeLists.txt

components/esp_modem/examples/modem_tcp_client/README.md

gabsuren · 2023-02-09T10:14:06Z

components/esp_modem/include/cxx_include/esp_modem_command_library.hpp

@@ -22,6 +22,14 @@ namespace dce_commands {
 * @{


Unrelated SPDX-FileCopyrightText: 2021-2022 -> SPDX-FileCopyrightText: 2021-2023

Not sure why the pre-commit hook for copyrights didn't trigger. Let me check.

components/esp_modem/examples/modem_tcp_client/main/modem_client.cpp

components/esp_modem/examples/modem_console/main/modem_console_main.cpp

david-cermak · 2023-03-16T12:54:52Z

Thanks for the review @euripedesrocha @gabsuren @espressif-abhikroy

MAJOR CHANGE: Enable DTE to redefine on_read() and write(cmd) directly

Similar to pppos-client, but without PPP mode. This uses standard mqtt client and a loopback service that forwards the TCP traffic from localhost to the modem and vice-versa. (next step is to create a dedicated tcp-transport layer for modem) Console example uses DCE commandable to demontrate how to handle URC

Plus rework a little to support implementation of multiple devices

This was referenced Nov 24, 2022

URC Reception DRAFT (IDFGH-8813) #179

Closed

Implement URC Handling. (IDFGH-8812) #180

Closed

david-cermak force-pushed the feature/modem_example_at_mqtt branch from 74d9b95 to 4d2bd34 Compare January 23, 2023 09:12

david-cermak marked this pull request as ready for review January 23, 2023 09:15

david-cermak requested review from espressif-abhikroy, euripedesrocha and gabsuren January 23, 2023 09:15

euripedesrocha reviewed Feb 7, 2023

View reviewed changes

gabsuren reviewed Feb 9, 2023

View reviewed changes

components/esp_modem/examples/modem_console/main/my_module_dce.cpp Show resolved Hide resolved

components/esp_modem/examples/modem_console/main/my_module_dce.cpp Outdated Show resolved Hide resolved

gabsuren reviewed Feb 9, 2023

View reviewed changes

components/esp_modem/examples/modem_console/main/CMakeLists.txt Show resolved Hide resolved

gabsuren reviewed Feb 9, 2023

View reviewed changes

components/esp_modem/examples/modem_tcp_client/README.md Outdated Show resolved Hide resolved

gabsuren reviewed Feb 9, 2023

View reviewed changes

david-cermak commented Feb 9, 2023

View reviewed changes

components/esp_modem/examples/modem_tcp_client/main/modem_client.cpp Outdated Show resolved Hide resolved

david-cermak commented Feb 9, 2023

View reviewed changes

components/esp_modem/examples/modem_tcp_client/main/modem_client.cpp Outdated Show resolved Hide resolved

david-cermak commented Feb 9, 2023

View reviewed changes

components/esp_modem/examples/modem_console/main/modem_console_main.cpp Outdated Show resolved Hide resolved

david-cermak force-pushed the feature/modem_example_at_mqtt branch 3 times, most recently from 9e3770b to 68d6bf1 Compare February 14, 2023 18:43

euripedesrocha approved these changes Mar 15, 2023

View reviewed changes

gabsuren approved these changes Mar 15, 2023

View reviewed changes

espressif-abhikroy approved these changes Mar 16, 2023

View reviewed changes

david-cermak force-pushed the feature/modem_example_at_mqtt branch from 68d6bf1 to 64f178c Compare March 16, 2023 12:53

david-cermak self-assigned this Mar 16, 2023

david-cermak added the modem label Mar 16, 2023

david-cermak and others added 5 commits March 16, 2023 19:39

feat(modem): DTE to support sending and receiving raw data

bf114d3

MAJOR CHANGE: Enable DTE to redefine on_read() and write(cmd) directly

fix(modem): AT-only example: initial adaptation to BG96

ceedcfc

Plus rework a little to support implementation of multiple devices

fix(modem): AT-only example: added into CI

97831c6

fix(modem): AT-only example: support MQTT over TLS on BG96

31d4323

david-cermak force-pushed the feature/modem_example_at_mqtt branch from 64f178c to 31d4323 Compare March 16, 2023 19:04

david-cermak merged commit e762ada into espressif:master Mar 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(esp_modem): Add mqtt example in AT-only mode #156

feat(esp_modem): Add mqtt example in AT-only mode #156

david-cermak commented Oct 7, 2022

jonathandreyer commented Oct 10, 2022

david-cermak commented Oct 13, 2022

jonathandreyer commented Oct 16, 2022

diplfranzhoepfinger commented Oct 19, 2022

david-cermak commented Oct 21, 2022

diplfranzhoepfinger commented Nov 24, 2022

jonathandreyer commented Jan 13, 2023

david-cermak commented Jan 13, 2023

jonathandreyer commented Jan 15, 2023 •

edited

Loading

jonathandreyer commented Jan 22, 2023

david-cermak commented Jan 23, 2023

david-cermak commented Jan 24, 2023

euripedesrocha Feb 6, 2023

david-cermak Feb 10, 2023

euripedesrocha Feb 7, 2023

david-cermak Feb 10, 2023

euripedesrocha Feb 7, 2023

david-cermak Feb 10, 2023

gabsuren Feb 9, 2023

david-cermak Feb 10, 2023

david-cermak commented Mar 16, 2023


		using namespace esp_modem;

		command_result net_open(CommandableIf *t)

feat(esp_modem): Add mqtt example in AT-only mode #156

feat(esp_modem): Add mqtt example in AT-only mode #156

Conversation

david-cermak commented Oct 7, 2022

jonathandreyer commented Oct 10, 2022

david-cermak commented Oct 13, 2022

jonathandreyer commented Oct 16, 2022

diplfranzhoepfinger commented Oct 19, 2022

david-cermak commented Oct 21, 2022

diplfranzhoepfinger commented Nov 24, 2022

jonathandreyer commented Jan 13, 2023

david-cermak commented Jan 13, 2023

jonathandreyer commented Jan 15, 2023 • edited Loading

jonathandreyer commented Jan 22, 2023

david-cermak commented Jan 23, 2023

david-cermak commented Jan 24, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david-cermak commented Mar 16, 2023

jonathandreyer commented Jan 15, 2023 •

edited

Loading