Failed to start turtlebot3_ros.vxe on TurtleBot3 burger

[sdoerr20]

Following the installation guide from the README I installed the ROS2 library on my TurtleBot3 burger robot based on an Raspberry Pi 3B plus. As the trace attached below shows, I used an external USB-Stick with a fat32 filesystem storing the ROS2 library and started the program timer_lambda.vxe which works fine.

Executing the program turtlebot3_ros.vxe in contrast fails, because the serial for the OpenCR Board is expected to be at /dev/ttyACM0, but the serial device for the OpenCR board is abstracted as /usb2ttyS/0 .

Is there an option to explicitly set the serial for the program turtlebot3_ros.vxe?

Trace recorded on the RPI3b plus with the VxWorks Image of the WindRiver RPI SDK:
-> cmd
[vxWorks *]# devs
drv refs name
1 [ 3] /
2 [ 3] /bd0:1 ==> /bd0a
5 [ 3] /bd0a
2 [ 3] /bin ==> /sd0a/sysroot/bin
2 [ 3] /boot ==> /sd0a/sysroot/boot
2 [ 3] /dev ==> /
2 [ 3] /etc ==> /sd0a/sysroot/etc
10 [ 3] /fifos
13 [ 3] /input/event
2 [ 3] /lib ==> /sd0a/sysroot/lib
0 [ 3] /null
12 [ 3] /random
8 [ 3] /stdio_pty_0xffff8000004e2fc0.M
7 [ 3] /stdio_pty_0xffff8000004e2fc0.S
6 [ 3] /tmp
3 [ 3] /ttyS0
2 [ 3] /tyCo/0 ==> /ttyS0
12 [ 3] /urandom
16 [ 3] /usb2ttyS/0
16 [ 3] /usb2ttyS/1
2 [ 3] /usr ==> /sd0a/sysroot/usr
11 [ 3] /zero
15 [ 3] host:
[vxWorks *]# set env LD_LIBRARY_PATH="/bd0a/lib"
[vxWorks *]# cd /bd0a/llvm/bin/
[vxWorks *]# rtp exec -u 0x20000 timer_lambda.vxe
Launching process 'timer_lambda.vxe' ...
Process 'timer_lambda.vxe' (process Id = 0xffff800000309010) launched.
[INFO] [minimal_timer]: Hello, world!
[INFO] [minimal_timer]: Hello, world!
[INFO] [minimal_timer]: Hello, world!
[INFO] [minimal_timer]: Hello, world!

[vxWorks *]# rtp exec -u 0x020000 turtlebot3_ros.vxe
Launching process 'turtlebot3_ros.vxe' ...
Process 'turtlebot3_ros.vxe' (process Id = 0xffff800000336b00) launched.
[INFO] [turtlebot3_node]: Init TurtleBot3 Node Main
[INFO] [turtlebot3_node]: Init DynamixelSDKWrapper
[PortHandlerVxworks::SetupPort] Error opening serial port!
[ERROR] [DynamixelSDKWrapper]: Failed to open the port(/dev/ttyACM0)!
[ERROR] [DynamixelSDKWrapper]: Failed to initialize SDK handlers
Cannot ioctl flush on /dev/ttyACM0
[ERROR] [turtlebot3_node]: Failed connection with Devices
[INFO] [turtlebot3_node]: Add Motors
[INFO] [turtlebot3_node]: Add Wheels
[INFO] [turtlebot3_node]: Add Sensors
terminating with uncaught exception of type rclcpp::exceptions::ParameterNotDeclaredException: sensors.bumper_1
0xffff800000b30ad0 (iTurtlebot3_ros): RTP 0xffff800000336b00 has been deleted due to signal 6.

[razr]

@sdoerr20, thanks for reporting it.
Turtlebot3 description is missing in the README.
could you please try
[vxWorks *]# rtp exec -u 0x80000 /bd0a/llvm/bin/turtlebot3_ros.vxe -- -i /usb2ttyS/0 __params:=/bd0a/llvm/bin/burger.yaml &
[vxWorks *]# rtp exec -u 0x30000 /bd0a/llvm/bin/hlds_laser_publisher.vxe -- __node:=hlds_laser_publisher &

You probably need to copy burger.yaml to the USB stick as well. I don't have TB3 at hands at the moment, so please keep me posted on your progress :)

[sdoerr20]

As first use case I tried to start the turtlebot3_ros programm with the suggested paramters. This leads to the completion of the TurtleBot3 setup routine which is indicated by the Message [INFO] [turtlebot3_node]: Run!. As the following trace also shows, VxWorks itself seems to stop the TurtleBot3 process immediate afterwards. Is there an option to increase the number of open files as VxWorks OS parameter?

Trace including setup routine as a reference:
-> devs
drv refs name
1 [ 3] /
2 [ 3] /bd0:1 ==> /bd0a
5 [ 3] /bd0a
2 [ 3] /bin ==> /sd0a/sysroot/bin
2 [ 3] /boot ==> /sd0a/sysroot/boot
2 [ 3] /dev ==> /
2 [ 3] /etc ==> /sd0a/sysroot/etc
10 [ 3] /fifos
13 [ 3] /input/event
2 [ 3] /lib ==> /sd0a/sysroot/lib
0 [ 3] /null
12 [ 3] /random
8 [ 3] /stdio_pty_0xffff8000004e9130.M
7 [ 3] /stdio_pty_0xffff8000004e9130.S
6 [ 3] /tmp
3 [ 3] /ttyS0
2 [ 3] /tyCo/0 ==> /ttyS0
12 [ 3] /urandom
16 [ 3] /usb2ttyS/0
16 [ 3] /usb2ttyS/1
2 [ 3] /usr ==> /sd0a/sysroot/usr
11 [ 3] /zero
15 [ 3] host:
value = -140737484992496 = 0xffff800000335010
-> cmd
[vxWorks *]# set env LD_LIBRARY_PATH="/bd0a/lib"
[vxWorks *]# cd /bd0a/llvm/bin/
[vxWorks *]# rtp exec -u 0x20000 timer_lambda.vxe
Launching process 'timer_lambda.vxe' ...
Process 'timer_lambda.vxe' (process Id = 0xffff8000005181a0) launched.
[INFO] [minimal_timer]: Hello, world!
[INFO] [minimal_timer]: Hello, world!
[INFO] [minimal_timer]: Hello, world!
[INFO] [minimal_timer]: Hello, world!
[vxWorks *]# rtp exec -u 0x80000 /bd0a/llvm/bin/turtlebot3_ros.vxe -- -i /usb2ttyS/0 __params:=/bd0a/llvm/bin/burger.yaml &
Launching process '/bd0a/llvm/bin/turtlebot3_ros.vxe' ...
Process '/bd0a/llvm/bin/turtlebot3_ros.vxe' (process Id = 0xffff8000005736e0) launched.
Attachment number for process '/bd0a/llvm/bin/turtlebot3_ros.vxe' is %1.
[vxWorks *]# [INFO] [turtlebot3_node]: Init TurtleBot3 Node Main
[INFO] [turtlebot3_node]: Init DynamixelSDKWrapper
[INFO] [DynamixelSDKWrapper]: Succeeded to open the port(/usb2ttyS/0)!
[INFO] [DynamixelSDKWrapper]: Succeeded to change the baudrate!
[INFO] [turtlebot3_node]: Start Calibration of Gyro
[INFO] [turtlebot3_node]: Calibration End
[INFO] [turtlebot3_node]: Add Motors
[INFO] [turtlebot3_node]: Add Wheels
[INFO] [turtlebot3_node]: Add Sensors
[INFO] [turtlebot3_node]: Succeeded to create sensor state publisher
[INFO] [turtlebot3_node]: Succeeded to create joint state publisher
[INFO] [turtlebot3_node]: Add Devices
[INFO] [turtlebot3_node]: Succeeded to create motor power server
[INFO] [turtlebot3_node]: Succeeded to create reset server
[INFO] [turtlebot3_node]: Succeeded to create sound server
[INFO] [turtlebot3_node]: Run! # Initializes completely, but fails immediately
terminating with uncaught exception of type std::system_error: pipe_select_interrupter: too many open files
0xffff800000b0c860 (iTurtlebot3_ros): RTP 0xffff8000005736e0 has been deleted due to signal 6.
[vxWorks *]#

[sdoerr20]

As second use case I also tried to test the LiDAR sensor driver independently from the TurtleBot3 bringup routine. Here again the initialization seems to work which is than followed by an internal ROS2 library error. Between these two states the test driver runs for approximately 2-3 min where I suspected to see some measurements, but did not see a related log output.

Therefore I started debugging on our Ubuntu reference setup, but can not find any LiDAR related hardware issues.

Trace with LiDAR initialization in the following:
-> devs
drv refs name
1 [ 3] /
2 [ 3] /bd0:1 ==> /bd0a
5 [ 3] /bd0a
2 [ 3] /bin ==> /sd0a/sysroot/bin
2 [ 3] /boot ==> /sd0a/sysroot/boot
2 [ 3] /dev ==> /
2 [ 3] /etc ==> /sd0a/sysroot/etc
10 [ 3] /fifos
13 [ 3] /input/event
2 [ 3] /lib ==> /sd0a/sysroot/lib
0 [ 3] /null
12 [ 3] /random
8 [ 3] /stdio_pty_0xffff80000030c480.M
7 [ 3] /stdio_pty_0xffff80000030c480.S
6 [ 3] /tmp
3 [ 3] /ttyS0
2 [ 3] /tyCo/0 ==> /ttyS0
12 [ 3] /urandom
16 [ 3] /usb2ttyS/0
16 [ 3] /usb2ttyS/1
2 [ 3] /usr ==> /sd0a/sysroot/usr
11 [ 3] /zero
15 [ 3] host:
value = -140737484992496 = 0xffff800000335010
-> cmd
[vxWorks *]# set env LD_LIBRARY_PATH="/bd0a/lib"
[vxWorks *]# cd /bd0a/llvm/bin/
[vxWorks *]# rtp exec -u 0x20000 timer_lambda.vxe
Launching process 'timer_lambda.vxe' ...
Process 'timer_lambda.vxe' (process Id = 0xffff80000030cec0) launched.
[INFO] [minimal_timer]: Hello, world!
[INFO] [minimal_timer]: Hello, world!
[INFO] [minimal_timer]: Hello, world!
[INFO] [minimal_timer]: Hello, world!
[INFO] [minimal_timer]: Hello, world!
[vxWorks *]# rtp exec -u 0x30000 /bd0a/llvm/bin/hlds_laser_publisher.vxe -- __node:=hlds_laser_publisher &
Launching process '/bd0a/llvm/bin/hlds_laser_publisher.vxe' ...
Process '/bd0a/llvm/bin/hlds_laser_publisher.vxe' (process Id = 0xffff800000565810) launched.
Attachment number for process '/bd0a/llvm/bin/hlds_laser_publisher.vxe' is %1.
[vxWorks *]# [INFO] [hlds_laser_publisher]: Init hlds_laser_publisher Node Main
[INFO] [hlds_laser_publisher]: port : /usb2ttyS/1 frame_id : laser
terminating with uncaught exception of type shared_library::LibraryLoadException: _rtld_exclusive_exit :: called # Runs for 3 min, than fails due to this exception

0xffff800000afed80 (iHlds_laser_publisher): RTP 0xffff800000565810 has been deleted due to signal 6.

[vxWorks *]#

[razr]

[INFO] [hlds_laser_publisher]: port : /usb2ttyS/1 frame_id : laser terminating with uncaught exception of type shared_library::LibraryLoadException: _rtld_exclusive_exit :: called # Runs for 3 min, than fails due to this exception
0xffff800000afed80 (iHlds_laser_publisher): RTP 0xffff800000565810 has been deleted due to signal 6.

This happens most probably that still not all libraries are copied to the filesystem. Let me look, it could be a defect in the copy script somewhere

[razr]

[INFO] [turtlebot3_node]: Run! # Initializes completely, but fails immediately terminating with uncaught exception of type std::system_error: pipe_select_interrupter: too many open files

@sdoerr20, are you using your own VxWorks compiled kernel or one from the SDK?
yes, there is an option, please try to set e.g.
NUM_FILES 2000
PIPE_MAX_OPEN_FDS 500

[sdoerr20]

@razr Yes, I use the pre-build VxWorks Image. Regarding my own Image I haven't finished that task, yet. From my perspctive it's better to get the TurtleBot3 with the pre-build image and than with the costume image in order to exclude further dependency issues.

NUM_FILES 2000
PIPE_MAX_OPEN_FDS 500 <

Can I set these options before the compilation of the VxWorks Image or can I export this variables (bash) analogously to the README command: > set env NUM_FILES=2000 <

[razr]

@sdoerr20, yes, I agree it makes sense to fix your problem with the RPI3 SDK image and then move on. so, I have asked my colleague to test it as well and he was able to run it.
Could you please confirm that you use this SDK https://labs.windriver.com/downloads/wrsdk-vxworks7-raspberrypi3b-1.4.tar.bz2 ?

Next, downloaded release 1.4 of RPi3 from labs.windriver.com and updated SD card and started the turtlebot3 app ...

U-Boot 2020.04-rc3-00037-g8aad16916d (Mar 16 2020 - 21:40:03 +0900)

DRAM:  948 MiB
RPI 3 Model B+ (0xa020d3)
MMC:   mmc@7e202000: 0, sdhci@7e300000: 1
Loading Environment from FAT... OK
In:    serial
Out:   vidconsole
Err:   vidconsole
Net:   No ethernet found.
starting USB...
Bus usb@7e980000: scanning bus usb@7e980000 for devices... 7 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found
Hit any key to stop autoboot:  0 
6274568 bytes read in 272 ms (22 MiB/s)
## Booting kernel from Legacy Image at 08000000 ...
   Image Name:   vxworks
   Image Type:   AArch64 VxWorks Kernel Image (uncompressed)
   Data Size:    6274504 Bytes = 6 MiB
   Load Address: 00100000
   Entry Point:  00100000
   Verifying Checksum ... OK
   Loading Kernel Image
   !!! WARNING !!! Using legacy DTB
## Starting vxWorks at 0x00100000, device tree at 0x00000000 ...
Target Naarget 
Instantiating /tmp as rawFs,  device = 0x1
 
 _________            _________
 \77777777\          /77777777/
  \77777777\        /77777777/
   \77777777\      /77777777/
    \77777777\    /77777777/
     \77777777\   \7777777/
      \77777777\   \77777/              VxWorks 7 SMP 64-bit
       \77777777\   \777/
        \77777777\   \7/     Core Kernel version: 3.1.2.1
         \77777777\   -      Build date: Apr 21 2020 10:28:21
          \77777777\
           \7777777/         Copyright Wind River Systems, Inc.
            \77777/   -                 1984-2020
             \777/   /7\
              \77\
               -   -------

                   Board: Raspberry Pi 3 Model B+ - ARMv8
               CPU Count: 4
          OS Memory Size: ~947MB
        ED&R Policy Mode: Deployed
     Debug Agent: Not started
         Stop Mode Agent: Not started
              BSP Status: *** UNSUPPORTED ***

usrNetDevNameGet: no network device
usrNetDevNameGet: no network device

 Adding 12334 symbols for standalone.

-> 
Find USB-Ethernet device: Lan78xx USB Ethernet
Waiting for IP attach...
Find USB-to-Serial adapter device: CP210x USB-to-Serial Adapter
Added new USB-to-Serial adapter device as /usb2ttyS/0
Find USB-to-Serial adapter device: Wind River Systems USB CDC Serial emulator
Added new USB-to-Serial adapter device as /usb2ttyS/1
Attached TCP/IP interface to usb2End unit 0
Connecting to the Network...
-> 
-> 
-> 
-> 
-> cmd
[vxWorks *]# set env LD_LIBRARY_PATH="/bd0a/RPi3/lib"
[vxWorks *]#                
[vxWorks *]# cd /bd0a/RPi3/llvm/bin/
[vxWorks *]# rtp exec -u 0x80000 turtlebot3_ros.vxe -- -i /usb2ttyS/1 __params:=/bd0a/RPi3/burger.yaml &
Launching process 'turtlebot3_ros.vxe' ...
Process 'turtlebot3_ros.vxe' (process Id = 0xffff8000003af7e0) launched.
Attachment number for process 'turtlebot3_ros.vxe' is %1.
[vxWorks *]# [INFO] [turtlebot3_node]: Init TurtleBot3 Node Main
[INFO] [turtlebot3_node]: Init DynamixelSDKWrapper
Cannot ioctl flush on /usb2ttyS/1
[INFO] [DynamixelSDKWrapper]: Succeeded to open the port(/usb2ttyS/1)!
Cannot ioctl flush on /usb2ttyS/1
[INFO] [DynamixelSDKWrapper]: Succeeded to change the baudrate!
[INFO] [turtlebot3_node]: Start Calibration of Gyro
[INFO] [turtlebot3_node]: Calibration End
[INFO] [turtlebot3_node]: Add Motors
[INFO] [turtlebot3_node]: Add Wheels
[INFO] [turtlebot3_node]: Add Sensors
[INFO] [turtlebot3_node]: Succeeded to create sensor state publisher
[INFO] [turtlebot3_node]: Succeeded to create joint state publisher
[INFO] [turtlebot3_node]: Add Devices
[INFO] [turtlebot3_node]: Succeeded to create motor power server
[INFO] [turtlebot3_node]: Succeeded to create reset server
[INFO] [turtlebot3_node]: Succeeded to create sound server
[INFO] [turtlebot3_node]: Run!
[INFO] [diff_drive_controller]: Init Odometry
[INFO] [diff_drive_controller]: Run!
[WARN] []: Messages of type 0 arrived closer (16666667) than the lower bound you provided (75000000) (will print only once)

[vxWorks *]# 

[sdoerr20]

After debugging and rebuilding my TurtleBot3 infrastructure with the most recent RPI3 SDK (wrsdk-vxworks7-raspberrypi3b-1.4.tar.bz2) I was able to start the TurtleBot3 setup as the following trace shows. Furthermore I managed to initialize the hlds sensor publisher as well.

-> cmd
[vxWorks *]# set env LD_LIBRARY_PATH="/bd0a/RPi3/lib"
[vxWorks *]# cd /bd0a/RPi3/llvm/bin/
[vxWorks *]# rtp exec -u 0x80000 turtlebot3_ros.vxe -- -i /usb2ttyS/0 __params:=/bd0a/RPi3/burger.yaml &
Launching process 'turtlebot3_ros.vxe' ...
Process 'turtlebot3_ros.vxe' (process Id = 0xffff800000e37420) launched.
Attachment number for process 'turtlebot3_ros.vxe' is %1.
[vxWorks *]# [INFO] [turtlebot3_node]: Init TurtleBot3 Node Main
[INFO] [turtlebot3_node]: Init DynamixelSDKWrapper
[INFO] [DynamixelSDKWrapper]: Succeeded to open the port(/usb2ttyS/0)!
[INFO] [DynamixelSDKWrapper]: Succeeded to change the baudrate!
[INFO] [turtlebot3_node]: Start Calibration of Gyro
[INFO] [turtlebot3_node]: Calibration End
[INFO] [turtlebot3_node]: Add Motors
[INFO] [turtlebot3_node]: Add Wheels
[INFO] [turtlebot3_node]: Add Sensors
[INFO] [turtlebot3_node]: Succeeded to create sensor state publisher
[INFO] [turtlebot3_node]: Succeeded to create joint state publisher
[INFO] [turtlebot3_node]: Add Devices
[INFO] [turtlebot3_node]: Succeeded to create motor power server
[INFO] [turtlebot3_node]: Succeeded to create reset server
[INFO] [turtlebot3_node]: Succeeded to create sound server
[INFO] [turtlebot3_node]: Run!
[INFO] [diff_drive_controller]: Init Odometry
[INFO] [diff_drive_controller]: Run!

[razr]

@sdoerr20 this is really good to hear. Thank you very much for chasing this problem until the end.
Could you please tell me where were the difficulties? it is in the setup instructions or in the build process or somewhere else?

[sdoerr20]

@razr To give an overall answer I would say it is simply the complete technology stack required to run a TurtleBot3. Sometimes you deal longer with certain issues than expected or something little is missing. Respectively, the whole USB-Drive installation caused several problems where in the end one of my devices became unusable.

At the moment I try to note down for me which steps are elementary. You can have a look at my
Draft if you like.