122 lines
6.5 KiB
Plaintext
122 lines
6.5 KiB
Plaintext
/**
|
|
*************************************************************************************************
|
|
* @file readme.txt
|
|
* @author MCD Application Team
|
|
* @brief Description of the Artificial Intelligence Hand Writing Character Recognition example.
|
|
*************************************************************************************************
|
|
*
|
|
* Copyright (c) 2019 STMicroelectronics. All rights reserved.
|
|
*
|
|
* This software component is licensed by ST under BSD 3-Clause license,
|
|
* the "License"; You may not use this file except in compliance with the
|
|
* License. You may obtain a copy of the License at:
|
|
* opensource.org/licenses/BSD-3-Clause
|
|
*
|
|
*************************************************************************************************
|
|
*/
|
|
|
|
This project demonstrate a complex application that is running on both CPU1(CA7) and CPU2(CM4).
|
|
The application is a launcher that recognize hand writing character drawn on the touch screen in order
|
|
to execute specific actions.
|
|
|
|
CPU1 (CA7) control the touch event and the Graphic User Interface.
|
|
CPU2 (CM4) is used to offload the processing of a Cube.AI pre-build Neural Network.
|
|
|
|
The communication between the CPU1(CA7) and the CPU2(CM4) is done through a Virtual UART to create an
|
|
Inter-Processor Communication channel seen as a TTY device in Linux.
|
|
The implementation is based on:
|
|
* RPMSG framework on CPU1(CA7) side
|
|
* and OpenAMP MW on the CPU2(CM4) side
|
|
|
|
OpenAMP MW uses the following HW resources
|
|
* IPCC peripheral for event signal (mailbox) between CPU1(CA7) and CPU2(CM4)
|
|
* MCUSRAM peripheral for buffer communications (virtio buffers) between CPU1(CA7) and CPU2(CM4)
|
|
Reserved shared memeory region for this example: SHM_ADDR=0x10040000 and SHM_SIZE=128k.
|
|
It is defined in platform_info.c file
|
|
|
|
A communication protocol has been defined between the CPU1(CA7 and the CPU2(CM4).
|
|
The data frames exchanged have the follwowing structure:
|
|
----------------------------------------------------------------
|
|
| msg ID | data Length | data Byte 1 | ... | data Byte n | CRC |
|
|
----------------------------------------------------------------
|
|
|
|
- 3 types of message could be received by CPU2(CM4):
|
|
* Set the Neural Network input type (0x20, 0x01, data, CRC)
|
|
* data = 0 => NN input is letter or digit
|
|
* data = 1 => NN input is letter only
|
|
* data = 2 => NN input is digit only
|
|
|
|
* Provide the touch screen coordinate (0x20, n, data_x1, data_y1, ... , data_xn, data_yn, CRC)
|
|
* n => the number of coordinate points
|
|
* data_xn => x coordinate of the point n
|
|
* data_yn => y coordinate of the point n
|
|
|
|
* Start ai nn processing (0x22, 0x00, CRC)
|
|
|
|
- 4 types of acknowledges could be received on CPU1(CA7) side:
|
|
* Bad acknowledge (0xFF, 0x00, CRC)
|
|
|
|
* Good acknowledge (0xF0, 0x00, CRC)
|
|
|
|
* Touch screen acknowledge (0xF0, 0x01, n, CRC)
|
|
* n => number of screen coordinate points acknowledged
|
|
|
|
* AI processing result acknowledge (0xF0, 0x04, char, accuracy, time_1, time_2, CRC)
|
|
* char => this is the recognized letter (or digit)
|
|
* accuracy => this is the confidence expressed in percentage
|
|
* time_1 => upper Bytes of the time (word) expressed in ms
|
|
* time_2 => lower Bytes of the time (word) expressed in ms
|
|
|
|
On CPU2(CM4) side:
|
|
- CPU2(CM4) initialize OPenAMP MW which initializes/configures IPCC peripheral through HAL
|
|
and setup openamp-rpmsg framwork infrastructure
|
|
- CPU2(CM4) creates 1 rpmsg channels for 1 virtual UART instance UART0
|
|
- CPU2(CM4) initialize the Character Recognition Neural Network
|
|
- CPU2(CM4) is waiting for messages from CPU1(CA7) on this channels
|
|
- When CPU2(CM4) receives a message on 1 Virtual UART instance/rpmsg channel, it processes the message
|
|
to execute the associated action:
|
|
* set the NN input type to the desire value
|
|
* or register the touch event coordinate to generate the picture that will be processed by the NN
|
|
* or start the NN processing and wait for the results
|
|
- On every previous action, the CPU(CM4) is sending back to the CPU1(CA7) and acknowledge already defined
|
|
above.
|
|
|
|
On CPU1(CA7) side:
|
|
- CPU1(CA7) open the input event to register the touch events generated by the user's finger drawing
|
|
- CPU1(CA7) configure the input type (Letter only) of the Neural Network running on the CPU2(CM4) by
|
|
sending a message throught the virtual TTY communication channel
|
|
- when the drawing is finished, CPU1(CA7) process the touch event data and send it to the CPU2(CM4)
|
|
- CPU1(CA7) start the Neural Network processing wait for the result and display the recognized character on
|
|
the diplay
|
|
|
|
Some information about the Character Recognition Neural Network:
|
|
The Character Recognition Neural Network used is a Keras model processed by Cube.AI to generate the executable
|
|
that can be run on the CPU2(CM4).
|
|
The Keras model used is located in the root directory of this project:
|
|
model-ABC123-112.h5
|
|
This model has been used in Cube.AI to generate the Neural Network binary.
|
|
The model accept as input a 28x28 picture encoded with float in black and white (black = 0.0 or White = 1.0).
|
|
The output layer of the Neural Network contains 36 neurons (A -> Z and 0 -> 9).
|
|
|
|
Notes:
|
|
- It requires Linux console to run the application.
|
|
- CM4 logging is redirected in Shared memory in MCUSRAM and can be displayed using following command:
|
|
cat /sys/kernel/debug/remoteproc/remoteproc0/trace0
|
|
|
|
Following command should be done in Linux console on CA7 to run the example :
|
|
> /usr/local/demo/bin/ai_char_reco_launcher /usr/local/demo/bin/apps_launcher_example.sh
|
|
|
|
You are ready to draw letter on the touch screen
|
|
|
|
Hardware and Software environment:
|
|
- This example runs on STM32MP157CACx devices.
|
|
- This example has been tested with STM32MP157C-DK2 and STM32MP157c-EVAL board and can be
|
|
easily tailored to any other supported device and development board.
|
|
|
|
Where to find the M4 firmware source code:
|
|
The M4 firmware source code is delivered as demonstration inside the STM32CubeMP1.
|
|
For the DK2 board:
|
|
<STM32CubeMP1>/Firmware/Projects/STM32MP157C-DK2/Demonstrations/AI_Character_Recognition
|
|
For the EV1 board:
|
|
<STM32CubeMP1>/Firmware/Projects/STM32MP157C-DK2/Demonstrations/AI_Character_Recognition
|