DIY microcontroller and happily "steal data"!

DIY a microcontroller and elegantly "steal" data!#

Background#

Recently, due to a personal project, I needed to send data from a host with no internet connection to a remote server. Considering that the host cannot connect to the internet, I decided to use a microcontroller as an intermediary. The data would be sent to the microcontroller via serial communication, and then the microcontroller with internet access would send the data to the server. This way, I could perfectly achieve the operation of "stealing" data! Let's get started!

Host Data Capture#

Since I needed to send data to the microcontroller, the first step was to obtain the data. Since the host is offline and has the highest level of permission, I decided to install a Python environment (this way, I could analyze the data, distinguish between abnormal values and valid data, and choose what to send) and use pyserial and pymssql for data capture. I won't list the detailed code here, but I'll mention a pitfall I encountered during the process.

Initially, because the Python program did not include a delimiter for sending data, the microcontroller couldn't determine where the data ended during testing. This resulted in fragmented data being sent to the remote server. However, after adding a delimiter in the main program, this problem was perfectly solved.

import serial
import json
import pymssql

def get_data():
    """
    """
    return(data)

# Add delimiter for sending data
def deliver_data():
    myChar='#'
    # Set the corresponding serial port number and communication baud rate
    ser = serial.Serial(port="COM4", baudrate=115200)

    # Convert the data to JSON format and send it
    data=get_data()
    comdata=json.dumps(data)
    write_len = ser.write(comdata.encode())
    ser.write(myChar.encode())
    print("Sent {} bytes through the serial port.".format(write_len)) 
    ser.close()
    
def main():
    deliver_data()
    print('Transmission completed')

if __name__=='__main__':
    main()

By the way, since the host cannot connect to the internet and is running on the Windows 7 operating system, configuring the Python environment was particularly difficult! After overcoming various obstacles, when I successfully ran the data capture script, my eyes were filled with tears.

Microcontroller Program Development#

Since the ESP8266 can be easily developed using the Arduino framework, it greatly reduces the difficulty for beginners like me to get started. Referring to the article "arduino(2) - Serial Data Reception and Local Area Network Communication Using TCP Protocol with ESP8266 Module" and making some modifications, I successfully implemented the reading and sending of serial data.

#include "ESP8266WiFi.h"
#define led 2
const char *ssid     = "wifi name";
const char *password = "wifi password";
const char *host = "server address";
WiFiClient client;
const int tcpPort = server port;
static String comdata = "";
char myChar='#';
void setup()
{
    Serial.begin(115200);    
    pinMode(led,OUTPUT);
    delay(10);
    Serial.println();
    Serial.print("Connecting to ");
    Serial.println(ssid);
    WiFi.begin(ssid, password);// Start
     // Check if successfully connected to the target network, block if not connected.
    while (WiFi.status() != WL_CONNECTED)
    {
        delay(500);
    }
 // Some prompts
    Serial.println("");
    Serial.println("WiFi connected");
    Serial.println("IP address: ");
    Serial.println(WiFi.localIP());
}
void loop()
{
/******************Serial Data Reception************************/
  while (Serial.available() > 0 ) // If there are characters received through the serial port.
  {
    comdata = Serial.readStringUntil(myChar);
  }
 /******************Serial Data Printing************************/ 
  if(comdata!="")// If data is received
  {
    client.print(comdata);// Send data to the server
   }
   comdata="";// Clear data
  if (client.connected()) // Try to access the target address, if connected, turn off the LED, as this kind of thing should be low-key
    digitalWrite(led, HIGH);
  else
    digitalWrite(led, LOW);

  while (!client.connected())// If not connected to the server, the client attempts to connect.
  {
    if (!client.connect(host, tcpPort))// In fact, this step is to connect to the server. If connected, this function returns true
    {
       Serial.println("Connecting....");
       delay(500);
    }    
  }
}

At this point, the entire process has been successfully completed, and data "stealing" has been achieved. However, it is not elegant to have a data cable connected to the host and then connected to a large NodeMCU development board. I cannot accept this!

I checked Taobao (a popular online shopping platform in China) to see if there was a more compact solution. It seems that a simple USB to TTL communication board + ESP-01s can solve this problem. I promptly bought it to replace the original solution.

Upgrade

However, the new solution had a fatal problem: unstable communication. This resulted in the host's data not being correctly sent to the microcontroller. To investigate, I even tested it on two different hosts, and one worked while the other consistently failed. In comparison, the old solution was much more stable and successfully "stole" data. I initially suspected that the instability was due to the power supply of the new solution, but I didn't continue to verify it. If any experts understand this issue, please let me know. Thank you!

DIY Microcontroller#

Since the existing solutions couldn't meet my requirements, there was only one way to go! I had to make a small and stable microcontroller myself. As a beginner like me, it was impossible to start from scratch, so I decided to check if any experts had already done similar projects on open hardware platforms. If I could successfully replicate one, then problem solved!

usb-esp8266 - JiaLiChuang EDA Open Hardware Platform (oshwhub.com)

Thanks to the project shared by the expert moguiliue, I directly made the PCB and got to work!

Iron Plate

Replication

The expert's solution worked well! The finished product ran smoothly and perfectly met my requirements. However, during the process, I found that the original design of the USB connector on the PCB had interference issues (I'm not sure if it was due to the components I purchased 🤦‍). I had to file down the interfering parts before soldering the USB connector. It was a bit laborious.

Furthermore, this solution only met the basic data communication needs and did not expose other GPIO pins, making it impossible to connect sensors or displays in the future. Although it was elegant, the playability was limited. What should I do?

DIY Microcontroller 2.0#

To achieve both elegance and expandability, why not refer to the expert's ideas and design a new version from scratch!

New Design

I designed a new development board from scratch, replacing the original CH340G solution with CH340C to save space for the external crystal oscillator circuit. I also added a 4-pin female header to expose the I2C communication interface. Since it was difficult to modify the original wiring and layout, I had to completely restructure the design. Currently, I have completed the PCB layout, but it has not been manufactured yet. I hope that after the materials arrive, this microcontroller, which I designed from scratch for the first time, will run successfully!

2023/4/23 update:#

Today, I finally made the microcontroller I designed from scratch. Everything is running smoothly, and I'm happy~
New Design