2941

Maximize Arduino serial output

Question:

There are a multiple questions intersperced throughout this post. Kindly read carefully and answer the pieces you can for an upvote.

Use Case

Create a time series csv file of pressure reads. These reads need to be of maximum frequency, but I only need it to last less than 10 seconds or so.

Material

<ol><li>Arduino Uno Clone (unalterable)</li> <li>serial over USB 2.0 (alterable)</li> <li>pyserial (Python 3)</li> <li>SSD</li> </ol>

Problem

Identify and fix the bottleneck preventing frequency from maximum reads/s.

The code

<h2>Arduino</h2> void setup() { //speed over the usb serial interface Serial.begin(450000); } void loop() { //get sensor data: read the input on analog pin 0: int sensorValue = analogRead(A0); //returns value 0-1023 //send over serial Serial.println(sensorValue); } <h2>Python</h2> ... ser.serial() ser.baudrate=450000 ser.open() while True: try: serialData = float(ser.readline().decode().replace('\n', '')) except(ValueError): continue #I ran the code without this try/except and got 100 reads higher than when this is included in here. Caused no change in order of magnitude. now = datetime.datetime.now() serialData2 = serialData * voltageOffset #a couple of these happen outfile.write(now+', '+str(serialData2)+'\n') except(KeyboardInterrupt): ... os.sys.exit(0)

Bandwidths

Some calculations to figure out where the bottleneck resides.

<a href="https://www.arduino.cc/en/Reference/AnalogRead" rel="nofollow">100 µs to read analog pin</a>

1 packet = 10 bits

serial wire speed: 450000/10 = 45000 packets / second

<em>Arduino samples-into-serial/second rate:</em>

100 µs * 1s/(10^-6 µs) = 1 x 10^4 samples / second

arduino silicon adapter: cheap cloned chip. Assumed to be underperforming as it is not an arduino-licensed product, but this is deemed acceptable loss as will be explained later. Assumed to have unlimited bandwidth, though this obviously will increase performance by an order of magnitude.

<em>USB 2.0</em>

packets/s = 480 mb/s * 10^6 b/mb * 1 packet/10 b = 48 x 10^6 packets / second

<em>PC file write speed</em>

400 MB/s = 320 x 10^6 packets/s

<em>PC python script</em>

Unknown. Presumed to be infinite? the datetime.now() query is known to take ~ 8 µs.

<strong>Conclusion:</strong> The bottleneck is the serial baud rate.

Adjusting the baud rate confirms this. I'm finding that el-cheapo arduino uno only supports a maximum 450000 baud rate, when the <a href="https://arduino.stackexchange.com/questions/296/how-high-of-a-baud-rate-can-i-go-without-errors" rel="nofollow">arduino actually supports 2-7 million</a> from what I can gather online. That being said, something is still wrong.

The linchpin question

Recall that at this baud rate, we should be seeing pressure reads generated at a theoretical 45000 packets / second.

<em>baudrate-data length-no parity-1 end bit present</em>: 450000-8-n-1

Let sensorValue = 1023. Data looks like this going into println: 1023\n

When I run serial.println(sensorValue) in the arduino, how many packets are sent over the wire to the PC? How is this serialized into packets? How many packets will be sent over the wire? Which one of the following is true:

A. 16 bit int requires 2 packets to be transmitted. The \n exacts 2 more. Total = 4

B. sensorValue is converted to a string and sent over the wire with separate packets: 1 for 1, 1 for 0, 1 for 2, 1 for 3, 1 for \, and 1 for n. Total = 6

C. Other (<strong>help me optimize??</strong>)

Tests

I'm not seeing results as I would expect.

Let baud = 450000.

Changing value in the arduino code serial.println(value) produces different reads/second in my txt file:

value = readPin\n >> 5417 pressure reads/s

value = 1 >> 10201 reads/s

value = 1023 >> 4279 reads/s

This tells me a couple things. There is overhead when reading the analogPin's value. I can also conclude the arduino also only sends over multiple packets of data when it has to (1 > 8 bit of data, 1023 > 16 bit of data). <em>Wait, what?</em> That doesn't make very much sense. <strong>How does that happen?</strong>

Adjusting the baud rate causes the number of reads to change until the arduino maxes out. But lets find the throughput I should be expecting.

450000 b/s * (1 packet / 10b) * [1 read / (4 or 6 packets)] = 11250 reads OR 7500 theoretical reads/s

Actual = 5417 reads/s

<strong>Where did 2,000 reads disappear to?</strong>

Answer1:

You are right that the bottleneck is the serial port (in the sense that you transmit data inefficiently). However, most of your other assumptions are wrong.

<strong>What does baud rate stand for</strong>

Baud rate is the number of distinct symbol changes per second. Here it would be equivalent for bits per second. The baud rate would include all bits transmitted, not only data (start, stop, parity).

<strong>How 10-bit value is transmitted</strong>

As you have 8n1 transmission, you cannot send exactly 10 bits of data. It must be a multiple of 8, so 8, 16, 24, etc. 10 bits will be sent in two 8 bit parts, just like they are stored in Arduino memory as an int.

<strong>How does println() work</strong>

println() converts numbers to string. You can specify base for this conversion (DEC, BIN, HEX, OCT) - default is DEC, so 1023 will be transmitted as 4 bytes + \n which is a single byte (ASCII 10) and \r (ASCII 13) which is also a single byte. Total of 6 bytes. 1 will require 3 bytes - 1 for data and 2 for newline and carriage return.

<strong>How to make it faster</strong>

<ol><li>Without changing almost anything - use println(val, HEX) - you will need 5 bytes maximum for numbers greater than 255.</li> <li>Using Serial.write() - this function puts raw binary data into serial, so if you want to send a value 10, it will send just one byte of value 10. However, when you want to send 10-bit variable, things get complicated - you need 2 bytes for that, and PC needs to know which is the first part and which is second to stitch it back together. You will need to come up with some form of simple transmission protocol to handle this, so probably some start/stop character.</li> <li>Do you need 10-bits resolution? If you could get on with 8-bits you could transmit just plain data, without any additional characters. Also, as suggested in the comments, you could use faster ADC clock then.</li> <li>Also suggested by KIIV - you could use ADC conversion interrupt that would trigger instantly when the measurement is completed.</li> </ol>

If you still go with 10-bit, I would suggest 3-byte frame: 1 start byte, and 2 bytes of data. This way you could theoretically reach ~18 000 frames per second, which is above the maximum analogRead() frequency.

Answer2:

One option is to sample x times then average those before sending back to the PC. Or take on the challenge of compressing on the 328p! You might be fine with some RLE if your pressure readings are mostly consistent.

Don't use 10 bits. The ADC may have 10 bits of resolution but the accuracy is +/- 2 LSB and is very dependent on a stable vref. So drop the 2 LSB and use the top 8 bits for fast Arduino-to-PC readings or multi-sample and average out the noise.

It's a noisy, shared, low resolution ADC. So don't leave the other analog pins floating as that contributes to the noise (just ground them). Also run the Arduino in a grounded enclosure if possible since the ADC has a MUX causing it to have longer traces. Switching ADC modes or pins also generates a bit of noise (not that you're doing that here). Battery power is great if you can keep a stable input voltage otherwise AC/DC noise might cause unexpected vref fluctuations.

The main loop on the Arduino runs both serial IO between the 328p and the 8u as well as your analog reads. So timing bottlenecks are a bit more complex.

The UNO serial-to-USB conversion is being handled by the ATMEGA8u2 (another chip on the board) which can communicate at 115.2kbaud (not 450000).

There are buffers on the 8u and the 328p just to add to the complexity. And the two chips run on different clocks. The ATMEGA8u2 runs from the crystal and the ATMEGA328p (primary processor) runs from the resonator (both 16MHz).

Depending on your needs, the <a href="http://www.pjrc.com/teensy/" rel="nofollow">Teensy</a> has a slightly more accurate ADC (an external one will give better reads) and a faster CPU, more RAM, etc. and it can max the USB port speed. Plus Paul has great info on some low level techniques to make things very fast.

Recommend

  • How to pass multiple output from function into cell array
  • cuda: warp divergence overhead vs extra arithmetic
  • Huge difference in netcat and iperf results for a 10G link
  • Measuring end user latency or performance on the client side?
  • How do I clone a single branch in Git?
  • Azure Media Services encoded mp4 file size is 10x the original
  • The file size of jQuery
  • How to load file contents from another domain using just JS?
  • Use Google font or host font myself [closed]
  • Resource usage increases abnormally in Google App Engine
  • AVAudioUnitEQ / .BandPass filter doesn't work
  • STFT Clarification (FFT for real-time input)
  • Update All Rows in DataBase with a hash value [closed]
  • determining internet speed in android
  • Adapting diverging stacked bar chart to use general update pattern
  • str_getcsv not parsing the data correctly
  • strategy for syncing Android app database with a remote API
  • updating and compacting sqlite database in android
  • How does MemberWiseClone create a new object with the cloned properties?
  • How to Upload a file in GWT
  • App Memory Usage differs between Devices
  • How Can I Prevent Activation For Some ListView Items When The Selection Mode Is MultiChoiceModal?
  • SIP Makefile fail (gnuwin and mingw)
  • How to get file download speed (transfer rate) with php?
  • Mongodb update() vs. findAndModify() performace
  • How to add learning rate to summaries?
  • It is possible use the same sql azure instance from two different cloud service of two different sub
  • Get the number 18437736874454810627
  • Write output of for loop to multiple files
  • Plotting densities in R
  • Consuming a WCF service in a Java Client using wsHttpBinding
  • configure: error: no acceptable C compiler found in $PATH
  • HTTP/2 streams vs HTTP/1.1 connections
  • Is there a way to do normal logging with EureakLog?
  • Magento Fatal error: Maximum execution error solution, on WAMP
  • Row Count Is Returning the incorrect number using RaptureXML
  • Initializer list vs. initialization method
  • formatting the colorbar ticklabels with SymLogNorm normalization in matplotlib
  • Is my CUDA kernel really runs on device or is being mistekenly executed by host in emulation?
  • Data Validation Drop Down Box Arrow Disappearing