Commit 9222c499 authored by Mark Wijzenbroek's avatar Mark Wijzenbroek
Browse files

Initial commit

parents
# Compiling the firmware
The firmware can be compiled with a "standard" CodeSourcery ARM EABI toolchain. Note, this package is unfortunately no longer available through the website of the company who provided this. Luckily, the files are still available on their server. Check https://elinux.org/ARMCompilers for download links; the direct download link to the 2014 CodeSourcery stack is available at https://sourcery.mentor.com/GNUToolchain/package12774/public/arm-none-eabi/arm-2014.05-28-arm-none-eabi-i686-pc-linux-gnu.tar.bz2
Adapt Makefile.posix to the place where you unpackaged the ARM toolchain. In principle you should be able to use another ARM EABI toolchain, BUT note that these other toolchains do not have libcs3*.a, which is the CodeSourcery loader, on which the firmware relies.
# The download protocol
The download node translates the stored data on the proximity sensors into something that can be displayed on your terminal using base64:
>1234
000000:abCD452fGJ896Kl+24/356
000001:8535/3jhh+3759ghenJRfh
000002:Mhg8046/83hfe6+59H6f//
000000-hurFJDv645074HDbvyf0+2
000001-the+574ggtu3bHJEfh93bh
000002-//////////////////////
<1234
First of all, each line has len( base64_encoded ) = 22 and len( raw ) = 16. To decode from base64 to raw data the data should be padded with two =s. Note: each line is individually encoded and thus corresponds to exactly one "record" of 16 bytes! Second, we notice a few types of records:
>1234 : node 1234 is now being read
<1234 : end of data from node 1234
xxxxxx: : this is line xxxxxx from the system log
yyyyyy- : this is line yyyyyy from the detection log
# The system log
Each record has a checksum (1 byte) at the beginning so that the ( sum of each byte ) modulo 256 is 0. Empty lines (all 0xFF, base64 encoded a lot of slashes) may have a wrong checksum and can be ignored. The system log always has as first line "EXPERIMENT_DATA\0". Note: this line has no checksum! All numbers unless otherwise indicated are stored little-endian.
Let us look more closely at how data is stored in the system log:
>%2018-11-05T10:51:11.127487+01:00%0952
000000-RVhQRVJJTUVOVF9EQVRBAA translates to: 4558504552494d454e545f4441544100
000001-3AJJwAcAAABTqgUAAEnABw translates to: dc0249c00700000053aa05000049c007
000002-igHvQAsAAABTqgQAAO9ACw translates to: 8a01ef400b00000053aa040000ef400b
000003-rwJTQA0AAABTqhIAAFNADQ translates to: af0253400d00000053aa12000053400d
000004-5wI2wA0AAABTqhQAADbADQ translates to: e70236c00d00000053aa14000036c00d
000005-UwJ1wBEAAABTqiIAAHXAEQ translates to: 530275c01100000053aa22000075c011
000006-5wKgwBQAAABTqjIAAKDAFA translates to: e702a0c01400000053aa320000a0c014
000007-nwK5wBcAAABTqkIAALnAFw translates to: 9f02b9c01700000053aa420000b9c017
000008-6wKIwBoAAABTqlIAAIjAGg translates to: eb0288c01a00000053aa52000088c01a
000009-DQJswB0AAABTqmIAAGzAHQ translates to: 0d026cc01d00000053aa6200006cc01d
000010-QQLLwCAAAABTqmoAAMvAIA translates to: 4102cbc02000000053aa6a0000cbc020
Each line contains exactly one log with the following structure:
00 01 020304050607 0809 0A0B0C 0D0E0F
a b c d e f
a: checksum
b: density
c: time
d: network authentication
e: reset value (8th bit in byte 0A), state (6th and 7th bits in byte 0A), index in detection log (1-5th bits of byte 0A, byte 0B and byte 0C)
f: rtc
# The detection log
Each line has a checksum (1 byte) at the beginning so that the ( sum of each byte ) modulo 256 is 0. Empty lines can be ignored. There is no header as in the system log. Each line stores 6 entries at 20 bits each.
# What's what?
There are a number of different devices and cables that are relevant:
* nRF51 Development Kit (Nordic Semi)
* Proximity sensors (SallandElectronics)
* Checkpoints (SallandElectronics)
* 10 pin - 6 pin "Plug of Nails" cable (Tag-Connect TC2030)
* USB to micro USB cable
To flash a proximity sensor you will need: the nRF51 development kit, a proximity sensor and the plug of nails cable. To download data from the proximity sensors you will need: a download node (checkpoint) and a USB to micro USB cable.
Each of these items will be discussed in detail below.
# nRF51 Development Kit
The nRF51 Development Kit (DK) is built by Nordic Semi and is used to flash the proximity sensors. The nRF51 DK is connected to your computer with the USB to micro USB cable and connected to a proximity sensor with the "Plug of Nails" cable. The "Plug of Nails" cable (10 pin side) is connected to the "Debug Out" port of the board, the 6 pin side to the appropriate port on the proximity sensor.
When connecting the development kit to the computer, turn the board on with the on/off switch, and LD5 will start blinking briefly after which it will remain on. If LD5 remains blinking, something is wrong (try a different USB to micro USB cable). The firmware for the proximity sensors can be loaded using nrfjprog (https://www.nordicsemi.com/index.php/eng/nordic/Products/nRF51822/nRF5x-Command-Line-Tools-Linux64/51386) which is available for Linux as binary .tar.gz or perhaps J-Link (https://www.segger.com/downloads/jlink/#J-LinkSoftwareAndDocumentationPack) which is available on Linux in RPM and DEB format. In this repository a wrapper .py script is incorporated which does the calls to nrfjprog. Note that you need to make sure that nrfjprog is on your path. After use, turn the development board off and disconnect from your PC, otherwise the battery will eventually be depleted.
# Proximity sensors
The proximity sensors were built by SallandElectronics, have no on/off switch and are powered by a CR2032 battery. To flash the firmware, you connect it to the development kit with the "Plug of Nails" cable (see above).
Each proximity sensor has two LEDs (green and red). Depending on which firmware is loaded, these LEDs will blink in a particular pattern or remain on or off. These patterns are known:
* Checkin node: green flickering (almost on all the time)
* Checkout node: red flickering (almost on all the time)
* Regular node: green or red with slow, short blinks (about 1/sec), depending on whether it is checked in (green) or not (red). To download data from the device, check it out and hold it near the downloader checkpoint. After download, all LEDs on the proximity sensor are turned off (no data on device).
* Tester node: red and green quick blinking
* Anchor node: green blinking (about 4/sec)
Only the regular nodes register proximity data and can be checked in or out. Furthermore note that:
* The 2018 firmware registers every 6 seconds to the system log and the detection log;
* The final firmware from the Kumbh-Mela project registers every x seconds to the system log and every y seconds to the detection log.
# Checkpoints
The checkpoints were built by SallandElectronics and are connected to a computer with the USB to micro USB cable. The device acts as a serial device (ftdi chip). The ftdi_sio driver is used under Linux to connect, which may or may not be present on your system (CentOS 7 provides this driver out of the box). On newer kernels you may need to load this driver manually.
There are three types of checkpoints: download nodes, which when connected have a slowly flashing red LED, time synchronization nodes, which when connected have a quickly flashing orange LED, and sniffer nodes, which have a slowly flashing or continuously on orange LED. The download nodes are used to download data to your computer, whereas the time nodes are used to synchronize the time. The sniffer nodes can be used to investigate the network in real time.
To activate the download node you should use the kumbhserial code that is also included in this repository. Offer the checked out nodes to the download node one after another, and make sure you are not too fast! The proximity sensor will have its green LED on while the download is in progress. After this, the sensor may blink its red LED for some time (erasing?). Do not offer a second proximity sensor to the download node before the first node has stopped blinking. Otherwise it may get confused and your data may be lost or muddled together as it wipes the data from the checked out node when you download. Also, make sure you keep only one proximity sensor close to the download node, otherwise it may attempt to start the download on both.
# 10 pin - 6 pin "Plug of Nails" cable
This cable is from Tag-Connect and is used to connect a proximity sensor to the nRF51 DK for flashing the firmware (see above).
# USB to micro USB cable
Any standard USB to micro USB cable should work.
data/*
*.pyc
__pycache__
.idea
*.egg-info
language: python
python:
- "3.4"
- "3.5"
install:
- pip install -U pip
- pip install -r requirements.txt
- pip install -r test_requirements.txt
before_script:
- mkdir -p data/raw
script:
- test $(pyflakes kumbhserial setup.py | tee /dev/stderr | wc -l) -eq 0
- nosetests
# Test the processor, which does not need any serial device
- touch data/raw/test_raw.txt
- kumbhprocessor
- test ! -e data/raw/test_raw.txt
- test -e data/raw/processed/test_raw.txt
- test -e data/detection/detection-test_raw-*.json
- test -e data/system/system-test_raw-*.json
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "<>"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2016 Zoltan Beck, Netherlands eScience Center, and
University of Amsterdam
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
# Kumbh Mela serial devices #
[![Build Status](https://travis-ci.org/indodutch/kumbhserial.svg?branch=master)](https://travis-ci.org/indodutch/kumbhserial)
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/36476556fdc94aa1b4a15ea978924e93)](https://www.codacy.com/app/github_4/kumbhserial?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=indodutch/kumbhserial&amp;utm_campaign=Badge_Grade)
Python 3 implementation for extracting data from devices in Kumbh Mela with a serial device driver.
## Installation ##
Use virtualenv or virtualenvwrapper to set up a suitable Python 3.4 or 3.5 environment.
Install the project with
```shell
pip install -r requirements.txt
```
## Usage ##
To read out the data from the download devices, run:
```shell
kumbhdownload
```
Use `kumbhdownload -h` to see usage information. In particular, use the `-J` option to disable the parser and only write raw text.
Unless otherwise specified, the binary data ends up in the `data/raw/processed` and the JSON data ends up in the `data/detections` and `data/system` directories.
To use read out the time from the sniffer, run:
```shell
kumbhsniffer
```
Unless otherwise specified, sniffer JSON data ends up in the `data/sniffer` directory.
To process any unprocessed files, run `kumbhprocessor`.
To read the Thinture GPS devices with `kumbhgps`. Again, see the options with `-h` flag.
# Indo-Dutch Kumbh Mela experiment serial device reader
#
# Copyright 2015 Zoltan Beck, Netherlands eScience Center, and
# University of Amsterdam
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Kumbh Mela serial device reader
"""
from .ports import serial_ports, choose_serial_port
from .interpreter import (TrackerInterpreter,
SeparatedTrackerEntrySetJsonConverter)
from .main import download, sniffer, processor
from .reader import SerialReader, run_reader, read_file
from .sniffer import SnifferInterpreter
from .appenders import (JsonListAppender, Dumper, RawPrinter, Duplicator,
ThreadBuffer)
from .version import __version__, __version_info__
__all__ = [
'serial_ports',
'choose_serial_port',
'TrackerInterpreter',
'SeparatedTrackerEntrySetJsonConverter',
'download',
'sniffer',
'processor',
'SerialReader',
'run_reader',
'read_file',
'SnifferInterpreter',
'JsonListAppender',
'Dumper',
'RawPrinter',
'Duplicator',
'ThreadBuffer',
'__version__',
'__version_info__',
]
# Indo-Dutch Kumbh Mela experiment serial device reader
#
# Copyright 2015 Zoltan Beck, Netherlands eScience Center, and
# University of Amsterdam
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" Pipe-like appenders, to dump data with different methods and formats. """
import json
import threading
import queue
class Dumper(object):
"""
Dumps bytes data to file.
"""
def __init__(self, path, flush=True):
"""
:param path: file path to write to.
:param flush: flush to file after writing. Slower but safer.
"""""
self.file = open(path, 'wb')
self.flush = flush
def append(self, data):
if len(data) > 0 and self.file:
self.file.write(data)
if self.flush:
self.file.flush()
def done(self):
self.file.close()
self.file = None
class Duplicator(object):
"""
Send the same incoming data to a sequence of appenders.
"""
def __init__(self, appenders):
self.appenders = appenders
def append(self, data):
for a in self.appenders:
a.append(data)
def done(self):
for a in self.appenders:
a.done()
class RawPrinter(object):
"""
Print incoming data to console.
"""
def append(self, data):
print(data)
def done(self):
pass
class JsonListAppender(object):
"""
Convert incoming data to a list of json objects. Each call to append will
add one JSON object to a JSON list.
"""
def __init__(self, appender):
self.appender = appender
self.first = True
self.appender.append(b'[')
def append(self, data):
if self.first:
self.first = False
else:
self.appender.append(b',')
self.appender.append(bytes(json.dumps(data, separators=(',', ':')),
encoding='ascii'))
def done(self):
self.appender.append(b']')
self.appender.done()
class ThreadBuffer(threading.Thread):
"""
Put a thread between the sender and the receiver, so that these can act
concurrently. This is helpful if both the sender and receiver do
independent I/O operations.
"""
def __init__(self, appender, **kwargs):
self.appender = appender
self.queue = queue.Queue()
self.is_done = False
super().__init__(**kwargs)
self.start()
def append(self, data):
self.queue.put(data)
def run(self):
while not self.is_done:
try:
data = self.queue.get(timeout=1)
self.appender.append(data)
except queue.Empty:
pass # no input yet, check if is_done
def join(self, **kwargs):
self.done()
try:
super().join(**kwargs)
finally:
self.appender.done()
def done(self):
if not self.is_done:
self.is_done = True
self.join()
# Indo-Dutch Kumbh Mela experiment serial device reader
#
# Copyright 2015 Zoltan Beck, Netherlands eScience Center, and
# University of Amsterdam
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Facilities to read out the Kumbh Mela GPS devices.
"""
import threading
import serial
from .reader import SerialReader
class GpsReaderThread(threading.Thread):
"""
Thread to read out a GPS device
"""
def __init__(self, port, appender, clear=True, **kwargs):
super().__init__()
self.interpreter = GpsInterpreter(port, appender)
self.reader = SerialReader(port, self.interpreter, wait_time=1,
baud_rate=115200, terminator=b'#',
insert_timestamp_at=(), **kwargs)
self.clear = clear
def run(self):
try:
writer = GpsWriter(self.reader.comm)