2to3, Python 2 to Python 3 made easy!

Do I need to upgrade from Python 2 to Python 3?

I started to develop my ADS-B tracker software in Python 2 and I delayed the upgrade to Python 3 for a long time. I was thinking it would be pretty difficult! Now that PIP is soon to drop support of Python 2.7, I finally got to it. It was much easier than I expected thanks to 2to3!

The magic of 2to3

You don’t really need to know much about Python 3 to do the upgrade! Just copy all your files in a new Python 3 directory, make sure you have installed all the Python 3 packages you need and use 2to3 and it is done!

cp project_master project_master_py3
cd project_master_py3
sudo apt-get install python3 python3-pip <<your Python 3 packages>>
python3 -m pip install <<your packages>>
2to3 -w *.py

It should almost work…

A little bit of tuning is needed

You may still have some errors due to 2 things:
– Python 3 does not like when mixing space and tab for indentation. It is quite easy to solve by being consistant. Use only spaces or only tabs, whatever is best for you (I don’t have a religion there!).
– Python 3 has a different way to manage strings and bytes. That one is trickier. I solved it by adding 2 functions to change the variable to bytes or to string when needed:

#fn to_bytes
def to_bytes(arg):
if hasattr(arg, 'encode'): return arg.encode('utf-8')
if hasattr(arg, 'decode'): return arg
return repr(arg).encode('utf-8')

#fn to_string
def to_string(arg):
if hasattr(arg, 'encode'): return arg
if hasattr(arg, 'decode'): return arg.decode('utf-8')
return repr(arg).decode('utf-8')

I just ran my program until it detected all the instances where bytes and strings were mixed.

That’s it, it did the job. My trackers are now running Python 3, and still connect to my online database, tweet when detect new or interesting planes, connect through SigFox when off grid,… as they were doing in Python 2. I didn’t notice an impact on the error rate so far.

12 free tools for website optimization

After developing my ADS-B trackers and the central database of aircraft, I wanted to share the results to the world. What’s best than having my own website for this? Over the years, I discovered these 12 tools to test and optimise my website. There are few directions which are important for website optimization: make sure your website is known by the main search engines, make sure your SEO meta data is right, make sure your website load fast enough and solve all your technical issues.

Website Optimization for SEO

Browseo
Browseo is a great free tool to make all the SEO data visible. It will show the title, meta tags, and other invisible data that are key for SEO just to make sure you got them right. At first, I didn’t see what it was useful for, because I was just checking my main index page, but when I used it to test my php pages, with dynamic meta tags, that was super useful to see what’s wrong with those tags!


Seositecheckup
Seositecheckup is a great free tool to analyse all the SEO data of your website and highlight what is missing. The free version can only scan your website once a day, but that let’s you some time to solve all the issues before checking them again.


Brokenlinkcheck
Broken links are impacting your SEO ranking, but are hard to find. Brokenlinkcheck will find them for you. Free for 2000 links. If you are using WordPress there are some dedicated plugins for this.

Make sure you are known to main search engines

Bing, Google, Yandex are the one to start with. Create an account, give them your URL and they will start to crawl your website. Check the result of those crawls regularly as they will report errors. Browseo (see above) is a great tool to understand canonical url issues. Regarding sitemaps, there are 2 philosophy: let them discover your site by themselves or use a very detailed one. I went with the first one and only put the main pages in my sitemap.

Website Optimisation for Speed

gtmetrix and Google Pagespeed Insight are two great free tools to understand what is making your website slow to load. Solving the issues requires some technical knowledge, but some tricks are very useful: optimise your images and videos, use a CDN, use Gzip compression and optimize PHP code.

Solve technical issues

Dnschecker
Have you recently switched web host or started a new website, then you are in the right place! DNS Checker provides free DNS lookup service for checking domain name server records against a randomly selected list of DNS servers in different corners of the world. Do a quick DNS propagation lookup for any domain name and check DNS data collected from all location for confirming that the website is completely propagated or not worldwide.


Immuniweb
Website security testing (GDPR compliance, CSH and HTTP headers check, common Javascript libraries version check…). Make sure you have all the security basics right!


Onlineconverter
A free tool to convert any image into the format you need for your website.


DownNotifier
A free tool to check if your website is offline.

Have fun!

3 tips to optimize PHP code

How I divided my server errors by 40 with PHP code optimization

I have been running several Raspberry Pi based trackers for a while, and I wanted to create a common database of detected aircraft and their positions. I looked for online database solution and how I could connect my Pi to them directly in Python, but that was not convenient.

I finally decided to get an hosting plan at OVH and create my own databases, with API to add more aircraft and positions. I wrote all the APIs of my server in PHP, as it was pretty straightforward. I got it work pretty well, but I was still having quite a lot of errors. At first, it was simple, I played a little with the timeout, I add retries, but I was still not able to go below 2% of errors. For a long time I lived with that, my system still worked well, but I finally decided to work on that and optimize PHP code.

The usual PHP tips to optimize PHP code…

A quick search on php optimization got me a list of things to do:

  • use single quote instead of double one: 'text' is better than "text"
  • calculate variable only once in a loop
  • close databases
  • order switch by order of use
  • select specific columns instead of * in mySQL SELECT queries
  • use foreach instead of while
  • use echo $a,$b,$c instead of echo $a.$b.$c
  • use sprintf to add strings: $string = sprintf('today is %s', $date)
  • use === instead of == in if: if ($a===$b) { action}
  • INSERT multiple values in database instead of multiple single INSERT

It seems that many of them were useful for older PHP versions, but not really for PHP versions over 7…

… and what really worked for me!

The 3 actions that had a tremendous effect on my API have been around database optimisation: first, I rewrote the few INSERT loops in order to INSERT multiple values at once. This was already much better. Then I checked my databases structures, to make sure I had all the right indexes corresponding to my SELECT queries. And finally, I switched all the heavy ones to InnoDB (I am not sure why OVH use MyISAM as default structure…).

These 3 actions reduced my error rate from 2% to 0.05% by reducing the load of my server (I am using a very basic hosted configuration, nothing fancy here). On the other tips I found, most of them had no or only very marginal impact. The only one I will keep, is the sprintf function to create my strings with variables as this makes my code much easier to read (it was easy to get lost in the quotes when you write mySQL queries with all that ‘ and ” and .).

How did I find that was my issues? What is important is to find where you PHP requests are spending time. For that, I simply implemented some time measurement inside the code (with the microtime function: $start_time=microtime(True); $stop_time=microtime(True); phpduration = $stop_time-$start_time;), that is coming back to my Pi with the result of the API. I could then dig into the one loading the server for more than 1s and reduced their load. I did the same in my Pi, to track which mySQL queries were taking more than 0.001s to run, and that was very efficient.

I was able to focus on optimising the most important ones (mainly those on huge database, such as positions, which rapidly contain millions of entries), and optimise their structure and indexes. There are many tools available to profile your PHP code, but I didn’t find any that was easy enough to start with compared to what I needed them for.

With 0.05% error rate, I can think about adding more trackers and maybe even have a way for others to share their tracking data with me! Let me know if you would share yours or if you have more tips to optimise PHP code…

What else can help reduce connection errors?

I also did several actions to reduce connection errors to my API server:
– make sure you don’t have any redirection
– make sure you updated requests and certifi on the Raspberry Pi, specially if you are using a secured connection. It can easily be done by:
pip install --upgrade requests certifi urllib3 idna chardet
– use session if you are calling the API regularly:
api_session = requests.Session()
resp = api_session.get([request])
will replace requests.get([request])
– set a different timeout for connection and response:
api_session.get([request], timeout=(5,10))

SigFox for Raspberry Pi

The hardware

First you need to find a Sigfox hat for your Raspberry Pi. Just Google “Sigfox for Raspberry Pi” and you will find the latest solution available. I am using a RPISIGFOX, made my SNOC. Pretty easy to setup and use. Just plug it in, change the serial port parameters, and it is ready to work. It comes with 1 year SigFox subscription that requires activation on Sigfox website. Depending on where you intend to use your Pi, you may need a different version of Sigfox radio (that one is mainly for Europe – US and Asia have different frequency bands for non regulated spectrum).


The Software

  1. Disable Raspberry Pi terminal on serial port with raspi-config utility: sudo raspi-config. Go to Interfacing Options then choose Serial then NO and OK
  2. Install pyserial: sudo apt-get install python-serial
  3. Download scripts: git clone https://github.com/SNOC/rpisigfox.git
  4. Enable script execution: chmod +x sendsigfox.py
  5. Plug antenna with its cable and send your first message: ./sendsigfox.py 0123

Setting up the Callbacks and your server

You also need to setup the callbacks on Sigfox system, and your server to manage the data. This is pretty straightforward with the documentation for the uplink messages. You are limited to 6 uplink messages per hour. On reception of the uplink message, the server will call your API, as you have defined in Sigfox interface. What you do with the data on your server is up to you!

Downlink messages are a little more complex to manage. You are limited to 4 downlink messages per day, and it was not very easy to make sure my Rasberry Pi application was remaining inside this budget, specially with a solar powered Pi and without a RTC … It took me some time to understand that the server was in fact the one making sure you stay in the budget, as if there is no downlink message to send, it won’t count into the 4 messages per day.

To request a downlink message, you just need to send ack=true with your uplink message.

On reception of the uplink request through the above API, your server will have to provide the answer: "[device ID]":{"downlinkData":"[data]"} if there is a downlink payload or "[device ID]":{"noData":true} when there is no payload to send. In that case, the downlink message will not count in your daily budget.

You can find more details on this great post on Sigfox downlink messages.

Keeping the Pi on time

As my Raspberry Pi is solar powered (with a 20W solar panel, which is enough for the Pi, but not for the RTL-SDR dongle that has a much bigger power consumption), I don’t necessarily have enough power to keep the Pi on 24/24. When it is off, the time stops to progress, as the Pi doesn’t have a RTC by default. After few days, time and date are completely wrong. That makes it difficult to control the downlink messages budget from the Pi. At the same time, SigFox is a great solution to get the network time to and update the time of the Pi. With the downlink messages, I was able to get the network time when the Pi is starting and update the date and time of the Pi with the simple command sudo date --set="9 JUN 2017 19:15:00".

How I use it?

With the payload limitations at 6 uplink messages per hour (12 bytes) and 4 downlink messages per day (8 bytes), Sigfox doesn’t really replace a Wifi connection, I cannot report all the detected aircrafts and their positions as I am normally doing. As, this is for an ADS-B tracker, I decided to focus on new aircraft and special ones (such as military) for the uplink direction, transmitting only the ModeS hex code and the timestamp at which it was detected, the remaining 4 bytes are used for maintenance purpose (HHHHHHYYYYMMDDHHMMSSXXXX).

To avoid going over 6 uplink messages per hour, each detection will go into a buffer, that is checked every 10 minutes. I have been working on some more compressed way to encode these data but practically I don’t need it, as I am very seldom over budget (meaning I am sending a message every 10 minutes as long as my Pi is powered by the battery). We will see if this is still the case after COVID-19 and the airlines start to have a more international flights. Each time my server receive an uplink message it will automatically generate a tweet (with the #Sigfox hashtag).

Downlink messages are used only to keep the time of the Pi in sync, as described before. As sometimes the Pi doesn’t receive the downlink message or is switched off again when battery is low and weather is grey, it may need more than one downlink per day for time sync. In addition of the network timestamp, I still have 2 bytes available to send other data to the Pi. I am using this for maintenance purpose (YYYYMMDDHHMMSSXX). Note that the uplink message must be exactly 8 bytes in size or it will not go through.

It has been a long time I wanted to play with a Sigfox hat, and it has been quite fun: not that difficult to make it work, but enough issues to solve to make that interesting!

Having fun with my ADS-B tracker project

How it started

Some years ago, I bumped into an article on Mashable about Jeremy Merrill using a Raspberry Pi ADS-B tracker to detect aircraft flying over his house and display the origin or destination of that plane (see article here and github there). As my place is just on a busy plane corridor, that gave me an idea. I decided to work on a similar project based on Piaware. Looking at how he did it and at the long literature you can find on the web on ADS-B, I created this site, my own software in Python to run on my Pi and other fun stuffs. I also ran into this site by SonicGoose and that gave me many new ideas (including the basic structure of this site – I never had any experience of html or php before).

One key part is getting access to aircraft databases to get more info from the ModeS hex code you get from the Piaware tracker. I am sharing my own database of planes, airports and routes I detected with one of my trackers (I have a fixed one at home, another one in Briancon in France (solar powered and connected by SigFox!) and a mobile one I take with me during my trips over the world).

When I started few years ago, I didn’t know that it will become such a fun project!

Updates

[2019/02] Update of installation script and new links
[2019/05] Update of the Radar page that now use my own positions database. Added aircraft silhouettes and Airlines logo.
[2019/05] Changed the domain to FoxtrotCharlie.ovh !!!
[2019/06] Changed landing page and reworked index for Google
[2019/07] Added Great Circle tracks between origin, destination and seen positions on Radar
[2019/09] Added Airports data, API page, better sitemap for Google index
[2020/01] Added a solar powered ADS-B tracker in Briancon, with a SigFox interface by SNOC to send new / special planes to this database when Wifi is off – works great!
Running an ADS-B tracker on a solar panel is quite challenging. After some research, I believed a 20W solar panel would be enough, but the stick consumption is very high, and solar energy production is far from stable… after few month using it, I would need about 80W to be able to power the system for 24h (Raspberry Pi 4 + Flightaware Pro Stick Plus + SNOC SigFox). SigFox is a great solution to replace a 4G modem which would also take too much energy. The limitation to 6 messages per hour is manageable, as I am only reporting new or special planes to this website, while the detailed data are stored and shared when the device has access to wifi.
The biggest issue I didn’t plan for was that the Pi doesn’t have a RTC, so time was wrong (it only progress when the Pi is on, about 8 to 12h per day… after few days, it was completely off). Hopefully, I could also use the downlink SigFox messages to get the network time and update the Pi time at each boot. Here the limitation is at 4 messages per day. It is controlled by the callback feature so it is quite easy to be sure you will not exceed the budget, even with a Pi with no idea what is current date or time. Next step is to work on shuting down properly the Pi when battery is low in order to avoid memory issue. So far it works with multiple wild shutdowns per day, but I don’t know for how long.
[2020/02] Updated the stats page to show more details
[2020/03] Added tweets on Sigfox interface with Briancon tracker
[2020/07] Working on adding MoPi-2 on the solar powered tracker to allow clean power down. Changed all the time in UTC. With trackers everywhere, it doesn’t make sens anymore to be on Hong Kong time.
[2020/09] Added a new blog. Let’s hope I will post regularly from now on![2020/10] API optimisation – reduced the errors from 2% to 0.05%
[2020/11] New landing page with a RSS feed reader to get up to date links to my blog posts. Let’s see the impact on Search ranking!
[2020/12] Upgraded all my Raspberry Pi code to Python3. I have been delaying that for a long time, but it was much easier than expected. Further optimisation of the API to reduce the error rate further. Some trackers are now at 0 errors, some still at 0.05%, still struggling to understand why.

Interesting links on ADS-B trackers

Sharing my database of Aircraft, Airports and Routes

I have been gathering data for few years now, from multiple sources (databases found online, some special aircraft found manually online, …). I now have more than 32,500 entries in my Aircraft database, including 6,000 that I flagged as “special”, meaning military or any other interesting features (helicopters, rare aircrafts …).

I also have more than 16,000 airports and 160,000 routes.

They are all accessible through dedicated API.

Search aircraft by Mode S code or tail number

You can access my database of aircraft through the following API:
https://www.foxtrotcharlie.ovh/API/public/api?ModeS=XXXXXX
where XXXXXX is the ModeS code of the aircraft, on 6 digits in hexadecimal.
https://www.foxtrotcharlie.ovh/API/public/api?Registration=XXXXXX
where XXXXXX is the tail number (or registration) of the aircraft (in some countries, tail numbers have a – like in China B-8638, it should be put here).
The output is in json format (RFC4627).
Working samples of the API:
https://www.foxtrotcharlie.ovh/API/public/api?ModeS=78102f
https://www.foxtrotcharlie.ovh/API/public/api?Registration=B-8638

Returned data:

      {
          "ModeS":            "78102f",
          "ModeSCountry":     "China",
          "Registration":     "B-8638",
          "ICAOTypeCode":     "A321",
          "Type":             "Airbus A321-211",
          "SerialNo":         "---",
          "RegisteredOwners": "China Southern Airlines",
          "OperatorFlagCode": "CSN",
          "FirstSeen":        "2017-03-25 23:58:09",
          "LastSeen":         "2019-09-06 07:25:41"
      }
  
DesignatorDescriptionFormat
ModeSModeS code in HEX format of the Aircraft6 characters
ModeSCountryCountry of registration of the Aircraftstring
RegistrationTail Number of the Aircraftstring
ICAOTypeCodeICAO Type Code of the Typestring
TypeType of Aircraftstring
SerialNoSerial Number of the Aircraftstring
RegisteredOwnersRegistered Owners of the Aircraftstring
OperatorFlagCodeCode of the Airline if anystring
FirstSeenTime stamp when my trackers first detected the AircraftTimestamp
LastSeenTime stamp when my trackers last detected the AircraftTimestamp
Json structure description for Aircraft API

Search airports by ICAO code

You can also access my database of airports through the following API:
https://www.foxtrotcharlie.ovh/API/public/api?ICAO=XXXX
where XXXX is the ICAO code of the airport on 4 characters.
The output is in json format (RFC4627).
Working samples of the API:
https://www.foxtrotcharlie.ovh/API/public/api?ICAO=VHHH

Returned data:

      {
          "ICAO":       "VHHH",
          "IATA":       "HKG",
          "Name":       "Hong Kong International Airport",
          "Location":   "Hong Kong",
          "Region":     "Hong Kong",
          "Latitude":   22.3089,
          "Longitude":  113.915
       }
DesignatorDescriptionFormat
ICAOICAO code of the Airport4 characters
IATAIATA code of the Airport3 characters
NameName of the Airportstring
LocationLocation of the Airport in clearstring
RegionRegion or Country of the Airportstring
LatitudeLatitude of the Airportfloat
LongitudeLongitude of the Airportfloat
Json structure description for Airport API

Search routes by Flight Number (ICAO format)

You can also access my database of routes through the following API:
https://www.foxtrotcharlie.ovh/API/public/api?FNB=XXXXXX
where XXXXXX is the flight number in ICAO format.
The output is in json format (RFC4627).
Working samples of the API:
https://www.foxtrotcharlie.ovh/API/public/api?FNB=AFR188

Returned data:

      {
          "FNB":            "AFR188",
          "Airline":        "AFR",
          "FlightNumber":   "188",
          "Origin":         "LFPG",
          "Destination":    "VHHH",
          "Via":            ""
       }
DesignatorDescriptionFormat
FNBFlight number in ICAO formatstring
AirlineICAO code of the airline4 characters
FlightNumberFlight numberstring
OriginOrigin airport (ICAO format)string
DestinationDestination airport (ICAO format)string
ViaVia airport – if any (ICAO format)string
Json structure description for Airports API

Disclaimer

All the information provided by these API are provided as is and for information only. They are coming from different sources and may contain mistakes or be out of date. Please use with caution and at your own risk.

You are welcome to comment if you see mistakes. Mode S codes can be re-allocated to new aircraft, and keeping the database up to date is not an easy task.

ADS-B tracking with an off grid Raspberry Pi

I have been looking for a solution to do ADS-B tracking off grid for quite a while. I have several Piaware trackers in different setup (fixed one at home, mobile one with GPS in the car, mobile one that I take with me when I travel abroad), and I wanted to add some in places where I don’t have power or wifi. After some research I decided to use a solar panel for power and a SigFox hat for connection. The SigFox hat has limited data throughput (6 messages per hour in uplink), but it is enough to report new or interesting aircraft to my server.

There is a lot of literature on solar power for the Pi, but I still ended up underestimating the size of the panel. 20W just gives me 8h per day in average. I will need to upgrade to 100W to make sure I run 24/24 7/7 365/365. The BIG power consumption comes from the RTL-SDR dongle, and you can not toggle it on and off to save power as it needs to be on for an extended period of time to do its job. Having my tracker going on and off created an unexpected issue that I didn’t see during my debug session with wifi: as the Pi does not have a RTC, the Pi time is losing 16h per day in average.

Rapidly, my timestamp for the planes detected were off. Hopefully, as SigFox allows also 4 downlink messages per day, I am able to request a time update through this channel. With some adjustment to make sure I don’t use my downlink messages budget when the Pi powers up just for few minutes when battery is low but sun is high, I was able to make that work. Next step is to work on improvements for the battery management and I will have a fully independent solution. I now have a Piaware tracker running fully off grid!


You can find more details on my setup and the data reported by my trackers on https://www.foxtrotcharlie.ovh


When I started this project I didn’t know how far it would bring me! My long term goal is still to share all my software on Github, but that will need some cleaning!

Also published on www.raspberrypi.org

How to build an ADS-B tracker

In order to build your ADS-B tracker you will need just few items: a Raspberry Pi, a RTL-SDR dongle, an antenna, and some free time! For the tracker, I am using a Raspberry Pi.

I started with the A version, but it was a little limited (it was often at 100% CPU load, running dump1090, Piaware and my own software). I then moved to the B+ version, and I was down to 10% which was much better. I now run a Raspberry Pi version 4.

To setup your tracker, follow these steps:

WARNING: AS THINGS MAY EVOLVE, YOU MAY NEED TO ADAPT THE TUTORIAL TO YOUR HARDWARE AND SOFTWARE VERSIONS

– First you need to download the latest NOOBS from Raspberry Pi website, onto a SD card formated in FAT (your Raspberry Pi is not recognizing FAT32 format). I do recommand to use minimum 32 GB of flash, specially if you want to have some space for recording positions of the aircraft you will track. If not, 8 GB is enough.

– Insert the SD card into the slot on the Raspberry Pi and power the board. Then follow the instructions. For this step, you will need a keyboard, a mouse and a TV with HDMI input.

– When it is done, upgrade the Raspberry Pi software and firmware:

        sudo apt-get update
        sudo apt-get upgrade
        sudo rpi-update

– Install tightvnc:

        sudo apt-get install tightvncserver
        tightvncserver

– You now have a working Pi, that you can manage from a remote computer with a VNC software (I am using tightvncjviewer.jar on Mac)

– Next step is to install the ADSB tracker. For this, you will need a SDR (Software Define Radio) dongle, that you will plus to your Pi. I am using both that one and Piaware Pro-stick. The pro-stick has an integrated amplifier that will give you some additional range. Now there is a new version with integrated filter which is working great. I recommend to switch to one of the Piaware Pro-sticks as they consistently show better results than other sticks that I have tried over the years.

Install Piaware – that will give you a free access to cool features from Flightaware at the same time. You can first create credentials on Flightaware, to get a USERNAME and a PASSWORD.

        #install piaware
        wget https://flightaware.com/adsb/piaware/files/packages/pool/piaware/p/piaware-support/piaware-repository_4.0_all.deb
        sudo dpkg -i piaware-repository_4.0_all.deb
        sudo apt-get update
        sudo apt-get install piaware
        sudo piaware-config allow-auto-updates yes
        sudo piaware-config allow-manual-updates yes
        sudo apt-get install dump1090-fa

– You now have a working tracker and you can see your data on Flightaware. This is already quite fun.

See what else you can do from there on foxtrotcharlie.ovh.