Featured

Having fun with my ADS-B tracker project

How it started

Some years ago, I bumped into an article on Mashable about Jeremy Merrill using a Raspberry Pi ADS-B tracker to detect aircraft flying over his house and display the origin or destination of that plane (see article here and github there). As my place is just on a busy plane corridor, that gave me an idea. I decided to work on a similar project based on Piaware. Looking at how he did it and at the long literature you can find on the web on ADS-B, I created this site, my own software in Python to run on my Pi and other fun stuffs. I also ran into this site by SonicGoose and that gave me many new ideas (including the basic structure of this site – I never had any experience of html or php before).

One key part is getting access to aircraft databases to get more info from the ModeS hex code you get from the Piaware tracker. I am sharing my own database of planes, airports and routes I detected with one of my trackers (I have a fixed one at home, another one in Briancon in France (solar powered and connected by SigFox!) and a mobile one I take with me during my trips over the world).

When I started few years ago, I didn’t know that it will become such a fun project!

Updates

[2019/02] Update of installation script and new links
[2019/05] Update of the Radar page that now use my own positions database. Added aircraft silhouettes and Airlines logo.
[2019/05] Changed the domain to FoxtrotCharlie.ovh !!!
[2019/06] Changed landing page and reworked index for Google
[2019/07] Added Great Circle tracks between origin, destination and seen positions on Radar
[2019/09] Added Airports data, API page, better sitemap for Google index
[2020/01] Added a solar powered ADS-B tracker in Briancon, with a SigFox interface by SNOC to send new / special planes to this database when Wifi is off – works great!
Running an ADS-B tracker on a solar panel is quite challenging. After some research, I believed a 20W solar panel would be enough, but the stick consumption is very high, and solar energy production is far from stable… after few month using it, I would need about 80W to be able to power the system for 24h (Raspberry Pi 4 + Flightaware Pro Stick Plus + SNOC SigFox). SigFox is a great solution to replace a 4G modem which would also take too much energy. The limitation to 6 messages per hour is manageable, as I am only reporting new or special planes to this website, while the detailed data are stored and shared when the device has access to wifi.
The biggest issue I didn’t plan for was that the Pi doesn’t have a RTC, so time was wrong (it only progress when the Pi is on, about 8 to 12h per day… after few days, it was completely off). Hopefully, I could also use the downlink SigFox messages to get the network time and update the Pi time at each boot. Here the limitation is at 4 messages per day. It is controlled by the callback feature so it is quite easy to be sure you will not exceed the budget, even with a Pi with no idea what is current date or time. Next step is to work on shuting down properly the Pi when battery is low in order to avoid memory issue. So far it works with multiple wild shutdowns per day, but I don’t know for how long.
[2020/02] Updated the stats page to show more details
[2020/03] Added tweets on Sigfox interface with Briancon tracker
[2020/07] Working on adding MoPi-2 on the solar powered tracker to allow clean power down. Changed all the time in UTC. With trackers everywhere, it doesn’t make sens anymore to be on Hong Kong time.
[2020/09] Added a new blog. Let’s hope I will post regularly from now on![2020/10] API optimisation – reduced the errors from 2% to 0.05%
[2020/11] New landing page with a RSS feed reader to get up to date links to my blog posts. Let’s see the impact on Search ranking!
[2020/12] Upgraded all my Raspberry Pi code to Python3. I have been delaying that for a long time, but it was much easier than expected. Further optimisation of the API to reduce the error rate further. Some trackers are now at 0 errors, some still at 0.05%, still struggling to understand why.

Interesting links on ADS-B trackers

The Mashable article that started everything.
Flyover github, a great place to start.
Piaware from Flightaware.
Raspberry Pi
SonicGoose, that inspired me to create my own database of aircraft, flights, airports…
ADSB-Exchange which has great APIs
The very good adsb-receiver on github
The awesome OpenSky, with it’s great interface
The excellent AboveTustin, if you want to tweet the planes flying over
And for sure Foxtrotcharlie!

2to3, Python 2 to Python 3 made easy!

Do I need to upgrade from Python 2 to Python 3?

I started to develop my ADS-B tracker software in Python 2 and I delayed the upgrade to Python 3 for a long time. I was thinking it would be pretty difficult! Now that PIP is soon to drop support of Python 2.7, I finally got to it. It was much easier than I expected thanks to 2to3!

The magic of 2to3

You don’t really need to know much about Python 3 to do the upgrade! Just copy all your files in a new Python 3 directory, make sure you have installed all the Python 3 packages you need and use 2to3 and it is done!

cp project_master project_master_py3
cd project_master_py3
sudo apt-get install python3 python3-pip <<your Python 3 packages>>
python3 -m pip install <<your packages>>
2to3 -w *.py

It should almost work…

A little bit of tuning is needed

You may still have some errors due to 2 things:
– Python 3 does not like when mixing space and tab for indentation. It is quite easy to solve by being consistant. Use only spaces or only tabs, whatever is best for you (I don’t have a religion there!).
– Python 3 has a different way to manage strings and bytes. That one is trickier. I solved it by adding 2 functions to change the variable to bytes or to string when needed:

#fn to_bytes
def to_bytes(arg):
if hasattr(arg, 'encode'): return arg.encode('utf-8')
if hasattr(arg, 'decode'): return arg
return repr(arg).encode('utf-8')

#fn to_string
def to_string(arg):
if hasattr(arg, 'encode'): return arg
if hasattr(arg, 'decode'): return arg.decode('utf-8')
return repr(arg).decode('utf-8')

I just ran my program until it detected all the instances where bytes and strings were mixed.

That’s it, it did the job. My trackers are now running Python 3, and still connect to my online database, tweet when detect new or interesting planes, connect through SigFox when off grid,… as they were doing in Python 2. I didn’t notice an impact on the error rate so far.

A DC-3 landing in Hong Kong

Another interesting catch by my ADS-B trackers in Hong Kong has been a Douglas DC-3 with tail number N41CQ on December 4th, 2019. This rare aircraft made a special trip from Hong Kong to Beijing via Guangzhou and Shanghai, to celebrate the 70th anniversary of the CNAC (China National Aviation Corporation) initial flight.

DC-3 details
Hex codeA4D208
Tail NumberN41CQ
TypeDouglas DC-3

12 free tools for website optimization

After developing my ADS-B trackers and the central database of aircraft, I wanted to share the results to the world. What’s best than having my own website for this? Over the years, I discovered these 12 tools to test and optimise my website. There are few directions which are important for website optimization: make sure your website is known by the main search engines, make sure your SEO meta data is right, make sure your website load fast enough and solve all your technical issues.

Website Optimization for SEO

Browseo
Browseo is a great free tool to make all the SEO data visible. It will show the title, meta tags, and other invisible data that are key for SEO just to make sure you got them right. At first, I didn’t see what it was useful for, because I was just checking my main index page, but when I used it to test my php pages, with dynamic meta tags, that was super useful to see what’s wrong with those tags!


Seositecheckup
Seositecheckup is a great free tool to analyse all the SEO data of your website and highlight what is missing. The free version can only scan your website once a day, but that let’s you some time to solve all the issues before checking them again.


Brokenlinkcheck
Broken links are impacting your SEO ranking, but are hard to find. Brokenlinkcheck will find them for you. Free for 2000 links. If you are using WordPress there are some dedicated plugins for this.

Make sure you are known to main search engines

Bing, Google, Yandex are the one to start with. Create an account, give them your URL and they will start to crawl your website. Check the result of those crawls regularly as they will report errors. Browseo (see above) is a great tool to understand canonical url issues. Regarding sitemaps, there are 2 philosophy: let them discover your site by themselves or use a very detailed one. I went with the first one and only put the main pages in my sitemap.

Website Optimisation for Speed

gtmetrix and Google Pagespeed Insight are two great free tools to understand what is making your website slow to load. Solving the issues requires some technical knowledge, but some tricks are very useful: optimise your images and videos, use a CDN, use Gzip compression and optimize PHP code.

Solve technical issues

Dnschecker
Have you recently switched web host or started a new website, then you are in the right place! DNS Checker provides free DNS lookup service for checking domain name server records against a randomly selected list of DNS servers in different corners of the world. Do a quick DNS propagation lookup for any domain name and check DNS data collected from all location for confirming that the website is completely propagated or not worldwide.


Immuniweb
Website security testing (GDPR compliance, CSH and HTTP headers check, common Javascript libraries version check…). Make sure you have all the security basics right!


Onlineconverter
A free tool to convert any image into the format you need for your website.


DownNotifier
A free tool to check if your website is offline.

Have fun!

US NAVY using ADS-B code hopping

US Navy P-8 Poseidon taking off at Perth Airport Darren Koch (GFDL 1.2 or GFDL 1.2), via Wikimedia Commons

Recently, I started to detect many P8 Poseidon aircraft above South China Sea, like if a dozen of different P8 were flying in the area each day. The only explanation I see is that they are doing ADS-B code hopping, switching to a new TACTICAL code every few minutes.

Here are some of the codes I detected in the past few days: AE686A, AE6879, AE6889, AE67B3, AE6784, AE687D, AE6850, AE6854, AE6809, AE6793, AE6807, AE6845, AE6819, AE6836, AE67C2, AE67DC, AE6885, AE67D6, AE6813, AE687E, AE689E, AE6884, AE67DE, AE6803, AE682D, AE687C, AE6849, AE68A2.

Few more codes detected recently: AE6848, AE67DF, AE6870, AE6883, AE683D, AE6822, AE6814

3 tips to optimize PHP code

How I divided my server errors by 40 with PHP code optimization

I have been running several Raspberry Pi based trackers for a while, and I wanted to create a common database of detected aircraft and their positions. I looked for online database solution and how I could connect my Pi to them directly in Python, but that was not convenient.

I finally decided to get an hosting plan at OVH and create my own databases, with API to add more aircraft and positions. I wrote all the APIs of my server in PHP, as it was pretty straightforward. I got it work pretty well, but I was still having quite a lot of errors. At first, it was simple, I played a little with the timeout, I add retries, but I was still not able to go below 2% of errors. For a long time I lived with that, my system still worked well, but I finally decided to work on that and optimize PHP code.

The usual PHP tips to optimize PHP code…

A quick search on php optimization got me a list of things to do:

  • use single quote instead of double one: 'text' is better than "text"
  • calculate variable only once in a loop
  • close databases
  • order switch by order of use
  • select specific columns instead of * in mySQL SELECT queries
  • use foreach instead of while
  • use echo $a,$b,$c instead of echo $a.$b.$c
  • use sprintf to add strings: $string = sprintf('today is %s', $date)
  • use === instead of == in if: if ($a===$b) { action}
  • INSERT multiple values in database instead of multiple single INSERT

It seems that many of them were useful for older PHP versions, but not really for PHP versions over 7…

… and what really worked for me!

The 3 actions that had a tremendous effect on my API have been around database optimisation: first, I rewrote the few INSERT loops in order to INSERT multiple values at once. This was already much better. Then I checked my databases structures, to make sure I had all the right indexes corresponding to my SELECT queries. And finally, I switched all the heavy ones to InnoDB (I am not sure why OVH use MyISAM as default structure…).

These 3 actions reduced my error rate from 2% to 0.05% by reducing the load of my server (I am using a very basic hosted configuration, nothing fancy here). On the other tips I found, most of them had no or only very marginal impact. The only one I will keep, is the sprintf function to create my strings with variables as this makes my code much easier to read (it was easy to get lost in the quotes when you write mySQL queries with all that ‘ and ” and .).

How did I find that was my issues? What is important is to find where you PHP requests are spending time. For that, I simply implemented some time measurement inside the code (with the microtime function: $start_time=microtime(True); $stop_time=microtime(True); phpduration = $stop_time-$start_time;), that is coming back to my Pi with the result of the API. I could then dig into the one loading the server for more than 1s and reduced their load. I did the same in my Pi, to track which mySQL queries were taking more than 0.001s to run, and that was very efficient.

I was able to focus on optimising the most important ones (mainly those on huge database, such as positions, which rapidly contain millions of entries), and optimise their structure and indexes. There are many tools available to profile your PHP code, but I didn’t find any that was easy enough to start with compared to what I needed them for.

With 0.05% error rate, I can think about adding more trackers and maybe even have a way for others to share their tracking data with me! Let me know if you would share yours or if you have more tips to optimise PHP code…

What else can help reduce connection errors?

I also did several actions to reduce connection errors to my API server:
– make sure you don’t have any redirection
– make sure you updated requests and certifi on the Raspberry Pi, specially if you are using a secured connection. It can easily be done by:
pip install --upgrade requests certifi urllib3 idna chardet
– use session if you are calling the API regularly:
api_session = requests.Session()
resp = api_session.get([request]) will replace requests.get([request])
– set a different timeout for connection and response:
api_session.get([request], timeout=(5,10))

SigFox for Raspberry Pi

The hardware

First you need to find a Sigfox hat for your Raspberry Pi. Just Google “Sigfox for Raspberry Pi” and you will find the latest solution available. I am using a RPISIGFOX, made my SNOC. Pretty easy to setup and use. Just plug it in, change the serial port parameters, and it is ready to work. It comes with 1 year SigFox subscription that requires activation on Sigfox website. Depending on where you intend to use your Pi, you may need a different version of Sigfox radio (that one is mainly for Europe – US and Asia have different frequency bands for non regulated spectrum).


The Software

  1. Disable Raspberry Pi terminal on serial port with raspi-config utility: sudo raspi-config. Go to Interfacing Options then choose Serial then NO and OK
  2. Install pyserial: sudo apt-get install python-serial
  3. Download scripts: git clone https://github.com/SNOC/rpisigfox.git
  4. Enable script execution: chmod +x sendsigfox.py
  5. Plug antenna with its cable and send your first message: ./sendsigfox.py 0123

Setting up the Callbacks and your server

You also need to setup the callbacks on Sigfox system, and your server to manage the data. This is pretty straightforward with the documentation for the uplink messages. You are limited to 6 uplink messages per hour. On reception of the uplink message, the server will call your API, as you have defined in Sigfox interface. What you do with the data on your server is up to you!

Downlink messages are a little more complex to manage. You are limited to 4 downlink messages per day, and it was not very easy to make sure my Rasberry Pi application was remaining inside this budget, specially with a solar powered Pi and without a RTC … It took me some time to understand that the server was in fact the one making sure you stay in the budget, as if there is no downlink message to send, it won’t count into the 4 messages per day.

To request a downlink message, you just need to send ack=true with your uplink message.

On reception of the uplink request through the above API, your server will have to provide the answer: "[device ID]":{"downlinkData":"[data]"} if there is a downlink payload or "[device ID]":{"noData":true} when there is no payload to send. In that case, the downlink message will not count in your daily budget.

You can find more details on this great post on Sigfox downlink messages.

Keeping the Pi on time

As my Raspberry Pi is solar powered (with a 20W solar panel, which is enough for the Pi, but not for the RTL-SDR dongle that has a much bigger power consumption), I don’t necessarily have enough power to keep the Pi on 24/24. When it is off, the time stops to progress, as the Pi doesn’t have a RTC by default. After few days, time and date are completely wrong. That makes it difficult to control the downlink messages budget from the Pi. At the same time, SigFox is a great solution to get the network time to and update the time of the Pi. With the downlink messages, I was able to get the network time when the Pi is starting and update the date and time of the Pi with the simple command sudo date --set="9 JUN 2017 19:15:00".

How I use it?

With the payload limitations at 6 uplink messages per hour (12 bytes) and 4 downlink messages per day (8 bytes), Sigfox doesn’t really replace a Wifi connection, I cannot report all the detected aircrafts and their positions as I am normally doing. As, this is for an ADS-B tracker, I decided to focus on new aircraft and special ones (such as military) for the uplink direction, transmitting only the ModeS hex code and the timestamp at which it was detected, the remaining 4 bytes are used for maintenance purpose (HHHHHHYYYYMMDDHHMMSSXXXX).

To avoid going over 6 uplink messages per hour, each detection will go into a buffer, that is checked every 10 minutes. I have been working on some more compressed way to encode these data but practically I don’t need it, as I am very seldom over budget (meaning I am sending a message every 10 minutes as long as my Pi is powered by the battery). We will see if this is still the case after COVID-19 and the airlines start to have a more international flights. Each time my server receive an uplink message it will automatically generate a tweet (with the #Sigfox hashtag).

Downlink messages are used only to keep the time of the Pi in sync, as described before. As sometimes the Pi doesn’t receive the downlink message or is switched off again when battery is low and weather is grey, it may need more than one downlink per day for time sync. In addition of the network timestamp, I still have 2 bytes available to send other data to the Pi. I am using this for maintenance purpose (YYYYMMDDHHMMSSXX). Note that the uplink message must be exactly 8 bytes in size or it will not go through.

It has been a long time I wanted to play with a Sigfox hat, and it has been quite fun: not that difficult to make it work, but enough issues to solve to make that interesting!

Breitling Jet Team

Breitling Jet Team, operated by Apache Aviation is based in Dijon, a city in France few hundreds kilometres south east of Paris. My mobile tracker picked most of the team when I spent few weeks in Chalon our Saone early this year.

Breitling Jet Team is flying on Aero L39C Albatros, with tails ES-YLX, ES-YLN, ES-YLF, ES-YLP, ES-YLR, ES-TLF, ES-TLG and ES-YLI. You can see more details in the search page.

Breitling has decided in 2019 not to renew its sponsorship as sole and exclusive partner of Apache Aviation, but is open to staying on board as co-sponsor. Apache Aviation and Breitling are in search of co-sponsors to operate Jet Team around the globe.

Founded by Jacques Bothelin in 1980, the team had several sponsors (and names) during the years: Patrouille Martini, Patrouille Ecco, Patrouille Addecco, Khalifa Jet Team before to be named Breitling Jet Team in 2003.

Let’s hope they will find a new partner to continue this adventure!

US military planes in South China Sea

There are a lot of US military activity in the area between Taiwan, Hong Kong and the South China Sea. Transports, SIGINT, reconnaissance, protection of US carriers in the area… mainly conducted by the US Air Force and the US Navy. Here is a list of planes seen in past few months (click on the link to see the radar map when the trackers got a MLAT position – these aircraft generally do not broadcast their position on ADSB):

AE049B – Tail number: 61-0313 – Boeing KC-135R – US Air Force

AE1492 – Tail number: 96-0042 – Boeing E-8C Joint Stars – US Air Force

AE1D8A – Tail number: 15-6511 – Lockheed EP-3E Aries 2 Orion – US Navy

AE1D95- Tail number: 15-9893 – Lockheed EP-3E Orion – US Navy

AE1D91- Tail number: 15-7318 – Lockheed EP-3E Aries2 Orion – US Navy

AE1D5D – Tail number: 16-1586 – Lockheed P-3C-II.2 Orion – US Navy

AE17EF- Tail number: 05-0730 – Boeing C-40C – US Air Force

AE04FD – Tail number: 58-0095 – Boeing KC135T – US Air Force

AE01CE – Tail number: 62-4134 – Boeing RC135W Stratotanker SIGINT – US Air Force

You also can see the latest detected military and other aircraft of interest and the pick of the day or search my database on foxtrotcharlie.ovh.

SAM727 flying over Guangzhou with Secretary Pompeo

Today (2020-10-30), the Boeing C-32A with tail 99-0003 was spotted by my trackers over Guangzhou with the CallSign SAM727.

A quick search shows a departure from Hanoi, and a current position over Japan. Probably used by Secretary Pompeo who was visiting Hanoi yesterday.