I don’t like warnings in log files. Even if there’s no real impact—I guess it’s kind of an OCD thing. IT-OCD. I just invented it.
I see something in yellow or red, I want to fix it. I have to. MUST…FIX…
You know what bugged me for ages? Time sync-related errors and warnings, primarily on my Domain Controllers. Event ID 129 and its friends.
There are loads of ways to solve it easily, but only temporarily because of the dreaded time drift. A permanent solution is a network time-server, also called a Stratum 1 source. I looked into these turnkey NTP boxes and thought, “You want how much for that thing?”
Obviously, not an option for a home lab. So, what now?
Oh look, there’s a Pi looking bored out of the window
I did some research on how to build my own and felt totally empowered. But I wasn’t prepared for the nightmare incoming. This turned into quite some project, which took me weeks to get where I am now, hence the long blog post.
The basics are easy: you need a Raspberry Pi, a GPS antenna, and a bit of software. Nice. LDT!
(pic of the final product)
My first attempt was the lazy way; I just bought an all-in-one USB GPS antenna for 15 Euro, plugged it in, and followed one of the million tutorials on the internet. Didn’t work. Another tut, didn’t work. Rinse, repeat. More research.
And that’s where the first challenge starts. Multiple different Pi versions exist, as well as antennas, so loads of the tutorials are outdated, or just for a different setup. Mine is for a Pi4, btw.
Further, I discovered Pis don’t come with real-time clocks because it would make them too expensive. So, I need something with an RTC, battery-driven, plus the GPS receiver.
And that USB thing never worked; I blamed myself for buying cheap.
So, the next thing I bought is called “Ultimate GPS Hat,” and it does, in fact, come with a built-in battery-buffered RTC and an antenna connector. Obviously, I needed a new antenna, too.
Next, I learnt about different antenna connectors, like SMA, RP-SMA, U.FL, and others—primarily because I kept buying the wrong adapters. If anyone need spare parts, let me know.
But one day I finally had everything together, but it still didn’t work. What?!
I couldn’t get “a fix.”
In this context, that’s the satellite link required to pick up the time. In one of the tutorials, I read about placing the antenna towards a window if possible. By that time I was already so desperate, I just placed it outside an open window.
And guess what, I got my fix.
Obviously, that’s no permanent solution. I walked around my apartment from outside and noticed a vent by the living room, which I never noticed from inside, as it’s almost inside the window bar. Good enough for me, so I ordered an antenna extension, of course the wrong one first, but the second try worked.
I just glued it to the outside wall. Doesn’t look nice, but whatever!
Long story short, I have a perfectly fine, running Stratum 1 time source in my network, and here are the steps.
First, we need to prep the Raspberry
I used the minimal version for headless deployment and copied an empty file called SSH into the root of the SD card.
Once booted up, check your DHCP for the latest IP and connect via SSH, and run
sudo raspi-config
I changed the hostname (option S4) and time zone (L2), minimised the GPU memory (P2) and extended the file system (A1) as usual, but we need an additional step here: the Serial Port (option P6) needs to be changed, too, with a “no” followed by “yes,” to make sure the GPS hat gets exclusive access.
Now reboot the Raspberry… The first of many times.
The next thing is to adjust some locale settings to prevent issues with packages later:
sudo nano /etc/default/locale # File generated by update-locale LANG=en_GB.UTF-8 LANGUAGE=en_GB:en LC_ALL=en_GB.UTF-8
Now it’s time to give the device a static IP:
sudo nano /etc/dhcpcd.conf interface eth0 static ip_address=10.0.20.9/24 static routers=10.0.20.1 static domain_name_servers=10.0.20.10 10.0.20.20
It doesn’t hurt to add the search domain:
sudo nano /etc/systemd/resolved.conf Domains=home.XXXXX.XX
And make sure it’s pointing to itself as a time source:
sudo nano /etc/systemd/timesyncd.conf NTP=10.0.20.9
Time for another reboot. Don’t forget to connect to the new IP address from now on!
Now, if you followed the steps so far, you’ll notice Wi-Fi is still disabled, and that’s great as we don’t need it. Let’s do the same with Bluetooth:
sudo rfkill block bluetooth
And run updates:
sudo apt-get update && sudo apt-get upgrade -y sudo rpi-update
Reboot number three.
You probably want to monitor that thing, don’t you? So let’s set up SNMP as well, plain and simple:
sudo apt install -y snmpd snmp sudo rm /etc/snmp/snmpd.conf && sudo nano /etc/snmp/snmpd.conf agentAddress udp:161,udp6:[::1]:161 view all included .1 rocommunity public default -V all rocommunity6 publicdefault -V all sysLocation Berlin sysContact not@yourbusiness
So far, we did everything with the default Pi user. That’s fine, but if you prefer root, you’ll need these steps:
sudo passwd root
Give yourself a password and run a change here:
sudo nano /etc/ssh/sshd_config #PermitRootLogin without-password —> PermitRootLogin yes
Remember me talking about OCD? Let’s clean up:
sudo apt autoremove -y
And shutdown!
No reboot this time. Yay!
We have a perfectly fine, prepared Raspberry, now we need to attach the hat—don’t forget to insert the battery first—and plug in the antenna, then boot it up again.
Install the GPS clients:
sudo apt-get install gpsd gpsd-clients
And you should see something in
cat /dev/serial0
But the content won’t make much sense, so we need to reroute the serial traffic.
sudo systemctl stop gpsd.socket && sudo systemctl disable gpsd.socket sudo gpsd /dev/serial0 -F /var/run/gpsd.sock
Next, we install the precision time packages:
sudo apt-get install python-gps pps-tools
File change:
sudo nano /boot/config.txt #GPS time changes enable_uart=1 #Disable Bluetooth for serial dtoverlay=pi3-disable-bt #Get 1PPS from HAT pin - your pin could be a different one dtoverlay=pps-gpio,gpiopin=4
In addition, check this file and delete any entry that looks like “console=serialXXXX”
sudo nano /boot/cmdline.txt
And verify there’s an entry called DEVICES="/dev/serial0" in
sudo nano /etc/default/gpsd
Enable gpsd at boot:
sudo systemctl enable gpsd
Next: Reboot number…Idk.
Now, the moment of truth.
Run this guy:
sudo gpsmon
For the first time, you will see loads of ,,,, between the lines, but they’ll get less and less depending on how quickly the fix comes in. With the antenna outside, it shouldn’t take more than a couple of minutes.
Another nice way to check is:
sudo cgps -s
Now you know where I’m located. Bring red wine if you ever visit me—I prep pasta!
Back to checking the time source:
sudo ppstest /dev/pps0
At this point, we have a local time source, but we need to make it available for other devices. You still with me?
sudo apt-get install chrony -y && sudo nano /etc/chrony/chrony.conf
Now, inside the file, look at the default NTP pool servers. We want to change it to something as close to us as possible for fallback. Here is a list https://www.ntppool.org/zone/@—just click continent, country, city, etc., to find a suitable pool and change it. Also, we need to point to our own clock as reference and allow network access.
My file looks like this:
# Welcome to the chrony configuration file. See chrony.conf(5) for more # information about usuable directives. pool de.pool.ntp.org iburst refclock SHM 0 poll 3 refid GPS refclock PPS /dev/pps0 refid PPS lock GPS allow 10.0.0.0/8 # This directive specify the location of the file containing ID/key pairs for # NTP authentication. keyfile /etc/chrony/chrony.keys # This directive specify the file into which chronyd will store the rate # information. driftfile /var/lib/chrony/chrony.drift # Uncomment the following line to turn logging on. #log tracking measurements statistics # Log files location. logdir /var/log/chrony # Stop bad estimates upsetting machine clock. maxupdateskew 100.0 # This directive enables kernel synchronisation (every 11 minutes) of the # real-time clock. Note that it can’t be used along with the 'rtcfile' directive. rtcsync # Step the system clock instead of slewing it if the adjustment is larger than # one second, but only in the first three clock updates. makestep 1 3
Reboot. Yes, I know. Then run:
chronyc sources
The *proves that we have a precise time source. Nanoseconds, phew! We also see the variation to the regular GPS signal, and some of the public NTP servers located in Germany.
Time is on my side!
In theory we’re done now, but how do we distribute the time and verify?
For network devices, it’s usually a setting somewhere, and all you have to do is to enter the chosen IP address. The same applies to vCenter and ESXi. My vCenter was a little stubborn, but there’s a KB over at VMware.
In Linux, Ubuntu as an example, we need to edit one line in:
nano /etc/systemd/timesyncd.conf
Uncomment NTP and add your IP address, then restart and verify the service with:
systemctl restart systemd-timesyncd systemctl status systemd-timesyncd
Good, but what about Windows?
The quick solution is this command:
w32tm /config /syncfromflags:manual /update /reliable:yes /manualpeerlist:”10.0.20.9”
But we won’t run this on each individual machine. So let’s do it the proper way with group policies!
Unfortunately, it requires a few steps if there’s more than one DC. There’s a great explanation on the topic here for you to follow:
https://theitbros.com/configure-ntp-time-sync-group-policy/
In a nutshell, it requires two GPOs and a WMI filter. No rocket science, it just takes a while for the steps and the policy replication.
To verify, run this command on a domain member a little later with the IP address of your DC:
w32tm /stripchart /computer:10.0.20.10 /dataonly /samples:5
The difference in time between the local machine and the DC should be neglectable, and is mostly caused by network latency. I can live with that :)
Oh boy, I was so happy as it finally worked, I drank a bit of the precious bottle Midleton Very Rare which I took with me as I left Ireland. I deserved it.
Take care!
