About nw3weather
Weather Station
Technical
I run a Davis Vantage Pro 2 station and use Weather Display software to collect the data in real time from the sensors, which is then broadcast to this website. Each sensor polls for data every 3-20s, and is available live on nw3weather with a delay of approx 5s, with aggregated data (e.g. daily mean, monthly max, graphs) calculated and cached every 1-5 minutes. At midnight, daily aggregations of the data are saved to the permanent historical data-sets using high-tech csv databases.
It is, however, important to note that these various updates will not always be carried out due to the large number of
potential failure nodes in the system - sensors (power, wireless signal), local PC (HDD failure, software crash, internet connection),
and the web server. I have been lucky not to suffer any serious hardware failures (most failures result in no physical data loss), but undoubtedly the time will come.
The current uptime status can view on the system page.
Accuracy
The data comes from a personal automatic weather station (PWS/AWS) so there can be no guarantees of accuracy or reliability,
and there is no immediate quality control, but I do intervene to fix glitches
or suspected misreadings and the setup shown below is designed to provide the most accurate readings possible.
For temperature and humidity readings this means shielding the sensors from
insolation and providing them with good airflow;
for rain data the sensor needs to be in as open a place as possible to avoid rain shadow, but not too open since a high wind can cause problems;
and for wind readings the sensors should be 10m above ground and clear from obstructions (to provide a 'clean' flow of air).
In an urban environment these conditions are difficult to meet, and this must be considered when using data from such sites,
especially when comparing them to non-urban sites.
A more in-depth guide to siting weather stations can be found here.
It should also be noted that at no point do I quote figures for the errors associated with my readings, though of course they technically do exist. This is partly due to conventions in displaying weather data, but also because it can be very difficult to ascertain such values, especially for urban sites: For a given sensor the errors can vary widely from day to day and are sensitive to quite specific factors. All I can say is that I have tried to minimise and correct for errors where possible, but some large errors may well exist.
Location
The nw3weather station is situated on the south-eastern edge of London's Hampstead Heath, a 790 acre ancient park just a few miles north of central London. Nearby areas include Hampstead village to the west, Highgate to the north, Belsize Park to the south-west, and Kentish Town to the south.
The station co-ordinates are approx. 51.556, -0.155, which is ~57m (195ft) above mean sea level.
Setup
The station comprises several sensors:- A combined thermometer/hygrometer housed in a home-built Stevenson Screen (a radiation shield)
- A combined anemometer & wind vane, located atop a 6m pole attached to the chimney of the house
- A self-tipping rain gauge with a tip resolution of 0.2mm, which is attached to the roof of the Stevenson Screen
- A barometer integrated into the receiving unit (indoors).
The external sensors are wired to a solar-powered transmission unit (which sits atop the radiation shield box). This wirelessly transmits the data from all sensors to the internal console every few seconds. Battery-backup and a data-logger ensure continuous operation
Weather station albums are also available, with more complete and detailed pictures of past and present setups (note that the above pics are of a previous station).
Additionally, there are two webcams used to monitor sky and ground conditions, which along with traditional techniques (pending AI assistance) I use to collect data for extra weather variables - sun hours, cloud cover, wet hours, snowfall, lying snow, fog, hail, and thunder(storms).
History
I started collecting data at this site on 28th July 2010. Prior to that I had been running the same setup in East Finchley, a town 5 km north of here, since February 2009, but I have kept daily temperature records since March 2008 (albeit not integrated into this website). Consequently, any records from before July 2010 are for East Finchley, though given the small distance involved, any data comparisons are largely valid.
The sky-facing webcam (Logitech C300) was added to the station on 1st August 2010; The ground-facing one on 19th December 2010.
In 2016 the original Oregon Scientific WMR-200 sensors began to fail due to age. In Feb '16 the wind sensor failed, followed by the T/H sensor in Sept '16 (replaced a month later), and the rain sensor in March '17. In Sept '17 I therefore replaced the entire station with a Davis VP2, a better model.
In June 2018 I installed a new high-resolution network camera to replace the original webcam.
Technical history
- The rain gauge was modified in March 2009 - the original had a resolution of only 1mm. This involved fitting a funnel to the collecting channel so that less rain is needed to cause a 'tip' in the internal bucket (each tip of 10ml results in a signal being sent to the receiving unit, which registers the rain).
- In July 2009 I built the radiation shield for the temp/hum sensor to improve the accuracy of the readings: The walls are double louvred hardwood, the roof is double insulated plywood with a cavity for air flow, and the whole thing is painted in white gloss. Maintenance is carried out twice a year, mainly to keep the paint glossy for maximal radiation reflection.
- The combined wind sensor unit was mounted on a 6.1m aluminium-alloy aerial pole in August 2009.
- A manual rain gauge was installed on 4th September 2011 - this is for the verification and calibration of the automatic gauge data.
- The automatic rain gauge was further modified on 30th March 2014 to a new resolution of 0.18 mm.
- The base and roof of the radiation shield were replaced on 30th March 2014 owing to decay of the originals.
- The entire weather station was replaced on 5th September 2017 due to failure of the original.
Website
The site was developed in Firefox but other modern browsers have been tested and found to display most things correctly.
nw3weather.co.uk launched on 10th September 2010, I having acquired the domain name soon after moving here in July.
The most recent version (v3) launched on 22nd May 2013.
Within a month or so, I submit manual observations for prior days to the system, as well as fixing glitches and correcting misreads.
These changes are reflected on the site almost immediately.
Technical
The process involved in converting readings from the sensors to the data viewable on this website can be summarised thus:
Each sensor wireless sends data to the base station, which transmits it to my home PC (a dedicated low-power machine running Windows 7).
Weather Display software renders and uploads this data via
FTP to a web server -
which broadcasts the PHP-rendered web pages to the World Wide Web so anyone making a valid HTTP request can view them.
The web server runs on a Linux VM which I manage, but is provisioned by Digital Ocean.
All web pages and data procedures are written in PHP (inc. graphs powered by the JPGraph library),
with some Javascript used to provide a basic level of dynamism.
Data storage is a combination of CSV (long-term),
PHP-serialised (short-term), and PHP variable-exported.
History
All old site versions are still available to view, but are no longer maintained and may not display the latest data.
Site version 2 (10th Sep 2011 - 20th May 2013).
This involved a change in the procedure for getting data, using
PHP, which
was also used extensively across the site to enable most of the new features and site-areas: obtaining all the historical data, displaying when
a record is set, the unit selection module, records for the current day (in the detailed pages), and a lot more besides.
A small amount of Javascript and AJAX were also introduced to develop the auto-updating option.
Site version 1 (10th Sep 2010 - 10th Sep 2011).
My first foray into web development.
No scripting was used at all - all pages came pre-rendered from the local PC using Weather Display's inbuilt functionality to insert live data
into prepared templates, which I wrote but were based on carterlake.org's.
Site version 0 (pre-nw3weather, Aug 2009 - Sep 2010).
A 90s throwback. It was simply a webpage pre-generated by Weather Display and hosted on
a free, ad-ridden web server. I only set it up so I could view my data away from home.
Site Traffic
These figures are approximations based on my site logs, and are for interest only. Figures for 2011 are from 01 Sep onwards.| Nw3weather Annual Traffic | ||||||
| Year | Total Visits | growth | Mean | Median | Max | Min |
| Total | 1,585,000 | 3329 | ||||
| 2011 | 7,000 | 57 | 54 | 240 () | () | |
| 2012 | 49,000 | 133 | 127 | 667 (04 Feb) | 39 (10 Jan) | |
| 2013 | 63,000 | + 29% | 174 | 152 | 1562 (18 Jan) | 67 (03 Sep) |
| 2014 | 75,000 | + 19% | 206 | 188 | 599 (14 Feb) | 106 (06 Sep) |
| 2015 | 84,000 | + 12% | 231 | 210 | 679 (01 Jul) | 116 (17 Oct) |
| 2016 | 85,000 | + 1% | 232 | 213 | 813 (23 Jun) | 121 (25 Dec) |
| 2017 | 98,000 | + 15% | 269 | 240 | 1486 (10 Dec) | 126 (24 Dec) |
| 2018 | 118,000 | + 20% | 324 | 282 | 1256 (28 Feb) | 173 (22 Dec) |
| 2019 | 143,000 | + 21% | 393 | 338 | 1661 (25 Jul) | 193 (23 Mar) |
| 2020 | 184,000 | + 29% | 504 | 429 | 1917 (04 Oct) | 224 (12 Sep) |
| 2021 | 183,000 | + -1% | 503 | 435 | 1552 (18 Jun) | 215 (25 Sep) |
| 2022 | 209,000 | + 14% | 573 | 453 | 3329 (19 Jul) | 240 (28 May) |
| 2023 | 287,000 | + 37% | 787 | 683 | 2594 (02 Nov) | 352 (04 Mar) |
Additionally, it is found that, as expected, traffic is higher on days of precipitation - particularly snowfall, which can drive traffic up by more than 500% - record site traffic for a long time was over 1500 visits on 18th Jan 2013. This was only surpassed in 2019 with a day of record-breaking heat bringing 1661 visits, then again in 2020 with a record-breaking 3-day rain storm in early Oct.
Acknowledgements
Thanks go to TNET weather and Wildwood Weather for script ideas which helped in the development of the historical data tables.
Thanks also to carterlake for CSS and HTML templates that I used in nw3weather's early years (pre-2013).
Special thanks to my weather assistant Juliet, and the various other family members who have helped keep the station running over the years
Finally, enormous thanks to any frequent visitors, and especially those have contacted me with kind words -
I really appreciate this as it makes all the effort and resources I devote to building and maintaining this website especially worthwhile.
Other Notes and Thoughts
Forecasting
The accuracy of the local forecast on the home page is considered to be low, since it is based entirely on pressure trends - falling pressure means cloud, and if it continues to fall, then rain; rising pressure signals sunny intervals then clear conditions. This is obviously very simplistic. The forecasting done by national meteorological offices requires the amassing of a huge amount of data from observation sites on land (like this one), on the sea, in the air and even from satellites in space. The data is then fed through supercomputers running numerical models that aim to simulate extremely complex atmospheric processes. The model outputs then need processing and interpreting before a forecast is produced. The whole operation costs many millions of pounds to set up and run. Barometric prediction, on the other hand, is very cheap but has severe limitations: it cannot offer regional guidance, it is only relevant for the next ~24-48 hours, and it breaks down completely in more complex atmospheric set-ups.
Data Quality, Differences, and Requests
As discussed, I take steps to ensure data quality, but this is not guaranteed and I accept that my data probably does not hold up
to international standards for an urban site. However, I am active in comparing my data to, and ensuring consistency with,
data produced by London's "official" stations, these being at all airports near-ish nw3 (Northolt, Heathrow, City),
as well as St James's Park, and one at Whitestone Pond in Hampstead.
A detailed discussion on my blog exists about rainfall differences, and similar (unpublished) analysis has been done for Temperature.
As a general comment, it should be stated that there can be many reasons for differences in weather across the large metropolis that is London.
Permanent reasons include elevation (lapse rate), hills (frost pockets, orographic lift),
buildings (UHI effect, turbulence).
Day-to-day causes are down to the localised nature of cloud and rain cells (a thunderstorm in 2009 dropped 36 mm on East Finchley, whilst Heathrow remained dry).
Thus, differences can be perfectly natural (though averaged out are usually small).
At any rate, my analysis seems to support the conclusion that as far my own data is concerned, its accuracy is within
the deviations due to the given natural reasons in the majority of cases. The notable exception is wind speeds, which under-report by 25-50% at nw3weather.
I will continue to strive to maintain good quality data as far as reasonably possible for a hobbyist site.
I am happy to share my raw data files with anyone who requests them from me. I can provide most data from Jan 2009, and temperature data from March 2008, at minutely or greater intervals, in CSV format. Additionally I am happy to answer questions and provide analysis of this data to individuals or groups with non-commercial interests. However, due to limited resources, I cannot provide extensive or repeated analysis; for this you should seek professional guidance (from the Met Office, probably).
Rain Gauge - Trace Rain, and Snowfall
The rain gauge works by water falling through a meshed funnel and tipping an internal bucket. The mesh serves to filter out grit and leaves, but also means snow gets trapped until it melts or is blown away by the wind. The equipment is sensitive to approx. 0.2mm of rain - so an hour's worth of light drizzle or a brief shower may not register, though I will manually record the day as having had a trace of rainfall (seen as 0.1mm).
Consequently, I measure snowfall by a combination of snow depth (1cm of snow approx. corresponds to 1mm of liquid precipitation) and high-resolution radar which can pick up all types of precipitation. To my knowledge, this is also how the Met Office do it. In any case, once I've gathered this information after a snowfall, I manually enter the approximate rain-equivalent figure into my records, so it does show up in the daily and monthly totals, but not immediately (typically by 9am the following day).
About me
I am a weather fanatic with a particular interest in data analysis, and have been since around 2007 when I first started to collect data (in Excel) from a very basic weather station I had. I enjoy extremes of weather, whether that be cold, heat, rain, dry, sun, cloud, snow, storms, hail, heavy showers, sleet, fog...
I studied Physics and Computer Science, both of which I was largely motivated to study by my love of the weather, and went on to work as a software engineer. In 2015 I left London and moved to New York. I currently live and work in the San Francisco Bay Area, California. The weather station back home in London is lovingly cared for by my mother and other relatives, and me when I come back every year or so (pandemics-permitting).
For more insight into the world of a weather hobbyist,
take a look at the article on me in the local paper.
I was featured
in another article, about the July 2019 heatwave, five years later.
And again for the record-breaking July 2022 heatwave.