Webstuff.Inblighty.Com

Website DIY - tricks and solutions

PHP: Automate the update of your Maxmind Geolite 2 database files

A PHP script to download Geolite 2 tar.gz files from Maxmind every week or month, and update your Country, City, or ASN mmdb file.

Prerequisites: Basic knowledge and experience of PHP. Ability to schedule script (cron job/task manager). Server running at least PHP 5.4.

The PHP Script:


Introduction:


updatemaxemailOld image, the new script similarly reports the result of mmdb file updates.

Maxmind updates its GeoLite data files every month. Manually installing the latest data file on my site was one of those (never) do later tasks; so I decided to write my own PHP script to automate the process and share it here.

Please read the Warning/Disclaimer (below), and test before live use. You can set NOTIFICATION_MODE to “3” (screen display), to avoid bombarding yourself with emails during initial testing.



I’d appreciate feedback, especially on its use with Windows Servers (not tested in this environment); and what, if any, modifications are required.


About the Script:


The script is intended to:

  • get a tar.gz file from Maxmind and copy it to your server
  • extract the .mmdb database to a file in the same directory as the script (or other specified directory)
  • check for errors and if necessary revert back to previous database file
  • provide information to identify the cause of errors during testing/live running
  • notify you of errors by email

To set up, just change the constants at the top of the script to suitable values for your site (see Configuring Script – below). If the script is installed and run in your site’s Maxmind “database” folder then you don’t even need to specify any paths or directories.

The script backs-up the current gzip and database file before update (prior back-ups are overwritten).  On error, the database file (only) is recovered from the back-up.

Although designed for Maxmind, it can be modified to obtain and extract gzips and tars from other remote sites.

See below for scheduling, configuring and issues.


Scheduling the script




This is not a tutorial, you should already be competent using your server’s scheduler; if necessary seek advice from your provider.  Useful guides: CronJobs | Windows Task Scheduler.

Cron job emails:

Cron notification emails do not differentiate between a script’s success and failure; and according to my host, most users do not read them.

cronmail

The script provides its own email to more effectively alert you of errors, but obviously can’t identify errors in the cron job command itself; so you may still want to set the cron job to email you for stderr.

Example commands that may work for you:

php /path/to/update_maxmind2_dbfile.php or;

wget -O – http://example.com/update_maxmind2_dbfile.php

You may also need to insert a path in front of the php/wget command.
Some hosts block use of wget (but may allow curl).

Related articles: PHP one liners: How to allow a script to be run from a Cron Job but not from a Browser


Configuring the Script


To set up, you just have to change the values of the constants to ones appropriate to your site.

032| // the download URL for Maxmind database file 033| DEFINE ('FROM_URL', 'http://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz'); 034| //DEFINE ('FROM_URL', 'http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz'); 035| //DEFINE ('FROM_URL', 'http://geolite.maxmind.com/download/geoip/database/GeoLite2-ASN.tar.gz'); 036| 037| // the directory location on your site where the Maxmind mmdb file should be installed: 038| DEFINE ('MY_MAXMIND_DATA_DIR', realpath( __DIR__ ) . '/'); 039| 040| // settings needed for notification of the result of update 041| DEFINE ('ADMIN_EMAIL_ADDRESS','bob@example.com'); // your email address 042| DEFINE ('THIS_SITE', 'example.com'); // domain name of site (for use in notification email) 043| DEFINE ('NOTIFICATION_MODE', 1); // 1 = email, 2 = screen + email, 3 = screen

Lines 33 to 35 (Maxmind download URL):

You can use the script to update either the Country, City or ASN mmdb file.

Un-comment the line with the URL you need and “//” comment out the other 2 lines. e.g. if you want the script to update the City mmdb you would uncomment line 34 and comment out lines 33 & 35. The URLs are current as @ 6 July 18 but might be changed at some point by Maxmind. The script will email you if it is unable to connect to Maxmind.

Line 38 (folder location to which Maxmind mmdb file should be extracted):

038DEFINE ('MY_MAXMIND_DATA_DIR', realpath( __DIR__ ) . '/');

MY_MAXMIND_DATA_DIR is the folder path where the mmdb file will be extracted. The path value MUST end with a “/”. In the above line the path “string” is set to be the current folder (the one the script runs in). If you want to use another folder then replace “realpath( __DIR__ )” with your own path “string” in one of the usual ways e.g. hard coded path “‘/myhome/public_html/maxdata/‘” or use $_SERVER[‘DOCUMENT ROOT’] etc etc.

If the Maxmind Data directory you specify does not exist, the the script will create it for you. If the directory containing the script has 775 permissions then the new directory will also be given set with 775 permission; otherwise it will default to setting as 755.

Other constants:

Set “NOTIFICATION_MODE” to 2 (display messages), and run the script from a browser to test before changing NOTIFICATION_MODE back to 3 and scheduling it as a cron job.
The remainder of the constants are self explanatory.


Issues


Permission Denied (Unix/Linux and users, owners, permissions):

The script may report that it is unable to create/write to the files in your Maxmind Directory this is probably an ownership/permissions problem.

If you originally created a file via say by anonymous FTP it will be “owned” by that user; this probably won’t be the same user that your (CLI or Web Served) scripts run under. So 644 permissions (Owner: read/write; Group and World: read) would prevent the script writing to the file; likewise a script cannot access a directory without execute permissions.

In these circumstances, you can either change to less secure permissions, or delete the dir & file (make sure there are no other files you need in the dir) and use the script to create from scratch.

This forum answer provides more info on “Permission Denied” Problems.

Problems with getting the file (cURL)

File “links” on some sites may take too long, redirect, or require cURL to provide a user agent or referer etc etc (at the time of writing this is not the case with Maxmind). You can  set curl options to cater for these type of problems.


Warning and disclaimer.


Use the script and suggestions on this page at your own risk.

I don’t claim to be an expert; I write about problems I’ve encountered and how I resolved them in my server environment. I’m human and make mistakes. Any code, content and suggestions are intended as a rough guide only.  I can’t guarantee that the solutions on this site will work for you.  Nor can I guarantee that they won’t be harmful to your particular site or business.


As always comments are welcome.

Andy Wrigley+ has worked in IT and Computer Audit for 30 years, and loves independent travel.


Enhanced by Zemanta

18 Comments

  1. mark

    Hi There,

    Would be helpful if you could post a sample file with basic variables filled in for maxmind lite with sample paths filled in. I keep getting “Could not open input file: /path/to/update_maxmind_dbfile.php”

    Thanks

    Mark

    • AW

      (edited) Legacy update has been replaced with a v2 update script – with self explanatory constants as per revised article. N.B. your error is from the server (not script) and indicates you haven’t scheduled the (cron) job correctly. I’ve emailed you about it.

  2. ip checker

    this method use full for update database from maxmind database.. many thanks for share this article..

    @Mark – for cron job set to :
    5 8 * * 3
    /usr/bin/wget -O /dev/null http://example.com/update_maxmind_dbfile.php

    caused GeoLite2 databases are updated on the first Tuesday. I set run script at Wednesday of the month at 8:05am. for hostgator hosting I use command in above.

    • AW

      Glad it works for you and thanks for providing an example cron used to schedule your script

  3. Andy

    Can you also adapt the script to the GEOIP2 database? Here the database is double packed. Thank you very much.

    • AW

      Edit: 6 July 18 the script has been replaced with a new one for Maxmind Geolite2

      see code fix in the reply to your next question

  4. Andy

    With this script i get with the free geoip2-lite-database the file: GeoLite2-Country.tar

    When i change this line:
    $ourGeoDataFile = preg_replace(‘/\.tar.gz\z|\.gzip\z/i’, ‘.mmdb’, $ourGzFile);
    it seems to be ok?

    • AW

      Hi Andy,

      Update: 6 July 18: The article has been updated and a new script written for Maxmind Geolite 2.

      The “gz” functions used in the code don’t untar and I assume that Maxmind code does not untar on the fly so I’d be surprised but delighted if your suggestion works (instead of just renaming the .tar file as .mmdb). However this code fix should work:

      <code fix for old version of script removed from this comment>

  5. Andy

    the problem is, that the mmdb-file (and 2 txt-files) is in a folder with timestamp (GeoLite2-Country_20180206). This function extract the hole folder and put it into “GeoLite2-Country_20180206”-folder

    • AW

      Script now completely re-written for GeoIP2.
      The only problem for Geolite2 files is the tar for the mmdb database which has the file in an ever name changing folder. I’ve edited the temporary quick and dirty enhancement code in my reply above. Untested, but hopefully will do the trick.

  6. Andy

    OK. That’s it. It’s running so. Without backup of the mmdb-file and error handle; but is works. Edit by AW:This article now has a new replacement script which includes back-up and error handling.

    thank you

  7. Tim

    AW,

    Edit: (this comment refers to the now replaced OLD script)

    Thank you for the post and the additional details to get this working with the new database files. I am running into an issue when adding the additional piece of code. I get a FAIL message with this additional message when I try to run the script: Last reported error: 8192 | Directive ‘register_long_arrays’ is deprecated in PHP 5.3 and greater | Unknown | 0

    Is there a solution to make this work for newer PHP versions?

    • AW

      Hi Tim

      I have edited the article and replaced the obsolete Legacy Maxmind script with a new one specifically designed for use with Maxmind’s Geolite2 files. Could you please let me know if this works (a different method is used for the extract so it might work on your server).

      Issue with old code:
      I’ve tested this code yesterday (including the v2 fix in comments) and it ran without error on PHP 7.1.

      The only Googled “solutions” I could find for this error refer to servers having the out of date directive “register_long_arrays” in their configuration files (e.g. php.ini) and that the directive should be removed from your servers configuration files. In your case the fwrite/gzread function is returning false indicating failure to write the mmdb file; the script then uses PHP’s error_get_last() to display the “last” error. I have no idea why, or whether the error really is related to “register_long_arrays”.

  8. Tim

    AW,

    This is great work! It now works as expected with the new database files for GeoLite2 files. I also have access to the (paid) GeoIP2 country download and was able to modify the script slightly to also work with that too. If anyone is interested just modify the line:
    $this->v2_gz = basename(FROM_URL);
    to this:
    $this->v2_gz = basename(‘GeoIP2-Country.tar.gz’);

    Reason for that is the filename is not in the URL for the paid databases.

    Thank you again AW. This script is a great resource.

    • AW

      Glad it works for you, and thanks for the info that a single edit enables it to be used with paid files as well. As I don’t use Maxmind paid services I had no knowledge of the relevant URLs or file names.

  9. Andy

    Hey. Thank you for the updated script. Works perfect.

  10. Andy

    Hy
    There’s a new problem with the script. To download the database you need now a key. So the URL is now:
    https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country&license_key=YOURKEY&suffix=tar.gz

    So now you get the files:
    geoip_download?edition_id=GeoLite2-Country&license_key=YOURKEY&suffix=tar.gz
    and:
    geoip_download?edition_id=GeoLite2-Country&license_key=YOURKEY&suffix=tar.tar

    and then the error:
    Failure: update of geoip_download?edition_id=GeoLite2-Country&license_key=YOURKEY&suffix=tar.mmdb at xxx.xx
    Error uncompressing tar.gz: phar “/home/www/xxx/html/xxx/geoip_download?edition_id=GeoLite2-Country&license_key=YOURKEY&suffix=tar.tar” exists and must be unlinked prior to conversion

    Thanks for your help

  11. Andy

    OK. I got it.
    I changed: $this->v2_gz = basename(FROM_URL);
    to:
    $this->v2_gz = basename(‘GeoLite2-Country.tar.gz’); (in proper quotation marks)
    and it runs!

Leave a Reply

Your comment will appear after its approved; usually within 12 hours but can be up to a week.
Email is optional and never published. It will only be used to contact you if clarification of your comment is needed.

Copyright © 2018-2024 Webstuff.Inblighty.Com
This site recommends and is hosted by: Kualo Web Hosting.    
Theme: hemingway
 

Blog home  |  ↑ Top of Page ↑