About "Validate External Links"
developed by Iritscen (http://iritscen.oni2.net)

Introduction
Validate External Links ("ValExtLinks" for short) is a Bash shell script for validating large numbers of external links on a wiki.  It was developed on a Mac, but hopefully any Bash shell from v3.2 onward can run it.  Non-Bash shells won't work, as there is a lot of Bash-specific syntax in this script.  The script invokes Unix binaries that are mostly pretty standard, but you might want to make sure that you have "curl" and "expect" installed.

The purpose of this read-me is not to tell you how to use the script.  Running the script with the --help option should give you that information.  This read-me intends to draw your attention to the following items that you will need to adapt to your system before you can use ValExtLinks, plus information for developing and testing it.

Execution
The following files are in the main directory, in reverse-alphabetical order:
	• validate_external_links.sh is the main script.
		• The AGENT variable should contain a reasonably up-to-date user agent string, preferably generated from Google Chrome since that is the browser used to take screenshots of pages.  There are different web sites you can visit in the browser to learn the user agent string, or you can simply upload and then visit the supplied print_user_agent.php (see Development folder).
		• WIKI_CURL and WIKI_HTTP need to be set to the location of the pages that contain "curl" error codes and HTTP response codes, respectively.  See the Documentation folder for copies of these pages.
		• WIKI_MAIN should be set to the location of the main documentation.
	• validate_external_links.command is intended to be the means of running ValExtLinks.  The idea is to make sure this file is executable and then to double-click it (or invoke it with "cron") when you want to run ValExtLinks.
		• You can use the _LOCAL variables to supply the exact links and exception files that you want the script to process.  Sample alternate invocations of ValExtLinks are also provided.  Currently, the extlinks.csv file hosted on oni2.net refreshes twice a day, at 06:20 and 14:20 GMT.
	• val_expect_sftp.txt is an "expect" script which performs the actual upload of the ValExtLinks report.  ValExtLinks was written to use SFTP because Oni2.net does not support regular FTP.
	• sftp_login.txt should be populated with your SFTP login info and the path that you want the report uploaded to.  The ValExtLinks script uses this file when using val_expect_sftp.txt to invoke "sftp".

For your reference, the Sample files folder contains:
	• exceptions.txt is a sample of how the exceptions list should look, formatted as a MediaWiki page.  Anything before "BEGIN LIST" and after "END LIST" is ignored, so a local plain-text file that only contains those keywords and a list of exceptions should work just as well.
	• extlinks.csv is a sample of oni2.net's external link table dump.
	• ValExtLinks report.htm/.rtf/.txt is a sample of the output you should get from ValExtLinks.

The Documentation folder contains:
	• curl codes.txt lists the possible error codes that "curl" can return, formatted for MediaWiki.
	• HTTP codes.txt lists the HTTP response codes that ValExtLinks understands, formatted for MediaWiki.
	• License.txt is a standard copy of the MIT license, and applies to the whole project.
	• Read Me.rtf is this file!

Development
For testing and development, the Development folder contains:
	• Get script line count.command tells you how big the script is.
	• print_user_agent.php can be used for setting the AGENT variable, as mentioned above.
	• Sample Archive response.txt is a sample of the output of the Internet Archive's snapshot availability API.
	• Sample header - OK.txt is a sample of what "curl" sees when it is run with the --head option and gets an OK (200) response.
	• Sample header - redirect.txt is a sample of what "curl" sees when it is run with the --head option and gets a redirection (302) response.
	• ValExtLinks to-do.rtf is the development to-do list.
	• YouTube - video_*.txt are files with sample page source from videos that are NG, used to teach Val how to recognize bad YT links.
	• YouTube bad link detection.rtf contains the links for the videos that the page source samples are from.

All right, that's all.  Have fun fixing those external links!