Justified commits

Fix permissions setting in admin.

The upgrade to jQuery 3 broke the fetching of val() on the roles
multiple select input, as the code assumed this returned null if
no role was selected, whereas now it returns an empty array.

Fix issue searching for numeric IDs.

There is an issue with Number::Phone that means it is matching e.g.
402609 as a Jersey phone number. Reported and behaviour improved at
https://github.com/DrHyde/perl-modules-Number-Phone/issues/101, but
until a new release, move the searches around so a ref search takes
precedence.

[Waste] Show garden sub link even if no reporting.

The code was only showing the garden waste subscribe link if its own
terms were met, plus if any request or reporting was allowed. It was
not always showing when needed, e.g. if no reporting was allowed and
you had requested new containers for every service.

[Highways England] Move fallback lookup to sender.

Due to not setting cobrand handler for HE (for correct email templates)
the munge function was not being called if the report was made on .com.
As HE has its own SendReport handler, we can put the same code in there
instead for both sites.

If all categories selected, show none in URL/ajax.

Having all categories selected is the same as having none
selected (as it shows all by default), so have that carry
through to the URL and ajax request.

[Open311] Consistent protected field ordering.

Without this sort, the protected fields could be pushed on in any order.
If more than one of those protected fields then had the same sort order,
they could be sorted in that pushed-on order, which could change between
runs.

CSV export, improve Site Used and add Device Type.

Since we started recording mobile/desktop website use in service, Site
Used now contains that, rather than the cobrand used. Rename Site Used
to Device Type (and only show service column), and add a new Site Used
column that always contains the cobrand used.

Upgrade Test::Differences.

Test::Differences 0.64 had a Build.PL that required Text::Diff 0.35, but
a Makefile.PL that required 1.43. This can confuse installation, so make
sure we have a more recent version of Test::Differences to resolve this.

Pre-lookup some data for /report/new/ajax call.

The call was doing repeated loops of contacts to look up bodies and
hint texts, repeated database calls to check anonymous allowed, and
making lists of body names it did not need to in templates.

Store Gaze result to prevent subsequent call.

For some reason as part of 66f2160d, I removed the stashing of the Gaze
call (made as part of nearby report looking up) to save it being called
again by the map zoom code.

Send email reports in staff/inactive categories.

The email SendReport code was treating any non-confirmed category as
unconfirmed, which meant it would fail to send reports in staff-only
categories.

Pin GitHub Actions to ubuntu-18.04.

GitHub are moving to 20.04 in the near future, and while I think that
should be fine (if we change the cache keys), don’t want it happening
unexpectedly.

[Bromley] Add waste reporting service.

This creates a non-map property-based reporting flow for reporting
missed collections and requesting new containers. Such reports are
always private, in categories that are hidden from the map filters
and never shown on .com.

[Cypress] Have full login on temperamental tests.

These tests sometimes fail on Travis; e.g. fetching /report/1 returns a
non-logged in version of the page, or the Inspector login works but you
are still logged in as the previous user. See if we can stabilise it by
doing a full login rather than a cy.request() one, and logging out / in
rather than going directly to re-login.

Don’t strip all spaces from Open311 categories.

If a category’s send method is Open311, only strip spaces from the
ends of the code. We are aware of active Open311 servers that have
codes with spaces in the middle.

Move some templating functions to ::Template.

Most of the functions called by them are in that module, so seems
easiest to keep them together, plus this fixes lack of Web module
import in the Template module.

Zoom in to asset if selected when changing asset.

If you click the change asset button on a report page that was loaded
inline from a map list page, you want to see assets around the report
and not the centre of the map.

Do not error in export if report has bad category.

If category groups are enabled, the CSV export includes a subcategory,
but it was only fetching a valid groups value if the row's contact was
found.

Spot user set on update not at object creation.

The changes in 67823bc78 create the necessary metadata at the creation
of a Comment object, but if the user is not included at that point but
later (as is done by the normal public update flow), then it would not
be created. Wrap the user accessor so we can make sure it is set.

Make correct selection from single filter choice.

If a single filter item, that was in multiple groups, was selected,
then the entry in the last group was being selected in the category
dropdown when starting a new report. Make sure we check for a match
in the group first, the same behaviour as when there is an existing
category.

Fix lookups in templates of categories with &s.

A hash lookup in a template is escaping the key used, meaning the lookup
is failing anywhere we are using a category containing an ampersand as a
key. A continuation of the changes made in 527d763b.

Do not have bootstrap run sudo commands.

Your site user may not have access to sudo, and your admin user may
cause permission issues if used to do the git checkout, or the Perl
module installation. Document the separate script to install system
packages instead.

Co-authored-by: Sam Pearson <sam@mysociety.org>

Centralise update creation to include fields.

Given the user, we can infer the name if not provided, and the extra
data if a staff user. We can also provide defaults for various other
fields. Always have superuser take precedence over from_body.

[UK] Ignore Highways England extra data.

If a report is made on a HE road, but it is not to be sent to HE (so the
user has picked it's not on the HE road) and the category name is shared
between council and HE, then the extra HE data (area, road, & sector) is
still saved on the report, and our open311-adapter rejects it due to the
additionnal information present. So make sure we remove this data before
sending.

Disable rename on unprotected Open311 categories.

If you rename an Open311 category when it is not protected, it will only
get reinstated/deleted by the Open311 populate script shortly after, and
can cause issues due to the problem category renaming that occurs.

In category admin, group already shown elsewhere.

Groups have their own special admin section above, so no need to show it
in the extra data list (especially as if it is in more than one group it
only shows a reference).

Fix dashboard report CSV export.

The change to join in e16054150 did not include a collapse that the
previous prefetch was doing, so multiple rows per result were being
returned.

Move per-row Contact lookup to the database.

On admin report lists, and in front-end lists when an inspector, each
row was querying the database for `category_display`. We create a new
relationship for this query, and join/prefetch it wherever we request
this data.

Include staff joins on /around page, copying what happens on /reports
to prevent more lookups there too. Also add some joins for user email
in admin report list.

Maintain category group on pin move.

If the same category is in multiple groups, upon moving the pin and
fetching new categories from the server it would then pick the last
instance of that category in the list. Instead of using val(), pick
the correct entry in the previously selected optgroup.

[Peterborough] Improve extra question handling.

Ignore (for Open311-sending purposes) any questions whose code starts
"PCC-", so that it is easier to use the admin to add extra questions,
and move the removal of the extra questions to a more normal location
so answers are not lost when extra fields is reset post-send.

[UK] Fix Highways England categories on pin move.

If you clicked on a non-HE location (so showing non-HE categories), then
moved the pin to a HE location, the HE message box would show and update
categories (but there would be no HE ones at the time). The server would
return the new categories (including HE ones), but as the issue location
had now not changed on this second call, the categories would not update
to hide non-HE ones (it would show both HE and non-HE categories).

So, instead of going off a location, check the road name - if it has not
changed, do not recreate the whole HE message box, but do call the right
HE selected/not selected function to update the categories.

Note that this, if "Somewhere else" is selected, then causes an infinite
loop, because the highways_change event triggers a category_change event
which fires the layer's road found function, and then repeat.

The highways_change->category_change event firing, according to the code
comment, is so that asset selection can take place, or to remove stopper
messages being shown. Previously, HE did not have its own categories but
overrode the body to be sent to; HE now does have categories of its own,
so I think this code can be removed.

[UK] Show Highways England categories on cobrands.

Without allowing through the Highways England body, clicking on a
Highways England road in these cobrands would leave no categories
to be selected.

Initial service worker.

This basic service worker behaves identically to the existing appcache -
some static scripts and CSS are cached, any HTML offline instead returns
a static HTML page that knows how to show data on stored problems out of
localStorage (stored there when /my/planned was visited online). Inspect
form submissions will be captured and can be synced back when online.

Once feature parity is established, we will then remove appcache, switch
from using localStorage to the cache API, and hopefully move all offline
support into the service worker.

Default logging to logged in user.

Historically, the FixMyStreet admin was accessed via HTTP Basic Auth and
a secure proxy, so the auth user was used for logging purposes. Nowadays
all admin use is by FixMyStreet user, and any Basic Auth is used for the
purpose of hiding staging sites from bots. This configuration means that
on those staging sites, all admin is logged by the one staging user, not
the actual user, so let's swap that round.

Only trigger one change event on initial popstate.

When going back to the initial state with popstate, a change event was
being triggered on every single option of the filter selects. This led
to a lot of change events running on the category/status multi-selects
which then needlessly repeated the same activities over and over. This
locked up the browser for seconds in locations with many categories.

Allow update signup even if updates disallowed.

It is possible for a report to be closed to public updates, but it still
could receive updates (eg updates only from problem reporter, or updates
only by staff), so we should still allow people to sign up.

[TfL] Fix reference ID in logged email.

If report made on .com, `body->get_cobrand_handler` does not return TfL
as it covers many bodies. Now we have the body handler, we can use that
instead.

[UK] Fix links in submit email.

For an email to somewhere for which the report has a body handler, we
want to use that cobrand's domain for the link - we already are using
their cobrand for the email template, after all.

Switch to default-escaped in templates.

This means any variable used in a template is automatically
HTML-escaped, unless it is marked as safe either in code by
using a SafeString, or in the template with the `mark_safe`
function or the `safe` filter.

[Mobile] Make sure disable message findable by app

The mobile app is looking at the old extra.datatype_description location
rather than the new disable_form location for per-question disabling (it
does look there for all-category disabling). So we need to make sure the
JSON includes the message in the place where it will be looking.

Note if an extra data question has two answers that disable the form and
they have different messaging, the app will only take one of them.

Only add filter strategy if HTTP WFS being used.

If GET-based WFS is being used, no server-side filtering takes place, so
we need a client side filter to perform the same job. But if normal POST
WFS is being used, the server-side filtering has already been performed,
so no need to do it again (in fact, no need to return the data necessary
to do such filtering either).

Fix issue loading inspector duplicates on /around.

As there are two category drop-downs in this situation (one in the
hidden reporting form, one in the inspect form), the correct entry
needs to be specified.

Fix issue loading duplicates when not to be shown.

If the duplicates.js file is loaded, then its mechanisms come into play,
and it fetches duplicates and hides the reporting form if there are any.
So we only want this to happen in the same situations as it being loaded
elsewhere, on a relevant cobrand and not just after signing in.

Rename variable that shows missing body message.

As the around page includes a copy of the reporting form (when JS is
available), it runs the same templating code that the reporting form
does. Part of this code compares the bodies and bodies_to_list stash
variables to see whether it needs to display a message about missing
body information. Previously both variables were empty on the around
page itself, so no message would be shown, but a new bodies variable
was added in 4668f81a, which meant the message was now output in the
HTML and shown briefly when the map was clicked - until updated by a
server response). Rename the variable so that this no longer occurs.

Stop empty Open311 group causing duplicate history

It arises that <group></group> now passes to the code as a one-element
list containing undef, which compared differently to what was stored -
the code assumed lists would contain things. A new changelog entry was
created each time. Hopefully resolve this once and for all by treating
groups more formally and making sure we always have lists to compare.

Factor out reporting form h1 heading.

form_heading.html was a misleading name for a partial containing the
optional message that some cobrands display after the reporting form
title, so we’ve now renamed it to make it clearer. This also allowed
us to spin out the actual “Report your problem” heading into its own
partial template, called form_heading.html.

Make sure determine_contact_type called later.

The request cobrand is set on the schema during the app's
setup_request(), which is called by the root auto action.
So anything in a controller's begin action will access an
incorrect cobrand (if one has already been set).

Allow contact send method to be unset always.

If a body was set to Open311, and a contact set to Email, it was
impossible to unset the contact's send method and set an Open311
code, because it would always demaned a valid email address.

Fix bug with body public display text.

The CSV utils function returns an array of arrays, not just an array,
which this code did not realise. All this affected was sometimes both
bodies were shown in a two-tier area when the code meant to show one.

Make asset layer select/hover controls use arrays.

This way, we will then be able to pass in multiple layers for only one
pair of select/hover controls. Storing the controls on the layer makes
it easier to handle, but it does mean the controls need to be switched
on/off with visibility, as otherwise the RootContainer layer is always
present, making things unselectable. This also means the controls need
to be added to the map first, otherwise control activation won't work.

Make sure existing attribute answers set up.

If you come to /report/new with a part-filled report (say you've gone
via an OAuth flow), then the stash needs to get the category from the
report, not a query parameter, and report_meta needs initializing, as
that is what the template uses to fill in existing attribute answers.

Set default mock area to default browser test area

This should make it easier to add browser tests as you won't have to
get the click precisely on the right pixel in order to have it match
the correct area.

Ignore cache-busting keyword on appcache.

`params` changed in Catalyst 5.90116 to always include a sole keyword
parameter (ie. with no equals sign) as a key with an undefined value;
the changelog entry is only "improved test cases for query keywork".

[UK] Switch a few config options to feature.

You don't want base_url specified apart from on live sites, so this
means all the non-live site code can be removed. Contact name/email
you probably want to be clear when not the live site, and it should
be easier to change example_places.

Fix bug going between report/new pages client side

A few bits weren't being correctly hidden, as the code didn't account
for someone going directly from a report page to a reporting page via
the Report another problem here button.

Improve non_public photo handling.

Clear the photo cache if the non_public flag is switched on, do not
cache non_public or LOGIN_REQUIRED photos, remove non_public photos
from memcached recent lists, pass through any cookies on non_public
reports/updates, and check the non_public flag on photo lookup.

Fix some incorrect timezone code.

Times in the database should be stored in the application server's local
timezone, by e.g. using `current_timestamp`, or by setting that timezone
explicitly before storage (the database columns are all without timezone
so any timezone info is silently ignored). Reports & updates fetched via
Open311 and offline updates were being put into the TIME_ZONE setting if
present, meaning they were stored incorrectly for future usage.

[UK] Deal with setup where county may be missing.

Currently, Dorset Council and BCP Council are bodies linked with
the relevant areas of the now-gone district councils; this means
there is no body linked with the area of the old county council,
which causes an error.

Make sure raw RABX column is utf8-encoded.

Without doing this, a call to e.g. $contact->set_extra_fields(@meta)
in PopulateServiceList.pm, with an unchanged meta that contains some
Unicode values, can write to the database (and cause an unneeded row
in the history table), because the column from the database is UTF-8
decoded, whilst the new text is UTF-8 encoded.

It looks like an attempt was made in filter_from_storage to fix this
issue, but the column comparison for marking a column as dirty takes
place without this being called.

Stream dashboard CSV output.

Rather than compile it all into on big string for output, write the
output as we go. Also output headers to encourage intermediaries to
stream as well.

Use relative report links where possible.

On some UK council cobrands, some reports listed might not have been sent
to that council, so links to those must go to the national site. However,
using absolute URLs for all these reports means that sometimes you change
domain when you don't need to (eg. if you’re on osm.fixmystreet.com or an
aliased version of the site), which can cause confusion. State when we’re
happy to use a relative link (ie. web-facing report links, not emails, or
share links) and do that when we can.

Improve inline checkbox spacing.

Reduce the space between checkbox and label, so the text isn't lost if
the page is zoomed in. Fixes #2411.

Replace the creator_fixed question with segmented control to match the
full questionnaire, and the admin verified checkboxes with inline-text
as they need no padding. Having done that, all uses of .inline are now
on their own lines, so no post-padding is needed any longer, but we’ll
leave some in case of future use.

Don't ask for email on alert signup if logged in.

The “Get updates” flow on a report page, if logged in, was showing
an input label but no input field (because one is not needed), but
then on submission asking for your email address. Add missing name
on submit button to fix this.

Improve use of OL ArgParser/missing zoom in URL.

Remove the server-side zoom-in-url fix, instead use an ArgParser
subclass to default to the provided data if nothing in URL. Then
we can switch to using short lat/lon in geocoder URLs.

Include moderation history in report updates.

If the user has moderating permission, or a cobrand allows the viewing
of moderation history, show moderation history within the updates on a
report page.

Compare assets on their ID field, not JS object.

As the selected feature is a clone (so it survives e.g. layer reloading
from server after a pan), we need to compare the asset ID field instead
to see if we already have a match, to prevent a double select().

This also fixes a bug where an auto-selected asset would, after picking
a different asset, reshow the selected pin image onmouseout, due to the
way OL was storing old hover state internally.

Activate test_mode on JS testing.

This is so front-end tests will not try and send email, or
call Gaze. Various co-ordinates have to be updated because
the default zoom level of the map has now changed.

Use standard email sending for handlemail.

The EmailUtil library expects "\n" line endings, otherwise it gets
confused. It also doesn't handle a different SMTP port, and so on.
We also make sure the return path is chomped.

[Zurich] Ignore deleted categories on change.

This led to an odd issue whereby a deleted category started being
used for a category change, but only when the other category with
the same name was edited (so thus changed the database's internal
ordering).

Fix history API bug.

replace_query_parameter() was stringifying select multiples, meaning
they were incorrectly recorded by the pushState and thus breaking on
navigation.

Speed up fetching lists of bodies.

Use a HashRefInflator wherever all_sorted is used, with consequential
changes to deal with it now not being an object (e.g. add some with_*
functions for manual fetching of extra data).

Check all contacts for metadata and non-public.

Previously it would only check the first matching entry, which could
miss out on information if two bodies had the same category and only
the second one looked at had metadata, or was marked as non-public.

Remove need for category_extras ajax call.

Add by_category output to the /report/new/ajax call, containing all the data
that /report/new/category_extras returns for one category. Then alter the JS
to use that data immediately when needed.

[Open311] No update text on fixed state move.

A move between fixed states (presumably from fixed-user to fixed-council)
should not count as a state change for the purposes of generating comment
text from templates.

Do update layer visibility on extras received.

Asset layer visibility depends on fixmystreet.bodies, which is updated from
ajax request by pin selection or category change, but the visibility change
happens before the ajax request - so if visibility is not also checked when
category_extras returns it can get out of sync.

[UK] Fix issue when body ID not equal to MapIt ID.

Hitherto when creating a body or ward alert on a UK site,
the MapIt area ID has been stored instead of the body ID.
This is okay for www.fixmystreet.com which for historical
reasons does have body IDs that match MapIt area IDs, but
other UK-based sites may well not. The alert lookup looks
for body ID, meaning those alerts will not work. Save the
body ID instead, plus fix some tests that were making the
same assumption.

Improve inspect form position and configurability.

Dealing with the map UI changes earlier leads to a better display
position, and we add a global variable check to allow cobrands to
not use this behaviour. Fixes #1863.

Improve handling of loading spinner display.

Going back to /around from /report/new can trigger loadend (and the
spinner hiding function) twice (if the strategy activation starts a
read), with triggerRead aborting that read and triggering the event
itself. This means the spinner is not displayed, as the count falls
below 0. We could pin the count above 0, but instead let’s log each
layer ID while ‘active’ and switch off when they’re all gone.

Fix front page critical CSS loading in IE.

The easy thing first - IE appears to 'stack' :before entries, so if a
stylesheet is loaded multiple times (e.g. due to critical CSS loading
on the front page), a font-size of 2.5em gets multipled. Switching to
rems fixes this.

The preload polyfill in IE8/9 loads the base stylesheet as if it were
after the layout stylesheet, which breaks the navigation display. The
polyfill has a new version which is somewhat simplified, updating the
existing <link> rather than creating a new one, so we upgrade to that
plus a PR to prevent browser hanging.

However, in IE8 this now incorrectly uses the print styles on screen,
meaning the navigation and footer disappear. So use an IE conditional
comment to just cut IE8 out entirely. This means we can also cut down
the polyfill somewhat.

Always set memcached namespace.

The namespace was only being set in the web loop, so cron scripts
were accessing different memcached keys. In particular, the state
list would be shared between instances on the same host.

Allow same-time state change from Open311 updates.

If we've pulled in a report and an update from an external source,
both may have exactly the same timestamp, and we do want to record
the problem_state in that case.

Set database session timezone on connection.

fixmystreet.com went down early Sunday morning because the database server had
been upgraded in the past year and was now set to UTC and not local time. This
confused the codebase when it encountered timestamps that could not exist, all
between 1-2am UTC.

Ideally, timestamps in the database should be 'with time zone' or be stored in
UTC, but for now let us set the time zone to the local one upon connection.

Fix issues with new report pin, improve cursor.

The "coerce pin ID to integer" behaviour to fix a previous issue was
coercing undefined to NaN, which then caused other problems with the
new report pin, such as fading out when hovered, and being the wrong
size.

Also improve the cursor handling, showing the grab/ grabbing cursors
where available.

Fix race condition making a new report.

If you started a new report by clicking the map before the
server had responded with the map list pins, when they did
arrive they would replace the new report pin.

Fix check for visible sub map links.

I don't think this check could ever have worked, because items are invisible
if their parent is set to `display: none`, so it would hide the links bar if
it was already hidden, and show it if it was already shown.

Use a js- class for the movement of the feed item into the sub map links.

[Zurich] Switch to more modern database states.

The Zurich code was written a long time ago, and used overriding so that
e.g. the hard-coded 'investigating' state referred to Wunsch (wish). Now
that states are stored in the database, we can create ones specially for
Zurich and use them instead. Hooray!

Remove category_options_copy.

This variable existed to keep a copy of the categories before it was
overwritten by report_edit_location (if a bad location was provided)
calling the variable setup functions again. But really, the function
shouldn't be overwriting the stash, it's only testing the lookup and
so should restore the stash at its end.

Store questionnaire data as soon as opened.

This means the questionnaire is considered 'answered' as soon as
a (HTML) link is clicked, which I think is okay. Then filling in
the questionnaire form will update the same questionnaire.

Generate state_map later on, for use by bar chart.

The summary page calls generate_grouped_data, which requires state_map
in order to sort the states appropriately, so generate it there rather
than earlier.

Fix issues with send method category change.

Use the send method recorded at the time of sending, not the current one
(it may have changed since), and only resend if the new send methods are
not a subset of the old.

[UK] Make sure jQuery is loaded where it is needed

Bromley uses jQuery for its A-Z on all pages; only load Bristol map
JavaScript on map pages (same as Angus); drop Oxfordshire JS use of
jQuery; fix Zurich load order; and improve jQuery checking in other
JavaScript.

Fix back bug, from report after using list filters

The popstate event for a filter change assumed that the list was already
being displayed, but it was not if you were going back from a report.

The reports_list function to display the list page is idempotent (though
this commit makes it more so), so can happily be called whether the list
page is currently displayed or not.

Prevent multiple 'Expand map' links appearing.

On mobile, if you e.g. selected one report and then another, you'd get
another "Expand map" appearing. Check that one is needed before adding
it, and also make sure the handler is only attached once to prevent it
conflicting with itself (and so cancelling itself out).

Make sure returned geocoder data is UTF-8 decoded.

It turns out that HTTP::Message (as used by LWP::Simple::get) only
decodes text/* and 'XML', not application/json, as used by all the
geocoders. Work around this, hopefully in a future-proof way.

Make sure large-width base CSS is after preload.

If it is before, then it is loaded, the onload event doesn't fire, and
the base styling is lost if the browser is resized from large to small
width.

Fix fetching of duplicate reports.

c9998984 altered nearby.json to use on_map rather than around_map, to
prevent needless text appearing, but did not take account of the fact
that it expects Problem objects, not Nearby objects.

Update Module::ScanDeps dependency.

This should fix #1886, which was caused by the update of Module::Build
as part of f97088d6 removing Module::Build::ModuleInfo which was still
being required by Module::ScanDeps. This was passing on Travis because
Module::ScanDeps was already installed.

Better testing for tests run on live site.

When the tests are run normally, using bin/run-tests, 00-check-config.t
is pointless as the config will have been newly set up from the example
file. To prevent running except through run-tests, check when test_mode
is set, dying if the live config file has been used.

Stop loading spinner covering map.

A full modal made it hard to perform multiple zooms/pans, making
the site slower to use than necessary. Tell the user the site is
updating, but let them interact with the map at the same time.

Return page with "No results" if no ref found.

This lookup is often performed by inspectors with appcache switched on,
which means they were getting the intercepted Glitch page rather than a
404; give them a no results page instead.

In RSS, only escape the necessary five/ctrl chars.

By default encode_entities uses named entities, which don't exist
by default in XML. It seemed nicer to switch to only escaping the
minimum rather than switching to numeric entities.

Allow verify/change/add email/phone in profile.

This expands the 'change email' functionality to work with phone
number also, allow addition of missing details, and verifying of
unverified details.

Move some auth functions to their own controllers.

Move the social auth functions to Auth::Social, and the
change email/password functions to Auth::Profile. There
are no actual code changes.

Fix test calling external service.

This test would pass offline (as query would fail without error),
and online was passing because Nominatim was presumably returning
no results before today?

[Zurich] Fix map offset bug.

The upgrade to OpenLayers in 31b8337e3 was not accompanied by fixing
the tileOrigin manually tweaked as an old workaround. This moved the
map c. 100m out of alignment.

Bristol fixed this in a7f60abb by adding topLeftCorners, but this is
better done by fixing origin_x/y.

Fix issues with Open/Closed translated too early.

It was possible for _hardcoded_states to be reached with no locale
selected (e.g. bin/update-all-reports), causing an error trying to
translate Open/Closed. But translating those there wasn't correct,
because that would then be cached and returned even if a different
language was being used. Caching was ignored in testing, which did
not help.

We no longer translate the state names in their objects, only upon
display at the point that we will know the locale. We can't simply
return the translation of Open/Closed because there may be entries
in the translation table as well.

Fix issue editing category with '/' in it.

Simplify chaining of body/category admin URLs so that all
categories are treated the same, with `/` or without, and
the template will then always have its CSRF token.

Fix bug if first page after restart is admin.

If the first page looked at after server launch was an admin one, the
User object was getting a different schema attached than the one used
by everything else (so e.g. the cobrand was not then available to it,
causing a crash on a body page). Using auto instead of begin prevents
this from happening, as the setup_request auto always runs first.

Make sure pin ID is an integer.

The JavaScript assumes it is, but due to a string comparison in
DBIx::Class [1] when a has_many prefetch is in use (as it is on
a /reports page when a shortlist-using staff user is logged in)
all IDs bar the first were being given an internal string value
and JSON encoding was outputting them as a string.

[1] https://github.com/dbsrgits/dbix-class/blob/v0.08209/lib/DBIx/Class/ResultSet.pm#L1331

Add debug toolbar middleware.

If debug is enabled, using the CATALYST_DEBUG/FIXMYSTREET_APP_DEBUG
environment variables, add a debug toolbar to the output, including
request/response details and a database query log.

This uses Plack middleware, so works by switching our dev server to
use Starman with plack directly, rather than via the script runner.

We remove the GZip compression as this interferes, and take a local
copy of the QueryLog::AdoptPlack trait as it needs a tweak to work.
Make sure the CSP header is not output in debug mode, as that would
prevent the toolbar JavaScript from running.

Do not fetch pins server side if JS enabled.

The page currently fetches the pins again client-side (as we don't know
the extent of the map at the server point). I realise this isn't ideal,
but should speed up the normal case quite a bit.

Store duplicate reverse lookups upon dupe setting.

This avoids the need to search the whole table to find a report's
duplicates. And is easier than constructing an index or upgrading
PostgreSQL.

Prevent unnecessary string translation.

Calling the function 'translate' causes gettext-extract to add the
string to the .po file. Also, we no longer need an admin 'updated'
message as the main form has its own success message.

Fix issues with database migration script.

A reference to ->storage was missed as this script changed in a
different branch. The schema files contain transaction commands
themselves. A check for migration 52 was missing.

Update map size if an extra column has appeared.

Pulling in a report via JavaScript that has two columns (e.g. the
inspect form) changes the size of the map, so updateSize needs to
be called.

Stop update form toggle causing report submission.

If the report is loaded with JavaScript, the outer form (for a new
report) is present and would otherwise take this click and start a
broken new report submission.

Stop errors from JS validator due to form in form.

The outer form is using the JS validator, which means the validator is
called for any event on the inner form fields. If the inner form isn't
using the JS validator, it errors as it assumes the closest form has a
validator object on it.

Having the inner form use the JS validator shouldn't have any effect.

Allow static home page template override.

If an /about/homepage.html template exists in your cobrand, it
will be used as the front page, and the normal front page will
be available at /report.

[Oxfordshire] Separate defect creation from state.

Revert the behaviour from 36baff2d, so that everyone can use the 'action
scheduled' state, and instead if someone with report_instruct permission
has the state set to 'action scheduled', add an extra mandatory question
asking whether they want to raise a defect or not.

Don't resend if category change subsets body.

Instead of checking if the list of bodies has changed at all, mark the
report for resending if there is a body in the new list not present in
the old.

[UK] Remove requirement for fixed body IDs.

Historically in UK cobrands, bodies have had IDs the same as the MapIt
area ID they cover. This can be confusing (if you are setting up a dev
environment, say) and should not be necessary. This commit removes the
requirement entirely, by switching any ID checks to either the name of
the body, or the actual area it covers.

One note: the body name in the test has to match so that we do not get
two bodies both covering 2237 created. This will not be necessary when
the tests are compartmentalized in the next commit.

[fixmystreet.com] Improve fixture loading script.

Given a MapIt area ID, the script will create a body and categories,
users with different levels of permissions, and some random problems
and updates. It can optionally empty out the database first.

Remove background overrides from sass core.

Most cobrands have the same header at mobile and desktop, so it is
easier and simpler for the default to have this and override where
necessary.

[UK] Stop nearest request with scientific notation

If the longitude was very close to 0, it was being sent as e.g. 1e-6
in the request to MapIt. Mock out the Bing query so that this can be
tested (the closest.t tests were previously always being skipped).

[Vagrant] Update submodule on any provisioning.

On an initial clone, the Vagrantfile would check and make sure that the
submodule was present (and usable inside and outside the box), but on a
subsequent provisioning it wasn't making sure it was up to date.

Fix timing edge case test failure.

If the creation of this test report and the loading of the dashboard
took place within the same second, the test report would appear in a
different column than the test expected.

On /reports maps, only include reports in view.

Update the reports with a bounding box, similar to on around pages.
This is made slightly trickier because we don't want to do anything
on page load (we already have the pins), we need to reload when the
zoom changes, and we don't want the strategy to get confused by its
first redraw, e.g. on pin hover, We also need to turn off the zoom-
to-bounds if we've got a bounding box in the URL.

Fix issue with categories with regex characters.

As the templates were using `grep`, they failed to match on a category
such as "Footpaths (right of way)". Changing the stash variables to be
hashes instead of lists makes checking for a key simpler. Fixes #1688.

Don't error when devolved body, blank send methods

The check assumed that if a body was devolved, either the contact
or the body would have a send method. Instead, use a contact send
method if devolved and it exists, a body send method if it exists
or the fallback if neither. Fixes #1374.

Fix bug if test run c. 55 hours before BST starts.

The code was adding two days and seven/eight hours to a floating
DateTime, which from 5-7pm on 23rd March 2017 created a DateTime
between 1-2am on 26th March which does not exist in e.g. the UK.

Remove superfluous lists from Open311 JSON output.

The Open311 JSON output was wrapping each request parameter in a list.
This was presumably to prevent the XML output including the parameters
as attributes rather than children. A better way to fix this is to use
the NoAttr parameter to XMLout which forces them to be children.

Make sure csrf_time is deleted after use.

If an out-of-date token was passed to check_csrf_token, then no new
token would be output on the error page because csrf_time was still
present.

Better path for showing config git version.

The default cobrand does not have a template directory, so the
directory change to it was not working. Use the root directory
of the repository instead.

Store and use initial URL to fix JavaScript bugs.

If you were on a body page, selected a report, then another report, the
"Back to all reports" link then had an incorrect URL. Also, going "Back
to all reports" after one report would fetch an incorrect Ajax query of
reports to show (as the URL had not yet been updated).

Both of these are fixed by storing the URL on page load, and using that
for the "Back to all reports" link and the URL used by the Ajax call.

There is still an issue with the URL shown if you filter, then select a
report, then click "Back to all reports", but the contents are okay.

Fix map pin clicking in IE11.

It turns out that whilst IE9+ defines MouseEvent, it's not in the
form that can be used as a constructor until Edge. To fix this we
check that MouseEvent is a function before using it as one.

Determine location before possible redirect.

If you changed the around filters on a URL with latitude/longitude, the
co-ordinates were not passed through as determine_location needed to be
called first.

Don't show alert email box if signed in.

It will default to your logged in address, ignoring what is put in this
box. This should also fix an issue where the update email address field
is initially blank when you are logged in and using the "Provide update
as" functionality (due to both fields having the same name).

Use shared Template base for all templates.

Reduce some duplicated code, and as a side effect gives HTML emails our
internal html_para filter, allowing single line returns in contact form
submissions.

Fix photo preview display after submission.

If the form is submitted and reshown, due to an error or logging in, the
image preview is loaded from the server, rather than the client. We need
to get the image source data to get the Exif data out of it.

Fix updating email field when reporting as another

When the form is shown via JavaScript, we need the email box to start
off editable as it is if you hadn't used JavaScript, and we also need
the change() to be fired on the right field.

Add user planned reports.

A user with the appropriate permission can add/remove reports from their
list of planned reports using a button on a report page. The list can be
viewed at /my/planned.

[Zurich] Fix server side map tile URL generation.

@scales does not include the hidden zoom levels, so we need to offset
the matrix ID to be correct in the URL but not do so to calculate the
tile row and column.

Generate unique config per test run.

This allows multiple test suites to run simultaneously (beforehand, a
second run would overwrite the same config file and lose its database
connection).

Clean up the created config file on exit.

Stop using collapse filter in category template.

If a category had eg. a double space, the use of the filter meant it
was output in the HTML with only a single space and so woudn't match
the database entry. Switch to more explicit template space chomping.

Fix a few JavaScript issues with reports pages.

Don't stringify pin problem ID, as get_marker_by_id assumes it will be
an integer; only activate the bounding box strategy if it exists; make
sure a refresh of the markers happens when going back to the list; set
the Back link on a pulled-in report to the current URL.

Fix two XSS vulnerabilities.

The title in the OpenGraph header was not being properly escaped, and
the hide pins/all pins links were using single quotes which were able
to be broken out of.

Also remove the single quotes around rss_feed_uri, though this is not
a vulnerability as its contents were sanitised (postcode or co-ords).

Patch Dropzone to correctly orient preview images.

Include a patch file, applied to Dropzone as of version e524e03c, that
also includes our previous Opera patch from 445e9a3d. The new minified
file is our cut-down exif.js plus the patched dropzone.js.

Remove most IE8 special cases.

Document the couple that are left, and fix the display of the logo
with a simpler positioning/ less overriding. Drop a needless usage
of 'rem' to make things a bit easier there too.

Stop using Text::Wrap to wrap email bodies.

If we don't do this, the default quoted-printable encoding will make
sure the lines are physically wrapped anyway. format=flowed could be
an alternative, but I think "=" acting as a quasi-hyphen is somewhat
nicer than every line ending with "=20".

A better solution could be quoted-printable doing soft wrapping only
at word breaks, making each line be readable though end with a " =",
but that didn't seem worth the effort involved.

Improve error handling of Vagrant installation.

It should now show a failure message on failure, rather than always show
the success message, and not show a confusing error about general.yml if
something went wrong before creating that file.

Improve CSS compilation.

Move to using libsass via CSS::Sass, and stop using compass, supplying
any used mixins directly. This removes the need for any ruby/gem based
installation, and greatly increases the speed of compilation. make_css
is also enhanced, bringing in the file monitoring previously done by a
separate script and improving its dependency monitoring.

Use only one templating system for emails.

Historically, emails sent offline (alerts, questionnaires, etc) used a
different templating system from those sent by the website (e.g. login
emails), though the newer system was also being used for the site name
and signature of offline emails.

Store Open311 error in report on failure.

It will still output via cron on the first failure. However, reduce the
verbosity of a couple of SendReport modules which are reporting failure
every time.

Send unresponsive body data per category to app.

The app doesn't currently use the unresponsive body data returned
via /report/new/ajax, so make sure it's returned via the category
look up it does make.

Add PNG image support.

Store image type along with hash in photo column, and use that when
outputting images / generating URLs. Make sure all public photo URL
generation goes through appropriate functions, and change temp URLs
so the filename can be output directly from the list.

Fix test issue running on PostgreSQL>=9.2 on Mac.

Test::PostgreSQL sets up a new database with initdb and empties the
generated postgresql.conf; PostgreSQL 9.2+ sets its locale/timezone
information at the initdb stage, not when it is run. It is possible
therefore for the tests to be in one timezone but the test database
to be in another (GMT) – causing a test failure when e.g. an update
is created manually. Make sure the first comment is well before the
others being tested, so there is no chance the order gets confused.

[fixmystreet.com] Fix for two-tier unresponsive.

Make sure the unresponsive message shows, if you're in a two-tier area
and one of the areas is marked as unresponsive, for all the categories
of that area.

Don't double-decode geocoded addresses.

Perl 5.20 introduced a version of Encode that errors on decoding already
decoded content (rather than returning the same string). Whilst this can
be taken as a bug in our code (although the decoding exists because some
versions of FastCGI silently UTF-8 encode the content), in the changelog
for Perl the only reference to this change is the line: "Encode has been
upgraded from version 2.49 to 2.60.".

Add cobrand-specific custom reporting fields.

This adds a new cobrand variable, report_form_extras, which contains a
list of extra fields that will be saved in the 'extra' metadata of the
report. Fields may optionally be marked as required.

Make sure top of reporting form is shown.

If the list of reports was scrolled and then the map was clicked, the
form would be pre-scrolled down. It now makes sure to scroll the form
to the top. Fixes #787.

Show remembered uploaded photos when Back used.

The uploaded_files solution in cbdfcad6, to show already uploaded files
stored in upload_fileid, worked if you submitted the form to the server
and had it returned, but did not show remembered photos if you used the
browser's Back button to return to the form from another page.

Instead, always read the data from upload_fileid, which works whichever
way you have come to the page. Fixes #1332.

Switch to JSON::MaybeXS, remove JSON::XS.

Travis has Cpanel::JSON::XS preinstalled, which means that the build
would fail there, as JSON::MaybeXS would try and upgrade JSON::XS to
version 3 which the snapshot did not contain.

Remove risk of infinite loop in PhotoSet.

Use a separate attribute for data direct from a database row when
creating a PhotoSet, so that data will always contain one or more
photo IDs.

Prevent template errors from being logged twice.

Catalyst::View::TT calls both c->log->error and c->error; the former
logs immediately, while the latter stashes the error, and then calls
c->log->error with it during finalize.

Add Module::Pluggable to cpanfile snapshot.

Module::Pluggable is marked as deprecated in Perl 5.18, to be removed
from core in a later version. cpanminus, which is called from carton,
ignores core modules that have been deprecated - running a deployment
install under 5.18 therefore caused a warning to be shown saying that
it could not find Module::Pluggable as no version was included in the
snapshot.

Fix questionnaire image sizing in Firefox.

A 100% max-width img within an inline-block within a fixed width wrapper
behaves differently in Firefox to other browsers; other browsers use the
width of the fixed width wrapper, whereas Firefox uses the width set for
the img. Given an identically set-up div, all the browsers use the width
of the div rather than the fixed width wrapper, so Firefox appears to be
behaving more consistently.

The easiest solution is to change the inline-block to display: block, as
the inline-block is alone within a floated fixed width div. Fixes #1243.

Only have jurisdiction_id in request if present.

So don't request '...services.xml?jurisdiction_id=' as that can lead to
an error from an Open311 server that expects either no parameter at all
or a stated jurisdiction_id.

Better error if Open311 returns nil service list.

If an Open311 server responded with e.g. '<services></services>', you
got a strange hash error as the code treated it as one undef service.
Spot this at the same point we check anything was returned.

Also remove a couple of warnings due to lc of a maybe-undef variable.

Workflow simplifications for Zurich.

Allow DM/supers to assign new reports more obviously, by changing
the one thing they need to (category for different DM, one of two
body dropdowns for subdivisions or external body).

[UK] Default blank names to the empty string.

If you are running on a system with GDAL2.2 or higher, null entries are
now returned as null, not the empty string. Boundary-Line contains null
entries for its NCP areas, so treat those like an empty string.

Manually prefetch any matching many-many countries

iterator() and prefetch() can't be used together, so the prefetch of many-many
countries is not happening in area lists, causing a slowdown if many areas are
to be returned. Instead, as we already do with codes, we manually fetch all of
the relevant countries from the join table and store them on the model objects
as we iterate through them.

Improve test running with changed settings/order.

A couple of tests were e.g. altering mapit.models.countries and not
setting it back, and another two would only pass if COUNTRY was set
to 'GB'. Set those to skip, and restore original settings.

[UK] Use database for co-ordinate transformation.

If the application is running on a server with GDAL 1.9 (and maybe 1.8),
its OSGB co-ordinate transformation can be up to a few metres out. As we
use the transformation by the database when saving new coordinates, also
use it when fetching so we can match and display accurate information.

https://trac.osgeo.org/gdal/ticket/4597 is the GDAL ticket that explains
that it is fixed in GDAL 1.10.

Fix a couple of command line argument issues.

Integer/float command line arguments weren't correctly updated with the
move from optparse in 5f0f33c4, and a call to super() was missed in one
add_arguments. Thanks to Roman Zimmermann for spotting.

[UK] May 2016 Boundary-Line import.

Deal with the Welsh Assembly boundary issues caused by new data being
provided too early, a returning parish council that presumably should
not have been deleted, and a duplicate unit ID.

[UK] Boundary-Line October 2015 import.

Two incorrect IDs in the source needed patching, work around a parish
that had disappeared and is now back, and remove ONS codes from Welsh
Assembly boundaries from before this import.

Fix bug parsing KML with Unicode filename.

The xml.sax module in python2 prior to 2.7.4 cannot handle a Unicode
filename (this was Python issue https://bugs.python.org/issue11159).
Make sure that we pass a str to xml.sax, which should be fine in any
version of python. Fixes #206.

Swap heading and policies in SVG/PNG output.

The graduated transparency does not translate to the PNG, so lines are
cut off; move the logo/source to the top and let the main text flow to
the bottom.

Fix display of batch numbers in email alerts.

721c73bd worked for some alerts (e.g. speaker alerts), but did not work
for word alerts, which by this point no longer have brackets around the
list of batch numbers. So check for the list in two ways, once with the
brackets, once not.

Fix multiple ORs in email alerts.

Xapian has changed how it brackets its terms; this code assumed it would
be `(((B123 OR B124) OR B125) OR B126)` whereas now it is `(B123 OR B124
OR B125 OR B126)`. As the old regex still matched on the `B\d+` part, an
email alert would be left with lots of "OR"s in the alert titles.

Consistent display of division result.

Hide both or absent if 0 (0 absent implies no data there yet, as SF
always absent; 0 both is not interesting). Also be consistent about
not including tellers in totals.

Join paragraphs if the latter starts lowercase.

The parser, when removing column numbers from the source, can leave new
paragraph indicators where they were. The parser has been improved, but
rather than reparse everything old, spot the same sort of thing when it
is displayed.

Try not to do it with "iv."-starting amendment paragraphs.

Stop regular expression segfault.

The previous regular expression could easily run out of stack space when
run on something with a large number of HTML elements. The new one is a)
simpler, b) explained, and c) possessive, so hopefully won't use any.

Improve search highlighting/phrase replacing.

Search highlighting ignores HTML elements entirely now, and phrase
replacement happens before glossary/wikipedia, so that the phrases
are already links which are correctly detected.

Ignore Stripe plans/coupons that aren't ours.

If Stripe sends us an event for a plan that doesn't start with 'mapit',
ignore it; in the command line script, exclude any non-MapIt coupons or
plans.

Make sure CSV reads in whole UTF-8 characters.

This isn't paticularly nice, but neither seemed messing about with
pyexcel. As the codecs.getreader iterator reads underlying data 72
bytes at a time, check and read additional bytes to ensure that we
are not splitting a UTF-8 character.

Only parse CSV file for number/bad postcodes once.

Store the row number through the process, and use that as a proxy for
knowing the file has been scanned already. If there are any bad rows,
we have to alter the form data so the figures are then passed through
to the next submission.

Prevent needless revalidation at end.

All the forms have been through validation before calling done(),
and get_all_cleaned_data() would repeat all that, so instead just
construct the combined cleaned data dictionary ourselves.

Allow boundary changes on import as well as new.

Whether adding a new generation, or altering the current one, have a way for
boundaries to be updated if they've changed, rather than always creating new
areas.

If multiple codes found on import, pick one.

Due to an import issue previously (where a mix of two OSM dumps meant there
were multiple KML files for areas where their name/level had changed in the
interim), there might be some duplicate OSM IDs in the database.

Code finding on import should find any active area

This meant when a mixed import KML mistakingly contained multiple files
for the same relation/way (due to name/level change), duplicate entries
were created even when the boundary was identical.

Fix issue where configuration lost calling proxy.

I haven't worked out the details as to how this is happening, but a call
to $self->proxy always calls new() and creates a new object, which isn't
returned, but somehow ends up blanking out the config_filename then used
to lookup the server config.

If the config was looked up at all before calling proxy, it was then not
affected, so this turned out to only affect Symology as the Confirm code
always looks up stuff first.

I have 'fixed' this by removing the check for proxy being set, as we can
always set it to the configured variable.

Remove unneeded test gubbins.

Make service_request_content always a function, which then means
we can improve a couple of test files to test their integrations
directly, rather than via the main UK one, so then we can remove
unneeded jurisdiction_id functions.

Switch to JSON::MaybeXS, remove JSON::XS.

Travis has Cpanel::JSON::XS preinstalled, which means that the build
would fail there, as JSON::MaybeXS would try and upgrade JSON::XS to
version 3 which the snapshot did not contain.

Include speeches before heading and "in the Chair"

If we get a speech before we've had any heading output, create an initial
heading for the speech to be included under, including any "in the Chair"
heading we've already seen. If we get an "in the Chair" later on, include
it as a procedural paragraph.

Better handling of e.g. multiple member speeches.

If <Member> name not matched, try and use it in a new speech (allow a name to
be recorded without an ID). Also extract any italic text inside. This is most
commonly used for "Several hon. Members: rose-".

Fix start dates of people with one non-Lord name.

As the script did not record min/max of names unprefixed, for entries
where there was only one unique non-prefixed non-Lord name, it didn't
set the right start dates.

Update package installation details.

No special dateutil code needed any longer, lxml should always be installed
(it's included from the importer included from speeches.views), and we need
a more up-to-date psycopg2, django-bleach and django-nose.

Migration to bring up to date.

This includes the verbose name changes to instance fields from the
upgrade to django-subdomain-instances, and the dropping of default
speaker ordering from 9d1a53b.

Bugfix for Django 1.5 / Python 3.

reverse_lazy doesn't work passed to HttpResponseRedirect with Django 1.5
as https://code.djangoproject.com/ticket/10491 was only fixed in 1.6. We
don't need to use reverse_lazy in these places so just remove it.

Add option to use a particular MapIt generation.

In April, new councils/wards will go live on MapIt but we don't want to
use them until the elections to those councils in May. This commit lets
WriteToThem continue to use an old MapIt generation.

Make sure other admin searches are limited in size

3431daf added pagination to person search, but this had the
unwanted side effect of removing any limit from other queue
searches (e.g. recently changed), causing awful performance
issues. Make sure all searches always have a limit.

Forward through auto-reply bounces.

If a bounce to a VERP address can't be parsed and has a subject that
looks like an automated reply rather than a delivery notification of
some sort, forward it on to the original message sender. Fixes #1.

Parse display name for multiple quoted strings.

Tidy up handling of phrase (display name) to be consistent throughout; a
phrase will be treated as is unless it starts and ends with double quote
marks, in which case it will be treated as a quoted string, unquoted and
unescaped.

Fixes comment in #13.

Bring regexes more in line with RFC5322.

FWS is still brought in as \s+ for simplicity. The obs_phrase test in
comment passes fine, so remove that code; space backtracking problems
will be fixed in the next commit.

This commit actually fixes the original example in #10 as $comment is
much simplified.

Update test to pass; test is actually incorrect in that the "comment"
should not be removed (RT#80665).

Return empty string from list import.

The documentation never said that a list was returned; indeed the example
provided is broken in that it prints out ARRAY(...) if run. Let us change
its behaviour to do as expected by the docs.

Switch uri to RFC3986 by default, match/fix docs.

The documentation said 2.26 switched to RFC3986 by default, but it did not;
that "~" would be escaped, but it never has been; and did not document that
'"' would now be correctly escaped.

‘require’ Script module, not ‘use’.

By using ‘use’, the package was loaded at compile time when the Smidsy
cobrand package was registered, which meant it tried to load a cobrand
before all the cobrands were loaded, leading not to an error but a new
empty cobrand package being created. This causes confusion when it was
then used in place of the actual cobrand package (because it was lower
case, it appeared after the actual package in Module::Pluggable’s list
as long as the version of M::P was recent enough).

Show Stats19 on homepage.

This also includes a new ID on the relevant FAQ, so we can link
directly to it from the homepage, plus some styling to make the
FAQ stand out when it's the :target of the URL hash fragment.

Remove fallback to [config dir]/packages.

This file is really designed (and maintained) to work on our own
deployment system; it's not a generic file that will work on any
non-Debian/Ubuntu Linux, and might not work when called via this
script as it may contain package dependency lines. Fixes #25.

Fallback to generic package files if present.

This allows a codebase to use packages.ubuntu, packages.debian and/or
packages.generic to supply a list of packages to install, rather than
assuming it can use precise on Ubuntu, or squeeze on Debian.

Change to right dir on private timestamped deploy.

For config updates, private repos were switching to the name of the
repo, which on a timestamped deploy would either not exist (a first
deploy), or be the previous deploy.

Also use the right ref on private repos that use MySQL (don't think
there are any at present!).

Allow app restart without restarting webserver.

If an app can be restarted by touching a file (e.g. a python wsgi.py
file), then you can specify this as `touch_to_restart` configuration
option and that will happen rather than Apache being restarted.

If the Apache config has changed (either the /etc/apache2 or the app
specific httpd.conf file), Apache will still be restarted.

Always charset decode, regardless of content type.

For example, `application/json; charset=utf-8` is one obvious contender
that should be decoded. If no charset is given or detected, assume none
rather than ISO-8859-1.

Update field usage to modern Django.

Add a deconstruct function for migrations; call from_db_value from
to_python rather than the other way round, and return "" for empty
dates rather than None (fixes #7); add filtering by python date or
datetime by using get_prep_value instead of get_db_prep_value #11.

Remove a reference to south model inspectors; we support supported
Django versions only.

Thanks to John Lehmann and Aamir Adnan for reporting/tests.

Create Source generic relation.

If a model has two GenericRelations to the same model, there is no way
to differentiate between them. So we add a new Source model to be used
for sources.

Still generate uniqueness lookups when a uniqueness field is empty.

In our specific example, we have Sections, with uniqueness on the tuple
of (parent, instance). parent is optional and can be None, but we still
want uniqueness of the tuple when that is the case.

When copying, do not preserve timestamps.

As currently written, this copies across a compiled file if the source
hasn't changed since last time, but that file will not be used because
the FixMyStreet code checks timestamps and only uses compiled files if
newer than source files.

I think originally, the idea was to copy across the old compiled file,
then compare against the new source, but that won't work as the source
file will always be newer as it is a new checkout.

If we don't preserve timestamps, then if the source is the same as the
last deploy, the compiled will be the same and it can be copied across
as a new file.

Use childNodes rather than getElementsByTagName.

getElementsByTagName() returns all descendants, which means it iterates
over children of things you have already discounted. childNodes returns
only the immediate children.

Allow MP names that differ only in honourifics.

This means if WriteToThem has Sir Malcolm Rifkind but TheyWorkForYou only has
Malcolm Rifkind, we still consider that a match. This might well apply to all
types this script handles, but will restrict just to MPs for now, seems to be
most likely place it could happen.

Better handling of map when no location.

Don't display:none the map (so setup code runs); it will still be hidden
by being covered by the sidebar, and then we can make the CSS reveal the
map when the latitude updates on someone adding a location. Plus fix the
pin being loaded.

Fix #8264 - Do not override ctrl/cmd+click. (#8265)

This includes clicks on hashtags, mentions, display names and media in the
timeline; and usernames in reply-indicator, detailed status, and the boost
modal.

Rewrite all proxied links before the display.

We have a multi-site WordPress install at blogs.mysociety.org, running
an instance for FixMyStreet Pro at /fixmystreet-pro/. We would like to
see that instance at www.fixmystreet.com/pro. Now I would have thought
that, as WordPress has separate site and home URL settings, this would
involve setting the home URL to /pro while leaving the site URL alone.
I was sadly mistaken.

As well as using the home URL for outputting absolute links, WordPress
strips its path element from REQUEST_URI in order to work out the base
of your installation. As we proxy /pro through to our WP installation,
this means WP sees /fixmystreet-pro/ as its REQUEST_URI, cannot locate
/pro in order to strip it and thinks the root is the base, then trying
to find a fixmystreet-pro top level page and failing.

We could rewrite REQUEST_URI in wp-config.php, but then the multi-site
code would be unable to find the right instance. In the end, we buffer
all the output and rewrite any non-admin links to what they should be.
This also fixes any absolute links within text bodies.

Check TOCs when testing against time restrictions.

This is so e.g. /KTN/MAN/y/09:30 correctly returns the off-peak
ticket (because although the departure time is before 09:30, it
is not either CrossCounty or London Midland).