[Proj] Testing framework

Kristian Evers kreve at sdfe.dk
Tue May 29 13:35:47 EST 2018 

All,

As hinted in previous mails I have been working on a proposal to improve
testing in general. Here are my thougts on the test setup in PROJ and how we
can improve it. The testing situation in PROJ is not particularly good at the
moment. It is a lot better than a couple of years ago but there is still a lot
that can be improved. Even has already settled on a testing framework for the
new features related to WKT/WKT2- support and later on also the active use of
the EPSG database. That still leaves the rest, which could use a bit of spring
cleaning. I'll try to address that below.

Today we have:

    1. nad/
        Shell scripts that call cs2cs with various input and checks the
        output against a distributed "answer book". These tests does not work
        on Windows (unless using cygwin or something similar).

    2. src/
        Individual programs testing the threading capabilities of PROJ
        (multistresstest.c & test228.c) and the geodesic library (geodtest.c).

    3. test/
        The gie test-suite [0]. Gie was created to make it simpler to write
        test cases for PROJ. A number of test files for gie are present
        in test/gie/ and test/gigs. These are mostly comprised of the auto-
        generated tests from the 'proj -C' selftest that was introduced with
        PROJ 4.9.3 (and removed again in 5.0.0) placed in builtins.gie. A
        handful of other test files have been written adding more "sane" test
        data. As part of the gie test-suite the GIGS [1] test data is included
        as well. Gie is a good environment for testing coordinate conversions
        given a setup string that can go into the proj_create() function.

    4. gie.c
        gie incoorporates some remnants of the selftest of PROJ 4.9.3 that can
        not be translated to the gie test file format. They are mostly simple
        API function tests.

The above tests do a fairly good job of testing coordinate input and output of
transformations. They also directly test a smaller subset of the functions
exposed in proj_api.h and proj.h (mostly the latter). The rest of the functions
are to some extent covered indirectly by the tests in nad/ and gie/. This is
not a particularly good way of testing the functionality of these functions
since edge cases are likely to be missed.

With the adoption of Catch2, as suggested by Even, we should be able to move
the tests from src/ and gie.c to this dedicated framework, preferably somewhere
in the test/ directory to keep things in a logical place separated from the
source code.

I would also like to move the tests in nad/ to test/. Initially they can just
be moved more or less as is but eventually I would like to see them migrated to
the gie format. It is not totally straight-forward but with a bit of scripting
it should be possible to automate. Doing that has the unfortunate side-effect
of not testing the cs2cs application any more since gie works with API
functions. Therefore a dedicated test setup for that is needed. I regard that
as a good thing, since that can also be used with proj, cct and geod.

So in summation my proposed TODO list goes something like:

    1. Move tests from src/ to the Catch2 framework
    2. Move selftest remnants in gie.c to the Catch2 framework
    3. Move shell-script tests in nad/ to test/
    4. Add a test framework for the command line applications, also in test/

Thoughts? What have I missed? Any suggestions on how to test the command line
applications?

[0] https://proj4.org/apps/gie.html
[1] https://www.iogp.org/bookstore/product/geospatial-integrity-of-geoscience-software-part-1-gigs-guidelines/


On 29 May 2018, at 20:24, Kristian Evers <kreve at sdfe.dk<mailto:kreve at sdfe.dk>> wrote:

Mateusz,

That was a good assumption at the time :-) Things are a little bit different now.
It would seem that Catch2 is also good for C code, yes? Obviously compiled
with a C++ compiler. The documentation is a bit vague on this (it just says that
maybe it works).

I am working on a proposal to improve testing of the existing code. So far I have
been working under the assumption that Catch2 would work for testing the
C code we have now. If not I will change that to cunit but it would definitely be
nice to just use one framework.

/Kristian

On 29 May 2018, at 20:02, Mateusz Loskot <mateusz at loskot.net<mailto:mateusz at loskot.net>> wrote:

Kristian, as you know, I've been considering PROJ API tests.
I assumed it would must be in C, so an obvious choice was cunit (as used in PostGIS for instance). I haven't arrived with anything though.

It's just a comment, not recommendation to stick with C and cunit.

Mateusz Loskot, mateusz at loskot.net<mailto:mateusz at loskot.net>
(Sent from mobile)

On Tue, 29 May 2018, 19:26 Kristian Evers, <kreve at sdfe.dk<mailto:kreve at sdfe.dk>> wrote:
Even,

Your timing is perfect. I was literally just typing up a long email about improving
the test situation in PROJ right now! I was going to propose a larger restructuring
of the tests we already have as well as adding a dedicated testing framework for
testing API functions etc. You’ve taken care of the latter and most pressing issue.
I’ll address the rest of my proposal at a later time I think.

I have only looked briefly at catch2 but it seems good to me. I am happy that you’ve
put your tests in test/.

/Kristian

> On 29 May 2018, at 19:15, Even Rouault <even.rouault at spatialys.com<mailto:even.rouault at spatialys.com>> wrote:
>
> Hi,
>
> I was researching a framework to test my new code (and that could also be used
> to test the existing C API if needed). Currently src/gie.c has ad-hoc testing,
> but it is really limited feature-wise: no nice error message (returns error
> code), no way to make a testcase go on even if a test check fails, etc...
>
> So a dedicated framework seems a better idea, and I've found catch2 :
>       https://github.com/catchorg/Catch2/blob/master/docs/Readme.md
>
> One of its main feature it is a header only testing framework, which means we
> can embed it easily in proj source tree, which is practical compared to other
> frameworks I've considered ( googletest, cppunit,  etc... ).
> The tut framework used by GDAL and GEOS
> ( http://mrzechonek.github.io/tut-framework ) would also enter this category
> of header(s) only, but it has not as much as activity than catch2.
> There's also Boost.Test, but I was a bit afraid with the boost name in it
> (although apparently it has a standalone mode).
>
> Example of tests I've just written with catch2 (just a minimalistic use of the
> testing framework)
> https://github.com/rouault/proj.4/blob/iso19111_ptr/test/cpp/test_crs.cpp
>
> I'm not particularly calling for a flame debate on the suject, just wanting to
> mention this finding.
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com<http://www.spatialys.com/>
> _______________________________________________
> Proj mailing list
> Proj at lists.maptools.org<mailto:Proj at lists.maptools.org>
> http://lists.maptools.org/mailman/listinfo/proj

_______________________________________________
Proj mailing list
Proj at lists.maptools.org<mailto:Proj at lists.maptools.org>
http://lists.maptools.org/mailman/listinfo/proj
_______________________________________________
Proj mailing list
Proj at lists.maptools.org<mailto:Proj at lists.maptools.org>
http://lists.maptools.org/mailman/listinfo/proj

_______________________________________________
Proj mailing list
Proj at lists.maptools.org<mailto:Proj at lists.maptools.org>
http://lists.maptools.org/mailman/listinfo/proj

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.maptools.org/pipermail/proj/attachments/20180529/223ecc12/attachment-0001.htm

More information about the Proj mailing list