Research issues in context-aware retrieval: evaluation of straying outside a context-aware cache

Peter Brown  and Gareth Jones
Department of Computer Science, University of Exeter, Exeter EX4 4PT, UK
P.J.Brown@ex.ac.uk, G.J.F.Jones@ex.ac.uk

Summary

This experiment measures recall degradation when the user's path of locations strays outside the area covered by a cache.

Fixed parameters of the experiment

Assumed purpose of cache:

to cover disconnected operation.

Contextual fields used:

just location.

Document collection:

items relating to tourist attractions in the South West; consists of 626 items; reasonably evenly distributed over the geographical area, but with some concentration on the towns and cities. All attractions are on land, and the border of the area covered consists of coastline, plus county boundaries between those deemed to be in the South West and those not.

How the cache was built

the cache for each test was built using a context consisting of a square, 20 km by 20 km, centred on the starting point for the test. This cache was built using a threshold score of 50%. It was assumed that the user's device would be big enough to hold any of the caches generated (not a unrealistic assumption given the small side of the document collection, and hence the even smaller size of the caches).

Matching and scoring algorithms:

the default ones built into the Context Matcher. Scores for matching two locations decay roughly linearly as the locations move apart (actual algorithm: score = 2.0 * ((target - abs(target - query)) / target) ).

Experimental approach

The experiment consisted of a number of individual tests. Each test had a starting point, and the cache for the test was built using this starting point. The starting points were chosen by hand, some being in the middle of popular towns and cities and some being `in the middle of nowhere'. The choice was based on hunch of where a tourist aid would be most used, not on the basis of any deep analysis.

In each test the user was assumed to proceed steadily in a straight-line path, performing a retrieval at regular retrieval points, a fixed distance apart. There were 11 retrieval points: the first was at the starting point of the test; the sixth was on the very edge of the area covered by the cache, and the remainder were increasingly further outside this area. The straight line paths were chosen so that the user remained in the overall area covered by the document collection, i.e. the South West of England (and did not stray into the sea!). Todo: add picture of cache area and retrieval points.

Each retrieval used the cache; the results of this retrieval were compared with what would have been retrieved if the whole document collection had been used tather that the cache. Each retrieval retrieved all the documents whose score was greater than a given threshold (a document's score would, not of course, be affected by whether it came from the cache or from the original document collection). A count was made of the total number of documents "lost" over the 11 retrieval points (i.e. documents retrieved from the original document collection and not the cache), and of the total number of documents retrieved from the cache.

Each test was repeated -- we call this a sequence of sub-tests -- with different threshold (the same threshold was used for all 11 retrievals).

Results of individual tests

Test with starting point at Dartmoor

Commentary on this starting point: the cache is centred on a wild moorland area.

Test with starting point at Exeter

Commentary on this starting point: this may be a favourable case as the cache is centred on an area with lots of attractions, and the area outside the cache has fewer attractions.

Test with starting point at Plymouth

Commentary on this starting point: this may be a favourable case as the cache is centred on an area with lots of attractions, and the area outside the cache has fewer attractions.

Test with starting point at Somerton

Commentary on this starting point: the cache is centred on non-prime tourist area.