Comments on: Internal links, and search engine crawlers http://www.mysociety.org/2008/07/17/internal-links-and-search-engine-crawlers/ Relentless user-focus on civic websites Fri, 10 Feb 2012 01:00:32 +0000 hourly 1 http://wordpress.org/?v=3.0.5 By: Matthew http://www.mysociety.org/2008/07/17/internal-links-and-search-engine-crawlers/comment-page-1/#comment-10754 Matthew Mon, 08 Sep 2008 13:13:43 +0000 http://www.mysociety.org/2008/07/17/internal-links-and-search-engine-crawlers/#comment-10754 The problem is there is no quote - they simply say "I refer the hon. Gentleman to the previous answer I gave [Official Report, 29 February 2008, column 1425]" - so there's no way to know which speech/answer within that column they mean automatically (it could try and match e.g. speaker name, even subject, but that's hard graft for not much gain). The problem is there is no quote – they simply say “I refer the hon. Gentleman to the previous answer I gave [Official Report, 29 February 2008, column 1425]” – so there’s no way to know which speech/answer within that column they mean automatically (it could try and match e.g. speaker name, even subject, but that’s hard graft for not much gain).

]]> By: Mark Longair http://www.mysociety.org/2008/07/17/internal-links-and-search-engine-crawlers/comment-page-1/#comment-10286 Mark Longair Sun, 07 Sep 2008 15:47:26 +0000 http://www.mysociety.org/2008/07/17/internal-links-and-search-engine-crawlers/#comment-10286 For what it's worth, quotations in the Scottish Parliament text on TheyWorkForYou should mostly link to the quoted text by using a different mechanism - the parser tries to match substrings of the quotation in the speeches from the particular day that's referenced. This isn't generally useful in the same way, of course, but I thought it worked surprisingly well, and is perhaps worth mentioning in this context. <i>"Perhaps in future we’ll be able to add some crowd-sourcing game to match the reference to the exact speech"</i> I don't think you need to do this (in most cases, anyway) based on the experience of automatic matching of quotations in the SP parser. For what it’s worth, quotations in the Scottish Parliament text on TheyWorkForYou should mostly link to the quoted text by using a different mechanism – the parser tries to match substrings of the quotation in the speeches from the particular day that’s referenced. This isn’t generally useful in the same way, of course, but I thought it worked surprisingly well, and is perhaps worth mentioning in this context.

“Perhaps in future we’ll be able to add some crowd-sourcing game to match the reference to the exact speech”

I don’t think you need to do this (in most cases, anyway) based on the experience of automatic matching of quotations in the SP parser.

]]> By: Francis Irving http://www.mysociety.org/2008/07/17/internal-links-and-search-engine-crawlers/comment-page-1/#comment-1988 Francis Irving Thu, 17 Jul 2008 18:55:13 +0000 http://www.mysociety.org/2008/07/17/internal-links-and-search-engine-crawlers/#comment-1988 Seems Yahoo parses the robots file in an interesting way. Firstly, our file was wrong for not including the Disallow lines for Yahoo as well - explaining why Yahoo was browsing the /user. But not why it was going quickly still. Matthew's patch to our robots.txt: https://secure.mysociety.org/cvstrac/chngview?cn=12284 Seems Yahoo parses the robots file in an interesting way. Firstly, our file was wrong for not including the Disallow lines for Yahoo as well – explaining why Yahoo was browsing the /user. But not why it was going quickly still.

Matthew’s patch to our robots.txt:
https://secure.mysociety.org/cvstrac/chngview?cn=12284

]]>