[oae-production] Preview Processor High Availability

[Lipscomb, Gary]

Hi all,

        Thanks for the update.
        My understanding from this is that if we are running one preview processor server and this dies we can bring up another and it will continue processing from where the original stopped since all the information is stored in "/var/search/needsprocessing.json" from nakamura.

        The two issues I can see are:-
1. how do we re-process the failed jobs
2. processing failures as mentioned by Ray.

What is there in the log to tell us that the process has failed - for automatic monitoring and alerting purposes.

thanks
Gary

|-----Original Message-----
|From: oae-production-bounces at collab.sakaiproject.org [mailto:oae-
|production-bounces at collab.sakaiproject.org] On Behalf Of Ray Davis
|Sent: Thursday, 6 October 2011 3:03 AM
|To: oae-production at collab.sakaiproject.org
|Subject: Re: [oae-production] Preview Processor High Availability
|
|I haven't spent as much time as I'd like looking at it, but one thing I
|noticed about the Preview Processor early on is that "false negatives"
|(processing failures caused by temporary glitches rather than bad data)
|can permanently take documents out of the needsprocessing list. I don't
|know if there's a perfect way to deal with that, but it seems like
|something to watch out for when trying to enable multiple processors.
|
|Best,
|Ray
|
|On 10/5/11 5:35 AM, N. Matthijs wrote:
|> Hi Gary,
|>
|>> On Wed, Oct 05, 2011 at 01:26:37PM +1100, Lipscomb, Gary wrote:
|>>> *         How does the preview-processor keep track of what it has
|>>> processed?
|>>
|>> Other than gleaning what has happened from local logs the content
|>> preview service does not maintain state locally. However, nakamura
|>> itself does and presents that info at:
|>>
|>> /var/search/needsprocessing.json
|>
|> Yes, each content item has got a property which indicates whether
|> it has been processed yet or not. Every 20 seconds, the preview
|> processor will pull down the 10 oldest unprocessed files and set
|> the property to processed (or failed) when finished.
|>
|>>> *         Is it possible to have more than one running concurrently?
|>>
|>> Possible, yes, but they'd likely end up pulling down the same or a
|>> similar list of tasks to process and so even if the concurrent
|>> updates caused no actual problems you'd not gain much. I could see
|>> an extension whereby nakamura was made aware of how many preview
|>> nodes existed and presented different lists on different urls for
|>> each processing node, but in practice one node is pretty good at
|>> keeping up with a reasonable volume of ingest. The snapshots don't
|>> have to appear instantly, a small delay is fine and so the preview
|>> node can always catch up in quieter moments, etc.
|>
|> Yes, running multiple processors concurrently would not give you a
|> lot of benefit, as they'd be processing the same set of files all
|> the time. However, I think we can support this with a number of
|> minor changes, and we'd be happy to work on a patch with you if
|> you'd be interested in that.
|>
|>>> *         If we have to build a new preview-processor server how
|>>> does it know what has already been processed or does it start
|>>> from scratch. If so, does it overwrite the existing jpegs>
|>>
|>> As above, it fetches the list of outstanding items from
|>> /var/search/needsprocessing.json and starts chugging away.
|>
|> Thanks,
|> Nicolaas
|>
|> _______________________________________________
|> oae-production mailing list
|> oae-production at collab.sakaiproject.org
|> http://collab.sakaiproject.org/mailman/listinfo/oae-production
|>
|
|_______________________________________________
|oae-production mailing list
|oae-production at collab.sakaiproject.org
|http://collab.sakaiproject.org/mailman/listinfo/oae-production
Charles Sturt University

| ALBURY-WODONGA | BATHURST | CANBERRA | DUBBO | GOULBURN | ONTARIO | ORANGE | SYDNEY | WAGGA WAGGA |

Give Generously - Support Young Australians

You can help young Australians to go to University and succeed in their studies by giving generously to the Charles Sturt University Foundation. To find out more or to make a donation go to the Foundation web site http://www.csu.edu.au/special/foundation. Australian donations are tax deductible.

LEGAL NOTICE
This email (and any attachment) is confidential and is intended for the use of the addressee(s) only. If you are not the intended recipient of this email, you must not copy, distribute, take any action in reliance on it or disclose it to anyone. Any confidentiality is not waived or lost by reason of mistaken delivery. Email should be checked for viruses and defects before opening. Charles Sturt University (CSU) does not accept liability for viruses or any consequence which arise as a result of this email transmission. Email communications with CSU may be subject to automated email filtering, which could result in the delay or deletion of a legitimate email before it is read at CSU. The views expressed in this email are not necessarily those of CSU.

Charles Sturt University in Australia  http://www.csu.edu.au  The Chancellery, Panorama Avenue, Bathurst NSW Australia 2795  ABN: 83 878 708 551; CRICOS Provider Numbers: 00005F (NSW), 01947G (VIC), 02960B (ACT)

Charles Sturt University in Ontario  http://www.charlessturt.ca 860 Harrington Court, Burlington Ontario Canada L7N 3N4  Registration: www.peqab.ca

Consider the environment before printing this email.