Page Speed Optimization Libraries
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
Public Types | Public Member Functions | Static Public Member Functions | Protected Types | Protected Member Functions | Protected Attributes | Friends | List of all members
net_instaweb::SerfUrlAsyncFetcher Class Reference

#include "serf_url_async_fetcher.h"

Inheritance diagram for net_instaweb::SerfUrlAsyncFetcher:

Public Types

enum  WaitChoice { kThreadedOnly, kMainlineOnly, kThreadedAndMainline }

Public Member Functions

 SerfUrlAsyncFetcher (const char *proxy, apr_pool_t *pool, ThreadSystem *thread_system, Statistics *statistics, Timer *timer, int64 timeout_ms, MessageHandler *handler)
 SerfUrlAsyncFetcher (SerfUrlAsyncFetcher *parent, const char *proxy)
virtual void ShutDown ()
virtual bool SupportsHttps () const
virtual void Fetch (const GoogleString &url, MessageHandler *message_handler, AsyncFetch *callback)
int Poll (int64 max_wait_ms)
bool WaitForActiveFetches (int64 max_milliseconds, MessageHandler *message_handler, WaitChoice wait_choice)
void FetchComplete (SerfFetch *fetch)
void ReportCompletedFetchStats (const SerfFetch *fetch)
 Update the statistics object with results of the (completed) fetch.
void ReportFetchSuccessStats (SerfCompletionResult result, const ResponseHeaders *headers, const SerfFetch *fetch)
 Updates states used for success/failure monitoring.
apr_pool_t * pool () const
void PrintActiveFetches (MessageHandler *handler) const
virtual int64 timeout_ms ()
ThreadSystemthread_system ()
void set_list_outstanding_urls_on_error (bool x)
bool track_original_content_length () const
void set_track_original_content_length (bool x)
bool SetHttpsOptions (StringPiece directive)
void SetSslCertificatesDir (StringPiece dir)
const GoogleStringssl_certificates_dir () const
void SetSslCertificatesFile (StringPiece file)
const GoogleStringssl_certificates_file () const
- Public Member Functions inherited from net_instaweb::UrlAsyncFetcher
void set_fetch_with_gzip (bool x)
bool fetch_with_gzip () const
AsyncFetchEnableInflation (AsyncFetch *fetch) const

Static Public Member Functions

static void InitStats (Statistics *statistics)
static bool ValidateHttpsOptions (StringPiece directive, GoogleString *error_message)

Protected Types

typedef Pool< SerfFetchSerfFetchPool
typedef std::vector< SerfFetch * > FetchVector

Protected Member Functions

bool allow_https () const
 Determines whether https is allowed in the current configuration.
bool allow_self_signed () const
bool allow_unknown_certificate_authority () const
bool allow_certificate_not_yet_valid () const
void set_https_options (uint32 https_options)
void Init (apr_pool_t *parent_pool, const char *proxy) EXCLUSIVE_LOCKS_REQUIRED(mutex_)
bool SetupProxy (const char *proxy) EXCLUSIVE_LOCKS_REQUIRED(mutex_)
bool StartFetch (SerfFetch *fetch) EXCLUSIVE_LOCKS_REQUIRED(mutex_)
virtual bool AnyPendingFetches ()
int ApproximateNumActiveFetches ()
void CancelActiveFetches ()
void CancelActiveFetchesMutexHeld () EXCLUSIVE_LOCKS_REQUIRED(mutex_)
bool WaitForActiveFetchesHelper (int64 max_ms, MessageHandler *message_handler)
void CleanupFetchesWithErrors () EXCLUSIVE_LOCKS_REQUIRED(mutex_)
bool shutdown () const EXCLUSIVE_LOCKS_REQUIRED(mutex_)
void set_shutdown (bool s) EXCLUSIVE_LOCKS_REQUIRED(mutex_)
- Protected Member Functions inherited from net_instaweb::UrlAsyncFetcher
 UrlAsyncFetcher ()

Protected Attributes

apr_pool_t * pool_
SerfFetchPool completed_fetches_
SerfThreadedFetcher * threaded_fetcher_


class SerfFetch
 To access stats variables below.

Additional Inherited Members

- Static Public Attributes inherited from net_instaweb::UrlAsyncFetcher
static const int64 kUnspecifiedTimeout

Detailed Description

TODO(sligocki): Serf does not seem to act appropriately in IPv6 environments, fix and test this. Specifically: (1) It does not attempt to fall-back to IPv4 if IPv6 connection fails; (2) It may not correctly signal failure, which causes the incoming connection to hang.

Member Function Documentation

virtual bool net_instaweb::SerfUrlAsyncFetcher::AnyPendingFetches ( )

AnyPendingFetches is accurate only at the time of call; this is used conservatively during shutdown. It counts fetches that have been requested by some thread, and can include fetches for which no action has yet been taken (ie fetches that are not active).

int net_instaweb::SerfUrlAsyncFetcher::ApproximateNumActiveFetches ( )

ApproximateNumActiveFetches can under- or over-count and is used only for error reporting.

void net_instaweb::SerfUrlAsyncFetcher::CleanupFetchesWithErrors ( )

This cleans up the serf resources for fetches that errored out. Must be called only immediately after running the serf event loop.

virtual void net_instaweb::SerfUrlAsyncFetcher::Fetch ( const GoogleString url,
MessageHandler message_handler,
AsyncFetch fetch 

Asynchronously fetch a URL, set the response headers and stream the contents to fetch and call fetch->Done() when the fetch finishes.

There is an unchecked contract that response_headers are set before the response_writer or callback are used. Caution, several implementations do not satisfy this contract (but should).


TODO(sligocki): GoogleString -> GoogleUrl or at least StringPiece.

TODO(sligocki): Include the URL in the fetch, like the request headers.

Implements net_instaweb::UrlAsyncFetcher.

void net_instaweb::SerfUrlAsyncFetcher::FetchComplete ( SerfFetch fetch)

Remove the completed fetch from the active fetch set, and put it into a completed fetch list to be cleaned up.

int net_instaweb::SerfUrlAsyncFetcher::Poll ( int64  max_wait_ms)
TODO(morlovich): Make private once non-thread mode concept removed.
void net_instaweb::SerfUrlAsyncFetcher::set_list_outstanding_urls_on_error ( bool  x)

Indicates that Serf should enumerate failing URLs whenever the underlying Serf library reports an error.

bool net_instaweb::SerfUrlAsyncFetcher::SetHttpsOptions ( StringPiece  directive)

Indicates that direct HTTPS fetching should be allowed, and how picky to be about certificates. The directive is a comma separated list of these keywords: enable disable allow_self_signed allow_unknown_certificate_authority allow_certificate_not_yet_valid Returns 'false' if the directive does not parse properly.

virtual void net_instaweb::SerfUrlAsyncFetcher::ShutDown ( )

Stops all active fetches and prevents further fetches from starting (they will instead quickly call back to ->Done(false).

Reimplemented from net_instaweb::UrlAsyncFetcher.

bool net_instaweb::SerfUrlAsyncFetcher::StartFetch ( SerfFetch fetch)

Start a SerfFetch. Takes ownership of fetch and makes sure callback is called even if fetch fails to start.

virtual bool net_instaweb::SerfUrlAsyncFetcher::SupportsHttps ( ) const

Determine if the fetcher supports fetching using HTTPS. By default we assume a fetcher can.

Reimplemented from net_instaweb::UrlAsyncFetcher.

virtual int64 net_instaweb::SerfUrlAsyncFetcher::timeout_ms ( )

Returns a maximum time that we will allow fetches to take, or kUnspecifiedTimeout (the default) if we don't promise to timeout fetches.

Reimplemented from net_instaweb::UrlAsyncFetcher.

bool net_instaweb::SerfUrlAsyncFetcher::track_original_content_length ( ) const

Indicates that Serf should track the original content length for fetched resources.

static bool net_instaweb::SerfUrlAsyncFetcher::ValidateHttpsOptions ( StringPiece  directive,
GoogleString error_message 

Validates the correctness of an https directive. Exposed as a static method for early exit on mis-specified pagespeed.conf.

Member Data Documentation

UpDownCounter* net_instaweb::SerfUrlAsyncFetcher::active_count_

This is protected because it's updated along with active_fetches_, which happens in subclass SerfThreadedFetcher as well as this class.

ThreadSystem::CondvarCapableMutex* net_instaweb::SerfUrlAsyncFetcher::mutex_

mutex_ protects serf_context_, active_fetches_ and shutdown_. It's protected because SerfThreadedFetcher needs access.

The documentation for this class was generated from the following file: