Page Speed Optimization Libraries
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
Public Types | Public Member Functions | Static Public Member Functions | Static Public Attributes | Protected Member Functions | Friends | List of all members
net_instaweb::ServerContext Class Referenceabstract

#include "server_context.h"

Inheritance diagram for net_instaweb::ServerContext:
net_instaweb::SystemServerContext net_instaweb::ApacheServerContext

Public Types

enum  Format { kFormatAsHtml, kFormatAsJson }
typedef std::pair
< RewriteOptions *, bool > 

Public Member Functions

 ServerContext (RewriteDriverFactory *factory)
void SetDefaultLongCacheHeaders (const ContentType *content_type, StringPiece charset, StringPiece cache_control_suffix, ResponseHeaders *header) const
void set_filename_prefix (const StringPiece &file_prefix)
void set_statistics (Statistics *x)
void set_rewrite_stats (RewriteStats *x)
void set_lock_manager (NamedLockManager *x)
void set_enable_property_cache (bool enabled)
void set_message_handler (MessageHandler *x)
StringPiece filename_prefix () const
Statisticsstatistics () const
NamedLockManagerlock_manager () const
RewriteDriverFactoryfactory () const
ThreadSynchronizerthread_synchronizer ()
ExperimentMatcherexperiment_matcher ()
void ApplyInputCacheControl (const ResourceVector &inputs, ResponseHeaders *headers)
bool IsPagespeedResource (const GoogleUrl &url) const
 Is this URL a ref to a Pagespeed resource?
const RewriteFilterFindFilterForDecoding (const StringPiece &id) const
bool DecodeUrlGivenOptions (const GoogleUrl &url, const RewriteOptions *options, const UrlNamer *url_namer, StringVector *decoded_urls) const
 See RewriteDriver::DecodeUrl.
void ComputeSignature (RewriteOptions *rewrite_options) const
Hasherhasher () const
const Hasherlock_hasher () const
const Hashercontents_hasher () const
FileSystemfile_system ()
void set_file_system (FileSystem *fs)
UrlNamerurl_namer () const
void set_url_namer (UrlNamer *n)
RewriteOptionsManagerrewrite_options_manager () const
SHA1Signaturesignature () const
void SetRewriteOptionsManager (RewriteOptionsManager *rom)
 Takes ownership of RewriteOptionsManager.
StaticAssetManagerstatic_asset_manager () const
void set_static_asset_manager (StaticAssetManager *manager)
Schedulerscheduler () const
void set_scheduler (Scheduler *s)
bool has_default_system_fetcher () const
UrlAsyncFetcherDefaultSystemFetcher ()
virtual CacheUrlAsyncFetcherCreateCustomCacheFetcher (const RewriteOptions *options, const GoogleString &fragment, CacheUrlAsyncFetcher::AsyncOpHooks *hooks, UrlAsyncFetcher *fetcher)
Timertimer () const
void set_timer (Timer *timer)
 Note: doesn't take ownership.
HTTPCachehttp_cache () const
void set_http_cache (HTTPCache *x)
void MakePagePropertyCache (PropertyStore *property_store)
 Creates PagePropertyCache object with the provided PropertyStore object.
PropertyCachepage_property_cache () const
const PropertyCache::Cohortdom_cohort () const
void set_dom_cohort (const PropertyCache::Cohort *c)
const PropertyCache::Cohortbeacon_cohort () const
void set_beacon_cohort (const PropertyCache::Cohort *c)
const PropertyCache::Cohortdependencies_cohort () const
void set_dependencies_cohort (const PropertyCache::Cohort *c)
const PropertyCache::Cohortfix_reflow_cohort () const
void set_fix_reflow_cohort (const PropertyCache::Cohort *c)
CacheInterfacefilesystem_metadata_cache () const
void set_filesystem_metadata_cache (CacheInterface *x)
CacheInterfacemetadata_cache () const
void set_metadata_cache (CacheInterface *x)
CriticalImagesFindercritical_images_finder () const
void set_critical_images_finder (CriticalImagesFinder *finder)
CriticalSelectorFindercritical_selector_finder () const
void set_critical_selector_finder (CriticalSelectorFinder *finder)
UserAgentMatcheruser_agent_matcher () const
void set_user_agent_matcher (UserAgentMatcher *n)
SimpleRandomsimple_random ()
bool store_outputs_in_file_system ()
void set_store_outputs_in_file_system (bool store)
RewriteStatsrewrite_stats () const
MessageHandlermessage_handler () const
NamedLockMakeCreationLock (const GoogleString &name)
void TryLockForCreation (NamedLock *creation_lock, Function *callback)
 Attempt to obtain a named lock without blocking. Return true if we do so.
void LockForCreation (NamedLock *creation_lock, Sequence *worker, Function *callback)
void set_hasher (Hasher *hasher)
 Setters should probably only be used in testing.
void set_signature (SHA1Signature *signature)
void set_default_system_fetcher (UrlAsyncFetcher *fetcher)
bool HandleBeacon (StringPiece body, StringPiece user_agent, const RequestContextPtr &request_context)
RewriteOptionsglobal_options ()
const RewriteOptionsglobal_options () const
void reset_global_options (RewriteOptions *options)
RewriteOptionsNewOptions ()
 Makes a new, empty set of RewriteOptions.
bool GetQueryOptions (const RequestContextPtr &request_context, const RewriteOptions *domain_options, GoogleUrl *request_url, RequestHeaders *request_headers, ResponseHeaders *response_headers, RewriteQuery *rewrite_query)
void GetRemoteOptions (RewriteOptions *remote_options, bool on_startup)
RewriteOptionsGetCustomOptions (RequestHeaders *request_headers, RewriteOptions *domain_options, RewriteOptions *query_options)
GoogleString GetRewriteOptionsSignatureHash (const RewriteOptions *options)
RewriteDriverNewRewriteDriver (const RequestContextPtr &request_ctx)
RewriteDriverNewRewriteDriverFromPool (RewriteDriverPool *pool, const RequestContextPtr &request_ctx)
RewriteDriverNewUnmanagedRewriteDriver (RewriteDriverPool *pool, RewriteOptions *options, const RequestContextPtr &request_ctx)
RewriteDriverNewCustomRewriteDriver (RewriteOptions *custom_options, const RequestContextPtr &request_ctx)
void ReleaseRewriteDriver (RewriteDriver *rewrite_driver)
ThreadSystemthread_system ()
UsageDataReporterusage_data_reporter ()
void set_shutting_down ()
bool shutting_down () const
void ShutDownDrivers (int64 cutoff_time_ms)
void MergeNonCachingResponseHeaders (const ResourcePtr &input, const OutputResourcePtr &output)
void MergeNonCachingResponseHeaders (const ResponseHeaders &input_headers, ResponseHeaders *output_headers)
 Entry-point with the same functionality, exposed for easier testing.
QueuedWorkerPoolhtml_workers ()
 Pool of worker-threads that can be used to handle html-parsing.
QueuedWorkerPoolrewrite_workers ()
 Pool of worker-threads that can be used to handle resource rewriting.
QueuedWorkerPoollow_priority_rewrite_workers ()
size_t num_active_rewrite_drivers ()
void InitWorkers ()
virtual void PostInitHook ()
 To set up AdminSite for SystemServerContext.
bool response_headers_finalized () const
void set_response_headers_finalized (bool x)
RewriteDriverPoolstandard_rewrite_driver_pool ()
const GoogleStringhostname () const
 Returns the current server hostname.
void set_hostname (const GoogleString &x)
void set_central_controller (std::shared_ptr< CentralController > controller)
CentralControllercentral_controller ()
void AddOriginalContentLengthHeader (const ResourceVector &inputs, ResponseHeaders *headers)
virtual void ApplySessionFetchers (const RequestContextPtr &req, RewriteDriver *driver)
virtual bool ProxiesHtml () const =0
RequestPropertiesNewRequestProperties ()
 Makes a new RequestProperties.
void DeleteCacheOnDestruction (CacheInterface *cache)
void set_cache_property_store (CachePropertyStore *p)
void set_decoding_driver (RewriteDriver *rd)
virtual PropertyStoreCreatePropertyStore (CacheInterface *cache_backend)
 Creates CachePropertyStore object which will be used by PagePropertyCache.
const PropertyCache::CohortAddCohort (const GoogleString &cohort_name, PropertyCache *pcache)
const PropertyCache::CohortAddCohortWithCache (const GoogleString &cohort_name, CacheInterface *cache, PropertyCache *pcache)
const CacheInterfacepcache_cache_backend ()
js_tokenizer_patterns () const
void ShowCacheHandler (Format format, StringPiece url, StringPiece ua, bool should_delete, AsyncFetch *fetch, RewriteOptions *options)
virtual GoogleString FormatOption (StringPiece option_name, StringPiece args)

Static Public Member Functions

static bool IsExcludedAttribute (const char *attribute)
static GoogleString ShowCacheForm (StringPiece user_agent)

Static Public Attributes

static const int64 kGeneratedMaxAgeMs
 The lifetime for cache-extended generated resources, in milliseconds.
static const int64 kCacheTtlForMismatchedContentMs
 Short lifetime for generated resources fetched with mismatching hash.
static const char kResourceEtagValue []
static const char kCacheKeyResourceNamePrefix []

Protected Member Functions

void ManageRewriteDriverPool (RewriteDriverPool *pool)


class ServerContextTest

Detailed Description

Server-specific context and platform adaption: threads, file system, locking, and so on.

TODO(piatek): This file was renamed from resource_manager.h – there are some associated fixes outstanding –
  1. abc of includes and class declarations.
  2. Rename variables, data members, parameters, etc.
  3. Rename methods.

Member Function Documentation

const PropertyCache::Cohort* net_instaweb::ServerContext::AddCohort ( const GoogleString cohort_name,
PropertyCache pcache 

Establishes a new Cohort for this property. This will also call CachePropertyStore::AddCohort() if CachePropertyStore is used.

const PropertyCache::Cohort* net_instaweb::ServerContext::AddCohortWithCache ( const GoogleString cohort_name,
CacheInterface cache,
PropertyCache pcache 

Establishes a new Cohort to be backed by the specified CacheInterface. NOTE: Does not take ownership of the CacheInterface object. This also calls CachePropertyStore::AddCohort() to set the cache backend for the given cohort.

void net_instaweb::ServerContext::AddOriginalContentLengthHeader ( const ResourceVector &  inputs,
ResponseHeaders headers 

Adds an X-Original-Content-Length header to the response headers based on the size of the input resources.

void net_instaweb::ServerContext::ApplyInputCacheControl ( const ResourceVector &  inputs,
ResponseHeaders headers 

Computes the most restrictive Cache-Control intersection of the input resources, and the provided headers, and sets that cache-control on the provided headers. Does nothing if all of the resources are fully cacheable, since in that case we will want to cache-extend.

Disregards Cache-Control directives other than max-age, no-cache, no-store, and private, and strips them if any resource is no-cache or private. By assumption, a resource can only be no-store if it is also no-cache.

virtual void net_instaweb::ServerContext::ApplySessionFetchers ( const RequestContextPtr req,
RewriteDriver driver 

Provides a hook for ServerContext implementations to determine the fetcher implementation based on the request.

Reimplemented in net_instaweb::SystemServerContext.

virtual CacheUrlAsyncFetcher* net_instaweb::ServerContext::CreateCustomCacheFetcher ( const RewriteOptions options,
const GoogleString fragment,
CacheUrlAsyncFetcher::AsyncOpHooks hooks,
UrlAsyncFetcher fetcher 

Creates a caching-fetcher based on the specified options. If you call this with DefaultSystemFetcher() then it will not include any loopback fetching installed in the RewriteDriver.

UrlAsyncFetcher* net_instaweb::ServerContext::DefaultSystemFetcher ( )

Note: for rewriting user content, you want to use RewriteDriver's async_fetcher() instead, as it may apply session-specific optimizations.

void net_instaweb::ServerContext::DeleteCacheOnDestruction ( CacheInterface cache)

Puts the cache on a list to be destroyed at the last phase of system shutdown.

CacheInterface* net_instaweb::ServerContext::filesystem_metadata_cache ( ) const

Cache for storing file system metadata. It must be private to a server, preferably but not necessarily shared between its processes, and is required if using load-from-file and memcached (or any cache shared between servers). This class does not take ownership.

const RewriteFilter* net_instaweb::ServerContext::FindFilterForDecoding ( const StringPiece &  id) const

Returns a filter to be used for decoding URLs & options for given filter id. This should not be used for actual fetches.

virtual GoogleString net_instaweb::ServerContext::FormatOption ( StringPiece  option_name,
StringPiece  args 

Returns the format for specifying a configuration file option. E.g. for option_name="EnableCachePurge", args="on", returns: nginx: "pagespeed EnableCachePurge on;" apache: "ModPagespeed EnableCachePurge on" The base class simply returns "EnableCachePurge on".

Reimplemented in net_instaweb::ApacheServerContext.

RewriteOptions* net_instaweb::ServerContext::GetCustomOptions ( RequestHeaders request_headers,
RewriteOptions domain_options,
RewriteOptions query_options 

Returns any custom options required for this request, incorporating any domain-specific options from the UrlNamer, options set in query-params, and options set in request headers. Takes ownership of domain_options and query_options.

bool net_instaweb::ServerContext::GetQueryOptions ( const RequestContextPtr request_context,
const RewriteOptions domain_options,
GoogleUrl request_url,
RequestHeaders request_headers,
ResponseHeaders response_headers,
RewriteQuery rewrite_query 

Runs the rewrite_query parser for any options set in query-params or in the headers. If all the pagespeed options that were parsed were valid, they are available either in rewrite_query->options() or in request_context if they are not actual options. The passed-in domain options control how options are handled, notably whether we allow related options or allow options to be specified by cookies. If you don't have domain specific options, pass NULL and global_options() will be used.

True is returned in two cases:

  • Valid PageSpeed query params or headers were parsed
  • No PageSpeed query-parameters or headers were found. False is returned if there were PageSpeed-related options but they were not valid.

It also strips off the PageSpeed query parameters and headers from the request_url, request headers, and response headers respectively. Stripped query params are copied into rewrite_query->pagespeed_query_params() and any PageSpeed option cookies are copied into rewrite_query->pagespeed_option_cookies().

void net_instaweb::ServerContext::GetRemoteOptions ( RewriteOptions remote_options,
bool  on_startup 

Fetches the remote configuration from the url specified in the remote_configuration option, and applies the received options if cached. If not cached, the options will be cached, and applied on the next request. Query options should be applied after remote options, to be able to override any option set in the remote configuration for debugging purposes. This method calls a blocking fetch of the remote configuration file. Methods remote_configuration_url() and remote_configuration_timeout_ms() are called from *remote_options. If on_startup is true, the fetch is backgrounded and the result is ignored. Startup fetches are only used for populating the cache.

GoogleString net_instaweb::ServerContext::GetRewriteOptionsSignatureHash ( const RewriteOptions options)

Returns the RewriteOptions signature hash. Returns empty string if RewriteOptions is NULL.

RewriteOptions* net_instaweb::ServerContext::global_options ( )

Returns a pointer to the master global_options. These are not used directly in RewriteDrivers, but are Cloned into the drivers as they are created. We generally do not expect global_options() to change once the system is processing requests, except in Apache when someone does a cache-flush by touching a file "cache.flush" in the file-cache directory.

const RewriteOptions* net_instaweb::ServerContext::global_options ( ) const

Returns a pointer to the master global_options without modifying the ServerContext.

bool net_instaweb::ServerContext::HandleBeacon ( StringPiece  body,
StringPiece  user_agent,
const RequestContextPtr request_context 

Handles an incoming beacon request by incrementing the appropriate variables. Returns true if the url was parsed and handled correctly; in this case a 204 No Content response should be sent. Returns false if the url could not be parsed; in this case the request should be declined. body should be either the query params or the POST body, depending on how the beacon was sent, from the beacon request.

Hasher* net_instaweb::ServerContext::hasher ( ) const
TODO(jmarantz): check thread safety in Apache.
void net_instaweb::ServerContext::InitWorkers ( )

A ServerContext may be created in one phase, and later populated with all its dependencies. This populates the worker threads.

static bool net_instaweb::ServerContext::IsExcludedAttribute ( const char *  attribute)

Returns whether or not this attribute can be merged into headers without additional considerations.

void net_instaweb::ServerContext::LockForCreation ( NamedLock creation_lock,
Sequence worker,
Function callback 

Attempt to obtain a named lock. When the lock has been obtained, queue the callback on the given worker Sequence. If the lock times out, cancel the callback, running the cancel on the worker.

QueuedWorkerPool* net_instaweb::ServerContext::low_priority_rewrite_workers ( )

Pool of worker-threads that can be used to handle low-priority/high CPU portions of resource rewriting.

NamedLock* net_instaweb::ServerContext::MakeCreationLock ( const GoogleString name)

Allocate an NamedLock to guard the creation of the given resource. If the object is expensive to create, this lock should be held during its creation to avoid multiple rewrites happening at once. The lock will be unlocked when creation_lock is reset or destructed.

void net_instaweb::ServerContext::ManageRewriteDriverPool ( RewriteDriverPool pool)

Takes ownership of the given pool, making sure to clean it up at the appropriate spot during shutdown.

void net_instaweb::ServerContext::MergeNonCachingResponseHeaders ( const ResourcePtr input,
const OutputResourcePtr output 

Take any headers that are not caching-related, and not otherwise filled in by SetDefaultLongCacheHeaders or SetContentType, but were set on the input resource, and copy them to the output resource. This allows user headers to be preserved. This must be called as needed by individual filters, prior to Write().

Note that this API is only usable for single-input rewriters. Combiners will need to execute some kind of merge, union, or intersection policy, if we wish to preserve origin response headers.

Note: this does not call ComputeCaching() on the output headers, so that method must be called prior to invoking any caching predicates on the output's ResponseHeader. In theory we shouldn't mark the caching bits dirty because we are only adding headers that will not affect caching, but at the moment the dirty-bit is set independent of that.

TODO(jmarantz): avoid setting caching_dirty bit in ResponseHeaders when the header is not caching-related.
CacheInterface* net_instaweb::ServerContext::metadata_cache ( ) const

Cache for small non-HTTP objects. This class does not take ownership.

Note that this might share namespace with the HTTP cache, so make sure your key names do not start with http://.

RewriteDriver* net_instaweb::ServerContext::NewCustomRewriteDriver ( RewriteOptions custom_options,
const RequestContextPtr request_ctx 

Like NewUnmanagedRewriteDriver, but uses standard semi-automatic memory management for RewriteDrivers.

NOTE: This does not merge custom_options with global_options(), the caller must do that if they want them merged.

Filters allocated using this mechanism have their filter-chain already frozen (see AddFilters()).

Takes ownership of 'custom_options'.

RewriteDriver* net_instaweb::ServerContext::NewRewriteDriver ( const RequestContextPtr request_ctx)

Generates a new managed RewriteDriver using the RewriteOptions managed by this class. Each RewriteDriver is not thread-safe, but you can generate a RewriteDriver* for each thread. The returned drivers manage themselves: when the HTML parsing and rewriting is done they will be returned to the pool.

Filters allocated using this mechanism have their filter-chain already frozen (see AddFilters()).

RewriteDriver* net_instaweb::ServerContext::NewRewriteDriverFromPool ( RewriteDriverPool pool,
const RequestContextPtr request_ctx 

As above, but uses a specific RewriteDriverPool to determine the options and manage the lifetime of the result. 'pool' must not be NULL.

RewriteDriver* net_instaweb::ServerContext::NewUnmanagedRewriteDriver ( RewriteDriverPool pool,
RewriteOptions options,
const RequestContextPtr request_ctx 

Generates a new unmanaged RewriteDriver with given RewriteOptions, which are assumed to correspond to drivers managed by 'pool' (which may be NULL if the options are custom). Each RewriteDriver is not thread-safe, but you can generate a RewriteDriver* for each thread. The returned drivers must be explicitly deleted by the caller.

RewriteDrivers allocated using this mechanism have not yet frozen their filters, and so callers may explicitly enable individual filters on the driver – beyond those indicated in the options. After all extra filters are added, AddFilters must be called to freeze them and instantiate the filter-chain.

Takes ownership of 'options'.

size_t net_instaweb::ServerContext::num_active_rewrite_drivers ( )

Returns the number of rewrite drivers that we were aware of at the time of the call. This includes those created via NewCustomRewriteDriver and NewRewriteDriver, but not via NewUnmanagedRewriteDriver.

const CacheInterface* net_instaweb::ServerContext::pcache_cache_backend ( )

Returns the cache backend associated with CachePropertyStore. Returns NULL if non-CachePropertyStore is used.

virtual bool net_instaweb::ServerContext::ProxiesHtml ( ) const
pure virtual

Determines whether in this server, it makes sense to proxy HTML from external sources. If we're acting as a reverse proxy that talks to the backend over HTTP, it makes sense to set this to 'true'. The JavaScript loaded from the HTML on the origin domain will be given full access to cookies on the proxied domain.

For resource-proxying (e.g. ModPagespeedMapProxyDomain) this should be set to 'false' as that command is intended only for reosurces, not for HTML.

Implemented in net_instaweb::ApacheServerContext.

void net_instaweb::ServerContext::ReleaseRewriteDriver ( RewriteDriver rewrite_driver)

Puts a RewriteDriver back on the free pool. This is intended to be called by a RewriteDriver on itself, once all pending activites on it have completed, including HTML Parsing (FinishParse) and all pending Rewrites.

TODO(jmarantz): this cannot recycle RewriteDrivers with custom rewrite options, which is a potential performance issue for Apache installations that set custom options in .htaccess files, where essentially every RewriteDriver will be a custom driver. To resolve this we need to make a comparator for RewriteOptions so that we can determine option-equivalence and, potentially, keep free-lists for each unique option-set.
void net_instaweb::ServerContext::reset_global_options ( RewriteOptions options)

Note that you have to ensure the argument has the right type in case a subclass of RewriteOptions is in use. This should also not be called once request processing has commenced.

bool net_instaweb::ServerContext::response_headers_finalized ( ) const

Determines whether we can assume that the response headers we see in rewrite_drivers when filters are applied reflect the final form from the origin. In proxy applications, this is generally true. But in Apache, it depends when the output_filter is applied relative to mod_headers and mod_expires.

The default-value is 'true'.

void net_instaweb::ServerContext::set_decoding_driver ( RewriteDriver rd)

Set the RewriteDriver that will be used to decode .pagespeed. URLs. Does not take ownership.

void net_instaweb::ServerContext::set_shutting_down ( )

Calling this method will stop results of rewrites being cached in the metadata cache. This is meant for the shutdown sequence.

void net_instaweb::ServerContext::SetDefaultLongCacheHeaders ( const ContentType content_type,
StringPiece  charset,
StringPiece  cache_control_suffix,
ResponseHeaders header 
) const

Set time and cache headers with long TTL (including Date, Last-Modified, Cache-Control, Etags, Expires).

Also sets Content-Type headers if content_type is provided. If content_type is null, the Content-Type header is omitted.

Sets charset if it's non-empty and content_type is non-NULL.

If cache_control suffix is non-empty, adds that to the Cache-Control

static GoogleString net_instaweb::ServerContext::ShowCacheForm ( StringPiece  user_agent)

Returns an HTML form for entering a URL for ShowCacheHandler. If the user_agent is non-null, then it's used to prepopulate the "User Agent" field in the form.

void net_instaweb::ServerContext::ShowCacheHandler ( Format  format,
StringPiece  url,
StringPiece  ua,
bool  should_delete,
AsyncFetch fetch,
RewriteOptions options 

Shows cached data related to a URL. Ownership of options is transferred to this function. If should_delete is true, deletes the entry as well.

void net_instaweb::ServerContext::ShutDownDrivers ( int64  cutoff_time_ms)

Waits a bounded amount of time for all currently running jobs to complete. This is meant for use when shutting down processing, so that jobs running in background do not access objects that are about to be deleted. If there are long-running outstanding tasks, the drivers may stay running past this call.

TODO(jmarantz): Change New*RewriteDriver() calls to return NULL when run after shutdown. This requires changing call-sites to null-check their drivers and gracefully fail.
RewriteDriverPool* net_instaweb::ServerContext::standard_rewrite_driver_pool ( )

Returns the RewriteDriverPool that's used by NewRewriteDriver (so calling NewRewriteDriverFromPool(standard_rewrite_driver_pool()) is equivalent to calling NewRewriteDriver.

bool net_instaweb::ServerContext::store_outputs_in_file_system ( )

Whether or not dumps of rewritten resources should be stored to the filesystem. This is meant for testing purposes only.

Member Data Documentation

const char net_instaweb::ServerContext::kResourceEtagValue[]

This value is a shared constant so that it can also be used in the Apache-specific code that repairs our caching headers downstream of mod_headers.

The documentation for this class was generated from the following file: