Page Speed Optimization Libraries
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
Classes | Namespaces
url_to_filename_encoder.h File Reference
#include <cstddef>
#include "pagespeed/kernel/base/string.h"
#include "pagespeed/kernel/base/string_util.h"

Go to the source code of this file.


class  net_instaweb::UrlToFilenameEncoder
 Helper class for converting a URL into a filename. More...


 Unit-test framework for wget fetcher.

Detailed Description (Joshua Marantz)

URL filename encoder goals:

  1. Allow URLs with arbitrary path-segment length, generating filenames with a maximum of 128 characters.
  2. Provide a somewhat human readable filenames, for easy debugging flow.
  3. Provide reverse-mapping from filenames back to URLs.
  4. Be able to distinguish http://x from http://x/ from http://x/index.html. Those can all be different URLs.
  5. Be able to represent http://a/b/c and http://a/b/c/d, a pattern seen with Facebook Connect.

We need an escape-character for representing characters that are legal in URL paths, but not in filenames, such as '?'.

We can pick any legal character as an escape, as long as we escape it too. But as we have a goal of having filenames that humans can correlate with URLs, we should pick one that doesn't show up frequently in URLs. Candidates are ~`!#$%^&()-=_+{}[],. but we would prefer to avoid characters that are treated specially by tools like shells or build tools. It turns out that , is neither frequent in URLs nor special anywhere else, so we use that.

The escaping algorithm is: 1) Escape all unfriendly symbols as ,XX where XX is the hex code. 2) Add a ',' at the end (We do not allow ',' at end of any directory name, so this assures that e.g. /a and /a/b can coexist in the filesystem). 3) Go through the path segment by segment (where a segment is one directory or leaf in the path) and 3a) If the segment is empty, escape the second slash. i.e. if it was<a then we escape the second / like,2Fa, 3a) If it is "." or ".." prepend with ',' (so that we have a non- empty and non-reserved filename). 3b) If it is over 128 characters, break it up into smaller segments by inserting ,-/ (Windows limits paths to 128 chars, other OSes also have limits that would restrict us)

For example: URL File / /, /index.html /index.html, /. /., /a/b /a/b, /a/b/ /a/b/, /a/b/c /a/b/c, Note: no prefix problem /u?foo=bar /u,3Ffoo=bar, // /,2F, /./ /,./, /../ /,../, /, /,2C, /,./ /,2C./, /very...longname/ /very...long,-/name If very...long is about 126 long.