# How gargle gets tokens

This vignette explains the purpose and usage of `token_fetch()` and the functions it subsequently calls. The goal of `token_fetch()` is to secure a token for use in downstream requests.

The target audience is someone who works directly with a Google API. These people roughly fall into two camps:

- The author of an R package that wraps a Google API.
- The useR who is writing a script or app, without using such a wrapper, either because the wrapper does not exist or there’s a reason to avoid the dependency.

This vignette might also be useful to the user of a wrapper package who needs to influence the operations of `token_fetch()`, e.g. by telling it to try auth methods in a non-default order or to not try certain methods at all.

`token_fetch()` is aimed at whoever is going to manage the returned token, e.g., incorporate it into downstream requests. It can be very nice for users if wrapper packages assume this responsibility, as opposed to requiring users to explicitly acquire and manage their tokens. We give a few design suggestions here and cover this in more depth in `vignette("gargle-auth-in-client-package")`.

    library(gargle)

## `token_fetch()`

`token_fetch()` is a rather magical function for getting a token. The goal is to make auth relatively painless for users, while allowing developers and power users to take control when and if they need to. Most users will presumably interact with `token_fetch()` only in an indirect way, mediated through an API wrapper package. That is not because the interface of `token_fetch()` is unfriendly – it’s very flexible! The objective of `token_fetch()` is to allow package developers to take responsibility for managing the user’s token, without having to implement all the different ways of obtaining that token in the first place.

The signature of `token_fetch()` is very simple and, therefore, not very informative:

    token_fetch(scopes, ...)

Under the hood, `token_fetch()` calls a sequence of much more specific credential functions, each wrapped in a `tryCatch()` and returning `NULL` if unsuccessful. The only formal argument these functions have in common is `scopes`, with the rest being passed via `...`.

This gives a sense of the credential functions and reflects the order in which they are called:

    writeLines(names(cred_funs_list()))
    #> credentials_byo_oauth2
    #> credentials_service_account
    #> credentials_external_account
    #> credentials_app_default
    #> credentials_gce
    #> credentials_user_oauth2

It is possible to manipulate this registry of functions. The help for `cred_funs_list()` is a good place to learn more and we present a concrete example in the last section of this vignette.

For now, however, we assume you’re working with the default registry that ships with gargle.

Note also that these credential functions are exported and can be called directly.

## Get verbose output

To see more information about what gargle is up to, set the `"gargle_verbosity"` option to “debug”. Read more in the docs for `gargle_verbosity()`.

## `credentials_byo_oauth2()`

The first function tried is `credentials_byo_oauth2()`. Here’s how a call to `token_fetch()` might work:

    token_fetch(token = <TOKEN2.0>)

    credentials_byo_oauth2(
      token = <TOKEN2.0>
    )

`credentials_byo_oauth2()` provides a back door for a “bring your own token” workflow. This function accounts for the scenario where an OAuth token has been obtained through external means and it’s convenient to be able to put it into force.

`credentials_byo_oauth2()` checks that `token` is of class `httr::Token2.0` and that it appears to be associated with Google. A `token` of class `request` is also acceptable, in which case the `auth_token` component is extracted and treated as the input. This is how a `Token2.0` object would present, if processed with `httr::config()`, as functions like `googledrive::drive_token()` and `bigrquery::bq_token()` do.

If `token` is not provided or if it doesn’t satisfy these requirements, we fail and `token_fetch()`’s execution moves on to the next function in the registry.

## `credentials_service_account()`

The next function tried is `credentials_service_account()`. Here’s how a call to `token_fetch()` with service account inputs plays out:

    token_fetch(scopes = <SCOPES>, path = "/path/to/your/service-account.json")

    # credentials_byo_oauth2() fails because no `token`,
    # which leads to this call:
    credentials_service_account(
      scopes = <SCOPES>,
      path = "/path/to/your/service-account.json"
    )

The `scopes` are often provided by the API wrapper function that is mediating the calls to `token_fetch()` and `credential_service_account()`. The `path` argument is presumably coming from the user. It is treated as a JSON representation of service account credentials, in any form that is acceptable to `jsonlite::fromJSON()`. In the above example, that is a file path, but it could also be a JSON string. If there is no named `path` argument or if it can’t be parsed as a service account credential, we fail and `token_fetch()`’s execution moves on to the next function in the registry.

Here is some Google documentation about service accounts:

- Cloud Identity and Access Management > Understanding service accounts

For R users, a service account is a great option for credentials that will be used in a script or application running remotely or in an unattended fashion. In particular, this is a better approach than trying to move OAuth2 credentials from one machine to another. For example, a service account is the preferred method of auth when testing and documenting a package on a continuous integration service.

The JSON key file must be managed securely. In particular, it should not be kept in, e.g., a GitHub repository (unless it is encrypted). The encryption strategy used by gargle and other packages is described in the article Managing tokens securely.

Note that fetching a token for a service account requires a reasonably accurate system clock. This is of particular importance for users running gargle inside a Docker container, as Docker for Windows has intermittently seen problems with clock drift. If your service account token requests fail with “Bad Request” inside a container, but succeed locally, check that the container’s system clock is accurate.

## `credentials_external_account()`

The next function tried is `credentials_external_account()`. Here’s how a call to `token_fetch()` with an external account inputs plays out:

    token_fetch(scopes = <SCOPES>, path = "/path/to/your/external-account.json")

    # credentials_byo_oauth2() fails because no `token`,
    # credentials_service_account() fails because the JSON provided via
    #   `path` is not of type "service_account",
    # which leads to this call:
    credentials_external_account(
      scopes = <SCOPES>,
      path = "/path/to/your/external-account.json"
    )

`credentials_external_account()` implements something called workload identity federation and is available to applications running on specific non-Google Cloud platforms. At the time of writing, gargle only supports AWS, but this could be expanded to other providers, such as Azure, if there is a documented need.

Similar to `credentials_service_account()`, the `path` is treated as a JSON representation of the account’s configuration and it’s probably a file path. However, in contrast to `credentials_service_account()`, this JSON only contains non-sensitive metadata, which is, indeed, the main point of this flow. The secrets needed to complete auth are obtained “on-the-fly” from, e.g., the running EC2 instance.

`credentials_external_account()` will fail for many reasons: there is no named `path` argument, the JSON at `path` can’t be parsed as configuration for an external AWS account, we don’t appear to running on AWS, suggested packages for AWS functionality are not installed, or the workload identity pool is misconfigured. If any of that happens, we fail and `token_fetch()`’s execution moves on to the next function in the registry.

Here is some Google documentation about workload identity federation and the specifics for AWS:

- Blog post: Keyless API authentication — Better cloud security through workload identity federation, no service account keys necessary
- Documentation: Configuring workload identity federation

## `credentials_app_default()`

The next function tried is `credentials_app_default()`. Here’s how a call to `token_fetch()` might work:

    token_fetch(scopes = <SCOPES>)

    # credentials_byo_oauth2() fails because no `token`,
    # credentials_service_account() fails because no `path`,
    # credentials_external_account() fails because no `path`,
    # which leads to this call:
    credentials_app_default(
      scopes = <SCOPES>
    )

`credentials_app_default()` loads credentials from a file identified via a search strategy known as Application Default Credentials (ADC). The credentials themselves are conventional service account, external account, or user credentials that happen to be stored in a pre-ordained location and format.

The hope is to make auth “just work” for someone working on Google-provided infrastructure or who has used Google tooling to get started, such as the `gcloud` command line tool. A sequence of paths is consulted, which we describe here, with some abuse of notation. ALL_CAPS represents the value of an environment variable.

    ${GOOGLE_APPLICATION_CREDENTIALS}
    ${CLOUDSDK_CONFIG}/application_default_credentials.json

    # on Windows:
    %APPDATA%\gcloud\application_default_credentials.json
    %SystemDrive%\gcloud\application_default_credentials.json
    C:\gcloud\application_default_credentials.json

    # on not-Windows:
    ~/.config/gcloud/application_default_credentials.json

If the above search successfully identifies a JSON file, it is parsed and ingested either as a service account token, an external account configuration, or an OAuth2 user credential. In the case of an OAuth2 credential, the requested `scopes` must also meet certain criteria. Note that this will NOT work for OAuth2 credentials initiated by gargle, which are stored on disk in `.rds` files. The storage of OAuth2 user credentials as JSON is unique to certain Google tools – possibly just the `gcloud` CLI – and should probably be regarded as deprecated. It is recommended to use ADC with a service account or workload identity federation. If this quest is unsuccessful, we fail and `token_fetch()`’s execution moves on to the next function in the registry.

The main takeaway lesson:

- You can make auth “just work” by storing the JSON for a service account or an external account at one of the filepaths listed above. It will be automagically discovered when `token_fetch()` is called with only the `scopes` argument specified.

Again, remember that the JSON key file for a conventional service account must be managed securely and should NOT live in a directory that syncs to the cloud. The JSON configuration for an external account is not actually sensitive and this is one of the benefits of this flow, but it’s only available in a very narrow set of circumstances.

## `credentials_gce()`

The next function tried is `credentials_gce()`. Here’s how a call to `token_fetch()` might work:

    token_fetch(scopes = <SCOPES>)
    # or perhaps
    token_fetch(scopes = <SCOPES>, service_account = <SERVICE_ACCOUNT>)

    # credentials_byo_oauth2() fails because no `token`,
    # credentials_service_account() fails because no `path`,
    # credentials_external_account() fails because no `path`,
    # credentials_app_default() fails because no ADC found,
    # which leads to one of these calls:
    credentials_gce(
      scopes = <SCOPES>,
      service_account = "default"
    )
    # or
    credentials_gce(
      scopes = <SCOPES>,
      service_account = <SERVICE_ACCOUNT>
    )

`credentials_gce()` retrieves service account credentials from a metadata service that is specific to virtual machine instances running on Google Cloud Engine (GCE). Basically, if you have to ask what this is about, this is not the auth method for you. Let us move on.

If this seems to happening to you and it’s not what you want, see the last section for how to remove this auth method.

## `credentials_user_oauth2()`

The next and final function tried is `credentials_user_oauth2()`. Here’s how a call to `token_fetch()` might work:

    token_fetch(scopes = <SCOPES>)

    # credentials_byo_oauth2() fails because no `token`,
    # credentials_service_account() fails because no `path`,
    # credentials_external_account() fails because no `path`,
    # credentials_app_default() fails because no ADC found,
    # credentials_gce() fails because not on GCE,
    # which leads to this call:
    credentials_user_oauth2(
      scopes = <SCOPES>,
      app = <OAUTH_APP>,
      package = "<PACKAGE>"
    )

`credentials_user_oauth2()` is where the vast majority of users will end up. This is the function that choreographs the traditional “OAuth dance” in the browser. User credentials are cached locally, at the user level, by default. Therefore, after first use, there are scenarios in which gargle can determine unequivocally that it already has a suitable token on hand and can load (and possibly refresh) it, without additional user intervention.

The `scopes`, `app` (likely to be renamed `client` in a future version of gargle), and `package` are generally provided by the API wrapper function that is mediating the calls to `token_fetch()`. Do not “borrow” an OAuth client from gargle or any other package; always use credentials associated with your package or provided by your user. Per the Google User Data Policy https://developers.google.com/terms/api-services-user-data-policy, your application must accurately represent itself when authenticating to Google API services.

The wrapper package would presumably also declare itself as the package requesting a token (this is used in messages). So here’s how a call to `token_fetch()` and `credentials_user_oauth2()` might look when initiated from `THINGY_auth()`, a function in the fictional thingyr wrapper package:

    # user initiates auth or does something that triggers it indirectly
    THINGY_auth()

    # which then calls
    gargle::token_fetch(
      scopes  = <SCOPES_NEEDED_FOR_THE_THINGY_API>,
      app     = thingy_app(),
      package = "thingyr"
    )

    # which leads to this call:
    credentials_user_oauth2(
      scopes  = <SCOPES_NEEDED_FOR_THE_THINGY_API>,
      app     = thingy_app(),
      package = "thingyr"
    )

See `vignette("gargle-auth-in-client-package")` for design ideas for a function like `THINGY_auth()`.

What happens tomorrow or next week? Do we make this user go through the browser dance again? How do we get to that happy place where we don’t bug them constantly about auth?

First, we define “suitable”, i.e. what it means to find a matching token in the cache. `credentials_user_oauth2()` is a thin wrapper around `gargle2.0_token()` which is the constructor for the `gargle::Gargle2.0` class used to hold an OAuth2 token. And that call might look something like this (simplified for communication purposes):

    gargle2.0_token(
      email   = gargle_oauth_email(),
      app     = thingy_app(),
      package = "thingyr",
      scope   = <SCOPES_NEEDED_FOR_THE_THINGY_API>,
      cache   = gargle_oauth_cache()
    )

gargle looks in the cache specified by `gargle_oauth_cache()` for a token that has these scopes, this client, and the Google identity specified by `email`. By default `email` is `NA`, so we might find one or more tokens that have the necessary scopes and client. In that case, gargle reveals the `email` associated with the matching token(s) and asks the user for explicit instructions about how to proceed. That looks something like this:

    The thingyr package is requesting access to your Google account.
    Enter '1' to start a new auth process or select a pre-authorized account.
    1: Send me to the browser for a new auth process.
    2: janedoe_personal@gmail.com
    3: janedoe@example.com
    4: janedoe_work@gmail.com
    Selection:

If none of the tokens has the right scopes and client (or if the user declines to use a pre-existing token), we head to the browser to initiate OAuth2 flow de novo.

A user can reduce the need for interaction by passing the target `email` to `thingy_auth()`:

    thingy_auth(email = "janedoe_work@gmail.com")

or by specifying same in the `"gargle_oauth_email"` option. A value of `email = TRUE`, passed directly or via the option, is an alternative strategy: `TRUE` means that gargle is allowed to use a matching token whenever there is exactly one match.

The elevated status of `email` for `gargle::Gargle2.0` tokens is motivated by the fact that many of us have multiple Google identities and need them to be very prominent when working with Google APIs. This is one of the main motivations for `gargle::Gargle2.0`, which extends `httr::Token2.0`. The `gargle::Gargle2.0` class also defaults to a user-level token cache, as opposed to project-level. An overview of the current OAuth cache is available via `gargle_oauth_cache()` and the output looks something like this:

    gargle_oauth_sitrep()
    #> 14 tokens found in this gargle OAuth cache:
    #> ~/Library/Caches/gargle
    #'
    #' email                         app         scopes                         hash...
    #' ----------------------------- ----------- ------------------------------ ----------
    #' abcdefghijklm@gmail.com       thingy      ...bigquery, ...cloud-platform 128f9cc...
    #' buzzy@example.org             gargle-demo                                15acf95...
    #' stella@example.org            gargle-demo ...drive                       4281945...
    #' abcdefghijklm@gmail.com       gargle-demo ...drive                       48e7e76...
    #' abcdefghijklm@gmail.com       tidyverse                                  69a7353...
    #' nopqr@ABCDEFG.com             tidyverse   ...spreadsheets.readonly       86a70b9...
    #' abcdefghijklm@gmail.com       tidyverse   ...drive                       d9443db...
    #' nopqr@HIJKLMN.com             tidyverse   ...drive                       d9443db...
    #' nopqr@ABCDEFG.com             tidyverse   ...drive                       d9443db...
    #' stuvwzyzabcd@gmail.com        tidyverse   ...drive                       d9443db...
    #' efghijklmnopqrtsuvw@gmail.com tidyverse   ...drive                       d9443db...
    #' abcdefghijklm@gmail.com       tidyverse   ...drive.readonly              ecd11fa...
    #' abcdefghijklm@gmail.com       tidyverse   ...bigquery, ...cloud-platform ece63f4...
    #' nopqr@ABCDEFG.com             tidyverse   ...spreadsheets                f178dd8...

## Manipulate the credential function registry

Recall that you can get an overview of the credential functions that `token_fetch()` works through like so:

    writeLines(names(cred_funs_list()))
    #> credentials_byo_oauth2
    #> credentials_service_account
    #> credentials_external_account
    #> credentials_app_default
    #> credentials_gce
    #> credentials_user_oauth2

Sometimes more than one of these auth methods “work”, but only one of them actually “works” and, sadly, it’s not the first one. In this case, gargle successfully gets a token, but then you experience token-related failure in downstream work.

The most common example of this is someone who is working on Google Compute Engine (GCE), but they prefer to auth as a normal user, not as the default service account.

Let’s say you want to prevent `token_fetch()` from even trying one specific auth method, clearing the way for it to automagically use the method you want. You can remove a specific credential function from the registry. Here’s how to do this for the scenario described above, where you want to skip GCE-specific auth:

    gargle::cred_funs_add(credentials_gce = NULL)

Learn more in the docs for `cred_funs_list()`. You can even make narrowly scoped changes to the registry with `local_cred_funs()` and `with_cred_funs()`.