Cloud logs observability improvements

[dbalabka]

dask-cloudprovider integration might be tough unless the cloud is configured correctly. Often, the developer might see that the initial start of the cluster simply hung, and Dask waits infinitely for the scheduler to be started:

Creating scheduler instance
dask-5424b11a-scheduler
        Internal IP: 10.128.0.220
        External IP: 35.184.140.159
Waiting for scheduler to run at 35.184.140.159:8786

Usually, the problem is observed in cloud-init logs or docker container std output that can fetched if you connect to the instance directly via SSH and cat cloud-init logs:

cat /var/log/cloud-init-output.log

Or fetches the Docker container stdout:

docker logs <container ID>

I propose to output these logs into a distributed logger.

[RaphaelRobidas]

I am having this exact issue, I cannot get the scheduler to start when creating a GCPCluster, although everything works if I spawn the scheduler and workers manually.

In my case, I do not even have a /var/log/cloud-init-output.log file nor cloud logs. Some extra logging would be fantastic!

[jacobtomlinson]

Doesn't GCP automatically collect logs into stackdriver?

[RaphaelRobidas]

@jacobtomlinson It does, but the logs are related to the VM starting up. The first serial port (console) similarly has only logging of the system. I haven't figured out how to get any log from the scheduler.

[dbalabka]

@jacobtomlinson , you are right the logging is enabled already:

dask-cloudprovider/dask_cloudprovider/gcp/instances.py

Lines 161 to 162 in 8b5ab97

	"key": "google-logging-enabled",
	"value": "true",

The cloud-init logs and docker container output can be found in serial port output logs under each instance details tab:

However, ideally, I would like to have an error output in distributed logging to see the error message right away in my Jupyter enviroment. It will require to parse logs which is not trivial. Currently, DX isn't perfect we observer simply hung cluster startup without any message about the error.

[jacobtomlinson]

I think the challenge is that we cant get distributed logging until the client connects to the scheduler. If there is an error in scheduler bringup (a common thing folks need these logs for) there's no way to retrieve it.

[dbalabka]

@jacobtomlinson , I found that Serial Port logs are not available, if the instances is stopped. I, usually, start the instance to see the logs:

I'm looking into possibility to install GCP Ops Agent on instance creation to collect logs into GCP Logging system. In such case, logs can be retrievied in any time even if the scheduler/worker is stopped:
https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent

The similar idea can be applied to any cloud.