Hadoop Streaming: Where Are Application Logs?

My question is similar to : hadoop streaming: how to see application logs? (The link in the answer is not currently working. So I have to post it again with an additional question)

Solution 1:

Try this HDFS path: /yarn/apps/&{user_name}/logs/application_${appid}/

in general:

Where to store container logs. An application's localized log directory will be found in ${yarn.nodemanager.log-dirs}/application_${appid}. Individual containers' log directories will be below this, in directories named container_{$contid}. Each container directory will contain the files stderr, stdin, and syslog generated by that container.

If you print to stderr you'll find it in files under this dir I mentioned above. There should be one file per one node.

Solution 2:

You must be aware that Hadoop-streaming uses stdout to pipe data from mappers to reducers. So if your logging system writes in stdout, you will be in trouble, since it will very likely break your logic and your job. One way to log is to write in stderr, thus you will see your logs in errors logs.