Conversion from unix bash script to windows batch script

Question

I'm trying to convert a Linux bash script to its windows counter part.
This is the actual bash script:

#!/bin/bash
# Usage parse_log.sh caffe.log
# It creates the following two text files, each containing a table:
#     caffe.log.test (columns: '#Iters Seconds TestAccuracy TestLoss')
#     caffe.log.train (columns: '#Iters Seconds TrainingLoss LearningRate')


# get the dirname of the script
DIR="$( cd "$(dirname "$0")" ; pwd -P )"

if [ "$#" -lt 1 ]
then
echo "Usage parse_log.sh /path/to/your.log"
exit
fi
LOG=`basename $1`
sed -n '/Iteration .* Testing net/,/Iteration *. loss/p' $1 > aux.txt
sed -i '/Waiting for data/d' aux.txt
sed -i '/prefetch queue empty/d' aux.txt
sed -i '/Iteration .* loss/d' aux.txt
sed -i '/Iteration .* lr/d' aux.txt
sed -i '/Train net/d' aux.txt
grep 'Iteration ' aux.txt | sed  's/.*Iteration \([[:digit:]]*\).*/\1/g' > aux0.txt
grep 'Test net output #0' aux.txt | awk '{print $11}' > aux1.txt
grep 'Test net output #1' aux.txt | awk '{print $11}' > aux2.txt

# Extracting elapsed seconds
# For extraction of time since this line contains the start time
grep '] Solving ' $1 > aux3.txt
grep 'Testing net' $1 >> aux3.txt
$DIR/extract_seconds.py aux3.txt aux4.txt

# Generating
echo '#Iters Seconds TestAccuracy TestLoss'> $LOG.test
paste aux0.txt aux4.txt aux1.txt aux2.txt | column -t >> $LOG.test
rm aux.txt aux0.txt aux1.txt aux2.txt aux3.txt aux4.txt

# For extraction of time since this line contains the start time
grep '] Solving ' $1 > aux.txt
grep ', loss = ' $1 >> aux.txt
grep 'Iteration ' aux.txt | sed  's/.*Iteration \([[:digit:]]*\).*/\1/g' > aux0.txt
grep ', loss = ' $1 | awk '{print $9}' > aux1.txt
grep ', lr = ' $1 | awk '{print $9}' > aux2.txt

# Extracting elapsed seconds
$DIR/extract_seconds.py aux.txt aux3.txt

# Generating
echo '#Iters Seconds TrainingLoss LearningRate'> $LOG.train
paste aux0.txt aux3.txt aux1.txt aux2.txt | column -t >> $LOG.train
rm aux.txt aux0.txt aux1.txt aux2.txt  aux3.txt

And this is my converted version:

ECHO OFF
REM Usage parse_log.sh caffe.log
REM It creates the following two text files, each containing a table:
REM     caffe.log.test (columns: '#Iters Seconds TestAccuracy TestLoss')
REM     caffe.log.train (columns: '#Iters Seconds TrainingLoss LearningRate')

SET sed="tools/3rdparty/bin/sed.exe"
SET awk="tools/3rdparty/bin/awk.exe"
SET grep="tools/3rdparty/bin/grep.exe"
SET rm="tools/3rdparty/bin/rm.exe"
SET paste="tools/3rdparty/bin/paste.exe"

REM get the dirname of the script
SET DIR=%~dp0

IF [%1]==[] (
ECHO "Usage parse_log.bat path\to\your.log"
EXIT
)

SET LOG="basename %~n1"
ECHO "Dir = %DIR%"
ECHO "LOG = %LOG%"

%sed% -n "/Iteration .* Testing net/,/Iteration *. loss/p" %1 > aux.txt
%sed% -i "/Waiting for data/d" aux.txt
%sed% -i "/prefetch queue empty/d" aux.txt
%sed% -i "/Iteration .* loss/d" aux.txt
%sed% -i "/Iteration .* lr/d" aux.txt
%sed% -i "/Train net/d" aux.txt
%grep% "Iteration " aux.txt | %sed%  "s/.*Iteration \([[:digit:]]*\).*/\1/g" > aux0.txt
%grep% "Test net output #0" aux.txt | %awk% "{print %11}" > aux1.txt
%grep% "Test net output #1" aux.txt | %awk% "{print %11}" > aux2.txt

REM Extracting elapsed seconds
REM For extraction of time since this line contains the start time
%grep% "] Solving " %1 > aux3.txt
%grep% "Testing net" %1 >> aux3.txt
%DIR%/extract_seconds.py aux3.txt aux4.txt

REM Generating
ECHO "#Iters Seconds TestAccuracy TestLoss"> %LOG%.test
%paste% aux0.txt aux4.txt aux1.txt aux2.txt | column -t >> %LOG%.test
%rm% aux.txt aux0.txt aux1.txt aux2.txt aux3.txt aux4.txt

REM For extraction of time since this line contains the start time
%grep% "] Solving " %1 > aux.txt
%grep% ", loss = " %1 >> aux.txt
%grep% "Iteration " aux.txt | %sed%  "s/.*Iteration \([[:digit:]]*\).*/\1/g" > aux0.txt
%grep% ", loss = " %1 | %awk% "{print %9}" > aux1.txt
%grep% ", lr = " %1 | %awk% "{print %9}" > aux2.txt

REM Extracting elapsed seconds
%DIR%/extract_seconds.py aux.txt aux3.txt

REM Generating
ECHO "#Iters Seconds TrainingLoss LearningRate"> %LOG%.train
%paste% aux0.txt aux3.txt aux1.txt aux2.txt | column -t >> %LOG%.train
%rm% aux.txt aux0.txt aux1.txt aux2.txt  aux3.txt

The problem is first of all I don't know if this is the correct conversion and does exactly what the bash script does. And second of all when I run the script, the cursor starts blinking (waiting) when it reaches the ECHO "LOG = %LOG%" section and after a long time, it prints the following:

tools/3rdparty/bin/sed.exe: couldn't edit aux.txt: is a terminal
tools/3rdparty/bin/sed.exe: couldn't edit aux.txt: is a terminal
tools/3rdparty/bin/sed.exe: couldn't edit aux.txt: is a terminal
tools/3rdparty/bin/sed.exe: couldn't edit aux.txt: is a terminal
tools/3rdparty/bin/sed.exe: couldn't edit aux.txt: is a terminal

And it waits again!

What is wrong here?

UPDATE

After changing aux.txt to auxx.txt the first error is resolved. But now I get these errors:

awk: {print examples\cifar10\caffe.log1}
awk:                ^ backslash not last character on line
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
awk: {print examples\cifar10\caffe.log1}
awk:                ^ backslash not last character on line
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
tools/3rdparty/bin/grep.exe: writing output: Invalid argument
Usage: ./extract_seconds input_file output_file
'column' is not recognized as an internal or external command,
operable program or batch file.

score 2 · Answer 1 · answered Apr 03 '16 at 07:31

AUX is reserved word, see Naming Files, Paths, and Namespaces:

Do not use the following reserved names for the name of a file:

CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9. Also avoid these names followed immediately by an extension; for example, NUL.txt is not recommended. For more information, see Namespaces.

Use another different name instead, e.g auxx.txt.

Thanks, that solved the first problem, now I get some other errors, I updated the question. — Rika, Apr 03 '16 at 07:48

score 1 · Answer 2 · edited May 23 '17 at 11:48

Please read answers on How to set environment variables with spaces? and Why is no string output with 'echo %var%' after using 'set var = text' on command line? as you most likely don't know how to define variable values with spaces correct.

Don't use IF [%1]==[] as square brackets have no special meaning in Windows command scripts on IF and therefore the batch processing could exit because of a syntax error with left value containing 1 or more spaces. Use instead IF "%~1"=="" and execute in a command prompt window call /? for an explanation of %~1 (first argument with surrounding double quotes removed).

It is advisable to use always %~1, %~2, ... with explicitly embedding those argument references in double quotes.

On Windows the directory separator is the backslash \ character and not the forward slash / character. Windows command processor corrects this often made mistake automatically where it detects it, but it can't detect and correct / in all cases automatically.

awk is from *nix and gives a backslash a completely different meaning as Windows command processor. I have no experience with awk, but I think it is better to specify file names with paths passed as arguments to awk also on Windows with a forward slash as directory separator instead of a backslash. Therefore I suggest to replace the lines:

SET awk="tools/3rdparty/bin/awk.exe"
SET grep="tools/3rdparty/bin/grep.exe"

rem Other lines of your batch file.

%grep% "Test net output #0" auxx.txt | %awk% "{print %11}" > aux1.txt
%grep% "Test net output #1" auxx.txt | %awk% "{print %11}" > aux2.txt

by

set "awk=tools\3rdparty\bin\awk.exe"
set "grep=tools\3rdparty\bin\grep.exe"

rem Other lines of your batch file.

set "LogFileName=%~11"
set "LogFileName=%LogFileName:\=/%"
"%grep%" "Test net output #0" auxx.txt | "%awk%" "{print %LogFileName%}" > aux1.txt
"%grep%" "Test net output #1" auxx.txt | "%awk%" "{print %LogFileName%}" > aux2.txt

Using *nix tools on Windows requires thinking on what is interpreted by Windows native command line tools like cmd.exe and what is interpreted by the *nix tools like awk.exe and grep.exe to use correct syntax on arguments.

set "LogFileName=%LogFileName:\=/%" replaces all backslashes by forward slashes in log file name with relative path. Run in a command prompt window set /? and read all output help pages for details about string replacements / substitutions in values of environment variables.

And on line

%paste% aux0.txt aux3.txt aux1.txt aux2.txt | column -t >> %LOG%.train

the Windows command processor (cmd.exe) can't find an application column with a file extension defined in environment variable PATHEXT in current directory or any directory defined in environment variable PATH. This application must be also specified with file extension and (relative) path.

Conversion from unix bash script to windows batch script

UPDATE

2 Answers2