-
- Posts: 9
- Joined: Wed Nov 30, 2011 12:08 pm
Parsing log files (cygwin grep.... awk...)
Hello.... I am trying to create reports off of backup log files - using cygwin (bash-> grep, awk) on WinServer2008R2/backup4all-Pro-4.6 However the log files appear to have zero fills out to 16bits so grep awk does not work. Things like bash-> cat, tail, work. All windows stuff like wordpad, textpad even notepad work. I can even copy paste from the cygwin rxvt window and vi paste and the zeros get stripped. But I am trying to script. Here is a cygwin tail list of the file followed by a octel dump of characters (od -c): (randy.log is a copy)
[email protected] /cygdrive/c/users/administrator/appdata/Roaming/Softland/Backup4all Professional 4/Logs
$ tail -2 randy.log
Backup completed at 1/3/2012 9:12:17 PM with 0 error(s), 1 warning(s)
[email protected] /cygdrive/c/users/administrator/appdata/Roaming/Softland/Backup4all Professional 4/Logs
$ od -c randy.log
0024260 \0 B \0 a \0 c \0 k \0 u \0 p \0 \0 <- (\0 means all zero bits is those 8 bits)
0024300 c \0 o \0 m \0 p \0 l \0 e \0 t \0 e \0
0024320 d \0 \0 a \0 t \0 \0 1 \0 / \0 3 \0
0024340 / \0 2 \0 0 \0 1 \0 2 \0 \0 9 \0 : \0
0024360 1 \0 2 \0 : \0 1 \0 7 \0 \0 P \0 M \0
0024400 \0 w \0 i \0 t \0 h \0 \0 0 \0 \0
0024420 e \0 r \0 r \0 o \0 r \0 ( \0 s \0 ) \0
0024440 , \0 \0 1 \0 \0 w \0 a \0 r \0 n \0
0024460 i \0 n \0 g \0 ( \0 s \0 ) \0 \r \0 \n \0
0024500
I thought maybe it is 16bit unicode so I tried piconv and even copied to real linux and tried iconv
But as you can see the results are not as I was hoping for.
[email protected] ~]# iconv -f ISO-8859-1 -t UTF-8 randy.log -o randyconv.log
[[email protected] ~]# ls -ld *ran*
-rw-r--r-- 1 root root 7494 Jan 3 17:26 randyconv.log
-rwx------ 1 root root 7490 Jan 3 17:23 randy.log
[[email protected] ~]#
I will keep studying but maybe someone can give some pointers to help out on this?
Thank you..very much..
[email protected] /cygdrive/c/users/administrator/appdata/Roaming/Softland/Backup4all Professional 4/Logs
$ tail -2 randy.log
Backup completed at 1/3/2012 9:12:17 PM with 0 error(s), 1 warning(s)
[email protected] /cygdrive/c/users/administrator/appdata/Roaming/Softland/Backup4all Professional 4/Logs
$ od -c randy.log
0024260 \0 B \0 a \0 c \0 k \0 u \0 p \0 \0 <- (\0 means all zero bits is those 8 bits)
0024300 c \0 o \0 m \0 p \0 l \0 e \0 t \0 e \0
0024320 d \0 \0 a \0 t \0 \0 1 \0 / \0 3 \0
0024340 / \0 2 \0 0 \0 1 \0 2 \0 \0 9 \0 : \0
0024360 1 \0 2 \0 : \0 1 \0 7 \0 \0 P \0 M \0
0024400 \0 w \0 i \0 t \0 h \0 \0 0 \0 \0
0024420 e \0 r \0 r \0 o \0 r \0 ( \0 s \0 ) \0
0024440 , \0 \0 1 \0 \0 w \0 a \0 r \0 n \0
0024460 i \0 n \0 g \0 ( \0 s \0 ) \0 \r \0 \n \0
0024500
I thought maybe it is 16bit unicode so I tried piconv and even copied to real linux and tried iconv
But as you can see the results are not as I was hoping for.
[email protected] ~]# iconv -f ISO-8859-1 -t UTF-8 randy.log -o randyconv.log
[[email protected] ~]# ls -ld *ran*
-rw-r--r-- 1 root root 7494 Jan 3 17:26 randyconv.log
-rwx------ 1 root root 7490 Jan 3 17:23 randy.log
[[email protected] ~]#
I will keep studying but maybe someone can give some pointers to help out on this?
Thank you..very much..
Last edited by hurtlocker on Wed Jan 04, 2012 7:50 pm, edited 1 time in total.
-
- Posts: 9
- Joined: Wed Nov 30, 2011 12:08 pm
Re: Parsing log files (cygwin grep.... awk...)
SOLUTION:
Hello.... using a different "from type" worked ***UTF16***
[[email protected] ~]# iconv -f UTF16 -t UTF8 randy.log -o randyconv.log
[[email protected] ~]# ll *ran*
-rw-r--r-- 1 root root 5281 Jan 4 08:57 randyconv.log <- half the size
-rwx------ 1 root root 10560 Jan 4 08:56 randy.log
[[email protected] ~]# od -c randyconv.log | tail -7
0012120 c k u p : \r \n B a c k u p <- no more \0 zero fills
0012140 c o m p l e t e d a t 1 /
0012160 3 / 2 0 1 2 9 : 1 2 : 1 7 P
0012200 M w i t h 0 e r r o r ( s
0012220 ) , 1 w a r n i n g ( s ) \r
0012240 \n
0012241
[[email protected] ~]#
Borrowed a real linux for iconv - Now I just need to find iconv for cygwin (not easy)
Hope this helps someone someday
Hello.... using a different "from type" worked ***UTF16***
[[email protected] ~]# iconv -f UTF16 -t UTF8 randy.log -o randyconv.log
[[email protected] ~]# ll *ran*
-rw-r--r-- 1 root root 5281 Jan 4 08:57 randyconv.log <- half the size
-rwx------ 1 root root 10560 Jan 4 08:56 randy.log
[[email protected] ~]# od -c randyconv.log | tail -7
0012120 c k u p : \r \n B a c k u p <- no more \0 zero fills
0012140 c o m p l e t e d a t 1 /
0012160 3 / 2 0 1 2 9 : 1 2 : 1 7 P
0012200 M w i t h 0 e r r o r ( s
0012220 ) , 1 w a r n i n g ( s ) \r
0012240 \n
0012241
[[email protected] ~]#
Borrowed a real linux for iconv - Now I just need to find iconv for cygwin (not easy)
Hope this helps someone someday
-
- Posts: 9
- Joined: Wed Nov 30, 2011 12:08 pm
Re: Parsing log files (cygwin grep.... awk...)
LAST POST on Subject:
If you just search iconv in the cygwin install gui, it will pull only libiconv2 as iconv() function and you'll think you have to go down the compile route (ugh)
To get iconv for cygwin you need to go to libs and select libiconv
syntax a bit different from linux
$ iconv -f UTF-16 -t UTF-8 randy.log > randy8.log
$ ll *ran*
-rwx------+ 1 Administrator Domain Users 10560 Jan 4 07:12 randy.log
-rw-r--r--+ 1 Administrator Domain Users 5278 Jan 4 10:19 randy8.log
Didn't mean to turn this in to a cygwin thread but for those who want to write summary reports for many backup4all backups using their logs and you are a UTF-8 programmer this is very powerful
THANK YOU ! - complete
BTW: useful command:
$ cygcheck -p iconv.exe
Found 4 matches for iconv.exe
libiconv/libiconv-1.13.1-1 GNU character set conversion library and utilities
libiconv/libiconv-1.13.1-2 GNU character set conversion library and utilities
libiconv/libiconv-1.14-1 GNU character set conversion library and utilities
libiconv/libiconv-1.14-2 GNU character set conversion library and utilities
If you just search iconv in the cygwin install gui, it will pull only libiconv2 as iconv() function and you'll think you have to go down the compile route (ugh)
To get iconv for cygwin you need to go to libs and select libiconv
syntax a bit different from linux
$ iconv -f UTF-16 -t UTF-8 randy.log > randy8.log
$ ll *ran*
-rwx------+ 1 Administrator Domain Users 10560 Jan 4 07:12 randy.log
-rw-r--r--+ 1 Administrator Domain Users 5278 Jan 4 10:19 randy8.log
Didn't mean to turn this in to a cygwin thread but for those who want to write summary reports for many backup4all backups using their logs and you are a UTF-8 programmer this is very powerful
THANK YOU ! - complete
BTW: useful command:
$ cygcheck -p iconv.exe
Found 4 matches for iconv.exe
libiconv/libiconv-1.13.1-1 GNU character set conversion library and utilities
libiconv/libiconv-1.13.1-2 GNU character set conversion library and utilities
libiconv/libiconv-1.14-1 GNU character set conversion library and utilities
libiconv/libiconv-1.14-2 GNU character set conversion library and utilities