1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Removing specific lines from a text file?

Discussion in 'Off Topic' started by Dean, Aug 14, 2011.

  1. Dean

    Dean Well-Known Member

    I've go a reasonably long file, and 99% of it are lines like these:
    127.0.0.1 - - [01/Jul/2011:02:04:29 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:04:30 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:05:01 -0700] "GET /whm-server-status HTTP/1.0" 200 28577
    127.0.0.1 - - [01/Jul/2011:02:05:01 -0700] "GET /whm-server-status?auto HTTP/1.1" 200 446
    127.0.0.1 - - [01/Jul/2011:02:05:01 -0700] "GET /whm-server-status?auto HTTP/1.1" 200 446
    127.0.0.1 - - [01/Jul/2011:02:05:02 -0700] "GET /whm-server-status?auto HTTP/1.1" 200 446
    127.0.0.1 - - [01/Jul/2011:02:05:03 -0700] "GET / HTTP/1.0" 200 111
    127.0.0.1 - - [01/Jul/2011:02:05:53 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:05:54 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:05:59 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:06:56 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:07:13 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:07:51 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:07:51 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:07:51 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:07:51 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:07:51 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:07:51 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:07:51 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:07:51 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:07:51 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:07:51 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:07:51 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:07:51 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:07:51 -0700] "OPTIONS * HTTP/1.0" 200 -
    127.0.0.1 - - [01/Jul/2011:02:07:51 -0700] "OPTIONS * HTTP/1.0" 200 -
    Any ideas how I could get rid of all lines that start with 127.0.0.1 ?
    I'm sure there is some type of grep command, but I've no idea how to go about that. I cannot seem to do that with notepad++ on my pc locally either. All ideas welcome :)
    (fingers tired)
     
  2. ibnesayeed

    ibnesayeed Well-Known Member

    "-v, --invert-match" is the switch for invert match in grep. That means,

    will remove all the lines from "old-file" containing the term "useless" and save the output in "new-file". You can also use a regular expression in place of "useless". From your command line issue "man grep" for more details. :)
     
    Dean likes this.
  3. Digital Doctor

    Digital Doctor Well-Known Member

    Dean likes this.
  4. Dean

    Dean Well-Known Member

    You have made my fingers extremely happy :)

    Thank You!!!

    I may upgrade to that at some point. Thanks!
     
    ibnesayeed likes this.
  5. Dean

    Dean Well-Known Member

    Can I do the opposite?

    Take the results of that filtered file and have just the lines containing 2 different IPs of interest piped into another file?

    grep "IP1" "IP2" input_file >output_file

    ?
     
  6. ibnesayeed

    ibnesayeed Well-Known Member

    If you want to have an OR match then you can use something like this:

    this should grab all the lines containing "foo" OR "bar" (OR both in any order) from old-file and save it in new-file. Mind that -E switch is for --extended-regexp. Hence, you can go as creative as you like. :)
     
    Dean likes this.
  7. ibnesayeed

    ibnesayeed Well-Known Member

    I would like to add one more thing. If you want to filter it for IP1 and IP2 then you can apply the above grep expression on raw file as well. Invert match filter pre-processing will not make any difference. :)
     
  8. Dean

    Dean Well-Known Member

    Just to be clear, '\' is the OR? Or is the '|' the or...

    I think it must be \ as you typed, because the | did not seem to work...
    ?
     
  9. ibnesayeed

    ibnesayeed Well-Known Member

    It was pipe (|), I have tested it before posting here. I will verify later.
     
  10. Dean

    Dean Well-Known Member

    It probably did not work for me because I was asking for things like ~ and %
    I realized I could do the query / grep without using either of those, so I did that, and only needed to search for 1 unique item.
     
  11. Floris

    Floris Guest

    or

    if you just want 1 IP,

    cat access_log |grep "thatiphere" > access_log_ip

    that should work on most *nix systems.

    Cleaning up a file with inverse would be a solution if the ip you're looking for is too dynamic, or more than one or two.
     
    Dean likes this.

Share This Page