Sed replace with the output of a bash command using capture group as argument


I'm trying to make some base64 substitution with sed.

What I'm trying to do is this:

sed -i "s|\(some\)\(pattern\)|\1 $(echo "\2" | base64 -d)|g" myFile

In English that would be:

<ul><li>Math a pattern</li> <li>Capture groups</li> <li>Use the captured group in a bash command</li> <li>Use the output of this command as a replacement string</li> </ul>

So far my command doesn't work since \2 is known only by sed and not by the bash command I'm calling.

What elegant solution to I have to pass a capture group to a command of which I want to use the output?

<hr />


Here is a minimal example of what I'm trying to do:

I have the following file:

someline someline Base64Expression stringValue=&quot;Zm9v&quot; someline Base64Expression stringValue=&quot;YmFy&quot;

And I want to replace the base64 by plain text:

someline someline Base64Expression stringValue=&quot;foo&quot; someline Base64Expression stringValue=&quot;bar&quot;

In the future I'll have to do the backward operation (encoding string in base64 on the decoded file)

I've started using awk but I though it could get simpler (and much more elegant) with sed. So far with awk I have this (where $bundle is the file I'm editing):

#For each line containing "Base64Expression" #Put in the array $substitutions[]: # The number of the line (NR) # The encoded expression ($2) # The decoded expression (x) substitutions=($(awk -v bd=$bundle ' BEGIN { # Change the separator from default FS="&quot;" ORS="," OFS="," } /Base64Expression/ { #Decode the base64 lines cmd="echo -ne \""$2"\" | base64 -d" cmd | getline x if ( (cmd | getline) == 0 ){ print NR, $2, x } } ' $bundle)) # Substitute the encoded expressions by the decoded ones # Use the entries of the array 3 by 3 # Create a sed command which takes the lines numbers for ((i=0; i<${#substitutions[@]}; i+=3)) do # Do the substitution only if the string is not empty # Allows to handle properly the empty variables if [ ${substitutions[$((i+1))]} ] then sed -i -e "${substitutions[$i]}s#${substitutions[$((i+1))]}#${substitutions[$((i+2))]}#" $bundle fi done


You can use e in GNU sed to pass the substitution string to a shell for evaluation. This way, you can say:

printf "%s %s" "something" "\1"

Where \1 holds a captured group. All together:

$ sed -r 's#match_([0-9]*).*#printf "%s %s" "something" "\1"#e' <<< "match_555 hello" something 555

This comes handy when you want to perform some shell action with a captured group, like in this case.

So, let's capture the first part of the line, then the part that needs to be encoded and finally the rest. Once this is done, let's print those pieces back with printf triggering the usage of base64 -d against the second slice:

$ sed -r '/^Base64/s#(.*;)([^\&]*)(&.*)# printf "%s%s%s" "\1" $(echo "\2" | base64 -d) "\3";#e' file someline someline Base64Expression stringValue=&quot;foo&quot; someline Base64Expression stringValue=&quot;bar&quot;

Step by step:

sed -r '/^Base64/s#(.*;)([^\&]*)(&.*)# printf "%s%s%s" "\1" $(echo "\2" | base64 -d) "\3";#e' file # ^^^^^^^ ^^^ ^^^^^^ ^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^ # | first part | the rest encode the 2nd captured group | # | | | # | important part execute the command # | # on lines starting with Base64, do...

The idea comes from this <a href="https://stackoverflow.com/a/34080390/1983854" rel="nofollow">superb answer by anubhava</a> on <em>How to change date format in sed?</em>.


It sounds like this is what you're trying to do:

$ cat tst.awk BEGIN { FS=OFS="&quot;" } /^Base64Expression/ { cmd="echo -ne \""$2"\" | base64 -d" if ( (cmd | getline x) > 0 ) { $2 = x } close(cmd) } { print } $ awk -f tst.awk file someline someline Base64Expression stringValue=&quot;foo&quot; someline Base64Expression stringValue=&quot;bar&quot;

assuming your echo | base64 is the right approach.


  • GET curl call not returning JSON object
  • Express.js ERR_TOO_MANY_REDIRECTS error
  • GitLab site is showing the plesk default page. What am i doing wrong?
  • How do ES6 tagged template functions interpret their argument order?
  • Remove characters after a specific character in column
  • help('modules') crashing? Not sure how to fix
  • Specifying virtual keyboard type for EditText in XML
  • Dynamically add UI elements to StackPanel after button pressed
  • What is the default HTTP verb in WebApi ? GET or POST?
  • C# List of Panels
  • Can I use AllJoyn Framework for Wifi Direct in iOS?
  • How can I replace the server in Web Component Tester
  • How to use jQuery's $.post() method with async/await and typescript
  • Clear activity stack before launching another activity
  • TFS 2015 - Waiting for an agent to be requested
  • How to synchronize jQuery dialog box to act like alert() of Javascript
  • Querying row counts segregated by date ranges
  • Get specific string
  • Object and struct member access and address offset calculation
  • Jenkins: FATAL: Could not initialize class hudson.util.ProcessTree$UnixReflection
  • Firefox Extension - Monitor refresh and change of tab
  • How can I sort a a table with VBA with given text condition?
  • chrome.tabs.executeScript only fires when the Developer Console is open
  • Record samples being played with OpenAL
  • Checking free space on FTP server
  • Change multiple background-images with jQuery
  • Encrypt data by using a public key in c# and decrypt data by using a private key in php
  • Align navbar back button on right side
  • Window Size for Mac application
  • SSO with signing and signature validation doesn't work
  • Deserializing XML into class C#
  • Hibernate gives error error as “Access to DialectResolutionInfo cannot be null when 'hibernate.
  • AT Commands to Send SMS not working in Windows 8.1
  • VB.net deserialize, JSON Conversion from type 'Dictionary(Of String,Object)' to type '
  • Transpose CSV data with awk (pivot transformation)
  • Windows forms listbox.selecteditem displaying “System.Data.DataRowView” instead of actual value
  • apache spark aggregate function using min value
  • Sorting a 2D array using the second column C++
  • costura.fody for a dll that references another dll
  • How to push additional view controllers onto NavigationController but keep the TabBar?