update regex to support BSD sed (for macOS devs) and GNU sed

[hpccpatest]

Description

The sed parsing present in lib/shared_function.sh does not work correctly on macOS. I think it is because macOS by default uses BSD sed instead of GNU sed (for instance using \1 within the regex expression works on GNU sed but not on BSD sed, however I need to find documentation confirming this). Updated the regex so it works correctly for both BSD and GNU sed (tested it locally and worked fine on both).

One possible alternative to this PR could be to add a note in the guide for macOS users so they are aware of the issue and can workaround it by installing GNU sed and using that in the script, however I think this PR makes it easier and reduces the onus on new developers.

Apart from fixing #287, this PR also stops the regex from matching/including ending quotes as part of the captured group if they are present.

Related issues / Pull Requests

Fixes #287

Contributor Agreement

• I confirm I have signed the Contributor License Agreement

[das7pad]

Thanks for looking into this! Unquoting really is tricky 🙃

I'll check with the team on how to resolve this.

[das7pad]

This will not trim in case there are quotes inside the value, e.g. foo='Instance managed by "Department"'.

before:

$ name=foo; cat | sed -r "s/^$name=([\"']?)(.+)\1\$/\2/"
foo='Instance managed by "Department"'

// output
Instance managed by "Department"

after:

$ name=foo; cat | sed -r "s/^$name=([\"']?)([^\"']+)[\"']?$/\2/"
foo='Instance managed by "Department"'

// output
foo='Instance managed by "Department"'

Using a non-greedy matching group instead of the "non-quote"-group in the middle does not work either:

$ name=foo; cat | sed -r "s/^$name=([\"']?)(.+?)[\"']?$/\2/"
foo='Instance managed by "Department"'

// output
Instance managed by "Department"'
                                ^-- unmatched quote

[briangough]

How about using bash to do the parsing in a subshell?

function read_variable() {
  local name=$1
  (
    source "$TOOLKIT_ROOT/config/variables.env"
    echo "${!name:-}"
  )
}

[hpccpatest]

@das7pad good point, I didn't think values for environment variables would have quotes in most cases, but I agree it makes sense to support that as well. Thanks for pointing that out.

@briangough thank you for looking into this. I think your solution is much simpler and cleaner and works well for the above mentioned edge case as well. I'll update the PR with it.

[hpccpatest]

@das7pad just wanted to check-in, if we can merge this PR or is there anything else I need to do prior to that? Thanks.

[hachem-dd]

This should seriously be merged ^^'.

[hpccpatest]

Thanks for the review @briangough! I'm unable to merge the PR, is this due to some restrictions where only maintainers can merge a PR?

[briangough]

Thanks for the review @briangough! I'm unable to merge the PR, is this due to some restrictions where only maintainers can merge a PR?

We found that the change was breaking when the config file had unquoted strings with whitespace FOO=aaa bbb that were (unfortunately) being accepted by the original regex... we're a bit stuck on this one at the moment so we may just have to note it or issue a warning in this situation.