What I'm doing is trying to use diff's --ignore-matching-lines= to ignore lines that contain certain patterns. The reason for this is I have a bash script that uses HPE's RESTful API to check/patch BIOS settings on the hosts. I use the --ignore-matching-lines= to omit patterns in the BIOS settings when both the RegEx matches the BIOS setting on the host as well as a basic template for the settings that I have stored in a variable. If diff finds the settings do not match the "golden configuration" it shows the differences on the terminal and prompts to apply the correct config. The settings I'm using the RegEx for are the UEFI boot order and Intel SGX settings.
This particular RegEx question is around the SGX settings. HPE has occasionally set the Epoch to a default of all 0's. I need to ensure that the SgxEpoch value is not all 0's, but rather a random 32 character string like in the following instance or we have a problem securing our enclave secretes.
Broken:
"SgxEpoch": "00000000000000000000000000000000"
OK:
"SgxEpoch": "5FSQUWEED6XPC8PJ2CWZGQIS4WWKLKUI"
I did quite a bit of looking and found that Wiktor Stribiżew was extremely helpful in showing how you can use POSIX to "match everything but" (I found out diff doesn't support lookaheads as it is BRE) - Regex: match everything but
So I came up with the following ERE version which also looks for "SgxEpoch": "" as that is how my comparison template is defined - https://regex101.com/r/Upd1KL/1
SgxEpochRegEx='"SgxEpoch":\s*(\"([^0].{31}|.[^0].{30}|.{2}[^0].{29}|.{3}[^0].{28}|.{4}[^0].{27}|.{5}[^0].{26}|.{6}[^0].{25}|.{7}[^0].{24}|.{8}[^0].{23}|.{9}[^0].{22}|.{10}[^0].{21}|.{11}[^0].{20}|.{12}[^0].{19}|.{13}[^0].{18}|.{14}[^0].{17}|.{15}[^0].{16}|.{16}[^0].{15}|.{17}[^0].{14}|.{18}[^0].{13}|.{19}[^0].{12}|.{20}[^0].{11}|.{21}[^0].{10}|.{22}[^0].{9}|.{23}[^0].{8}|.{24}[^0].{7}|.{25}[^0].{6}|.{26}[^0].{5}|.{27}[^0].{4}|.{28}[^0].{3}|.{29}[^0].{2}|.{30}[^0].|.{31}[^0])\"|"")'
And then converted it to BRE so it would work with diff:
SgxEpochRegEx='"SgxEpoch":\s*\("\([^0].\{31\}\|.[^0].\{30\}\|.\{2\}[^0].\{29\}\|.\{3\}[^0].\{28\}\|.\{4\}[^0].\{27\}\|.\{5\}[^0].\{26\}\|.\{6\}[^0].\{25\}\|.\{7\}[^0].\{24\}\|.\{8\}[^0].\{23\}\|.\{9\}[^0].\{22\}\|.\{10\}[^0].\{21\}\|.\{11\}[^0].\{20\}\|.\{12\}[^0].\{19\}\|.\{13\}[^0].\{18\}\|.\{14\}[^0].\{17\}\|.\{15\}[^0].\{16\}\|.\{16\}[^0].\{15\}\|.\{17\}[^0].\{14\}\|.\{18\}[^0].\{13\}\|.\{19\}[^0].\{12\}\|.\{20\}[^0].\{11\}\|.\{21\}[^0].\{10\}\|.\{22\}[^0].\{9\}\|.\{23\}[^0].\{8\}\|.\{24\}[^0].\{7\}\|.\{25\}[^0].\{6\}\|.\{26\}[^0].\{5\}\|.\{27\}[^0].\{4\}\|.\{28\}[^0].\{3\}\|.\{29\}[^0].\{2\}\|.\{30\}[^0].\|.\{31\}[^0]\)"\|""\)'
Here's an example of the BRE version catching either a non-zero or blank version of the Epoch as well as not catching something.
[~]$ echo '"SgxEpoch": "5FSQUWEED6XPC8PJ2CWZGQIS4WWKLKUI"' | grep '"SgxEpoch":\s*\("\([^0].\{31\}\|.[^0].\{30\}\|.\{2\}[^0].\{29\}\|.\{3\}[^0].\{28\}\|.\{4\}[^0].\{27\}\|.\{5\}[^0].\{26\}\|.\{6\}[^0].\{25\}\|.\{7\}[^0].\{24\}\|.\{8\}[^0].\{23\}\|.\{9\}[^0].\{22\}\|.\{10\}[^0].\{21\}\|.\{11\}[^0].\{20\}\|.\{12\}[^0].\{19\}\|.\{13\}[^0].\{18\}\|.\{14\}[^0].\{17\}\|.\{15\}[^0].\{16\}\|.\{16\}[^0].\{15\}\|.\{17\}[^0].\{14\}\|.\{18\}[^0].\{13\}\|.\{19\}[^0].\{12\}\|.\{20\}[^0].\{11\}\|.\{21\}[^0].\{10\}\|.\{22\}[^0].\{9\}\|.\{23\}[^0].\{8\}\|.\{24\}[^0].\{7\}\|.\{25\}[^0].\{6\}\|.\{26\}[^0].\{5\}\|.\{27\}[^0].\{4\}\|.\{28\}[^0].\{3\}\|.\{29\}[^0].\{2\}\|.\{30\}[^0].\|.\{31\}[^0]\)"\|""\)'
"SgxEpoch": "5FSQUWEED6XPC8PJ2CWZGQIS4WWKLKUI"
[~]$ echo '"SgxEpoch": ""' | grep '"SgxEpoch":\s*\("\([^0].\{31\}\|.[^0].\{30\}\|.\{2\}[^0].\{29\}\|.\{3\}[^0].\{28\}\|.\{4\}[^0].\{27\}\|.\{5\}[^0].\{26\}\|.\{6\}[^0].\{25\}\|.\{7\}[^0].\{24\}\|.\{8\}[^0].\{23\}\|.\{9\}[^0].\{22\}\|.\{10\}[^0].\{21\}\|.\{11\}[^0].\{20\}\|.\{12\}[^0].\{19\}\|.\{13\}[^0].\{18\}\|.\{14\}[^0].\{17\}\|.\{15\}[^0].\{16\}\|.\{16\}[^0].\{15\}\|.\{17\}[^0].\{14\}\|.\{18\}[^0].\{13\}\|.\{19\}[^0].\{12\}\|.\{20\}[^0].\{11\}\|.\{21\}[^0].\{10\}\|.\{22\}[^0].\{9\}\|.\{23\}[^0].\{8\}\|.\{24\}[^0].\{7\}\|.\{25\}[^0].\{6\}\|.\{26\}[^0].\{5\}\|.\{27\}[^0].\{4\}\|.\{28\}[^0].\{3\}\|.\{29\}[^0].\{2\}\|.\{30\}[^0].\|.\{31\}[^0]\)"\|""\)'
"SgxEpoch": ""
[~]$ echo '"SgxEpoch": "00000000000000010000000000000000"' | grep '"SgxEpoch":\s*\("\([^0].\{31\}\|.[^0].\{30\}\|.\{2\}[^0].\{29\}\|.\{3\}[^0].\{28\}\|.\{4\}[^0].\{27\}\|.\{5\}[^0].\{26\}\|.\{6\}[^0].\{25\}\|.\{7\}[^0].\{24\}\|.\{8\}[^0].\{23\}\|.\{9\}[^0].\{22\}\|.\{10\}[^0].\{21\}\|.\{11\}[^0].\{20\}\|.\{12\}[^0].\{19\}\|.\{13\}[^0].\{18\}\|.\{14\}[^0].\{17\}\|.\{15\}[^0].\{16\}\|.\{16\}[^0].\{15\}\|.\{17\}[^0].\{14\}\|.\{18\}[^0].\{13\}\|.\{19\}[^0].\{12\}\|.\{20\}[^0].\{11\}\|.\{21\}[^0].\{10\}\|.\{22\}[^0].\{9\}\|.\{23\}[^0].\{8\}\|.\{24\}[^0].\{7\}\|.\{25\}[^0].\{6\}\|.\{26\}[^0].\{5\}\|.\{27\}[^0].\{4\}\|.\{28\}[^0].\{3\}\|.\{29\}[^0].\{2\}\|.\{30\}[^0].\|.\{31\}[^0]\)"\|""\)'
"SgxEpoch": "00000000000000010000000000000000"
[~]$ echo '"SgxEpoch": "00000000000000000000000000000000"' | grep '"SgxEpoch":\s*\("\([^0].\{31\}\|.[^0].\{30\}\|.\{2\}[^0].\{29\}\|.\{3\}[^0].\{28\}\|.\{4\}[^0].\{27\}\|.\{5\}[^0].\{26\}\|.\{6\}[^0].\{25\}\|.\{7\}[^0].\{24\}\|.\{8\}[^0].\{23\}\|.\{9\}[^0].\{22\}\|.\{10\}[^0].\{21\}\|.\{11\}[^0].\{20\}\|.\{12\}[^0].\{19\}\|.\{13\}[^0].\{18\}\|.\{14\}[^0].\{17\}\|.\{15\}[^0].\{16\}\|.\{16\}[^0].\{15\}\|.\{17\}[^0].\{14\}\|.\{18\}[^0].\{13\}\|.\{19\}[^0].\{12\}\|.\{20\}[^0].\{11\}\|.\{21\}[^0].\{10\}\|.\{22\}[^0].\{9\}\|.\{23\}[^0].\{8\}\|.\{24\}[^0].\{7\}\|.\{25\}[^0].\{6\}\|.\{26\}[^0].\{5\}\|.\{27\}[^0].\{4\}\|.\{28\}[^0].\{3\}\|.\{29\}[^0].\{2\}\|.\{30\}[^0].\|.\{31\}[^0]\)"\|""\)'
[~]$
The problem I have is I don't understand how the above RegEx works. It appears to me that it is all just a bunch of alternatives using the meta-character | where I am basically saying if x number of "any character except newline" (via .{x}) and then followed by a 0 and then again x number of any character except newline (via .{x}). It seems to me that this RegEx should match the following examples but it doesn't... and I don't understand why.
"SgxEpoch": "00000000000000010000000000000000"
"SgxEpoch": "00000000000000000000000001000000"
"SgxEpoch": "00000100000000000000000000000000"
Here is it working very similarly to how it is implemented in the bash script. In the first input fd <(curl ...) it pulls all BIOS settings from the host in JSON format. For the 2nd fd <(echo ...) I echo a variable with the desired BIOS settings in generic form. Then both are fed to | python -m json.tool | perl -00pe 's:\[.*?\]:($x=$&)=~s/\s//gs;$x:ges' to alphabetize both inputs to diff, pretty print format to separate the JSON settings by \n so --suppress-common-lines only displays the discrepancies to the user and perl removes \n characters for array type variables because I could not get diff to match across newlines as "one chunk" with something like ..
[awilk00@nvdejb-dc-2p ~]$ SgxEpochRegEx='"SgxEpoch":\s*\("\([^0].\{31\}\|.[^0].\{30\}\|.\{2\}[^0].\{29\}\|.\{3\}[^0].\{28\}\|.\{4\}[^0].\{27\}\|.\{5\}[^0].\{26\}\|.\{6\}[^0].\{25\}\|.\{7\}[^0].\{24\}\|.\{8\}[^0].\{23\}\|.\{9\}[^0].\{22\}\|.\{10\}[^0].\{21\}\|.\{11\}[^0].\{20\}\|.\{12\}[^0].\{19\}\|.\{13\}[^0].\{18\}\|.\{14\}[^0].\{17\}\|.\{15\}[^0].\{16\}\|.\{16\}[^0].\{15\}\|.\{17\}[^0].\{14\}\|.\{18\}[^0].\{13\}\|.\{19\}[^0].\{12\}\|.\{20\}[^0].\{11\}\|.\{21\}[^0].\{10\}\|.\{22\}[^0].\{9\}\|.\{23\}[^0].\{8\}\|.\{24\}[^0].\{7\}\|.\{25\}[^0].\{6\}\|.\{26\}[^0].\{5\}\|.\{27\}[^0].\{4\}\|.\{28\}[^0].\{3\}\|.\{29\}[^0].\{2\}\|.\{30\}[^0].\|.\{31\}[^0]\)"\|""\),'
[awilk00@nvdejb-dc-2p ~]$ hostsettings='{"ServicePhone":"","SgxEpoch": "00000000000000000000000000000000","SgxEpochControl":"SgxEpochNoChange","DefaultBootOrder":["PcieSlotNic","EmbeddedFlexLOM","EmbeddedStorage","PcieSlotStorage","Usb","Cd","UefiShell","Floppy"]}'
[awilk00@nvdejb-dc-2p ~]$ desiredsettings='{"ServicePhone":"","SgxEpoch": "","SgxEpochControl":"SgxEpochNoChange","DefaultBootOrder":["Floppy","Cd","Usb","EmbeddedStorage","PcieSlotStorage","EmbeddedFlexLOM","PcieSlotNic","UefiShell"]}'
[awilk00@nvdejb-dc-2p ~]$ echo "${hostsettings}" | python -m json.tool | perl -00pe 's:\[.*?\]:($x=$&)=~s/\s//gs;$x:ges'
{
"DefaultBootOrder": ["PcieSlotNic","EmbeddedFlexLOM","EmbeddedStorage","PcieSlotStorage","Usb","Cd","UefiShell","Floppy"],
"ServicePhone": "",
"SgxEpoch": "00000000000000000000000000000000",
"SgxEpochControl": "SgxEpochNoChange"
}
[awilk00@nvdejb-dc-2p ~]$ echo "${desiredsettings}" | python -m json.tool | perl -00pe 's:\[.*?\]:($x=$&)=~s/\s//gs;$x:ges'
{
"DefaultBootOrder": ["Floppy","Cd","Usb","EmbeddedStorage","PcieSlotStorage","EmbeddedFlexLOM","PcieSlotNic","UefiShell"],
"ServicePhone": "",
"SgxEpoch": "",
"SgxEpochControl": "SgxEpochNoChange"
}
[awilk00@nvdejb-dc-2p ~]$ diff --report-identical-files --suppress-common-lines --side-by-side --ignore-matching-lines="${SgxEpochRegEx}" <(echo "${hostsettings}" | python -m json.tool | perl -00pe 's:\[.*?\]:($x=$&)=~s/\s//gs;$x:ges') <(echo "${desiredsettings}" | python -m json.tool | perl -00pe 's:\[.*?\]:($x=$&)=~s/\s//gs;$x:ges')
"DefaultBootOrder": ["PcieSlotNic","EmbeddedFlexLOM","Emb | "DefaultBootOrder": ["Floppy","Cd","Usb","EmbeddedStorage
"SgxEpoch": "00000000000000000000000000000000", | "SgxEpoch": "",
[awilk00@nvdejb-dc-2p ~]$
[awilk00@nvdejb-dc-2p ~]$
[awilk00@nvdejb-dc-2p ~]$ hostsettings='{"ServicePhone":"","SgxEpoch": "5FSQUWEED6XPC8PJ2CWZGQIS4WWKLKUI","SgxEpochControl":"SgxEpochNoChange","DefaultBootOrder":["PcieSlotNic","EmbeddedFlexLOM","EmbeddedStorage","PcieSlotStorage","Usb","Cd","UefiShell","Floppy"]}'
[awilk00@nvdejb-dc-2p ~]$ echo "${hostsettings}" | python -m json.tool | perl -00pe 's:\[.*?\]:($x=$&)=~s/\s//gs;$x:ges'
{
"DefaultBootOrder": ["PcieSlotNic","EmbeddedFlexLOM","EmbeddedStorage","PcieSlotStorage","Usb","Cd","UefiShell","Floppy"],
"ServicePhone": "",
"SgxEpoch": "5FSQUWEED6XPC8PJ2CWZGQIS4WWKLKUI",
"SgxEpochControl": "SgxEpochNoChange"
}
[awilk00@nvdejb-dc-2p ~]$ echo "${desiredsettings}" | python -m json.tool | perl -00pe 's:\[.*?\]:($x=$&)=~s/\s//gs;$x:ges'
{
"DefaultBootOrder": ["Floppy","Cd","Usb","EmbeddedStorage","PcieSlotStorage","EmbeddedFlexLOM","PcieSlotNic","UefiShell"],
"ServicePhone": "",
"SgxEpoch": "",
"SgxEpochControl": "SgxEpochNoChange"
}
[awilk00@nvdejb-dc-2p ~]$ diff --report-identical-files --suppress-common-lines --side-by-side --ignore-matching-lines="${SgxEpochRegEx}" <(echo "${hostsettings}" | python -m json.tool | perl -00pe 's:\[.*?\]:($x=$&)=~s/\s//gs;$x:ges') <(echo "${desiredsettings}" | python -m json.tool | perl -00pe 's:\[.*?\]:($x=$&)=~s/\s//gs;$x:ges')
"DefaultBootOrder": ["PcieSlotNic","EmbeddedFlexLOM","Emb | "DefaultBootOrder": ["Floppy","Cd","Usb","EmbeddedStorage
[awilk00@nvdejb-dc-2p ~]$
NOTE: I'm almost certain I found a thread on stack overflow where someone reviewed diff's source code and found it does a compare on a line by line basis but cannot find that thread ATM.
NOTE2: I noticed that when diff finds differentiating lines immediately preceding a line where there would have been have a match in the --suppress-common-lines regex, diff will not remove the two lines that match the regex and show them as a difference immediately after the preceding non-regex matching line it found a difference in. Hope I didn't butcher that too bad. For example:
[~]$ SgxEpochRegEx='"SgxEpoch":\s*\("\([^0].\{31\}\|.[^0].\{30\}\|.\{2\}[^0].\{29\}\|.\{3\}[^0].\{28\}\|.\{4\}[^0].\{27\}\|.\{5\}[^0].\{26\}\|.\{6\}[^0].\{25\}\|.\{7\}[^0].\{24\}\|.\{8\}[^0].\{23\}\|.\{9\}[^0].\{22\}\|.\{10\}[^0].\{21\}\|.\{11\}[^0].\{20\}\|.\{12\}[^0].\{19\}\|.\{13\}[^0].\{18\}\|.\{14\}[^0].\{17\}\|.\{15\}[^0].\{16\}\|.\{16\}[^0].\{15\}\|.\{17\}[^0].\{14\}\|.\{18\}[^0].\{13\}\|.\{19\}[^0].\{12\}\|.\{20\}[^0].\{11\}\|.\{21\}[^0].\{10\}\|.\{22\}[^0].\{9\}\|.\{23\}[^0].\{8\}\|.\{24\}[^0].\{7\}\|.\{25\}[^0].\{6\}\|.\{26\}[^0].\{5\}\|.\{27\}[^0].\{4\}\|.\{28\}[^0].\{3\}\|.\{29\}[^0].\{2\}\|.\{30\}[^0].\|.\{31\}[^0]\)"\|""\),'
[~]$ desiredsettings='{"SgxEpoch": "","SgxEpochControl":"SgxEpochNoChange","DefaultBootOrder":["Floppy"]}'
[~]$ hostsettings='{"SgxEpoch": "5FSQUWEED6XPC8PJ2CWZGQIS4WWKLKUI","SgxEpochControl":"SgxEpochNoChange","DefaultBootOrder":["PcieSlotNic"]}'
[~]$ diff --report-identical-files --side-by-side --ignore-matching-lines="${SgxEpochRegEx}" <(echo "${hostsettings}" | python -m json.tool) <(echo "${desiredsettings}" | python -m json.tool)
{ {
"DefaultBootOrder": [ "DefaultBootOrder": [
"PcieSlotNic" | "Floppy"
], ],
"SgxEpoch": "5FSQUWEED6XPC8PJ2CWZGQIS4WWKLKUI", "SgxEpoch": "",
"SgxEpochControl": "SgxEpochNoChange" "SgxEpochControl": "SgxEpochNoChange"
} }
[~]$ diff --report-identical-files --side-by-side --ignore-matching-lines="${SgxEpochRegEx}" <(echo "${hostsettings}" | python -m json.tool | grep -v '[]],') <(echo "${desiredsettings}" | python -m json.tool | grep -v '[]],')
{ {
"DefaultBootOrder": [ "DefaultBootOrder": [
"PcieSlotNic" | "Floppy"
"SgxEpoch": "5FSQUWEED6XPC8PJ2CWZGQIS4WWKLKUI", | "SgxEpoch": "",
"SgxEpochControl": "SgxEpochNoChange" "SgxEpochControl": "SgxEpochNoChange"
} }
[~]$
Because of the sensitivity of the data, I wanted to be sure I had a clear understanding of how this RegEx worked.
I would also greatly appreciate any syntax verification that you can offer around the huge regex line since I don't really understand how it works.
The ask here is for someone to explain/verify the regex. I thought it would be best that I explain my end goal due to the XY Problem, but really I can't afford the time investment currently to re-write everything. I am open to that going forward but I have a date to meet.
Aaron
